Reviewerly/exHarmony huggingface dataset:
Vision-Based Reactive Planning and Control of Quadruped Robots in Unstructured Dynamic Environments

Quadruped robots have received increasing attention for the past few years. However, existing works primarily focus on static environments or assume the robot has full observations of the environment. This limits their practical applications since real-world environments are often dynamic and partially observable. To tackle these issues, vision-based reactive planning and control (V-RPC) is developed in this work. The V-RPC comprises two modules: offline pre-planning and online reactive planning. The pre-planning phase generates a reference trajectory over continuous workspace via sampling-based methods using prior environmental knowledge, given an LTL specification. The online reactive module dynamically adjusts the reference trajectory and control based on the robot's real-time visual perception to adapt to environmental changes.
Repnas: Searching for Efficient Re-Parameterizing Blocks

In the past years, significant improvements in the field of neural architecture search(NAS) have been made. However, in order to improve the performance of the model, many search spaces contain multi-branch architectures, which leads to the gap between the searched constraint and real inference time. In this work, we propose a re-parameterization (Rep) search space based on structural Rep techniques. In the Rep search space, the subnets are multi-branch in training and single-path in inference. Furthermore, RepNAS, a one-stage NAS approach, is present to efficiently search the optimal diverse branch block (ODBB) for each layer under the branch number constraint. Our experimental results show the searched ODBB can easily surpass the manual diverse branch block (DBB) with efficient training.
Run and Chase: Towards Accurate Source-Free Domain Adaptive Object Detection

Recently, there has been increasing interest in the Source-Free Domain Adaptive Object Detection task, which involves training an object detector on the unlabeled target data using a pre-trained source model without accessing the source data. Most related methods are developed from the mean-teacher framework, which aims to train the student model closer to the teacher model via a pseudo labeling manner, where the teacher model is the exponential-moving-average of the student models at different time-steps. Following this line of works, we propose a Run-and-Chase Mutual-Learning method to strengthen the interactions between the student model and the teacher model in both feature and prediction levels. In our method, the student model is optimized to run away from the teacher model at the feature level, while chasing the teacher model at the prediction level. In this way, the student model is forced to be distinguishable at different time-steps, so that the teacher model can acquire more diverse task-related information and produce higher-accuracy pseudo labels. As the training goes, the student and teacher models are updated iteratively and promoted mutually, which can prevent the model collapse problem. Extensive experiments are conducted to validate the effectiveness of our method.
Full/Regular Research Paper submission to (CSCI-RTCW): Multi Class Classification of Online Radicalization Using Transformer Models

Online Radicalization is a major security threat to a nation and has the power to influence young minds through online blogs and articles present on social media. It can occur in various forms such as political, social, criminal radicalization etc depending upon the intention and target of the propagator. Each type of radicalization aims to harm a different section of society and thus, requires a different type of treatment and mitigation plan. In our paper we have proposed the use of transformer based models such as BERT etc. to identify the type of radicalization in online text. We have also presented a comparative analysis of several transformers based classifiers for multi class classification of radicalization on social media. The results show that DistilBERT outperforms the other transformer models and has achieved an accuracy of 96.3 percent in this text classification task. As per our knowledge, this is a first of its kind study where the type of radical behaviour in text is being detected.
Learning Disentangled Representation with Mutual Information Maximization for Real-Time UAV Tracking

Efficiency has been a critical problem in UAV tracking due to limitations in computation resources, battery capacity, and unmanned aerial vehicle maximum load. Although discriminative correlation filters (DCF)-based trackers prevail in this field for their favorable efficiency, some recently proposed lightweight deep learning (DL)-based trackers using model compression demonstrated quite remarkable CPU efficiency as well as precision. Unfortunately, the model compression methods utilized by these works, though simple, are still unable to achieve satisfying tracking precision with higher compression rates. This paper aims to exploit disentangled representation learning with mutual information maximization (DR-MIM) to further improve DL-based trackers’ precision and efficiency for UAV tracking. The proposed disentangled representation separates the feature into an identity-related and an identity-unrelated features. Only the latter is used, which enhances the effectiveness of the feature representation for subsequent classification and regression tasks. Extensive experiments on four UAV benchmarks, including UAV123@10fps, DTB70, UAVDT and VisDrone2018, show that our DR-MIM tracker significantly outperforms state-of-the-art UAV tracking methods.
Dual-Expert Distillation Network for Few-Shot Segmentation

Few-shot segmentation has attracted growing interest owing to its value in practical applications. The primary challenge of few-shot segmentation lies in semantic information discovery, especially for query images. To tackle this issue, we propose a dual-expert distillation network (DEDN) made up of a scenario-level expert and an object-level expert to obtain semantic information from different perspectives. In DEDN, experts can learn from each other through online knowledge distillation with positive-guided Kullback-Leibler divergence. We innovate the Scenario Normalization and Object Continuity Guidance on dual experts to guarantee the various perspectives respectively. We further propose the Adaptive Weighted Fusion to adapt the trained experts to novel classes and obtain reliable fused predictions. Extensive experiments on Pascal-5i and COCO-20i show that our approach achieves state-of-the-art results.
Deep Unsupervised Momentum Contrastive Hashing for Cross-modal Retrieval

Unsupervised cross-modal hashing (UCMH) methods often start from the similarity of sample features and design a reconstruction loss to achieve similarity preservation. However, these methods suffer from inaccurate similarity problems, be-cause different feature representations may share similar semantic information. In this paper, we propose Deep Unsupervised Momentum Contrastive Hashing (DUMCH). Specifically, we introduce momentum contrastive learning for unsupervised cross-modal hashing, which allows us to flexibly define a robust loss by comparing positive and negative samples. Moreover, in order to achieve similarity retention of hash codes in Hamming space and fully utilize the potential of contrastive learning in Hamming space, we remove the L2 normalization corresponding to cosine similarity and design a novel normalization method called hash normalization, which has been proved to greatly improve the model performance. We conducted extensive experiments on three datasets, and the experimental results demonstrate the superiority of DUMCH.
Implementation Roadmap for Neural Networks in Array Databases

Relational databases lack behind when handling array data and thus array databases were created to fill this gap. Array databases provide optimized storage, retrieval, and processing of multidimensional discrete data (MDD), also known as array data. Just like relational array databases, data processing in array databases is handled declaratively through an array query language that offers enough expressible power to define a myriad of operations. However, despite the advancements in array database technology, there is still a gap in describing machine learning (ML) algorithms and in particular neural networks which, in recent years, have been adopted for predicting phenomena in science and engineering. In this contribution, we outline an implementation roadmap for defining neural networks in an array database. We first identify the necessary linear algebra operators present in a feed-forward neural network and use them to define the training and prediction operations of that network. We also define other operators that, though they are not part of linear algebra, are essential for a complete machine-learning implementation.
Adaptive and Robust Fourier-Mellin-Based Image Watermarking for Social Networking Platforms

According to the Buckets effect, the capacity of a bucket depends on the length of the shortest board. This principle also applies to social networking platform resilient (SNPR) image watermarking, which should be comprehensive and free from significant shortcomings. In the frequency domain, the watermarked region is formed using log-polar coordinate mapping (LPM) and has a ring-like structure. However, this structure cannot be stretched or compressed, and it causes a streaking effect at the edges of the watermarked image. These issues have been addressed in the proposed method. Specifically, an adaptive optimization framework is used to adjust the embedding strength and range of the watermark, and multiple synchronization strategies are adopted to correct flip and aspect ratio. Compared with state-of-the-art works, the proposed method significantly improves the imperceptibility of the watermarked image and its robustness to various distortions and lossy transmission on social networking platforms (SNPs).
Semantic Embedding Uncertainty Learning for Image and Text Matching

Image and text matching measures the semantic similarity for cross-modal retrieval. The core of this task is semantic embedding, which mines the intrinsic characteristics of visual and textual for discriminative representation. However, cross-modal ambiguity of image and text (the existence of one-to-many associations) is prone to semantic diversity. The mainstream approaches utilized the fixed point embedding to represent semantics, which ignored the embedding uncertainty caused by semantic diversity leading to incorrect results. To address this issue, we propose a novel Semantic Embedding Uncertainty Learning (SEUL), which represents the embedding uncertainty of image and text as Gaussian distributions and simultaneously learns the salient embedding (mean) and uncertainty (variance) in the common space. We design semantic uncertainty embedding for facilitating the robustness of the representation in the semantic diversity context. A combined objective function is proposed, which optimizes the semantic uncertainty and maintains discriminability to enhance cross-modal associations. Extended experiments are performed on two datasets to demonstrate advanced performance.
An Analysis of Public Sentiment around Face Mask During Covid-19

The impact of the covid-19 pandemic continues to be felt as we face the aftermath the pandemic has caused globally in terms of loss of life, economic damage, civil unrest, and political instability. Mixed messaging and partisanship from leaders worldwide have made proposed solutions a divisive political stance. The polarization of covid-19 and the sentiments around proposed solutions warrant investigation; thus, this study's objective is to examine the sentiments and topics in global public discourse around one of the proposed measures to contain the spread of covid-19; mask usage. This study looks to examine the sentiments around the global discourse of Twitter users on mask usage to investigate the relationship between public sentiment, fluctuations in covid-19 cases, and topics of discussion and influential events through sentiment analysis and topic modeling. Sentiment analysis and Topic Modeling are conducted on the dataset after it is preprocessed to analyze the abstract topics discussed over the time period. The results are then visualized to highlight key topics present in the discussion around mask usage and the accompanying sentiment that is then examined in relation to global events and case numbers during the same time. The goal is to understand how the nature of global discourse can impact the perceptions of mask usage and how that relates to behavior that continues the spread of the pandemic.
A Set of Machine-Learning Exercises

This paper represents several assignments for teaching a machine/deep-learning course. Students starts with simple assignments to implement neural networks in Python and progress toward more advanced assignments to perform image classification using Google TensorFlow. Students will use “Jupyter Notebook” platform to code the lab assignments and projects. Each assignment represents the main benefits of using neural networks. Visualization of the performance and final results are used to verify the correctness of the results.
Dual-level Consistency Learning for Unsupervised Domain Adaptive Night-time Semantic Segmentation

In autonomous driving, it is crucial to train a segmentation model that can perform well in poor illumination. Due to the lack of pixel-level annotated nighttime images and a large domain discrepancy between day-night image pairs, it could be tough to achieve encouraging performance for the nighttime model. In this paper, we propose a novel Dual-level Consistency Learning (DCL) method which fully utilizes these day-night correspondences across two domains. Specifically, our DCL regularizes the style and semantic difference consistency of day-night images at different feature levels. We conduct style difference consistency at the shallow feature level and facilitate contrastive semantic difference regression consistency at the deep feature level. These designs enable the proposed DCL to acquire pairwise pixel-level pseudo supervision, reducing the dependence on annotated nighttime images. Empirically, we conduct extensive experiments on standard benchmarks such as Dark Zurich and ACDC. The encouraging results show the effectiveness of our DCL method.
Non-rigid Registration Technique for Large Deformation Medical Image

Medical image registration faces significant challenges in dealing with non-rigid deformations. The problem such as folding of deformation displacement field and model degradation, all can lead to loss of registration accuracy. To address these problems, a deep learning-based unsupervised method(MulSc-Net) is proposed. On one hand, the MulSc-Net adopt a novel multi-scale registration strategy which can capture deformations at different scales with a larger receptive field via a fusion model of convolution and dilated convolution. On the other hand, an anti-folding constraint is introduced to ensure the continuity of displacement field, and a residual method is employed to prevent model degradation during training. In this work, the MulSc-Net model is evaluated on a lung CT dataset. The experimental results show that MulSc-Net can achieve better registration accuracy compared to current related methods.
A Lightweight Framework for Objection Detection in Adverse Lighting Conditions

Although the latest object detection methods have demonstrated strong performance on large-scale comprehensive datasets, various adverse lighting conditions limit their potential application in real-world scenarios, especially the deployment on mobile robots. In this study, we propose a novel and lightweight re-illumination framework for end-to-end adaptive enhancement for adverse lighting images. Specifically, we employ a set of differentiable image processing modules and pixel-wise curve parameter mapping to adapt to various lighting conditions. We use YOLOv3 detection loss to learn the curve parameters of the U-shaped parameter predictor (UPP) in a weakly-supervision manner. We further deploy the proposed framework to a low-power hardware platform. The experimental results demonstrate the effectiveness of our proposed method in various adverse lighting conditions(i.e. haze, low-light).
Discriminative Spatiotemporal Alignment for Self-Supervised Video Correspondence Learning

This paper focuses on self-supervised video correspondence learning, which learns effective representations from raw videos without manual annotations and exploits the learned representations for video visual tracking tasks. Previous methods extract temporal correspondence between two frames in fixed geometric structures, which easily leads to mismatches of pixels and overlooks the intra-frame semantic correspondence. To address these issues, we propose a Discriminative Spatiotemporal Alignment (DSA) framework to improve the tracking accuracy in the inference stage. DSA first discriminates representations of different instances for each reference frame through an Instance-Guided Spatial Alignment (IGSA) module. Then, it employs a Focused Temporal Alignment (FTA) module, which samples discriminative pixels from reference frames and propagates the labels of the sampled reference pixels to a target pixel. Experimental results show that DSA possesses flexibility and generalizability and has boosted previous approaches on three tracking tasks, including video object segmentation, human part segmentation, and pose keypoint tracking.
HPCNet: A Hybrid Progressive Coupled Network for Image Deraining

Convolutional neural networks (CNNs) and Transformers have significantly succeeded in low-level vision tasks. Although prominent complementary characteristics exist regarding the larger receptive field and better convergence, only some efforts have compacted them efficiently due to their individual and nonnegligible weakness. In this paper, we propose a hybrid progressive coupled network (HPCNet) for rain perturbation removal, which integrates the advantages of these two architectures while maintaining both effectiveness and efficiency. In particular, we achieve the progressive decomposition and association of rain-free and rain features, designed as an asymmetrical dual-path mutual representation network to alleviate the computational cost. Meanwhile, we equip the network with high-efficiency convolutions and resolution rescaling strategy to trade off the computational complexity. Extensive experiments show that our method outperforms MPRNet on average while saving 87.2% and 61.1% of the computational cost and parameters.
A Geometrical Characterization on Feature Density of Image Datasets

Recently, the interpretability and verification of deep learning have attracted enormous attention from both academic and industrial communities, aiming to gain users’ trust and ease their concerns. To guide learning procedures or data operations carried out in a more interpretable way, in this paper, we put a similar perspective on image datasets, the inputs of deep learning. Based on manifold learning, we work out an interpretable geometrical characterization on the curvity of manifolds to depict the feature density of datasets, which is represented with the ratio of the Euclidean distance and the geodesic distance. It is a noteworthy characteristic of image datasets and we take the dataset compression and enhancement problems as application instances via sample credit assignment with the geometrical information. Experiments on typical image datasets have justified the effectiveness and enormous prospect of the presented geometrical characteristic.
Designing Optics and Algorithm for Ultra-Thin, High-Speed Lensless Cameras

There is a growing demand for small, light-weight and low-latency cameras in the robotics and AR/VR community. Mask-based lensless cameras, by design, provide a combined advantage of form-factor, weight and speed. They do so by replacing the classical lens with a thin optical mask and computation. Recent works have explored deep learning based post-processing operations on lensless captures that allow high quality scene reconstruction. However, the ability of deep learning to find the optimal optics for thin lensless cameras has not been explored. In this work, we propose a learning based framework for designing the optics of thin lensless cameras. To highlight the effectiveness of our framework, we learn the optical phase mask for multiple tasks using physics-based neural networks. Specifically, we learn the optimal mask using a weighted loss defined for the following tasks-2D scene reconstructions, optical flow estimation and face detection. We show that mask learned through this framework is better than heuristically designed masks especially for small sensors sizes that allow lower bandwidth and faster readout. Finally, we verify the performance of our learned phase-mask on real data.
CSCI-RTNA:Research Paper: Security and Privacy Challenges in Voice-Based Social Networks

Voice-based social networks (VBSNs) like Clubhouse have explosively grown since they were introduced. During the pandemic, VBSNs allowed people to communicate with others via audio while many were isolated at home. Their services successfully met the need for social interactions, and they quickly established themselves as one of the major social network service providers alongside Facebook and Twitter. The unique features of VBSNs open the possibilities for diverse uses of them. As one of the application areas, we focus on emergency and disaster management. Using VBSNs, we can address the limitations of existing disaster response using online social networks. For the practical use of VBSNs during disasters, we, however, must resolve the security and privacy issues. To this end, in this paper, we define the characteristics of VBSNs and identify the security and privacy issues to be considered. The result of the user survey and threat scenarios show us that there is an urgent need for research on this topic. Then, we present potential solutions and challenges.
A Lightweight Dynamic Gesture Recognition Network with Spatio-Temporal Attention

As one of the simplest and most natural ways of human-computer interaction, gestures can effectively convey information through silent hand movements and facilitate our production and life in combination with intelligent devices. Accurate and effective gesture recognition technology is an effective way to improve the efficient work of intelligent human-computer interaction devices in complex environments. However, due to the influence of complex surroundings and image noise, accurate prediction of dynamic gestures still faces certain challenges. Recently, benefiting from the end-to-end feature extraction capabilities of convolutional neural networks (CNN), deep learning has achieved good results in different dynamic gesture recognition tasks. However, it still has certain shortcomings in learning spatio-temporal semantic features to identify effective regional features. In order to accurately recognize continuous dynamic gestures in video, a dynamic gesture recognition network based on spatio-temporal attention mechanism is proposed to realize end-to-end dynamic gesture recognition. Aimed at the problem of information loss in the process of spatial and temporal feature extraction of video frames, a 3D residual convolution neural network is used to simultaneously model the temporal relationship and spatial information between frames. In order to obtain discriminative feature information, a spatio-temporal attention module (STA) is proposed to improve the ability of the network to extract temporal and spatial features. A large number of experiments on two public datasets show that the proposed dynamic gesture recognition network based on spatio-temporal attention can obtain the best recognition performance compared with other state-of-the-art gesture recognition models, and further prove that the network is lightweight, feasible and practical method.