Reviewerly/exHarmony huggingface dataset:
RegionCL: Exploring Contrastive Region Pairs for Self-supervised Representation Learning

Self-supervised learning methods (SSL) have achieved significant success via maximizing the mutual information between two augmented views, where cropping is a popular augmentation technique. Cropped regions are widely used to construct positive pairs, while the remained regions after cropping have rarely been explored in existing methods, although they together constitute the same image instance and both contribute to the description of the category. In this paper, we make the first attempt to demonstrate the importance of both regions in cropping from a complete perspective and the effectiveness of using both regions via designing a simple yet effective pretext task called Region Contrastive Learning (RegionCL). Technically, to construct the two kinds of regions, we randomly crop a region (called the paste view) from each input image with the same size and swap them between different images to compose new images together with the remained regions (called the canvas view). Then, instead of taking the new images as a whole for positive or negative samples, contrastive pairs are efficiently constructed from the regional perspective based on the following simple criteria, i.e., each view is (1) positive with views augmented from the same original image and (2) negative with views augmented from other images. With minor modifications to popular SSL methods, RegionCL exploits those abundant pairs and helps the model distinguish the regions features from both canvas and paste views, therefore learning better visual representations. Experiments on ImageNet, MS COCO, and Cityscapes demonstrate that RegionCL improves MoCov2, DenseCL, and SimSiam by large margins and achieves state-of-the-art performance on classification, detection, and segmentation tasks. The code is publicly available at https://github.com/Annbless/RegionCL .
SDFA: Structure-Aware Discriminative Feature Aggregation for Efficient Human Fall Detection in Video

Older people are susceptible to fall due to instability in posture and deteriorating health. Immediate access to medical support can greatly reduce repercussions. Hence, there is an increasing interest in automated fall detection, often incorporated into a smart health-care system to provide better monitoring. Existing systems focus on wearable devices that are inconvenient or video monitoring that has privacy concerns. Moreover, these systems provide a limited perspective of their generalization ability as they are tested on datasets containing few activities that have wide disparity in the action space and are easy to differentiate. Complex daily life scenarios pose much greater challenges with activities that overlap in action spaces due to similar posture or motion. To overcome these limitations, we propose a fall detection model, called structure-aware discriminative feature aggregation, based on human skeletons extracted from low-resolution videos. The use of skeleton data ensures privacy and low-resolution videos ensures low hardware and computational cost. Our model captures discriminative structural displacements and motion trends using unified joint and motion features projected onto a shared high-dimensional space. Particularly, the use of separable convolution combined with a powerful graph convolutional network architecture provides improved performance. Extensive experiments on five large-scale datasets with a wide range of evaluation settings show that our model achieves competitive performance with extremely low computational complexity and runs faster than existing models.
Adversarial and Isotropic Gradient Augmentation for Image Retrieval With Text Feedback

Image Retrieval with Text Feedback (IRTF) is an emerging research topic where the query consists of an image and a text expressing a requested attribute modification. The goal is to retrieve the target images similar to the query text modified query image. The existing methods usually adopt feature fusion of the query image and text to match the target image. However, they ignore two crucial issues: overfitting and low diversity of training data, which make the feature fusion based IRTF task not generalizable. Conventional generation based data augmentation is an effective way to alleviate overfitting and improve diversity, but increases the volume of training data and generation model parameters, which is bound to bring huge computation costs. By rethinking the conventional data augmentation mechanism, we propose a plug-and-play Gradient Augmentation (GA) based regularization approach. Specifically, GA contains two items: 1) To alleviate model overfitting on the training set, we deduce an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">explicit adversarial gradient augmentation</i> from the perspective of adversarial training, which challenges the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">“no free lunch”</i> philosophy. 2) To improve the diversity of training set, we propose an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">implicit isotropic gradient augmentation</i> from the perspective of gradient descent-based optimization, which achieves the goal of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">big gain but no pain</i> . Besides, we introduce deep metric learning to train the model and provide theoretical insights of GA on generalisation. Finally, we propose a new evaluation protocol called Weighted Harmonic Mean (WHM) to assess the model generalisation. Experiments show that our GA outperforms the state-of-the-art methods by 6.2 and 4.7% on CSS and Fashion200 k datasets, respectively, without bells and whistles.
Hand Gesture Recognition Control for Infotainment System

In day-to-day life, the accidents occurring while driving an automobile is based on either human error or mechanical error. These human errors may occur due to control issues on the steering wheel, and the control of infotainment during driving on the parallel action. According to research conducted by the AAA Foundation, the cause for the distraction of drivers for more than forty seconds is controlling the infotainment system. While taking your eyes off the view road for two seconds could increase your chances of getting into the chance of an accident, taking forty seconds to control your entertainment system is exceedingly unsafe and poses a significant risk to both the driver as well as the other road users. To avoid such mishaps, a technique for managing infotainment equipment with hand gestures should be developed. There are multiple ways to control or access the Infotainment system and Entertainment system in Automobiles. Giving inputs through touch panels, buttons, and voice commands is one of those ways. Deep Learning algorithms should be developed to detect hand gestures and actuation based on gesture recognition. A novel approach of giving inputs to the infotainment system through hand gestures is introduced in this work. The system will recognize the hand gesture and actuate the devices in the infotainment system based on the gesture recognized. The hand gesture recognition system and infotainment system should be integrated in order to actuate the devices in the infotainment system by giving hand gestures as input. With the help of hand gesture recognition in infotainment control, the problem of distraction while driving infotainment will reduce.
Boosting Variational Inference With Margin Learning for Few-Shot Scene-Adaptive Anomaly Detection

Anomaly detection in surveillance videos aims to identify frames where abnormal events happen. Existing approaches assume that the training and testing videos are from the same scene, exhibiting poor generalization performance when encountering an unseen scene. In this paper, we propose a Variational Anomaly Detection Network (VADNet), which is characterized by its high scene-adaptation - it can identify abnormal events in a new scene only via referring to a few normal samples without fine-tuning. Our model embodies two major innovations. First, a novel Variational Normal Inference (VNI) module is proposed to formulate image reconstruction in a conditional variational auto-encoder (CVAE) framework, which learns a probabilistic decision model instead of a traditional deterministic one. Secondly, a Margin Learning Embedding (MLE) module is leveraged to boost the variational inference and aid in distinguishing normal events. We theoretically demonstrate that minimizing the triplet loss in MLE module facilitates maximizing the evidence lower bound (ELBO) of CVAE, which promotes the convergence of VNI. By incorporating variational inference with margin learning, VADNet becomes much more generative that is able to handle the uncertainty caused by the changed scene and limited reference data. Extensive experiments on several datasets demonstrate that the proposed VADNet can adapt to a new scene effectively without fine-tuning and achieve remarkable performance, which outperforms other methods significantly and establishes new state-of-the-art in the case of few-shot scene-adaptive anomaly detection. We believe our method is closer to real-world application due to its strong generalization ability. All codes are released in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/huangxx156/VADNet</uri> .
On Fooling Facial Recognition Systems using Adversarial Patches

Researchers are increasingly interested to study novel attacks on machine learning models. The classifiers are fooled by making small perturbation to the input or by learning patches that can be applied to objects. In this paper we present an iterative approach to generate a patch that when digitally placed on the face can successfully fool the facial recognition system. We focus on dodging attack where a target face is misidentified as any other face. The proof of concept is show-cased using FGSM and FaceNet face recognition system under the white-box attack. The framework is generic and it can be extended to other noise model and recognition system. It has been evaluated for different - patch size, noise strength, patch location, number of patches and dataset. The experiments shows that the proposed approach can significantly lower the recognition accuracy. Compared to state of the art digital-world attacks, the proposed approach is simpler and can generate inconspicuous natural looking patch with comparable fool rate and smallest patch size.
Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations

Sound is one of the most informative and abundant modalities in the real world while being robust to sense without contacts by small and cheap sensors that can be placed on mobile devices. Although deep learning is capable of extracting information from multiple sensory inputs, there has been little use of sound for the control and learning of robotic actions. For unsupervised reinforcement learning, an agent is expected to actively collect experiences and jointly learn representations and policies in a self-supervised way. We build realistic robotic manipulation scenarios with physics-based sound simulation and propose the Intrinsic Sound Curiosity Module (ISCM). The ISCM provides feedback to a reinforcement learner to learn robust representations and to reward a more efficient exploration behavior. We perform experiments with sound enabled during pre-training and disabled during adaptation, and show that representations learned by ISCM outperform the ones by vision-only baselines and pre-trained policies can accelerate the learning process when applied to downstream tasks.
Research on Abnormal Transaction Detection Method for Blockchain

Blockchain has become a hot topic in current academic research due to its decentralization, imtamability and anonymity. However it also makes blockchain a tool for money laundering, fraud, extortion and other illegal activities. Therefore, it is particularly important to supervise and detect abnormal transactions on blockchain. In our paper, graph data structure is used to express blockchain node transactions. Aiming at the problem of too few abnormal transaction samples of block chain, k-rate sampling and feature similarity are used to solve the problem of unbalanced data. Further more, we select some features for feature preprocessing according to the distribution of multiple features of blockchain transactions. Finally, the blockchain transaction topology is used for multi-graph convolutional neural network machine learning methods to train the blockchain transaction data. So the abnormal nodes and non-abnormal nodes classification model is obtained. And the abnormal transactions of blockchain is detected by using the classification model. Experimental results show that compared with traditional algorithms such as logistic regression (LR), multi-layer perceptron (MLP) and linear regression (LR), the anomaly detection algorithm based on sample equalization and feature engineering has significantly improved recall rate. This provides a theoretical and practical basis for our next work – tracing abnormal nodes of blockchain, which has a good prospect for industrial application.
Calling Cards: Concrete Visual End-User Programming

Creating a program that performs even a simple task and shows the result is unapproachable to most people, and even trained programmers face a burden to create a new program. In this paper, we introduce a prototype system and model for live visual dataflow programming where intermediate steps are visible and all components are tangible and manifest. Our system aims to allow a user to dive in and immediately have a working program that can be incrementally extended.
Countering Physical Eavesdropper Evasion with Adversarial Training

Signal classification is a universal problem in adversarial wireless scenarios, especially when an eavesdropping radio receiver attempts to glean information about a target transmitter’s patterns, attributes, and contents over a wireless channel. In recent years, research surrounding the idea of Machine Learning (ML)-based signal classification has focused on modulation classification, with the downstream objective of demodulation. However, while the computer vision data domain has made significant progress in ensuring robust classification of images despite crafted perturbations, this success has not been translated to secure modulation classification. In this work, we perform the first-ever physical test of an eavesdropping ML-based modulation classifier radio, which we trained offline using a ensemble of i.i.d. models. Each model is trained with a weighted mixture of data perturbed by iterative“, least likely” white box attacks and non-attacked data. We then tested the ensemble online using coaxial-connected Software Defined Radios (SDRs). We conducted a case study comparing our results to the state-of-the-art computer vision approaches to investigate the presence of “label leaking”, model capacity sensitivity, understand the viability of parallel and sequential variations on perturbation training, and assess the effectiveness of iterative attack training. Our results show that perturbations can result in guessing-level classification performance from eavesdroppers, and that varying levels of robustness can be achieved against all presented attacks. These findings confirm that any receiver presents a new attack vector by utilizing ML techniques for classification tasks, and can be vulnerable to evasion attacks at little-to-no cost to transmitters. Consequently, we argue for the use of our training scheme in all ML-based classifying radios where security is a concern.
Unit Testing for Concepts in Neural Networks

Abstract Many complex problems are naturally understood in terms of symbolic concepts. For example, our concept of “cat” is related to our concepts of “ears” and “whiskers” in a non-arbitrary way. Fodor (1998) proposes one theory of concepts, which emphasizes symbolic representations related via constituency structures. Whether neural networks are consistent with such a theory is open for debate. We propose unit tests for evaluating whether a system’s behavior is consistent with several key aspects of Fodor’s criteria. Using a simple visual concept learning task, we evaluate several modern neural architectures against this specification. We find that models succeed on tests of groundedness, modularity, and reusability of concepts, but that important questions about causality remain open. Resolving these will require new methods for analyzing models’ internal states.
Tunable polarization encoder capable of polarization conversion and separation based on a layered photonic structure

In this paper, a polarization encoder with a polarization conversion function is investigated using a layered photonic structure comprising nonlinear materials and indium antimonide. The electric field of the incident linear polarization waves considered are parallel to the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x-y</i> plane at an angle of 45° to the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</i> -axis, and can be decomposed into components directed along the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</i> - and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">y</i> -axes, generating two polarized waves: transverse electric and transverse magnetic waves. Due to the nonlinear effect and the temperature and magnetic flux density tunable properties of indium antimonide, the light intensity, temperature, and magnetic flux density can be used to adjust the reflection phase difference and polarization form. Discussing the influences of these three physical quantities on the phase difference, the incident linear polarization wave and reflected circular polarization wave conversion, the circular polarization conversion of two different rotations and the circular polarization separation of a certain bandwidth can be simultaneously realized with the specific modification of the light intensity or temperature. Moreover, four disparate input logic levels composed of temperature and magnetic flux density achieve specific encoding outputs for all polarization forms. This signifies that polarization selection through control of the logic codes can be easily attained. The consequences show that this form of polarization conversion and encoding is tailored and can be accurately manipulated. Hence, this provides a novel foundation for tunable and diverse polarization splitters and selectors.
Underwater Target Detection Based on Improved YOLOv4

As one of the underwater image processing methods, the underwater target detection algorithm is becoming increasingly important in daily life. An underwater target detection algorithm is proposed in this paper. The difference between this algorithm and the existing algorithms is that it is based on improved YOLOv4. K-means++ algorithm uses re-clustering method to obtain anchor boxes in underwater scene. Image enhancement is also performed using MSRCR with dark channel fusion, and the dataset is trained using a segmented training method. The experimental results show that the detection effect of this method is better than several other methods.
Boundary Defense Against Black-box Adversarial Attacks

Black-box adversarial attacks generate adversarial samples via iterative optimizations using repeated queries. Defending deep neural networks against such attacks has been challenging. In this paper, we propose an efficient Boundary Defense (BD) method which mitigates black-box attacks by exploiting the fact that the adversarial optimizations often need samples on the classification boundary. Our method detects the boundary samples as those with low classification confidence and adds white Gaussian noise to their logits. The method’s impact on the deep network’s classification accuracy is analyzed theoretically. Extensive experiments are conducted and the results show that the BD method can reliably defend against both soft and hard label black-box attacks. It outperforms a list of existing defense methods. For IMAGENET models, by adding zero-mean white Gaussian noise with standard deviation 0.1 to the logits of those images with classification confidence less than 0.3, the defense reduces the attack success rate to almost 0 while limiting the classification accuracy degradation to around 1 percent.
Self-supervised Vision Transformers with Data Augmentation Strategies Using Morphological Operations for Writer Retrieval

This paper introduces a self-supervised approach using vision transformers for writer retrieval based on knowledge distillation. We propose morphological operations as a general data augmentation method for handwriting images to learn discriminative features independent of the pen. Our method operates on binarized $$224\times 224$$ -sized patches extracted of the documents' writing region, and we generate two different views based on randomly sampled kernels for erosion and dilation to learn a representative embedding space invariant to different pens. Our evaluation shows that morphological operations outperform data augmentation generally used in retrieval tasks, e.g., flipping, rotation, and translation, by up to 8%. Additionally, we evaluate our data augmentation strategy to existing approaches such as networks trained with triplet loss. We achieve a mean average precision of 66.4% on the Historical-WI dataset, competing with methods using algorithms like SIFT for patch extraction or computationally expensive encodings, e.g., mVLAD, NetVLAD, or E-SVM. In the end, we show by visualizing the attention mechanism that the heads of the vision transformer focus on different parts of the handwriting, e.g., loops or specific characters, enhancing the explainability of our writer retrieval.
Practical Parallel of Autonomous Unmanned Aerial Vehicle by Mission Planner

Satellite imaging has always been a challenging task and at the current time being there are very few alternatives for capturing vertical images of Earth's vast landscape. Aerial photography by using multiple unmanned aerial vehicles (UAV) might be one of the solutions. This research aims to study, improve and demonstrate autonomy for UAVs by using specialized software and apparatus. The mission can be utilized to allow the UAVs to fly sequentially in the same pathway without colliding and creating the highest amount of coverage, which it is important for some remote sensing applications. Certain benefits can be obtained from using UAV rather than satellite imaging such as lower cost of overall equipment than a space satellite and higher resolution pictures due to flexibility in altitude operations. The results of implement the autonomous UAV has compare between path of actual flight and the setting way-points and it express exceptional results by presenting less of Root Mean Square Error.