Reviewerly/exHarmony huggingface dataset:
Dual-Path Reconstruction Guided Segmentation Network for Unsupervised Anomaly Detection and Localization

Visual anomaly detection methods with localization are critically important for industrial manufacturing quality control. Because of the rarity of anomalies and the irregular variation of anomaly patterns, unsupervised methods have been widely explored. For the anomaly detection and localization tasks, the image reconstruction-based approaches have shown competitive performance. However, reconstruction results of those methods are coarse and visually blurred, which leads to a high rate of false detection and false pixel-level localization. To address those issues, we propose a framework called Dual-Path Re-construction Guided Segmentation Network (DRGS-Net), which determines the abnormal regions by segmenting the anomaly image with its reconstructed result as the reference template. DRGS-Net mainly consists of a novel dual-path reconstruction sub-network and a specifically designed anomaly segmentation sub-network. They are jointly trained end-to-end, with the former fusing texture repair and image reconstruction information to obtain fine-grained reconstruction results and the latter learning a decision boundary between normal and anomalous regions based on reconstruction results. On the standard benchmark dataset MVTecAD and an additional dataset DAGM, DRGS-Net shows competitive performance in image-level detection and achieves outstanding improvement in pixel-level localization. Further experiments with the few samples setting demonstrate that DRGS-Net retains strong performance with only few anomaly-free training images.
Agile gesture recognition for capacitive sensing devices: adapting on-the-job

Automated hand gesture recognition has been a focus of the AI community for decades. Traditionally, work in this domain revolved largely around scenarios assuming the availability of the flow of images of the operator's/user's hands. This has partly been due to the prevalence of camera-based devices and the wide availability of image data. However, there is growing demand for gesture recognition technology that can be implemented on low-power devices using limited sensor data instead of high-dimensional inputs like hand images. In this work, we demonstrate a hand gesture recognition system and method that uses signals from capacitive sensors embedded into the etee hand controller. The controller generates real-time signals from each of the wearer's five fingers. We use a machine learning technique to analyse the time-series signals and identify three features that can represent 5 fingers within 500 ms. The analysis is composed of a two-stage training strategy, including dimension reduction through principal component analysis and classification with K-nearest neighbour. Remarkably, we found that this combination showed a level of performance which was comparable to more advanced methods such as supervised variational autoencoder. The base system can also be equipped with the capability to learn from occasional errors by providing it with an additional adaptive error correction mechanism. The results showed that the error corrector improve the classification performance in the base system without compromising its performance. The system requires no more than 1 ms of computing time per input sample, and is smaller than deep neural networks, demonstrating the feasibility of agile gesture recognition systems based on this technology.
How Does Knowledge Injection Help in Informed Machine Learning?

Informed machine learning describes the injection of prior knowledge into learning systems. It can help to improve generalization, especially when training data is scarce. However, the field is so application-driven that general analyses about the effect of knowledge injection are rare. This makes it difficult to transfer existing approaches to new applications, or to estimate potential improvements. Therefore, in this paper, we present a framework for quantifying the value of prior knowledge in informed machine learning. Our main contributions are threefold. Firstly, we propose a set of relevant metrics for quantifying the benefits of knowledge injection, comprising in-distribution accuracy, out-of-distribution robustness, and knowledge conformity. We also introduce a metric that combines performance improvement and data reduction. Secondly, we present a theoretical framework that represents prior knowledge in a function space and relates it to data representations and a trained model. This suggests that the distances between knowledge and data influence potential model improvements. Thirdly, we perform a systematic experimental study with controllable toy problems. All in all, this helps to find general answers to the question how knowledge injection helps in informed machine learning.
A Multi-feature Integration Descriptor for Instance Image Retrieval

Deep features can provide remarkable performance in image retrieval, but their domain adaptability and robustness are still going along. To our knowledge, some classical visual features are robust to changes in orientation, scale, and transformation. It is still challenging to combine their advantages. To address this issue, we propose a new multi-feature integration descriptor for instance image retrieval. The main highlights are threefold: (1) we propose a new method to obtain the salient region map that exploits the luminance information of L*a*b* color space and Tamura textural features together with deep features. (2) Based on the salient region map, a spatial fusion scheme is proposed to integrate multiple features, which can highlight the object region and suppress irrelevant regions. (3) A new channel balancing strategy is introduced to enhance channels with necessary or discriminative patterns and weaken channels with redundant patterns in deep feature maps. Detailed experimental results show that our proposed method outperforms several recent state-of-the-art methods on five public datasets.
Exploiting ResNeXt with Convolutional Shortcut for Signal Modulation Classification at Low SNRs

Most of the current ResNeXt-based AMC models only use identity mappings as skip connections, which cannot significantly enhance the representation abilities of the network. To address this problem, we proposed an improved ResNeXt-based AMC network (CSCA-ResNeXt) in this paper. Combining convolutional shortcuts with identity skip connections avoids network degradation and fully extracts the shallow and deep features. We also introduce the CA (Coordinate Attention) to highlight important features. It not only captures cross-channel information, but also extracts positional information and long-range correlations of signal sequences. In addition, the use of GAP (Global Average Pooling) classification enhances the association between feature maps and modulation modes, improves generalization of the network and reduces the number of parameters. CSCA-ResNeXt achieves state-of-the-art classification and generalization performance on the RML2016.10a and RML2016.10b datasets, with the highest classification accuracy of 94.45% and 95.18%, respectively. Moreover, the recognition accuracy at low SNRs of -20 to 0 dB is 39.84% and 42.49%, surpassing MCLDNN by 1.89% and 1.29% respectively. Furthermore, CSCA-ResNeXt reduces the misclassification between QAM16 and QAM64.
The Generalizability of Explanations

Due to the absence of ground truth, objective evaluation of explainability methods is an essential research direction. So far, the vast majority of evaluations can be summarized into three categories, namely human evaluation, sensitivity testing, and salinity check. This work proposes a novel evaluation methodology from the perspective of generalizability. We employ an encoding-decoding module to learn the distributions of the generated explanations and observe their learnability as well as the plausibility of the learned distributional features. First we briefly demonstrate the evaluation idea of the proposed approach at LIME, and then quantitatively evaluate multiple popular explainability methods. We also find that smoothing the explanations with SmoothGrad can significantly enhance their generalizability.
SynC: A Dense Retrieval Method based on Syntactical Contrastive Learning

Recently, dense retrieval method has significantly outperformed sparse retrieval technology. It becomes a main-stream approach of relevant passage retrieving task. Dense retrieval tasks encode query and passage into dense representation space and apply contrastive learning to obtain more representative vectors, then retrieve the most similar passage by the inner product of the dense vectors. However, previous achievements rely on extremely large batch size, epoch and large-scaled pre-trained models, which results in tremendous computational resource consumption. Also, the expensive equipment prerequisite and low training efficiency constrain the development of dense retrieval community. Therefore, we present an alternative solution to improve training efficiency and quality by syntactical contrastive learning methods with specially designed masking strategy. To alleviate the computational consumption problem, this paper proposes query-based and passage-based masking strategies to obtain syntactical-isolated representations. Besides, instead of only considering query-to-passage similarity while conducting contrastive learning, we additionally consider query-to-query and passage-to-passage similarity when training the dual-encoder retriever. The experiments show that the proposed approach achieved competitive results in small batch size and epoch comparing to previous state-of-the-art dense retrieval methods, and also to strong baseline.
Solving Continual Learning with Noisy Labels by Sample Selection and Replay

One of the major distinguishing features of Continual Learning(Cl)is that training tasks will change over time, so how to adjust the model to learn different tasks is a challenge. One of the promising solutions is to use stored historical task data to help the model retain old knowledge in the training process for new tasks. However, most existing methods do not take into account that there may be noisy labels in the training data, which will aggravate the forgetting of the old task. In this paper, we propose a replay-based method to solve the continual learning with noisy labels. We first filter the data through the consistency of labels and their feature distribution in the feature space and add it to the replay buffer for the model training. We use supervised contrastive learning to train the model. In order to avoid the loss of other data information, we use the distribution of samples in the feature space to add a pseudo label. To verify the effectiveness of our algorithm, we conducted experiments on four datasets, including three artificial noise datasets MINST, cifar10, cifar100, and a real-world noise dataset Webvision, our method can achieve the best results. The experimental results show that the way we filter data and the way we update buffer data have a very important impact on the performance of the model.
Zero-Shot Text Classification with Semantically Extended Textual Entailment

Zero-shot text classification (0SHOT-TC) aims to detect classes that the model never seen in the training set, and has attracted much attention in the research community of Natural Language Processing (NLP). The emergence of pre-trained language models has fostered the progress of 0SHOT-TC, which turns the task into a textual entailment problem of binary classification. It learns an entailment relatedness (yes/no) between the given sentence (premise) and each category (hypothesis) separately. However, the hypothesis generation paradigms need to be further studied, since the label itself or the label descriptions have limited ability to fully express the category space. Conversely, humans can easily extend a set of words describing the categories to be classified. In this paper, we propose a novel zero-shot text classification method called Semantically Extended Textual Entailment (SETE), which imitates the human's ability in knowledge extension. In the proposed method, three semantic extension methods are used to enrich the categories through a combination of static knowledge (e.g. expert knowledge, knowledge graph) and dynamic knowledge (e.g. language models), and the textual entailment model is finally used for 0SHOT-TC. The experimental results on the benchmarks show that our approach significantly outperforms the current methods in both generalized and non-generalized 0SHOT-TC.
Towards Spatio-temporal Collaborative Learning: An End-to-End Deepfake Video Detection Framework

With the rapid development of facial tampering techniques, the deepfake detection task has attracted widespread social concerns. Most existing video-based methods adopt temporal convolution to learn temporal discontinuities directly, where they might neglect to explore both local detail mutation and inconsistent global expression semantics in the temporal dimension. This makes it difficult to learn more discriminative forgery cues. To mitigate this issue, we introduce a novel deepfake video detection framework specifically designed to capture fine-grained traces of tampering. Concretely, we first present a Multi-layered Feature Extraction module (MFE) that constructs comprehensive spatio-temporal representations by stitching different levels of features together. Afterward, we propose a Bidirectional temporal Artifact Enhancement module (BAE), which exploits local differences between adjacent frames to enhance frame-level features. Moreover, we present a Cross temporal Stride Aggregation strategy (CSA) to mine inconsistent global semantics and adaptively obtain multi-timescale representations. Extensive experiments on several benchmarks demonstrate that the proposed method outperforms state-of-the-art performance compared to other competitive approaches.
Attentional Opponent Modelling for Multi-agent Cooperation

Opponent modelling (OM) can enable agents to reason behaviors of others, and hence act accordingly and interact effectively in multi-agent reinforcement learning (MARL). Existing OM approaches commonly assume the availability of opponents' trajectory, which needs incessant message exchange during execution in the partially observable environment. As a result, they are not cost-effective due to communication overhead, and are also inapplicable to many practical tasks with constraints on communication. To handle this problem, we propose attentional opponent modelling (ATOM) to infer the beliefs of neighboring entities with only local observation. ATOM does not require communication during execution and can be easily combined with existing MARL methods. As examples, we propose ATOM Actor-Critic and ATOM Q-learning architectures to facilitate multi-agent cooperation. Extensive experiments on several challenging tasks (i.e., Cooperative Navigation, Predator-Prey and Starcraft Multi-Agent Challenge) show the superior performance of our methods compared with benchmark OM approaches.
IC-CViT: Inverse-Consistent Convolutional Vision Transformer for Diffeomorphic Image Registration

Diffeomorphic registration plays a crucial role in medical image analysis due to the invertible and one-to-one mapping transformation. In recent years, with the development of deep learning technology, convolutional neural networks (CNNs) have been a broad focus of research in medical image registration, and CNN-based methods have made great progress. However, the results of most existing methods generally are not necessarily diffeomorphic, generating implausibly bijective mappings between images due to the interpolation and discrete representation. Furthermore, the performances of CNNs may be limited by a lack of precise comprehension of global and long-range cross-image spatial relevance. Vision Transformer (ViT) is capable of enhancing the long-distance information interaction ability to identify the semantically anatomically correspondences of medical images. Compared with CNN, ViT has weak local feature extraction ability due to less inductive bias, especially in small-scale training datasets, meaning that the samples between adjacent pixels cannot be exploited adequately. To address the above challenges, we propose a novel Inverse-Consistent Convolutional Vision Transformer (IC-CViT) network for diffeomorphic image registration. Specifically, image pairs can explicitly conduct bi-directional registration through the predicted deformation filed, generated within the diffeomorphic mappings space and restricted by the proposed inverse consistent loss term. We verify our method on two 3D brain MRI scan datasets including OASIS and LPBA40. Comprehensive results demonstrate that IC-CViT achieves state-of-the-art registration accuracy while maintaining desired diffeomorphic properties.
Fighting Attacks on Large Character Set CAPTCHAs Using Transferable Adversarial Examples

Over a long period, large character set CAPTCHAs are widely used to defend against automated attack programs on the Internet. However, with the development of deep learning techniques, some attacks for large character set CAPTCHAs have been proposed, proving that they are no longer secure. To defend against black-box attacks on these CAPTCHAs, we propose a novel defense method based on transferable adversarial example techniques. On the one hand, we defend against character recognition attacks by adding adversarial perturbations to the characters of CAPTCHAs combining three strategies: Gradient-based Attacks, Input Transformations and Attention Mechanism. On the other hand, we defend against character detection attacks by leveraging an ensemble method to generate adversarial perturbations on the background of CAPTCHAs. To the best of our knowledge, this is the first study to improve the security of large character set CAPTCHAs against black-box attacks based on transferable adversarial example techniques. Using the eight most popular Chinese CAPTCHA schemes as examples, we conduct comprehensive experiments. Results show that our method improves the security of large character set CAPTCHAs by making the average success rate of black-box attacks significantly drop from 53.33% to 3.49%. Overall, our method can be helpful to the design of more secure large character set CAPTCHAs.
Non-Generative Energy Based Models

Energy-based models (EBM) have become increasingly popular within computer vision. EBMs bring a probabilistic approach to training deep neural networks (DNN) and have been shown to enhance performance in areas such as calibration, out-of-distribution detection, and adversarial resistance. However, these advantages come at the cost of estimating input data probabilities, usually using a Langevin based method such as Stochastic Gradient Langevin Dynamics (SGLD), which bring additional computational costs, require parameterization, caching methods for efficiency, and can run into stability and scaling issues. EBMs use dynamical methods to draw samples from the probability density function (PDF) defined by the current state of the network and compare them to the training data using a maximum log likelihood approach to learn the correct PDF. We propose a non-generative training approach, Non-Generative EBM (NG-EBM), that utilizes the Approximate Mass, identified by Grathwohl et al. [1], as a loss term to direct the training. We show that our NG-EBM training strategy retains many of the benefits of EBM in calibration, out-of-distribution detection, and adversarial resistance, but without the computational complexity and overhead of the traditional approaches. In particular, the NG-EBM approach improves the Expected Calibration Error by a factor of 2.5 for CIFAR10 and 7.5 times for CIFAR100, when compared to traditionally trained models.
Bayesian Sharpness-Aware Prompt Tuning for Cross-Domain Few-shot Learning

Few-shot learning aims to learn a classifier to recognize novel classes with only few labeled images in each class. Fine-tuning is a promising tool to solve the few-shot learning problem, which pre-trains a large-scale model on source domains and then adapts it to target domains. However, existing methods have poor generalization when encountering the domain-shifting problem in the cross-domain scenario. Inspired by recent advances on domain generalization and prompt-based tuning methods, this paper proposes Bayesian Sharpness-Aware Prompt Tuning (BSAPT) for the cross-domain few-shot learning task. Instead of learning deterministic prompts like existing methods, our BSAPT learns a weight distribution over prompts to model the uncertainty caused by limited training data and resist overfitting. To improve the generalization ability, our BSAPT seeks the prompts which lie in neighborhoods having uniformly low loss by simultaneously minimizing the training loss value and loss sharpness. Benefiting from deterministic pre-trained training and Bayesian inference, our BSAPT has better generalization ability and less overfitting than existing fine-tuning methods. Extensive experiments on public datasets show that our BSAPT outperforms state-of-the-art methods and achieves new state-of-the-art performance in the cross-domain few-shot learning task.
Unsupervised Arabic Speech Embedding Model for Speaker Identification

Speech modality has recently been explored at a large scale with the advancement of computational capabilities, forensics applications, and the availability of big data. Automation of speaker identification via advanced machine learning approaches is gaining momentum to improve various applications, particularly forensics and information security. Due to the lack of large Arabic datasets, existing speaker identification models are trained on small amounts of speech inputs which is inefficient when applied in real-life. In the present study, an unsupervised Arabic speech embedding model is created, leveraging large amounts of unlabeled speech data to help streamline the identification process for various security areas. The proposed framework is divided into two sections: unsupervised speech feature extraction and supervised speaker identification. An unlabeled Arabic speech dataset was constructed with approximately 1 million samples to develop the deep feature embedding model. Various machine learning classifiers and a deep neural network were used for speaker identification on the labeled Emirati speech database, which contains utterances of eight traditional Emirati phrases. The K-nearest neighbor classifier achieved the best result with the speech embeddings for speaker identification with an average accuracy of 93.58% using five-fold cross-validation.