Reviewerly/exHarmony huggingface dataset:
Lightweight Attention Network for Very High-Resolution Image Semantic Segmentation

Semantic segmentation is one of the most challenging tasks for very high resolution (VHR) remote sensing applications.Deep convolutional neural networks (CNN) based on the attention mechanism have shown outstanding performance in VHR remote sensing images semantic segmentation.However, existing attention-guided methods require the estimation of a large number of parameters that are affected by the limited number of available labeled samples that results in underperforming segmentation results.In this paper, we propose a multi-scale feature fusion lightweight model (MSFFL) to greatly reduce the number of parameters and improve the accuracy of semantic segmentation.In this model, two parallel enhanced attention modules, i.e., the spatial attention module (SAM) and the channel attention module (CAM) are designed by introducing encoding position information.Then a covariance calculation strategy is adopted to recalibrate the generated attention maps.The integration of enhanced attention modules into the proposed lightweight module results in an efficient lightweight attention network (LiANet).The performance of the proposed LiANet is assessed on two benchmark datasets.Experimental results demonstrate that LiANet can achieve promising performance with a small number of parameters.
Semantic-aware knowledge distillation with parameter-free feature uniformization

Abstract Knowledge distillation aims to distill knowledge from teacher networks to train student networks. Distilling intermediate features has attracted much attention in recent years as it can be flexibly applied in various fields such as image classification, object detection and semantic segmentation. A critical obstacle of feature-based knowledge distillation is the dimension gap between the intermediate features of teacher and student, and plenty of methods have been proposed to resolve this problem. However, these works usually implement feature uniformization in an unsupervised way, lacking guidance to help the student network learn meaningful mapping functions in the uniformization process. Moreover, the dimension uniformization process of the student and teacher network is usually not equivalent as the mapping functions are different. To this end, some factors of the feature are discarded during parametric feature alignment, or some factors are blended in some non-parametric operations. In this paper, we propose a novel semantic-aware knowledge distillation scheme to solve these problems. We build a standalone feature-based classification branch to extract semantic-aware knowledge for better guiding the learning process of the student network. In addition, to avoid the inequivalence of feature uniformization between teacher and student, we design a novel parameter-free self-attention operation that can convert features of different dimensions into vectors of the same length. Experimental results show that the proposed knowledge distillation scheme outperforms existing feature-based distillation methods on the widely used CIFAR-100 and CINIC-10 datasets.
Integrating Spatial and Temporal Information for Violent Activity Detection from Video Using Deep Spiking Neural Networks

Increasing violence in workplaces such as hospitals seriously challenges public safety. However, it is time- and labor-consuming to visually monitor masses of video data in real time. Therefore, automatic and timely violent activity detection from videos is vital, especially for small monitoring systems. This paper proposes a two-stream deep learning architecture for video violent activity detection named SpikeConvFlowNet. First, RGB frames and their optical flow data are used as inputs for each stream to extract the spatiotemporal features of videos. After that, the spatiotemporal features from the two streams are concatenated and fed to the classifier for the final decision. Each stream utilizes a supervised neural network consisting of multiple convolutional spiking and pooling layers. Convolutional layers are used to extract high-quality spatial features within frames, and spiking neurons can efficiently extract temporal features across frames by remembering historical information. The spiking neuron-based optical flow can strengthen the capability of extracting critical motion information. This method combines their advantages to enhance the performance and efficiency for recognizing violent actions. The experimental results on public datasets demonstrate that, compared with the latest methods, this approach greatly reduces parameters and achieves higher inference efficiency with limited accuracy loss. It is a potential solution for applications in embedded devices that provide low computing power but require fast processing speeds.
Robust unsupervised feature selection via data relationship learning

Unsupervised feature selection robust to many outliers is a challenging task. The crucial difficulty is learning a robust subspace, which preserves local structure. The most common solution is to reduce fitting error by applying different robust norms. However, there are three shortcomings. Firstly, they are not robust enough when outliers distributed both randomly and concentratedly are widely present. Secondly, outlier removal is not considered. Thirdly, it is not easy to understand and choose an euclidean distance threshold that decides a sample as an outlier in different scenarios. The first two shortcomings make previous methods fail to achieve their expected learning results, and the third one increases the application difficulty in different fields. To address these issues, a robust unsupervised feature selection via data relationship learning (RUFSDR) is proposed in this paper. Specifically, scores representing the data's importance will be learned and assigned to each sample. Inliers will be given different positive scores. Outliers will be given 0 such that a subspace, which preserves the local structure better, can be learned without prior knowledge about the distance threshold. The experiments conducted on various datasets with several scenarios show the superiority of RUFSDR.
Verification of Recurrent Neural Networks with Star Reachability

The paper extends the recent star reachability method to verify the robustness of recurrent neural networks (RNNs) for use in safety-critical applications. RNNs are a popular machine learning method for various applications, but they are vulnerable to adversarial attacks, where slightly perturbing the input sequence can lead to an unexpected result. Recent notable techniques for verifying RNNs include unrolling, and invariant inference approaches. The first method has scaling issues since unrolling an RNN creates a large feedforward neural network. The second method, using invariant sets, has better scalability but can produce unknown results due to the accumulation of overapproximation errors over time. This paper introduces a complementary verification method for RNNs that is both sound and complete. A relaxation parameter can be used to convert the method into a fast overapproximation method that still provides soundness guarantees. The method is designed to be used with NNV, a tool for verifying deep neural networks and learning-enabled cyber-physical systems. Compared to state-of-the-art methods, the extended exact reachability method is 10 × faster, and the overapproximation method is 100 × to 5000 × faster.
Explainable AI In Education : Current Trends, Challenges, And Opportunities

Explainable Artificial Intelligence (XAI) has garnered significant attention in recent years to increase the transparency of AI models and systems and aid in decision-making. This is particularly important in the education sector, where AI systems are being increasingly used in LMS (Learning Management Systems) and EDM (Educational Data Mining) tools to support the learning and assessment of students. In this paper, based on the existing research and literature on XAI we assert that while XAI in education shares some characteristics with the wider application of explainability approaches in other domains, it also has its own distinctive stipulations that differ from other domains. We aim to address these disparate characteristics by overviewing the XAI approaches in literature from their inception to the current state-of-the-art. We then address various back-drops to appropriately adjust these methods to facilitate favorable explanations for students and finally we identify research gaps in various parameters and suggest potential future research avenues by tethering the identified gaps with advancements in the field.
Underwater Image Enhancement based on Improved Water-Net

Enhancement for underwater image has been gradually valued with the development of ocean engineering as well as remote operated vehicles. Multifarious methods are applied in underwater image enhancing recently. Particularly, varieties of convolutional neural networks (CNN) have been applied for this field. However, during distrinct conditions lightness and medium quality, underwater images are complex and variable, which makes it much more challenging for CNN models to enhance images in different environments. In this paper, improvements are made to the architecture in Water-Net, in which we increased the enhancement units (E-Units) in the backbone of the network and improved the output of the confidence map. Experimental results on Underwater Image Enhancement Benchmark (UIEB) indicate that our network achieved better capability than other algorithms.
A binocular vision-based underwater object size measurement paradigm: Calibration-Detection-Measurement (C-D-M)

In this study, we propose a binocular vision-based underwater object size measurement paradigm: Calibration-Detection-Measurement (C-D-M). In the C-procedure, we construct an underwater refraction model to calculate basic parameters and refractive parameters of an underwater binocular vision system in waterproof housing. These parameters are key to realising the conversion of 2D pixel coordinates to 3D world coordinates. In the D-procedure, we attempt to address the difficult problem of underwater object detection via multi-feature joint perception, and accurately segment the regions of underwater objects in binocular images. Subsequently, the object regions and 3D reconstruction parameters are input into the M-procedure to measure the sizes of underwater objects. Specifically, a novel contour curve matching strategy is presented instead of dense matching to acquire reliable matching points. Subsequently, a 3D contour curve is reconstructed to calculate the key sizes of objects. Finally, comprehensive evaluations are conducted to demonstrate the effectiveness of our C-D-M paradigm.
A Novel Improved Binary Harris Hawks Optimization For High dimensionality Feature Selection

Harris Hawks Optimization (HHO) is a relatively new meta-heuristic algorithm that has shown promise in solving various optimization problems. However, HHO suffers from some limitations that can affect its performance. The two main drawbacks of HHO are population diversity and local optima. To overcome these limitations and adapt it to solve feature selection issues, a new meta-heuristic, Chaotic Opposition Harris Hawks Optimization with Simulated Annealing (COHHS), is proposed in this paper. Two main improvements are proposed for the HHO. The first one, the chaotic opposition, is applied at the initialization phase of HHO to improve the population diversity of the search agents. The second one, simulated annealing, is applied to find the optimal solution during each iteration to improve HHO exploitation. The proposed method is a dynamic structure that was originally designed to solve problems with continuous optimization. In this paper, we propose a binary COHHS (BCOHHS) using X-shaped functions to boost the efficiency of feature selection. The KNN classifier is used to evaluate the classification accuracy. The performance of the proposed method is evaluated on nine high-dimensional medical datasets and compared with other optimization methods. The experimental results confirm the superiority of the BCOHHS method over the other methods on the majority of the datasets.
An Improved Modulation Recognition Algorithm Based on Fine-Tuning and Feature Re-Extraction

Modulation recognition is an important technology in wireless communication systems. In recent years, deep learning-based modulation recognition algorithms, which can autonomously learn deep features and achieve superior recognition performance compared with traditional algorithms, have emerged. Yet, there are still certain limitations. In this paper, aiming at addressing the issue of poor recognition performance at low signal-to-noise ratios (SNRs) and the inability of deep features to effectively distinguish among all modulation types, we propose an optimization scheme for modulation recognition based on fine-tuning and feature re-extraction. In the proposed scheme, the network is firstly trained with the signals at high SNRs; then, the trained network is fine-tuned to the untrained network at low SNRs. Finally, on the basis of the features learned by the network, deeper features with enhanced discriminability for confused modulation types are obtained using feature re-extraction. The simulation results demonstrate that the proposed optimization scheme can maximize the performance of the neural network in the recognition of signals that are easily confused and at low SNRs. Notably, the average recognition accuracy of the proposed scheme was 91.28% within an SNR range of −8 dB to 18 dB, which is an improvement of 8% to 17% in comparison with four existing schemes.
Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition

Learning a set of tasks in sequence remains a challenge for artificial neural networks, which, in such scenarios, tend to suffer from Catastrophic Forgetting (CF).The same applies to End-to-End (E2E) Automatic Speech Recognition (ASR) models, even for monolingual tasks.In this paper, we aim to overcome CF for E2E ASR by inserting adapters, small architectures of few parameters which allow a general model to be fine-tuned to a specific task, into our model.We make these adapters task-specific, while regularizing the parameters of the model shared by all tasks, thus stimulating the model to fully exploit the adapters while keeping the shared parameters to work well for all tasks.Our method outperforms all baselines on two monolingual experiments while being more storage efficient and without requiring the storage of data from previous tasks.
Extended Expectation Maximization for Under-Fitted Models

In this paper, we generalize the well-known Expectation Maximization (EM) algorithm using the α−divergence for Gaussian Mixture Model (GMM). This approach is used in robust subspace detection when the number of parameters is kept small to avoid overfitting and large estimation variances. The level of robustness can be tuned by the parameter α. When α → 1, our method is equivalent to the standard EM approach and for α < 1 the method is robust against potential outliers. Simulation results show that the method outperforms the standard EM when it comes to mismatches between noise models and their realizations. In addition, we use the proposed method to detect active brain areas using collected functional Magnetic Resonance Imaging (fMRI) data during task-related experiments.
A Multi-Stage Hierarchical Relational Graph Neural Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis targets at accurately perceiving the emotional states by incorporating related information from multiple sources. However, existing methods mostly neglect the unbalanced contributions and inherent relational interactions across distinct modalities. In this paper, we propose a multi-stage hierarchical relational graph neural network (MHRG), catering to intra- and inter-modal dynamics learning with modality calibration. In the first stage, modality-specific graph convolution modules are introduced to learn the intra-modal sequential semantics. In the second, we design a modality-adaptive modification module to determine the contribution of each modality based on the prediction confidence. Finally, diverse inter-modal dynamics are considered respectively by a novel hierarchical relational graph fusion method for further aggregation according to the type of interactions. Extensive experiments on benchmark datasets demonstrate that MHRG outperforms the existing methods and achieves the state-of-the-art performance.
Super-Resolution Information Enhancement for Crowd Counting

Crowd counting is a challenging task due to the heavy occlusions, scales, and density variations. Existing methods handle these challenges effectively while ignoring low-resolution (LR) circumstances. The LR circumstances weaken the counting performance deeply for two crucial reasons: 1) limited detail information; 2) overlapping head regions accumulate in density maps and result in extreme ground-truth values. An intuitive solution is to employ super-resolution (SR) pre-processes for the input LR images. However, it complicates the inference steps and thus limits application potentials when requiring real-time. We propose a more elegant method termed Multi-Scale Super-Resolution Module (MSSRM). It guides the network to estimate the lost details and enhances the detailed information in the feature space. Noteworthy that the MSSRM is plug-in plug-out and deals with the LR problems with no inference cost. As the proposed method requires SR labels, we further propose a Super-Resolution Crowd Counting dataset (SR-Crowd). Extensive experiments on three datasets demonstrate the superiority of our method. The code will be available at https://github.com/PRIS-CV/MSSRM.git.
Meta Learning for Domain Agnostic Soft Prompt

The prompt-based learning, as used in GPT-3, has become a popular approach to extract knowledge from a powerful pre-trained language model (PLM) for natural language understanding tasks. However, either applying the hard prompt for sentences by defining a collection of human-engineering prompt templates or directly optimizing the soft or continuous prompt with labeled data may not really generalize well for unseen domain data. To cope with this issue, this paper presents a new prompt-based unsupervised domain adaptation where the learned soft prompt is able to boost the frozen pre-trained language model to deal with the input tokens from unseen domains. Importantly, the meta learning and optimization is developed to carry out the domain agnostic soft prompt where the loss for masked language model is minimized. The experiments on multi-domain natural language understanding tasks show the merits of the proposed method.
Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-to-Speech

This paper proposes Virtuoso, a massively multilingual speech–text joint semi-supervised learning framework for text-to-speech synthesis (TTS) models. Existing multilingual TTS typically supports tens of languages, which are a small fraction of the thousands of languages in the world. One difficulty to scale multilingual TTS to hundreds of languages is collecting high-quality speech–text paired data in low-resource languages. This study extends Maestro, a speech–text joint pretraining framework for automatic speech recognition (ASR), to speech generation tasks. To train a TTS model from various types of speech and text data, different training schemes are designed to handle supervised (paired TTS and ASR data) and unsupervised (untranscribed speech and unspoken text) datasets. Experimental evaluation shows that 1) multilingual TTS models trained on Virtuoso can achieve significantly better naturalness and intelligibility than baseline ones in seen languages, and 2) they can synthesize reasonably intelligible and naturally sounding speech for unseen languages where no high-quality paired TTS data is available.
Improving aspect-based sentiment analysis with contrastive learning

As a fine-grained sentiment analysis task focusing on detecting the sentiment polarity of aspect(s) in a sentence, aspect-based sentiment analysis (ABSA) plays a significant role in opinion analysis and review analysis. Recently, a number of methods have emerged to leverage contrastive learning techniques to enhance the performance of ABSA by learning fine-grained sentiment representations. In this paper, we present and compare two commonly used contrastive learning approaches for enhancing ABSA performance: sentiment-based supervised contrastive learning and augmentation-based unsupervised contrastive learning. Sentiment-based supervised contrastive learning employs sentiment labels to distinguish between positive and negative samples. Augmentation-based unsupervised contrastive learning aims to utilize various data augmentation strategies to generate positive samples. Experimental results on three public ABSA datasets demonstrate that both contrastive learning methods significantly improve the performance of ABSA. Sentiment-based supervised contrastive learning outperforms augmentation-based unsupervised contrastive learning in terms of overall performance improvements. Furthermore, we conduct additional experiments to illustrate the effectiveness and generalizability of these two contrastive learning approaches. The experimental code and data are publicly available at the link: https://github.com/Linda230/ABSA-CL.
Analyzing Acoustic Word Embeddings from Pre-Trained Self-Supervised Speech Models

Given the strong results of self-supervised models on various tasks, there have been surprisingly few studies exploring self-supervised representations for acoustic word embeddings (AWE), fixed-dimensional vectors representing variable-length spoken word segments. In this work, we study several pre-trained models and pooling methods for constructing AWEs with self-supervised representations. Owing to the contextualized nature of self-supervised representations, we hy-pothesize that simple pooling methods, such as averaging, might already be useful for constructing AWEs. When evaluating on a standard word discrimination task, we find that HuBERT representations with mean-pooling rival the state of the art on English AWEs. More surprisingly, despite being trained only on English, HuBERT representations evaluated on Xitsonga, Mandarin, and French consistently outperform the multilingual model XLSR-53 (as well as Wav2Vec 2.0 trained on English).
Robust Data2VEC: Noise-Robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning

Self-supervised pre-training methods based on contrastive learning or regression tasks can utilize more unlabeled data to improve the performance of automatic speech recognition (ASR). However, the robustness impact of combining the two pre-training tasks and constructing different negative samples for contrastive learning still remains unclear. In this paper, we propose a noise-robust data2vec for self-supervised speech representation learning by jointly optimizing the contrastive learning and regression tasks in the pre-training stage. Furthermore, we present two improved methods to facilitate contrastive learning. More specifically, we first propose to construct patch-based non-semantic negative samples to boost the noise robustness of the pre-training model, which is achieved by dividing the features into patches at different sizes (i.e., so-called negative samples). Second, by analyzing the distribution of positive and negative samples, we propose to remove the easily distinguishable negative samples to improve the discriminative capacity for pre-training models. Experimental results on the CHiME-4 dataset show that our method is able to improve the performance of the pre-trained model in noisy scenarios. We find that joint training of the contrastive learning and regression tasks can avoid the model collapse to some extent compared to only training the regression task.