Reviewerly/exHarmony huggingface dataset:
SDBC: A Novel and Effective Self-Distillation Backdoor Cleansing Approach

Deep Neural Networks (DNNs) are vulnerable to backdoor attacks, which only need to poison a small portion of samples to control the behavior of the target model. Moreover, the escalating stealth and power of backdoor attacks present not only significant challenges to backdoor defenses but also enormous potential threats to the widespread adoption of DNNs. In this paper, we propose a novel backdoor defense framework, called Self-Distillation Backdoor Cleansing (SDBC), to remove backdoor triggers from the attacked model. For the practical scenario where only a very small portion of clean data is available, SDBC first introduces self-distillation to clean the backdoor in DNNs. Extensive experiments demonstrate that SDBC can effectively remove backdoor triggers under 6 state-of-the-art backdoor attacks using less than 5% or even less than 1% clean training data without compromising accuracy. Experimental results show that the proposed SDBC outperforms existing state-of-the-art (SOTA) methods, reducing the average ASR from 95.36% to 5.75% and increasing the average ACC by 1.92%.
Design of a Multimodal Short Video Classification Model

With the development of mobile Internet, a large amount of short video data is generated on the Internet. The urgent problem of short video classification is how to better fuse the information of different multimodal information. This paper proposes a short video multimodal fusion (SV-MF) scheme based on deep learning combined with pre-trained models to complete the classification task of short video. The main innovations of the SV-MF scheme are as follows: (1) We find that text modalities contain higher-order information and tend to perform better than audio and visual modalities, and with the use of pre-trained language models, text modalities have been further improved in multimodal video classification. (2) Due to the strong semantic representation ability of text. The SVMF scheme proposes a local fusion method based on Transformer for low-order visual and audio modal information to alleviate the information deviation caused by multi-mode fusion. (3) The SV-MF scheme proposes a post processing strategy based on keywords to further improve the classification accuracy of the model. Experimental results based on a multimodal short video classification dataset derived from social networks show that the performance of the SV-MF scheme is better than the previous video fusion scheme.
Retinex Meets Transformer: Bridging Illumination and Reflectance Maps for Low-Light Image Enhancement

Low-light image enhancement, which is also known as LLIE for short, aims to reconstruct the original normal image from its low-illumination counterpart. Recently, it has received increasingly attention in image restoration. In particular, with the success of deep convolutional neural network (CNN), Retinex-based approaches have emerged as a promising line of research in LLIE, since they can well transfer adequate prior knowledge from an image captured under sufficient illumination to its low-light version for image enhancement. However, existing Retinex-based approaches usually overlook the correlation between Illumination and Reflectance maps which are both derived from the same feature extractor, leading to sub-optimal reconstructed image quality. In this study, we propose a novel Transformer architecture for LLIE, termed Bridging Illumination and Reflectance maps Transformer which is shortly BIRT. It aims to estimate the correlation between Illumination and Reflectance maps derived from Retinex decomposition within a Transformer architecture via the Multi-Head Self-Attention mechanism. In terms of model structure, the proposed BIRT comprises Retinex-based and Transformer-based sub-networks, which allow our model to elevate the image quality by learning cross-feature dependencies and long-range details between Illumination and Reflectance maps. Experimental results demonstrate that the proposed BIRT model achieves competitive performance on par with the state-of-the-arts on the public benchmarking datasets for LLIE.
Scale Worlds: Iterative refinement, evaluation, and theory-usability balance of an immersive virtual learning environment

Feedback-based iterative refinement is important in the development of any human-computer interface. The present work aims to evaluate and iteratively refine an immersive learning environment called Scale Worlds (SW), delivered via a head-mounted display (HMD). SW is a virtual learning environment encompassing scientific entities of a wide range of sizes that enables students an embodied experience while learning size and scale. Five usability experts performed think aloud while carrying out four interactive tasks in SW and compared three different design options during A/B testing. Improvement features based on the feedback from an earlier SW usability evaluation as well as HMD-specific features were examined. Usability experts completed the post-study system usability questionnaire, the NASA task load index, and a bipolar laddering survey that collected subjective perception of specific SW features. Results show that the progress panel (an improvement feature) was informative while the instructions (another improvement feature) caused clutter. The experts indicated clear usability preferences during A/B testing, which helped resolve three sets of theory-usability conflicts. The overall assessment of SW paved a path for theory-usability balance and provided valuable insights for designing and evaluating usability in immersive virtual learning environments.
Unconstrained Feature Model and Its General Geometric Patterns in Federated Learning: Local Subspace Minority Collapse

Federated Learning is a decentralized approach to machine learning that enables multiple parties to collaborate in building a shared model without centralizing data, but it can face issues related to client drift and the heterogeneity of data. However, there is a noticeable absence of thorough analysis regarding the characteristics of client drift and data heterogeneity in FL within existing studies. In this paper, we reformulate FL using client-class sampling as an unconstrained feature model (UFM), and validates the soundness of UFM in FL through theoretical proofs and experiments. Based on the model, we explored the potential information loss, the source of client drifting, and general geometric patterns in FL, called local subspace minority collapse. Through theoretical deduction and experimental verification, we provide support for the soundness of UFM and observe its predicted phenomenon, neural collapse.
Temporal Task Graph Based Dynamic Agent Allocation for Applying Behavior Trees in Multi-agent Games

Behavior Trees have gained widespread applications across diverse domains, facilitating the decomposition of complex tasks into manageable subtasks. However, an inherent challenge in maximizing the performance of BTs lies in dynamically allocating agents to subtasks as time progresses. This allocation predicament is compounded by the intricate nature of game states and the temporal variations in subtask activation. In this paper, we propose a novel approach that combines temporal task graphs with reinforcement learning to dynamically allocate agents among subtasks in BT. We employ a temporal task graph to model the dynamic activation of subtasks, where encoded vectors are multiplied by the agent's encoded observation. This enables each agent to be assigned to a specific subtask while considering comprehensive information about all subtasks. Moreover, we aggregate the Q-values of selected subtasks for all agents, leveraging this information to compute a total loss for updating the entire network. To evaluate the efficacy of our approach, we conducted extensive experiments on the challenging benchmark provided by Google Research Football. The results clearly demonstrate a significant performance improvement in BTs when leveraging our proposed framework.
Time-Warp-Invariant Processing with Multi-spike Learning

Sensory signals are encoded and processed by neurons in the brain in a form of action potentials, also called spikes that carry clue information across both spatial and temporal dimensions. Learning of such a clue information could be challenging, especially considering the case of long-delayed reward. This temporal credit assignment problem has been solved by a new concept of aggregate-label learning that motivates the development of a family of multi-spike learning algorithms whose remarkable learning performance has been demonstrated. However, most of the current spike-based learning methods are developed without consideration of input temporal fluctuations that constitute a common source of variability in sensory signals such as speech. Therefore, robust spike-based learning under fluctuations of both compression and dilation remains intriguing for exploration. In this paper, we first show the time-warp invariant characteristic of a conductance-based neuron model, based on which we then develop a new multi-spike learning algorithm for time-warp-invariant processing. Experimental results for speech recognition highlight the outstanding robustness of our algorithm against temporal distortions as compared with other relevant spike-based methods. Therefore, our study successfully confirms the effectiveness of multi-spike learning for time-warp robustness, extending a new scope for spike-based processing and learning.
Improving SLDS Performance Using Explicit Duration Variables with Infinite Support

Switching Linear Dynamical Systems (SLDS) are probabilistic graphical models used both for self-supervised segmentation and dimensionality reduction. Despite their modeling capabilities, SLDS are particularly hard to train. They oftentimes over-segment the timeseries or completely ignore some of the states, reducing the usefulness of the acquired segmentation. To improve the segmentation in Switching Linear Dynamical Systems, we introduce explicit-duration variables with infinite support. We extend the Beam Sampling algorithm to perform the efficient inference allowing for a duration distribution with infinite support. We conduct experiments on three benchmarks (two already prevalent in the state-space model literature and one demonstrating behavior in a sparse setting) that test the correctness and efficiency of our solution.
FHSI-GNN: Fusion Hierarchical Structure Information Graph Neural Network for Extractive Long Documents Summarization

Extractive text summarization aims to select salient sentences from documents. However, most existing extractive methods struggle to capture inter-sentence relations in long documents. In addition, the hierarchical structure information of the document is ignored. For example, some scientific documents have fixed chapters, and sentences in the same chapter have the same theme. To solve these problems, this paper proposes a Fusion Hierarchical Structure Information Graph Neural Network for Extractive Long Documents Summarization. The model constructs a section node containing sentence nodes and global information according to the document structure. It integrates the hierarchical structure information of the text and uses position information to identify sentences. The section node acts as an intermediary node for information interaction between sentences, which better enriches the relationships between sentences and has higher computational efficiency. Our model has achieved excellent results on two datasets, PubMed and arXiv. Further analysis shows that the hierarchical structure information of documents helps the model select salient content better.
Feature Reconstruction Distillation with Self-attention

A recently proposed knowledge distillation method based on feature map transfer verifies that the intermediate layers of the teacher model can be used as effective targets for training the student model for better generalization. Existing researches mainly focus on how to efficiently transfer knowledge between the intermediate layers of teacher and student models. However, they ignore the increase in the number of channels in the intermediate layer, which will introduce redundant feature information, and there is also a lack of interaction between shallow features and deep features. To alleviate these two problems, we propose a new knowledge distillation method called Feature Reconstruction Knowledge Distillation (FRD). By reconstructing the features of the intermediate layer of the teacher model, the student model can learn more accurate feature information. In addition, to tackle the problem that feature maps have only local information, we use a self-attention mechanism to fuse shallow and deep features, thereby enhancing the model's ability to handle global details. Through extensive experiments on different network architectures involving various teacher and student models, we observe that the proposed method significantly improves performance.
A Lightweight Text Classification Model Based on Label Embedding Attentive Mechanism

This paper presents a lightweight model based on the self-attention mechanism for text classification tasks. In our model, we incorporate auxiliary information of the label through the label embedding method, enabling the model to capture the contextual language variations of the same word. Furthermore, we address the issue of misclassification of similar texts by introducing the contrastive loss function, in conjunction with the traditional cross-entropy loss function. Experimental evaluations are conducted on multiple datasets, comparing our model against others with similar parameter scales, thus demonstrating the effectiveness of the proposed approach.
Text-to-Image Synthesis with Threshold-Equipped Matching-Aware GAN

In this paper, we propose a novel Equipped with Threshold Matching-Aware Generative Adversarial Network (ETMA-GAN) for text-to-image synthesis. By filtering inaccurate negative samples, the discriminator can more accurately determine whether the generator has generated the images correctly according to the descriptions. In addition, to enhance the discriminative model's ability to discriminate and capture key semantic information, a word fine-grained supervisor is constructed, which in turn drives the generative model to achieve high-quality image detail synthesis. Numerous experiments and ablation studies on Caltech-UCSD Birds 200 (CUB) and Microsoft Common Objects in Context (MS COCO) datasets demonstrate the effectiveness and superiority of the proposed method over existing methods. In terms of subjective and objective evaluations, the model presented in this study has more advantages than the recently available state-of-the-art methods, especially regarding synthetic images with a higher degree of realism and better conformity to text descriptions.
Minimizing Distortion in Steganography via Adaptive Language Model Tuning

Linguistic steganography, a technique that hides secret information within normal text, possesses tremendous potential in various applications such as protecting user privacy. However, previous research in linguistic steganography has primarily focused on adjusting the probability distribution of steganographic text (stegotext) to minimize the difference with text generated by language models, thereby achieving indistinguishability between the two. Nonetheless, the significant gap between real text and generated text has often been overlooked. To address this issue, this paper proposes an innovative method: using an adaptive model tuning strategy, the generated stegotext becomes statistically closer to real text. We leverage a well-trained classifier in conjunction with a fundamental generative language model to produce stegotext that aligns closely with the distribution of real text. Consequently, we gain better control over the distortion between the stegotext and real text, while effectively embedding secret information. Compared to traditional methods, our approach reduces Kullback-Leibler divergence and steganography detection rates, demonstrating its enhanced effectiveness.
Spatial and Frequency Domains Inconsistency Learning for Face Forgery Detection

With the rapid development of face forgery technology, it has attracted widespread attention. The current face forgery detection methods, whether based on Convolutional Neural Network (CNN) or Vision Transformer (ViT), are biased towards extracting local or global features respectively. These methods are relatively one-sided, and the extracted features are not robust and general enough. In this work, we exploit intra-frame inconsistency as well as inter-modal inconsistency between spatial and frequency domains to improve performance and generalization for face forgery detection. We efficiently extract intra-frame inconsistency by utilizing the capabilities of Swin Transformer. Its self-attention mechanism and attention mapping between patch embeddings naturally represent the inconsistency relations, allowing for simultaneous modeling of both local and global features, making it our ideal choice. Meanwhile, we also introduce frequency information to further improve detection performance, and design a Cross-Attention Feature Fusion (CAFF) module to exploit the inconsistency between spatial and frequency modalities to extract more general feature representations. Extensive experiments demonstrate the effectiveness of the proposed method.
TextBFA: Arbitrary Shape Text Detection with Bidirectional Feature Aggregation

Scene text detection has achieved great progress recently, however, it is challenging to detect arbitrary shaped text in the scene images with complex background, especially for those unobvious and long texts. To tackle this issue, we propose an effective text detection network, termed TextBFA, strengthening the text feature by aggregating high-level semantic features. Specifically, we first adopt a bidirectional feature aggregation network to propagate and collect information on feature maps. Then, we exploit a bilateral decoder with lateral connection to recover the low-resolution feature maps for pixel-wise prediction. Extensive experiments demonstrate the detection effectiveness of the proposed method on several benchmark datasets, especially on inconspicuous text detection.
Restore Translation Using Equivariant Neural Networks

Invariance to spatial transformations such as translations is a desirable property and a basic design principle for classification neural networks. However, the commonly used convolutional neural networks (CNNs) are actually very sensitive to even small translations. There exist vast works to achieve exact or approximate transformation invariance by designing transformation-invariant models or assessing the transformations. These works usually make changes to the standard CNNs and harm the performance on standard datasets. In this paper, rather than modifying the classifier, we propose a pre-classifier restorer to recover translated inputs to the original ones which will be fed into any classifier for the same dataset. The restorer is based on a theoretical result which gives a sufficient and necessary condition for an affine operator to be translational equivariant on a tensor space.
Anomaly Detection of Fixed-Wing Unmanned Aerial Vehicle (UAV) Based on Cross-Feature-Attention LSTM Network

With the gradual penetration of unmanned aerial vehicle (UAV) technology and its related applications in people's lives, the safety of UAVs has become an important research focus. In this paper, we present an anomaly detection method based on cross-feature-attention LSTM neural networks. In an unsupervised setting, we use two types of networks to extract temporal and spatial features from flight data and predict future states to detect abnormal flight behavior. We conduct experiments on real flight data, the Air Lab Fault and Anomaly (ALFA) dataset, using multiple sets of different feature combinations. The results indicate that our method can maintain high performance across different feature combinations, achieving an average accuracy of 0.96 and a response time of 5.01 s.
Multi-view Consistency View Synthesis

Novel view synthesis (NVS) aims to synthesize photo-realistic images depicting a scene by utilizing existing source images. The core objective is that the synthesized images are supposed to be as close as possible to the scene content. In recent years, various approaches shift the focus towards the visual effect of images in continuous space or time. While current methods for static scenes treat the rendering of images as isolated processes, neglecting the geometric consistency in static scenes. This usually results in incoherent visual experiences like flicker or artifacts in synthesized image sequences. To address this limitation, we propose Multi-View Consistency View Synthesis (MCVS). MCVS leverages long short-term memory (LSTM) and self-attention mechanism to model the spatial correlation between synthesized images, hence forcing them closer to the ground truth. MCVS not only enhances multi-view consistency but also improves the overall quality of the synthesized images. The proposed method is evaluated on the Tanks and Temples dataset, and the FVS dataset. On average, the Learned Perceptual Image Patch Similarity (LPIPS) is better than state-of-the-art approaches by 0.14 to 0.16%, indicating the superiority of our approach.
Knowledge-Distillation-Warm-Start Training Strategy for Lightweight Super-Resolution Networks

In recent years, studies on lightweight networks have made rapid progress in the field of image Super-Resolution (SR). Although the lightweight SR network is computationally efficient and saves parameters, the simplification of the structure inevitably leads to limitations in its performance. To further enhance the efficacy of lightweight networks, we propose a Knowledge-Distillation-Warm-Start (KDWS) training strategy. This strategy enables further optimization of lightweight networks using dark knowledge from traditional large-scale SR networks during warm-start training and can empirically improve the performance of lightweight models. For experiment, we have chosen several traditional large-scale SR networks and lightweight networks as teacher and student networks, respectively. The student network is initially trained with a conventional warm-start strategy, followed by additional supervision from the teacher network for further warm-start training. The evaluation on common test datasets shows that our proposed training strategy can result in better performance for a lightweight SR network. Furthermore, our proposed approach can also be adopted in any deep learning network training process, not only image SR tasks, as it is not limited by network structure or task type.
Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models

The effectiveness of large language models (LLMs) and instruction learning has been demonstrated in different pre-trained language models (such as ChatGPT). However, current prompt learning methods usually use a unified template for the same tasks, and the template is difficult to capture significant information from different instances. To integrate the semantic attention dynamically on the instance level, We propose ISPrompt, an instance-semantic-aware prompt learning model. Specifically, the instance-driven prompt generated from the semantic dependency tree is introduced. Then, the proposed model would select a suitable semantic prompt from the prompt selection pool to motivate the prompt-based fine-tuning process. Our results show that the proposed model achieves state-of-the-art performance on few-shot learning tasks, which proves that ISPrompt integrating the instance semantics dynamically could assume as a better knowledge-mining tool for PLMs.