Reviewerly/exHarmony huggingface dataset:
Global-Temporal Enhancement for Sign Language Recognition

The continuous sign language recognition task is challenging which needs to identify unsegmented gloss from long videos in a weakly supervised manner. Some previous methods hope to extract information of different modalities to enhance the representation of features, which often complicates the network and focuses too much on visual features. Sign language data is a long video, as the time span increases, the model may forget the information of early time steps. The long-distance temporal modeling ability directly affects the recognition performance. Therefore, a Global-Temporal Enhancement (GTE) module is proposed to enhance temporal learning ability. Most of the current continuous sign language recognition networks have a three-step architecture, i.e., visual, sequence and alignment module. However, such architecture is difficult to get enough training under current Connectionist Temporal Classification (CTC) losses. So two auxiliary supervision methods are proposed, namely Temporal-Consistency Self-Distillation (TCSD) and GTE loss. TCSD uses two global temporal outputs from different depths to supervise local temporal information. GTE loss can provide a moderate supervision to balance the features extracted by deep and shallow layers. The proposed model achieves state-of-the-art or competitive performance on PHOENIX14, PHOENIX14-T datasets.
Neural Field Conditioning Strategies for 2D Semantic Segmentation

Abstract Neural fields are neural networks which map coordinates to a desired signal. When a neural field should jointly model multiple signals, and not memorize only one, it needs to be conditioned on a latent code which describes the signal at hand. Despite being an important aspect, there has been little research on conditioning strategies for neural fields. In this work, we explore the use of neural fields as decoders for 2D semantic segmentation. For this task, we compare three conditioning methods, simple concatenation of the latent code, Feature-wise Linear Modulation (FiLM), and Cross-Attention, in conjunction with latent codes which either describe the full image or only a local region of the image. Our results show a considerable difference in performance between the examined conditioning strategies. Furthermore, we show that conditioning via Cross-Attention achieves the best results and is competitive with a CNN-based decoder for semantic segmentation.
Contour-Augmented Concept Prediction Network for Image Captioning

Semantic information in images is essential for image captioning. However, previous works leverage the pre-trained object detector to mine semantics in an image, making the model unable to accurately capture visual semantics, and further making the generated descriptions irrelevant to the content of the given image. Thus, in this paper, we propose a Contour-augmented Concept Prediction Network (CCP-Net), which leverages two additional aspects of visual information, including high-level features (concepts) and low-level features (contours) in an end-to-end manner, to encourage the contribution of visual content in description generation. Furthermore, we propose a contour-augmented visual feature extraction module and equip it with elegantly designed feature fusion. Utilizing homogeneous contour features can better enhance visual feature extraction and further promote visual concept prediction. Extensive experimental results on MS COCO dataset demonstrate the effectiveness of our method and each proposed module, which can obtain 40.6 BLEU-4 and 135.6 CIDEr scores. Code will be released in the final version of the paper.
A Lightweight Multi-Scale Large Kernel Attention Hierarchical Network for Single Image Deraining

Many current single image deraining methods focus on increasing network depth and width, which results in higher computational overhead and parameters. To address this issue, we propose a lightweight multi-scale large kernel attention hierarchical network(LMANet). Our approach combines multi-scale and Large Kernel Attention(LKA) to create Multi-Scale Large Kernel Attention (MSLKA), where large kernel decomposition can effectively decouple large kernel convolution operations to capture long-term dependencies at different granularity levels in a light weight. We then use the Channel Attention Feed-Forward Network (CAFN), which employs channel attention and gating mechanisms to correct channel information and reduce redundant features, thus further reducing the network size. Finally, we enhanced the inter-layer information learning of the feature maps by adding top-down Self Attention Distillation (SAD) to the network training process, thus accelerating the network training and alleviating 38 $$\%$$ of the network parameters. Experimental results on benchmark datasets show that LMANet achieves satisfactory performance while significantly reducing the number of parameters.
Adaptive Embedding and Distribution Re-margin for Long-Tail Recognition

Visual recognition methods assume models will be evaluated on the same class distribution as training data, but real-world data is often heavily class-imbalanced. To address this, the essential idea is to provide discriminative fitting abilities for classes with different sample sizes, i.e., the model achieves better generalization on less frequent classes, while maintaining high classification ability on the recurring classes. In this work, we propose to unify representation learning and classification learning with robust margin adjustment, which enforces a suitable margin in logit space and regularizes the distribution of embeddings. This procedure reduces representation bias in the feature space and reduces classification bias in the logit space at the same time. We further augment the under-represented tail classes on the feature level via re-balanced sampling from the robust prototype, calibrated with the knowledge from well-represented head classes and adaptive embedding uncertainty estimation. We conduct extensive experiments on a common long-tailed benchmark CIFAR100-LT. Experimental results demonstrate the advantage of the proposed AMDRG for the long-tailed recognition problem.
Higher Target Relevance Parallel Machine Translation with Low-Frequency Word Enhancement

Non-autoregressive translation (NAT) has received a surge of interest due to its success in inference speed by predicting all tokens independently and simultaneously. However, it is difficult for this paradigm to model the conditional information between words in the target side, which means its translation accuracy is sacrificed and damaged. Although many advanced studies are proposed to improve its generation quality, they come at the cost of decoding speed compared to its counterpart. In this paper, we propose to introduce an evaluation module to evaluate the NAT generations during training to guide model parameter update, and as a fine-tuning module during inference to generate plentiful fluency and faithfulness predictions. This recipe can significantly improve the model performance on the basis of ensuring the decoding efficiency of NAT. Furthermore, to mitigate the large prediction error of low-frequency words caused by knowledge distillation (KD) in non-autoregressive generation, we supply an enhanced KD to train NAT students, which exploits the complementarity of bilingual and monolingual, and transfer both knowledge to the NAT model. We not only verify our ideas on widely-used WMT14 English-German and WMT16 Romanian-English tasks, but also make more amelioration on the low-resource national languages CCMT2019 Mongolian-Chinese and CWMT2017 Uyghur-Chinese.
Ms-AMPool: Down-Sampling Method for Dense Prediction Tasks

In recent years, CNN-based neural networks have continuously proven to have good performance in the field of computer vision. Dense prediction tasks (object detection and image segmentation), as fundamental tasks in the field of computer vision, the CNN models currently designed for them rely mainly on the construction of multi-scale information. Pooling layer, as a low-cost down-sampling component for building such models, effectively saves the number of model parameters and computational effort. The existing mainstream pooling methods not only lose a lot of effective feature information during down-sampling, but also do not perform multi-scale fusion of feature information during down-sampling, which greatly affects the final performance of the network. To this end, we propose an adaptive mixed pooling method. It is designed to allow the network to achieve adaptive mixing of pooling methods by learning the relevant weight parameters, so that the network can retain as much valid feature information as possible when down-sampling. Further, we construct a multi-scale adaptive mixed pooling method that can ensure that the network performs adaptive fusion of multi-scale feature information when down-sampling. We have conducted related experiments in yolo series object detection network as well as UNet and UNeXt image segmentation networks, and the results show the superiority of our method compared to other mainstream pooling methods for performance and parametric balance in dense prediction tasks.
Glancing Text and Vision Regularized Training to Enhance Machine Translation

Bilingual parallel sentences, combined with visual annotations, created an innovative machine translation scenario within the encoder-decoder framework, known as multimodal machine translation. In generally, it was encoded as an additional visual representation to enhance the dependent-time context vector when generating the target translation word by word. However, this approach only simulated the consistency between the visual annotation and the source language and did not sufficiently consider the consistency among the source language, the target language, and the visual context. To address this problem, we proposed a novel method that adds visual features to both the encoder and decoder. In the encoder, we designed a cross-modal correlation mechanism to effectively integrate textual and visual information. In the decoder, we designed a multimodal graph to enhance the related information of vision and text. Experimental results showed that the proposed approach significantly improved translation performance compared to strong baselines for the English-German/French language pairs. The ablation study further confirmed the effectiveness of the proposed approach in improving translation quality.
Limited Information Opponent Modeling

The goal of opponent modeling is to model the opponent policy to maximize the reward of the main agent. Most prior works fail to effectively handle scenarios where opponent information is limited. To this end, we propose a Limited Information Opponent Modeling (LIOM) approach that extracts opponent policy representations across episodes using only self-observations. LIOM introduces a novel policy-based data augmentation method that extracts opponent policy representations offline via contrastive learning and incorporates them as additional inputs for training a general response policy. During online testing, LIOM dynamically responds to opponent policies by extracting opponent policy representations from recent historical trajectory data and combining them with the general policy. Moreover, LIOM ensures a lower bound on expected rewards through a balance between conservative and exploitation. Experimental results demonstrate that LIOM is able to accurately extract opponent policy representations even when the opponent’s information is limited, and has a certain degree of generalization ability for unknown policies, outperforming existing opponent modeling algorithms.
Diversifying Non-dissipative Reservoir Computing Dynamics

The Euler State Network (EuSNs) model is a recently proposed Reservoir Computing methodology that provides stable and non-dissipative untrained dynamics by discretizing an appropriately constrained ODE. In this paper, we propose alternative formulations of the reservoirs for EuSNs, aiming at improving the diversity of the resulting dynamics. Our empirical analysis points out the effectiveness of the proposed approaches on a large pool of time-series classification tasks.
Joint Demosaicing and Denoising with Frequency Domain Features

Image demosaicing and denoising play important roles in modern digital camera image processing and have been studied in recent decades. Due to the interference of independent demosaicing and denoising causes color errors and the loss of image details, therefore, the joint demosaicing and denoising problem has been extensively studied. Although the simple combination of existing demosaicing and denoising methods can obtain recovery images, there are still considerable room for improvement. In this paper, we propose an end-to-end frequency domain features network (FFNet) to solve the ill-posed joint problem. Different from previous methods based on spatial domain features, FFNet uses frequency domain information to learn global and local image features, and it is based on the vision Transformer architecture. Our model contains mainly two frequency domain feature blocks: a global Fourier block (GFB) that learns a global frequency domain parameter weight to achieve global attention and a local Fourier block (LFB) based on multi-layer perceptron to enhance local feature learning. These two modules and channel attention are combined into the frequency domain attention block (FAB), which constitutes the core module of FFNet. Extensive experiments demonstrate that FFNet outperforms previous approaches and achieves state-of-the-art performance.
Gradient-Based Learning of Finite Automata

The inference of regular grammars from recurrent neural networks has usually been a process that is performed after the network has been trained. In this work, we propose two implementations of a recurrent neural network that model the behavior of weighted finite automata. These models are trained through automatic differentiation in order to learn the Tomita grammars, and regularization techniques are explored to ensure determinism. The learned automata are interpretable from the model weights, requiring little to no transformation to be obtained. This brings closer the training and inference phases, thus optimizing the learning process.
TS-GCN: A novel tumor segmentation method integrating transformer and GCN

&lt;abstract&gt;&lt;p&gt;As one of the critical branches of medical image processing, the task of segmentation of breast cancer tumors is of great importance for planning surgical interventions, radiotherapy and chemotherapy. Breast cancer tumor segmentation faces several challenges, including the inherent complexity and heterogeneity of breast tissue, the presence of various imaging artifacts and noise in medical images, low contrast between the tumor region and healthy tissue, and inconsistent size of the tumor region. Furthermore, the existing segmentation methods may not fully capture the rich spatial and contextual information in small-sized regions in breast images, leading to suboptimal performance. In this paper, we propose a novel breast tumor segmentation method, called the transformer and graph convolutional neural (TS-GCN) network, for medical imaging analysis. Specifically, we designed a feature aggregation network to fuse the features extracted from the transformer, GCN and convolutional neural network (CNN) networks. The CNN extract network is designed for the image's local deep feature, and the transformer and GCN networks can better capture the spatial and context dependencies among pixels in images. By leveraging the strengths of three feature extraction networks, our method achieved superior segmentation performance on the BUSI dataset and dataset B. The TS-GCN showed the best performance on several indexes, with Acc of 0.9373, Dice of 0.9058, IoU of 0.7634, F1 score of 0.9338, and AUC of 0.9692, which outperforms other state-of-the-art methods. The research of this segmentation method provides a promising future for medical image analysis and diagnosis of other diseases.&lt;/p&gt;&lt;/abstract&gt;
Bring the Noise: Introducing Noise Robustness to Pretrained Automatic Speech Recognition

Abstract In recent research, in the domain of speech processing, large End-to-End (E2E) systems for Automatic Speech Recognition (ASR) have reported state-of-the-art performance on various benchmarks. These systems intrinsically learn how to handle and remove noise conditions from speech. Previous research has shown, that it is possible to extract the denoising capabilities of these models into a preprocessor network, which can be used as a frontend for downstream ASR models. However, the proposed methods were limited to specific fully convolutional architectures. In this work, we propose a novel method to extract the denoising capabilities, that can be applied to any encoder-decoder architecture. We propose the Cleancoder preprocessor architecture that extracts hidden activations from the Conformer ASR model and feeds them to a decoder to predict denoised spectrograms. We train our preprocessor on the Noisy Speech Database (NSD) to reconstruct denoised spectrograms from noisy inputs. Then, we evaluate our model as a frontend to a pretrained Conformer ASR model as well as a frontend to train smaller Conformer ASR models from scratch. We show that the Cleancoder is able to filter noise from speech and that it improves the total Word Error Rate (WER) of the downstream model in noisy conditions for both applications.
Fuzzy knowledge spaces based on $ \beta $ evaluation criteria

&lt;abstract&gt;&lt;p&gt;In KST, it is always assumed that the knowledge state represents items that an individual can solve in ideal conditions. Namely, the answers of individuals to items can be encoded as either correct or incorrect. The correct answer indicates a complete mastery of the item, but the incorrect answer may indicate a partial mastery of the item. It is reasonable to use a fuzzy knowledge state to represent the partial mastery of items instead of complete mastery. The fuzzy knowledge state of an individual is represented by a fuzzy set in $ \mathcal{F}(Q) $ that the individual is capable of solving. For any fuzzy knowledge state, each item has a value that represents the level of individual mastery of the item. Fuzzy knowledge spaces and fuzzy learning spaces are generalizations of knowledge spaces and learning spaces. The generalization based on partial order is helpful to distinguish the equally informative items, which can directly induce a discriminative fuzzy knowledge structure. It is effective to use fuzzy knowledge spaces and fuzzy learning spaces to assess knowledge and guide further learning. A fuzzy knowledge space and a fuzzy learning space can be faithfully summarized by the fuzzy knowledge basis, since they are union-closed. Any fuzzy knowledge state of a fuzzy knowledge space can be generated by forming the union of some fuzzy knowledge states in the basis. A fuzzy knowledge basis is a generalization of the knowledge basis of a knowledge space.&lt;/p&gt;&lt;/abstract&gt;
Electronic Medical Record Recommendation System Based on Deep Embedding Learning with Named Entity Recognition

Electronic medical records (EMR) provide valuable insights into patients’ medical history, symptoms, and treatments. Similar EMRs can help clinicians make an accurate diagnosis and develop an appropriate treatment plan for their patients, which makes the EMR recommendation a hot topic. However, searching for similar cases in a database containing many EMRs would be labor-intensive, while a recommendation system can return results quickly. In order to improve the recommendation accuracy and reduce the time-consuming at the same time, a similar EMR recommendation framework is proposed, which consists of three parts: Data Preprocessing, Prefetching, and Similarity Assessment. In the preprocessing module, named entities are extracted by a Named Entity Recognition (NER) tool and sliced by a particular rule. In the prefetching module, a pretrained deep learning model is fine-tuned on a classification task and generates embeddings for EMRs to avoid redundant calculations and filter candidate samples by computing cosine similarity. Furthermore, Weight-DSC is proposed to assess the similarity of EMR pairs, which is calculated by entity frequency and outperforms other entity-based methods. Experiments on real data show that this framework demonstrates superior recommendation performance compared to previous approaches across all metrics and saves 2/3 of the query time, notably achieving a 2.89% increase in Mean Reciprocal Rank (MRR).
A Unified Video Semantics Extraction and Noise Object Suppression Network for Video Saliency Detection

Video salient object detection (VSOD) aims to segment the most attractive objects from a video sequence. Exploring video semantics and suppressing noise objects are two challenges in the VSOD. In this paper, we propose a unified end-to-end network with video Semantics Extraction and Noise Object suppression (SENO). SENO has two modules, including a video semantics module (VSM) and a contrastive learning module (CLM). VSM extracts video semantics by calculating global pixel correspondences, locating the video salient objects. CLM pulls close video foregrounds and pushes away interference objects, which enhances effective video salient features and suppresses noise objects. CLM is only applied during training, avoiding extra overhead during inference. Besides, our SENO does not use the pre-processing temporal modeling techniques such as optical flow methods, which avoids high computational costs and accumulated inaccuracies caused by these complex models. Experimental results on five benchmark testing datasets show that our SENO outperforms state-of-the-art methods. In addition, the proposed SENO can detect results in real-time.
Space-Time Video Super-Resolution Based on Long-Term Time Dependence

Space-Time Video Super Resolution (STVSR) is designed to recover low resolution (LR), low frame rate (LFR) video into video with higher resolution (HR) and high frame rate (HFR). Recently, good STVSR performance has been achieved by deformation convolution-based methods. However, all current STVSR models synthesize the embedding of missing frame features by the short-term temporal information of two adjacent frame features, ignoring the role of long-term temporal information of other consecutive input frames for missing frame synthesis. In addition, it is difficult for the existing STVSR to explicitly utilize space-time correlation features to assist in high-resolution frame reconstruction. To alleviate these problems. In this paper, we propose a long-term time-dependent space-time video super-resolution network (LTTDN) based for STVSR. The network design as a bidirectional deformable convolutional gated cyclic interpolation (BDCGCI) module for STVSR. the BDCGCI module is able to extract long-term temporal information from more input frames and effectively use the long-term temporal information of video frames for interpolation. In addition, we propose a spatial-temporal attention fusion module. This module adaptively aggregates the spatial-temporal features in video frames. In this way, it helps high-resolution frame reconstruction. Extensive experiments on several benchmark datasets show that our method outperforms the state-of-the-art STVSR method.