Reviewerly/exHarmony huggingface dataset:
Collaborative Spatial-Temporal Distillation for Efficient Video Deraining

In this paper, we propose a novel knowledge distillation framework to improve the efficiency of deep networks for video deraining. The knowledge is transferred from a large-scale powerful teacher network to a compact efficient student network via the proposed collaborative spatial-temporal distillation framework. The framework is equipped with three collaboration schemes of different granularities that make use of spatial-temporal redundancy in a complementary way for better distillation performance. First, the spatial alignment module applies distillation constraints at different spatial scales to achieve better scale invariance in transferred knowledge. Second, the temporal alignment module traces both temporal status between teacher and student separately and collaboratively, to comprehensively utilize inter-frame information. Third, these two alignment modules interact through a spatial-temporal adaptor, where spatial-temporal knowledge is transferred in a unified framework. Extensive experiments demonstrate the superiority of our distillation framework as well as the effectiveness of each module. Our code is available at: https://github.com/HuYuzhang/Knowledge-Distillation.
DDT: Dual-branch Deformable Transformer for Image Denoising

Transformer is beneficial for image denoising tasks since it can model long-range dependencies to overcome the limitations presented by inductive convolutional biases. However, directly applying the transformer structure to remove noise is challenging because its complexity grows quadratically with the spatial resolution. In this paper, we propose an efficient Dual-branch Deformable Transformer (DDT) denoising network which captures both local and global interactions in parallel. We divide features with a fixed patch size and a fixed number of patches in local and global branches, respectively. In addition, we apply deformable attention operation in both branches, which helps the network focus on more important regions and further reduces computational complexity. We conduct extensive experiments on real-world and synthetic denoising tasks, and the proposed DDT achieves state-of-the-art performance with significantly fewer computational costs.
Efficient Video Matting on Human Video Clips for Real-Time Application

This paper presents an efficient and effective matting framework for human video clips. To alleviate the inefficiency problem in existing models, we propose using a refiner dedicated to error-prone regions, and reduce the computation at higher resolutions, so the proposed framework can achieve real-time performance for 1080p 60fps videos. Also, with the recurrent architecture, our model is aware of temporal information and produces temporally more consistent matting results compared to models processing each frame individually. Moreover, it contains a module for capturing semantic information. That makes our model easy to use without troublesome setup, such as annotating trimaps or other additional inputs. Experiments show that our proposed method outperforms previous matting methods, and reaches the state of the art on the VideoMatte240K dataset.
A Robust Generative Image Steganography Method based on Guidance Features in Image Synthesis

The existing generative steganography methods have the limitations of the low capacity and poor stego quality. The target image which are served as guidance features and used to translate the image from the original to special one during the synthetic process. These guidance features are abundant, stable and do not contain the identity information which can be used as cover in steganography. This paper proposed a generative image steganography by using the guidance features in image synthesis, and a secret fusion algorithm is proposed to solve the problems of guidance features embedding and extraction errors. Due to the robustness of styles and attributes, the embedded guidance features can be extracted directly in receiver side from the synthesized image without code book or database. Compared with the existing generative steganography methods, the proposed method can achieve a higher security and quality while maintaining a larger embedding capacity.
Evaluating the Performance Acceleration of Generalized Linear Solver using Normal Equation on Three Architectures for Tall Skinny Datasets

In previous work, we effectively applied a Normal Equation method to solve the most draining task in the Generalized Linear Model training process on a tall-skinny real-world dataset. This paper generalizes this method by applying it to synthetic data in various sizes. Besides, we evaluated the method on a wider column of data to evaluate the scalability. In addition, we measured and made a comparison of the execution on three different architectures: Vector Machine, an up-to-date GPGPU, and x86 CPU, along with various compilers and BLAS implementations.
A Pseudo-Dual Self-Rectification Framework for Semantic Segmentation

Semantic segmentation has achieved remarkable success in various applications. However, the training process for such techniques necessitates a significant amount of labeled data. Although semi-supervised frameworks can alleviate this issue, traditional approaches typically require multiple baseline models to form a dual model. To allow a semi-supervised semantic segmentation framework to be used in robotic systems with precious computation and memory resources, we propose a framework utilizing a single baseline model only. The overall framework is composed of three parts: an encoder, a shallow decoder, and a deep decoder. It distills knowledge from the ensemble of two decoders to improve the encoder, which can implicitly form a pseudo-dual model. It also calculates class-wise likelihoods according to the similarity between features and class prototypes learned from different decoders and rectifies low-confidence pseudo-labels. Our framework outperforms state-of-the-art frameworks on benchmark datasets with a significant amount of decrease in using computing resources.
Serverless Cloud Functions - Opportunity in Chaos

Due to its cost-effectiveness and limited scope of administration, Serverless Computing has fast become a favorite cloud computing execution model. Meanwhile, with the rise of distributed cloud architectures and microservices in the last decade, many development teams have adopted the principles of Chaos Engineering. This allows them to assess the effects of random failures or delays on an application. In prior literature, serverless developers measured and reported cold-start penalties and transaction latency, whereas Chaos Engineers have studied security and resiliency in cloud infrastructure. In this work, we combine these approaches to measure the performance of a set of serverless cloud functions which implement common server-side file and database operations. We study each function's performance response under a set of controlled chaos experiments, wherein we emulate various client load conditions, as well as inject random delays into the function execution. We find that under heavy 1000-client load, the longest-latency operations can provide as much as 36.5% improvement to overall response time by failing early.
HPGER: Integrating Human Perception Into Group Emotion Recognition

Over the past few years, deep neural networks have been widely employed for representation learning and achieved remarkable success in many computer vision tasks, such as visual sentiment analysis and emotion recognition. However, identifying image sentiments similar to what humans do is challenging due to the complexity of raw images and the intangible nature of human visual perception. Besides, training a deep learning model from scratch for a complex task such as group emotion recognition is very time-consuming and requires a large amount of labeled data representing the total population which is practically infeasible. In addition, creating annotated data for each specific task is costly and sometimes impossible. So instead, we can use the knowledge extracted by a model trained on related existing labeled datasets. To address the above challenges, we propose an end-to-end group emotion recognition framework that integrates human perception learned from human eye fixation data to efficiently and effectively extract the sentiment of input images and classify them as positive, negative, or neutral. The proposed architecture aims to leverage the information in eye fixation data to learn how humans perceive and interpret the visual world in a free-viewing task. Our framework can be utilized in any free-viewing task, potentially reducing the need for a large amount of labeled data and speeding up the training phase. In the following sections, The following sections outline the initial steps and describe different parts of our architecture.
Medical Image Super-Resolution via Diagnosis-Guided Attention

Medical image super-resolution (SR) is an important medical image processing task and is often helpful for downstream medical analysis tasks. Most of the conventional SR methods tried to generate visually more convincing images whereas ignoring the following downstream tasks. In this paper, we take the Alzheimer’s disease diagnosis as the downstream task and propose a novel diagnosis-guided medical image SR network, which can make the SR and diagnosis be boosted by each other. The method contains two sub-networks, i.e., the SR network and the diagnosis network. To achieve better diagnosis performance, in the SR network, we apply the deformable convolution to capture the regions of interest (ROIs) with different and irregular sizes and shapes, which are important for diagnosis. Moreover, to integrate the two tasks, i.e., SR and diagnosis, more profoundly, we design a novel diagnosis-guided attention module, which makes the key regions for diagnosis can be reconstructed more clearly by the SR network. The extensive experiments on medical image data sets show that the proposed method often outperforms other state-of-the-art SR methods, which demonstrates its effectiveness. The codes of this paper are released in https://github.com/WJingwei/SRDA.
A Super-Resolution Method for Small Object Detection in Road Scenes

For an autonomous vehicle, early detection of small objects such as traffic signs or small road hazards is very important, especially when the vehicle is traveling at high speed. However, due to the limited size and low quality of small objects, many common object detection networks performed poorly. Detection networks specially designed for small objects are needed. In this paper, we propose a novel network with a super-resolution method consisting of four steps. First, a spatial prior distribution model is carried out to generate a focus region. Second, a localization block is designed to find potential small object positions in the focus region. Third, a super-resolution block for feature enhancement is applied to small object locations and outputs a super-resolution image. Finally, a classification block is designed to distinguish the super-resolution image's class. We test our network on Tongji Long-Distance Road Object (TJ-LDRO) and Tsinghua-Tencent 100K (TT100K) datasets, and the results demonstrate the effectiveness of our method.
LACL: Lesion-Aware Contrastive Learning Framework for Medical Image Classification

Recently, contrastive learning has received significant attention in various classification tasks of natural images. However, current contrastive learning frameworks display unsatisfactory performance on medical images due to the inability of obtaining fine-grained visual features. In this paper, we propose a Lesion-Aware Contrastive Learning (LACL) framework to learn more discriminative and comprehensive representations and reinforce the attention of diagnosis regions on medical images. LACL framework includes two phases of training procedures: the comprehensive feature-extracting phase and the contrastive learning enhancement phase. In the first phase, LACL fully captures meaningful deep features related to the training targets to form comprehensive visual representations, by training a novel lesion-aware module we proposed. In the second phase, we introduce the previous representation information into contrastive learning to guide the LACL framework in learning disease- related features. This approach provides more effective guidance than the traditional contrastive learning method of directly comparing features. Extensive experiments on several benchmark datasets demonstrate that our LACL framework significantly improves the performance of medical image classification and highlights the lesion areas for disease diagnosis.
An Integration Enhanced Repetitive Path Planning Scheme for Omnidirectional Mobile Robot Manipulators

The repetitive path planning (RPP) is an important issue in the research of mobile robot manipulators. By utilizing the integration enhanced method, this paper proposes a new RPP scheme for omnidirectional robot manipulators. Differing from the existing RPP schemes, such an integration-enhanced RPP (IERPP) scheme can realize the RPP of omni-directional mobile robot manipulators even in the presence of noise. Comparative simulation results under a specific omnidi-rectional mobile robot manipulator perturbed by constant and time-varying noises are presented to validate the effectiveness of the proposed IERPP scheme.
Intermediate-Task Learning with Pretrained Model for Synthesized Speech MOS Prediction

Mean Opinion Scores (MOS) prediction has been a crucial task for speech quality assessment. Recently, methods based on pre-training and fine-tuning achieved the start-of-the-art performance for MOS prediction. While these two-stage methods can improve the generalization ability of the model, there still exists a large gap between the pre-training and the downstream MOS prediction objectives in the input distribution and the output label space. In this paper, we proposed a three-stage intermediate task training scheme, then we tailored two possible intermediate tasks (classification and contrastive learning) for the speech MOS prediction task. The experimental results show that the proposed models achieve the SOTA results compared with the existing two-stage systems in most of the metrics for in-domain and out-of-domain datasets with less computational complexity.
Few-shot Semantic Segmentation by Exploiting Dynamic and Regional Contexts

Few-shot Semantic Segmentation (FSS) has received increasing interests recently. Modeling effective interaction be-tween support and query images is a crucial challenge in existing prototype based methods. In this paper, we propose a Dynamic and Regional Context Network (DRCNet) to achieve sufficient support-query interaction for accurate FSS. A Dynamic Context Module (DCM) is first proposed to capture the spatial details in query images by building dynamic convolutions in local views. To further alleviate the undesirable noises, a Regional Context Module (RCM) is proposed to mine and exclude the background and ambiguous objects in query images by modeling the prototypes for ambiguous regions. Experimental results on Pascal-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> datasets demonstrate that our proposed DRCNet performs significantly superior against state-of-the-art methods.
Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning

Multi-Teacher knowledge distillation provides students with additional supervision from multiple pre-trained teachers with diverse information sources. Most existing methods explore different weighting strategies to obtain a powerful ensemble teacher, while ignoring the student with poor learning ability may not benefit from such specialized integrated knowledge. To address this problem, we propose Adaptive Multi-teacher Knowledge Distillation with Meta-Learning (MMKD) to supervise student with appropriate knowledge from a tailored ensemble teacher. With the help of a meta-weight network, the diverse yet compatible teacher knowledge in the output layer and intermediate layers is jointly leveraged to enhance the student performance. Extensive experiments on multiple benchmark datasets validate the effectiveness and flexibility of our methods. Code is available: https://github.com/Rorozhl/MMKD.
Information-density Masking Strategy for Masked Image Modeling

Recent representation learning approaches mainly fall into two paradigms: contrastive learning (CL) and masked image modeling (MIM). Combining these two methods may boost the performance, but its learning process still heavily depends on the random masking strategy. We conjecture that the random masking may hinder learning the comprehensive relationship between concept and visual patches. To overcome these limitations, we propose an information-density masking (IDM) strategy for general visual transformers. Specifically, the IDM mask out the visual patches according to their activation values of attention maps. To obtain the attention maps before the reconstruction, a self-supervised training framework CAMAE is further proposed. In addition, in order to reduce the redundancy among different attention maps, we introduce a pattern-learning balance (PLB) sampling to adaptively adjust the learning progress in different attention spaces. Extensive experiments indicate that our method efficiently retains more comprehensive visual characteristics and achieves state-of-the-art performance.
Adversarial Attacks on Generated Text Detectors

Generated text detectors can effectively detect the machine-generated texts which aim to produce false information to destroy the credibility of the media platform. However, generated text detectors are vulnerable to adversarial example attacks which focus on char-level perturbations to produce many word errors. In this paper, we design a sentence granularity based black-box attack model Sentence-Keyword-Attack (SK-Attack), which can effectively generate semantics-preserved, fluent, and grammatical adversarial examples. SK-Attack adaptively truncates the input examples based on sentence granularity and searches for the essential sentences to apply a sequence of contextualized perturbations with strict constraints. SK-Attack also applies keyword protection to preserve the keywords from being perturbed. Experiments show that SK-Attack outperforms the baselines when attacking the RoBERTa detector with various challenging generated text datasets and also has strong transferability to other attack models.
A Lightweight Grouped Low-rank Tensor Approximation Network for 3D Mesh Reconstruction From Videos

Existing methods for 3D mesh reconstruction from videos suffer from increasingly large parameter counts and model sizes due to encoders such as multi-hidden state recurrence. Therefore many models become complex and more difficult to be applied in practice. Based on this problem, we propose a lightweight grouped low-rank tensor approximation network for 3D mesh reconstruction from videos. Specifically, firstly we propose a generalized grouped low-rank tensor approximation algorithm, which decomposes the original high-rank tensor into multiple weighted groups with different low-rank tensors to maximize the approximation of the high-rank tensor. Then we also design a lightweight selection rearranging strategy to reduce feature redundancy and focus on local fragment features. Notably, our method can be flexibly plugged into other 3D reconstruction tasks. Experiments show that our method not only improves the performance but also reduces the parameters of the entire network by about 90% compared with the existing SOTA method. Our grouped low-rank tensor approximation method reduces the parameters by about 99.8% in a single GRU. We demonstrate the outstanding generalization of our method in other 3D reconstruction tasks, eg. 3D face reconstruction and multi-view stereo.
Question Formulation and Transformer Model Resilience: Effects of Question-Wording Variants on the Robustness of Transformer-Based Question-Answer Models

Question-answer is a paradigm that seeks to provide automated responses to queries posed in natural language utilizing a body of textual content as the source of the answers. A key research challenge is how the changes in question formulation affect the stability of current question-answer transformer models. This paper conducts a preliminary analysis of the stability of question-answer transformer models in the medical domain when the same question is asked in different orders or with other semantically identical variations. The results from our experiments demonstrate that the arrangement of words influences the outcome and consistency of answers from transformer models.
Multi-level Deep Learning Kalman Filter

The well-known Kalman filter and its adaptive variants belong to model-based optimization, and their optimality depends on reliable prior information such as system models, which is sometimes hard to obtain. To reasonably introduce prior domain knowledge on the basis of offline data learning, a multi-level deep learning Kalman filter is designed in this paper with dynamic model parameter learning for evolution trend prediction, process noise covariance learning to obtain the optimal gain, and compensation term learning to correct the errors after the filtering update. The gated recurrent unit is used to construct offline learning modules, which endow the multi-level filter with nonlinear model fitting and memory iterative learning capabilities. The proposed algorithm is validated in maneuvering target tracking tasks, showcasing significant enhancements.
Scene Graph based Fusion Network for Image-Text Retrieval

A critical challenge to image-text retrieval is how to learn accurate correspondences between images and texts. Most existing methods mainly focus on coarse-grained correspondences based on co-occurrences of semantic objects, while failing to distinguish the fine-grained local correspondences. In this paper, we propose a novel Scene Graph based Fusion Network (dubbed SGFN), which enhances the images’/texts’ features through intra-and cross-modal fusion for image-text retrieval. To be specific, we design an intra-modal hierarchical attention fusion to incorporate semantic contexts, such as objects, attributes, and relationships, into images’/texts’ feature vectors via scene graphs, and a cross-modal attention fusion to combine the contextual semantics and local fusion via contextual vectors. Extensive experiments on public datasets Flickr30K and MSCOCO show that our SGFN performs better than quite a few SOTA image-text retrieval methods.
Unsupervised 3D Face Reconstruction with Reprogramming Skip Connections

In unsupervised 3D face reconstruction, existing methods modeling the canonical face typically exclude the skip connections between encoder-decoder pairs. Consequently, they have difficulty capturing appearance details necessary for the task. However, directly applying original skip connections only induces these methods to degrade to a trivial 2D texture reconstruction algorithm. In this paper, we propose novel Reprogramming Skip Connections (RSCs), which escape from bringing about degradation and improve the 3D face reconstruction quality. Specifically, the proposed method filters out inappropriate information causing degradation by aggregating the features from the encoder in spatial dimensions into several prototypes. These prototypes preserving beneficial information are subsequently combined with the corresponding decoder features with the help of expansion masks. Further, we design the masks reconstruction consistency loss to improve the quality of the expansion masks. Our experiments verify the superiority of our method compared to other competitors.