Reviewerly/exHarmony huggingface dataset:
DBIA: Data-Free Backdoor Attack Against Transformer Networks

Recently, transformer architecture has demonstrated its significance in both Natural Language Processing (NLP) and Computer Vision (CV) tasks. Although other network models are known to be vulnerable to the backdoor attack, which embeds triggers in the models and controls the models’ behavior when the triggers are presented, little is known about how such an attack performs on the transformer models. In this paper, we propose DBIA, a novel Data-free <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Backdoor Attack against the CV-oriented transformer networks, leveraging the inherent attention mechanism of transformers to generate triggers and injecting the backdoor using a poisoned substitute dataset. We conducted extensive experiments using three benchmark transformers, i.e., ViT, DeiT, and Swin Transformer, on four mainstream image classification tasks, i.e., ImageNet, CIFAR-10, GTSRB, and Youtube Face. The evaluation results demonstrate that, with fewer resources, our approach can embed backdoors with a high success rate and a low impact on the performance of the victim transformers.
BH-VQA: Blind High Frame Rate Video Quality Assessment

High frame rate (HFR) videos can provide consumers with a more immersive viewing experience in motion-rich scenes. However, they also pose a great challenge for video compression and transmission due to the increase in frame rates. Therefore, it is very important to choose proper frame rates and bit rates to achieve a trade-off between transmission bandwidth and visual quality. In this paper, we propose a novel Blind HFR Video Quality Assessment (BH-VQA) model by exploring the efficient and effective motion representation from the deep neural network (DNN). Concretely, we first train a baseline VQA model (i.e. a backbone network and a regressor) on a large-scale VQA database to derive a powerful quality-aware feature extractor for the spatial and motion feature extraction. Then, the HFR video is split into a sequence of video clips and the spatial features of each video clip are extracted just using the first frame of the video clip. To capture temporal distortions caused by frame rate variations and object and camera motion, we calculate deep structural similarities between continuous frames of each video clip as the motion features. Finally, the temporal quality dependencies between video clips are learned through a gated recurrent unit (GRU) network to obtain the perceptual video quality score. Experimental results show that BH-VQA achieves the best performance on two publicly available HFR VQA databases. The code of BH-VQA will be released.
What a drag! Streamlining the UAV design process with design grammars and drag surrogates

Unmanned Aerial Vehicles (UAVs) continue to pro-liferate, revolutionizing tasks such as cargo transport, surveillance, and search and rescue operations. With the discovery of novel use cases or specialized tasks for aerial vehicles, there is an increased need for improved design space exploration and performance estimation techniques for candidate UAV designs. Typical pipelines for this design process rely on time-consuming human efforts to identify productive design geometries or ex-pensive computational approaches for performance analysis to reconcile aerodynamic, electrical, and physical interactions. In this work-in-progress paper, we propose the use of a design process that uses a design grammar for UAV design generation and a Graph Neural Network (GNN)-based drag surrogate trained on simulation data for accelerated UAV design space exploration. We formulate a UAV design grammar and provide preliminary performance results from the GNN drag surrogate for randomly generated designs. We expect our approach to accelerate the exploration of UAV design geometries using a learned surrogate drag model to circumvent resource-hungry Computer-aided design (CAD) and simulation routines.
Counterfactual Inference for Visual Relationship Detection in Videos

Visual relationship detection in videos is a challenging task since it requires not only to detect static relationships but also to infer dynamic relationships. Recent progress has been made through enriching visual representations by appearance and motion fusion or spatial and temporal reasoning, but without exploring the intrinsic causality between representations and predictions. In this paper, we propose a novel counterfactual inference method for video relationship detection, which infers the causal effects of appearance, motion and language features on the predictions of static and dynamic relationships. Specifically, starting with building a causal graph to represent the causality between features and relationship categories, we then construct counterfactual scenes by intervening the features to infer their effects on prediction, and finally incorporate the inferred effects into the relationship categorization by adaptively learning the weights of appearance, motion and language. Extensive experiments on two benchmark datasets demonstrate the effectiveness of our method.
SSNet: Learning Self-Similarity for Few-Shot Semantic Segmentation

Few-shot Segmentation(FSS) refers to the task of segmenting newly introduced classes using only a limited number of closely marked examples. In the past, prototype learning based on metric and segmentation based on visual correspondence mostly ignore the matching relationship of query image itself. In this article, we present a novel self-similarity based hyperrelation network (SSNet), which introduces self-similarity generation module (SGM) and self-similarity mask module (SMM) to capture self-similarity information of target classes in query images and help the network better understand the internal similarity of query images. We also replace the feature mask with the input mask to eliminate the interference of background information in the support image. Experiments on the PASCAL-S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> show that SSNet achieves new state-of-the-art (SOTA), with a mIoU score of 64.6% in the 1-shot scenario and 68.1% in the 5-shot scenario, which is 0.62% higher than the SOTA method in the 1-shot scenario. This demonstrates that SSNet can achieve efficient and accurate few-shot segmentation with only a small number of samples.
Splittable Pattern-Specific Weight Pruning for Deep Neural Networks

Network pruning is one of the most dominant model compression methods today, which can be broadly divided into filter pruning and weight pruning. Unlike filter pruning that deletes the whole filters thus prone to cause unrecoverable accuracy loss, weight pruning removes the single weights at a fine-grained level, which effectively avoids this problem. However, weight pruning leads to unstructured sparsity and is thus incompatible with general platforms. To address such limitation, we propose Splittable Pattern-Specific Weight Pruning(SPWP) to achieve both compression and compatibility, consisting of Patterned Weight Searching(PWS) and Kernel Equivalent Splitting(KES). Specifically, we study the intrinsic features of convolution kernels and devise PWS to prune weights in regular shapes based on such features. During inference, KES equivalently splits the pruned sparse kernel into parallel branches according to the linear additivity of convolution, allowing the network to be accelerated on general platforms. Extensive experiments of different models on various datasets demonstrate the superior performance of our method. For example, SPWP can prune 60.1% total FLOPS of ResNet-56 on CIFAR-10 with even a 0.08% of top-1 accuracy increase.
Fusion of Human Cognitive Knowledge and Machine Inference for Breast Cancer Detection

With the rapid development of artificial intelligence technology, more and more breast ultrasound CAD systems have emerged. In view of the problem that the existing breast ultrasound diagnosis system relies too much on manually design features and needs a lot of time to annotate and correct the data, the proposal of “fully automated breast ultrasound CAD” solves this problem. It reduces the workload of doctors, and the performance of the diagnosis system has been significantly improved in terms of interpretability and automation. However, the fully automated diagnosis system, which is completely dependent on the machine, also faces some challenges: on the one hand, some BI-RADS features, such as “blood vessels”, cannot be observed in the ultrasound image, resulting in the machine being unable to extract such important information from the existing image for subsequent reasoning. On the other hand, some features that are not obvious enough are difficult to be recognized by machines in ultrasonic images. However, this kind of information may play a decisive role in the final judgment of the diagnosis results. The loss of this kind of information caused by factors such as image quality in the process of machine feature extraction will affect the final diagnosis performance. Therefore, this paper proposes a new diagnostic paradigm: a semi-automatic breast ultrasound diagnosis system based on human-machine fusion. First, some features automatically extracted by machine, while the other features that are difficult to be recognized by machine are marked and scored by experienced ultrasonic doctors. Further, the features of the two parts are fused into more informative features. These features are used as medical knowledge for diagnosis to build a human-machine fusion breast ultrasound knowledge graph, then diagnose through the KG reasoning algorithm, and output the final diagnosis results. The experimental results show that the proposed method can effectively improve the performance of the fully automated diagnosis system.
Learning Group Residual Representation for Group Activity Prediction<sup>*</sup>

The goal of group activity prediction is to infer the group activity involved multiple individuals before it is completely executed. Previous methods focused on capturing pair-wise relationships between individuals, but lacked the exploration of group-wise interactions which can provide global guidance from a macroscopic perspective. To further explore the group-wise interaction, we propose a Group Residual Module (GRM) which constructs a virtual leader node to summarize the group representation and designs a bidirectional message passing mechanism to build the bridge between group and individuals. To capture the spatial-temporal correlation jointly, we propose a Spatial-Temporal Group Residual Network composed of spatial GRMs and temporal GRMs. Different from existing methods that obtain additional information from the complete activity execution, temporal masks in the temporal GRMs are designed to enforce our network to excavate as much discriminative information as possible from the observed activity sequence. Moreover, experimental results show that our network achieves state-of-the-art performance on Volleyball Dataset and Collective Activity Dataset.
Making Adversarial Attack Imperceptible in Frequency Domain: A Watermark-based Framework

With the development of multimedia communication technology, the image information stored in electronic devices faces increasing privacy risks and requires processing for protection. However, it is found that adversarial perturbations added to images for semantic information protection may corrupt the frequency domain watermarks added for copyright statement. With such challenges, we propose an Adversarial Frequency domain Watermarking (AFW) framework to protect images from both copyright and semantic content. Specifically, the AFW framework constructs the images as adversarial examples by embedding crafted adversarial watermarks in the frequency domain, followed by an optimization algorithm to improve the visual quality. Notably, AFW can generally integrate with existing watermark and attack methods. Extensive experiments on five network models and the ImageNet dataset demonstrate that the AFW framework can achieve information hiding and adversarial attacking goals under visual quality assurance.
Deep Unsupervised Hashing with Selective Semantic Mining

Most of the existing unsupervised hashing methods usually construct semantic similarity structure to guide hashing learning. However, due to the lack of filtering of useless information, some wrong guiding information in the similarity structure may damage the retrieval performance. Besides, some works adopt the framework of contrastive learning to preserve the discriminative semantic information that is more important for the hashing task. But such a training strategy may incorrectly embed some semantically similar samples far away due to the absence of manual label supervision, thus producing sub-optimal hash codes. To solve the aforementioned problems, we propose a novel method named Deep Selective Semantic Mining Hashing (DSSMH). Specifically, with the prior knowledge obtained by clustering, we select semantically correct image pairs with high confidence to alleviate the guidance of wrong information and correct sampling bias in contrastive learning. Extensive experiments demonstrate that DSSMH outperforms existing state-of-the-art methods.
Transfer Learning Approach for Multilabel Fine Grain Image Attribute Extraction

In the era of artificial intelligence, industries are taking big leap towards transforming overall process of product manufacturing, business measures, retail & customer experience. AI is contributing to small as well as big businesses to enable the automation throughout the pipeline. For steps like product onboarding, attribute identification, PDP enrichment, catalogue generation. Small & mid-level eCommerce industries, in early stage, tends to adopt a low-cost solution for transforming the services. The Literature provides unique and affordable deep learning based multilabel classification approach fore extracting fine grained image level multiple attributes. Converting the objective into multilabel classification problem and using existing stable deep learning models to further train some fine level features, transfer learning is one of the affordable steps to consider. It provides efficient architecture with optimum results, validated on best resources. Extracting product image level fine grained properties is a complex task which requires dense networks with high computing resources. Existing state of the art architectures for classification problems, after tweaking few layers, used as a baseline models. These architectures have proven one of the best solutions for multilabel classification problems with minimal resources required.
Multi-view Network Embedding with Structure and Semantic Contrastive Learning

Multi-view network embedding aims to learn low-dimensional representation vectors for nodes while preserving multiple relationships between nodes. It can substantially reduce downstream network analysis tasks’ time and space complexity. Although previous works have achieved great performance, they suffer from two limitations: (1) they only preserve the network structure and ignore the semantic level information; (2) they only focus on intra-view signals and ignore the powerful influence of inter-view signals. These limitations highlight the need for more comprehensive approaches to multi-view network embedding that can effectively capture the structure and semantic information, as well as the influence of inter-view signals. A new framework, Multi-view Network Embedding with Structure and Semantic Contrastive Learning (MNE-SSCL), is proposed to address these limitations. It can learn high-quality low-dimensional node embeddings in both intra-veiw and interview, while preserving the structure and semantic information simultaneously. Extensive experiments on three real datasets show that MNE-SSCL outperforms the state-of-the-art methods.
Intensifying The Consistency of Pseudo Label Refinement for Unsupervised Domain Adaptation Person Re-Identification

Clustering-based unsupervised domain adaptation (UDA) for person re-identification aims to learn in the unlabeled target domain. However, the noise problem of clustering-based generated pseudo labels remains under-explored, and these wrong labels can mislead the feature learning process. In this paper, we propose a consistent and intensive pseudo label refinement method in which pseudo labels generated in two different feature spaces, local and global, refine each other to improve the pseudo label quality of the final clusters. Then we utilize a quantitative criterion to measure label inaccuracy and fine-tune the target domain to reduce the noise by an inaccuracy-guided pseudo label optimization scheme. On a strong benchmark, we demonstrate the superiority of the method with extensive experiments. Specifically, our method outperforms the baseline by 7.2% mAP on the Duke2Market task and 0.5% mAP on the Market2MSMT task over the state-of-the-art.
Online Action Detection with Learning Future Representations by Contrastive Learning

Online Action Detection (OAD), which predicts the ongoing human action from a streaming video, is an important task in multimedia analysis. Compared with offline action detection, OAD only uses historical and current frames without any access to future information, resulting in limited performances. Current OAD methods obtain future information by training an additional future anticipation task and suffer from low efficiency. To address the low efficiency problem, we propose to integrate future information into visual representations for OAD. Specifically, we learn future representations through a contrastive-learning-based method. Further, two novel sampling strategies, multiple scale sampling and timeline overlap sampling, are introduced to enhance our method. The experimental results show that our method significantly improves inference efficiency and achieves promising effectiveness performances on two popular OAD benchmarks.
Detecting Anomalous Energy Consumptions in Smart Buildings – an Overview of Two Unsupervised Techniques

In context of the increasing importance of monitoring our energy consumption due to our energy reserves running short, this paper gives an overview of existing possibilities to detect anomalous energy consumptions in smart cities based on unsupervised machine learning. After defining and presenting different types of anomalies, practical applications and methods for anomaly detection in context of smart cities are discussed.
Edgeformer: Edge-Enhanced Transformer for High-Quality Image Deblurring

Transformers in image deblurring have achieved significant progress recently under their outstanding ability to model the long-range information of features. However, the modeling in the blurry image is challenging due to the ambiguous correlation caused by the diversity of the blur degree. Whereas the modeling works well in clear regions, which can be represented by edges or textures. Therefore, we explore the effectiveness of edges and textures in image deblurring and present an effective and efficient edge-enhanced Transformer, called Edgeformer, to acquire high-quality images. Specifically, we develop an efficient edge-enhanced attention (EA) that focuses on the sharp areas when computing attention maps. Furthermore, we introduce an edge-enhanced feed-forward network (EFFN) that could discriminate edge and texture features preserved for latent clear image deblurring. Experimental results show that the proposed method performs favorably against the state-of-the-art methods.
Service Chatbots and International Students: A Systematic Review

Mental health has predominantly been a social stigma where suffering individuals are in a constant combat with themselves. Youth and young adult students who would otherwise be the future of a nation are succumbing to depression or anxiety with the rise of socio-economic pressures. International students in the US are no exceptions and they fall prey to psychiatric and psychological issues. In such cases, they need easily accessible resources at their disposal. This paper surveys the existing research that studies various aspects of difficulties faced by international students in the United States and offers ideas of solutions based on computing and artificial intelligence through the effectiveness of chatbot systems.
Swap-Reconstruction Autoencoder for Compositional Zero-Shot Learning

Compositional zero-shot learning (CZSL) aims to distinguish images from unseen compositional classes, which consist of state and object concepts that individually appear in some seen compositional images. The key challenge of CZSL is how to effectively mitigate the contextuality issue for achieving a desirable compositional transfer from seen classes to unseen ones. In CZSL, the visual appearances of the same state are inconsistent when combined with different objects. To address the above dilemma, we propose a swap-reconstruction autoencoder (SRA) to capture the intrinsic context of the ambiguous states. Specifically, SRA learns a consistent embedding space for multi-modal data. A swap-reconstruction mechanism is designed to disentangle the visual embedding of states and objects. The loss including a superclass-oriented state swap-reconstruction loss and object swap-reconstruction loss model the contextual relationship between states and objects. Extensive experiments demonstrate that SRA outperforms current state-of-the-art methods on the three benchmark datasets.
SST: Real-time End-to-end Monocular 3D Reconstruction via Sparse Spatial-Temporal Guidance

Real-time monocular 3D reconstruction is a challenging problem that remains unsolved. Although recent end-to-end methods demonstrate promising results, tiny structures and geometric boundaries are hardly captured due to their insufficient supervision neglecting spatial details and oversimplified feature fusion ignoring temporal cues. To address the problems, we propose an end-to-end 3D reconstruction network SST, which utilizes Sparse estimated points from visual SLAM system as additional Spatial guidance and fuses Temporal features via a cross-modal attention mechanism, achieving more detailed reconstruction results. We propose a Local Spatial-Temporal Fusion module to exploit more informative spatial-temporal cues from multi-view color information and sparse priors, as well a Global Spatial-Temporal Fusion module to refine the local TSDF volumes with the world-frame model from coarse to fine. Extensive experiments on ScanNet and 7-Scenes demonstrate that SST outperforms all state-of-the-art competitors, whilst keeping a high inference speed at 59 FPS, enabling real-world applications with real-time requirements.
Weakly-supervised Temporal Action Localization with Adaptive Clustering and Refining Network

Weakly-supervised temporal action localization task aims to localize temporal boundaries of action instances by using only video-level labels. Existing methods primarily adopt Multi-Instance-Learning (MIL) scheme to handle this task. The effectiveness of MIL scheme depends heavily on the selection of top-k action snippets, which is unstable and requires manual tuning. To address these deficiencies, we propose an Adaptive Clustering and Refining Network (ACRNet). Specifically, we present an action-aware clustering strategy that is adaptable and requires no manual tuning to separate action and background snippets of diverse videos based on intra-class activation distribution. And a cluster refining step is included to eliminate false action snippets by considering inter-class activation distribution, which greatly improves robustness and localization accuracy. Extensive experiments on THUMOS14, ActivityNet 1.2&1.3 benchmarks show that our method achieves state-of-the-art performance.