Reviewerly/exHarmony huggingface dataset:
Patch-Wise Auto-Encoder for Visual Anomaly Detection

Anomaly detection without priors of the anomalies is challenging. In the field of unsupervised anomaly detection, traditional auto-encoder (AE) tends to fail based on the assumption that by training only on normal images, the model will not be able to reconstruct abnormal images correctly. On the contrary, we propose a novel patch-wise auto-encoder (Patch AE) framework, which aims at enhancing the reconstruction ability of AE to anomalies instead of weakening it. Each patch of image is reconstructed by corresponding spatially distributed feature vector of the learned feature representation, i.e., patch-wise reconstruction, which ensures anomaly- sensitivity of AE. Our method is simple and efficient. It advances the state-of-the-art performances on Mvtec AD benchmark, which proves the effectiveness of our model. It shows great potential in practical industrial application scenarios.
An Inter-Observer Consistent Deep Adversarial Training for Visual Scanpath Prediction

The visual scanpath represents the fundamental concept upon which visual attention research is based. As a result, the ability to predict them has emerged as a crucial task in recent years. It is represented as a sequence of points through which the human gaze moves while exploring a scene. In this paper, we propose an inter-observer consistent adversarial training approach for scanpath prediction through a lightweight deep neural network. The proposed method employs a discriminative neural network as a dynamic loss that better models the natural stochastic phenomenon while maintaining consistency between the distributions related to the subjective nature of scanpaths traversed by different observers. The competitiveness of our approach against state-of-the-art methods is shown through a testing phase.
Efficient Prediction of Model Transferability in Semantic Segmentation Tasks

How to efficiently select highly transferable pretrained models remains a challenging problem in few-shot semantic segmentation tasks. Existing transferability metrics for classification tasks are difficult to compute on segmentation data due to the high-dimensional output of the segmentation model. In this work, we generalize existing transferability metrics to efficiently predict the transferability of semantic segmentation models, by calculating transferability scores over the sampled pixel-wise features. Then with the help of transferability, we propose a transferability-weighted finetuning method which puts more importance on those low-transferability regions to improve the overall transfer accuracy on the target task. Experiments on a challenging benchmark show that the transferability scores produced by our adaptation method are highly correlated with the groundtruth transfer accuracy, achieving 0.718 Spearman’s correlation coefficient on average and at least 67× gain on efficiency. In addition, our transferability-weighted finetuning method outperforms vanilla fine-tuning by 4% in transfer accuracy.
Efficient Transfer by Robust Label Selection and Learning with Pseudo-Labels

Semi-supervised techniques have been successful in reducing the amount of labels needed to train a neural network. Often these techniques focus on making the most out of the given labels and exploiting the unlabeled data. Instead of considering the labels as a given, we start with focusing on how to select good labels for efficient semi-supervised learning with pseudo-labels. We propose CLaP: Clustering, Label selection and Pseudo-labels. It clusters an unlabeled dataset on an embedding, that was pretrained on large scale dataset (ImageNet1k). We use the cluster centers for querying initial labels that are both representative and diverse. We use samples consistent over multiple clustering runs as pseudo-labels. We propose a mixed loss on both the initial labels and the pseudo-labeled data to train a neural network. With CLaP the samples to be annotated are automatically selected, reducing the load by a human annotator to search for these samples. We demonstrate that CLaP outperforms of state-of-the-art methods in few-shot transfer tasks on full datasets by 20% on 1-shot to 3% on 5-shot. It also outperforms the accuracy of the state-of-the-art in the BSCD-FSL benchmark up to 23%, depending on the dataset and amount of labels.
Improving Nerf with Height Data for Utilization of GIS Data

Neural Radiance Fields (NeRF) has been applied to various tasks related to representations of 3D scenes. Most studies based on NeRF have focused on a small object, while a few studies have tried to reconstruct large-scale scenes although these methods tend to require large computational cost. For the application of NeRF to large-scale scenes, a method based on NeRF is proposed in this paper to effectively use height data which can be obtained from GIS (Geographic Information System). For this purpose, the scene space was divided into multiple objects and a background using the height data to represent them with separate neural networks. In addition, an adaptive sampling method is also proposed by using the height data. As a result, the accuracy of image rendering was improved with faster training speed.
Lightweight Network Towards Real-Time Image Denoising On Mobile Devices

Deep convolutional neural networks have achieved great progress in image denoising tasks. However, their complicated architectures and heavy computational cost hinder their deployments on mobile devices. Some recent efforts in designing lightweight denoising networks focus on reducing either FLOPs (floating-point operations) or the number of parameters. However, these metrics are not directly correlated with the on-device latency. In this paper, we identify the real bottlenecks that affect the CNN-based models’ runtime performance on mobile devices: memory access cost and NPU-incompatible operations, and build the model based on these. To further improve the denoising performance, the mobile-friendly attention module MFA and the model reparameterization module RepConv are proposed, which enjoy both low latency and excellent denoising performance. To this end, we propose a mobile-friendly denoising network, namely MFDNet. The experiments show that MFDNet achieves state-of-the-art performance on real-world denoising benchmarks SIDD and DND under real-time latency on mobile devices. The code and pre-trained models will be released.
Global-Local Awareness Network for Image Super-Resolution

Deep-net models based on self-attention, such as Swin Transformer, have achieved great success for single image super-resolution (SISR). While self-attention excels at modeling global information, it is less effective at capturing high frequencies (e.g., edges etc.) that deliver local information primarily, which is crucial for SISR. To tackle this, we propose a global-local awareness network (GLA-Net) to effectively capture global and local information to learn comprehensive features with low- and high-frequency information. First, we design a GLA layer that combines a high-frequency-oriented Inception module with a low-frequency-oriented Swin Transformer module to simultaneously process local and global information. Second, we introduce dense connections in-between GLA blocks to strengthen feature propagation and alleviate the vanishing-gradient problem, where each GLA block is composed of several GLA layers. By coupling these core designs, GLA-Net achieves SOTA performance on SISR.
Can Human Attribute Segmentation be More Robust to Operational Contexts Without New Labels?

Human Attribute Segmentation (HAS) describes, pixel-wise, the different semantic parts of people in an image. This fine-grained description is useful for several applications (e.g. security, fashion). However, despite the good performance reached by supervised Semantic Segmentation (SS) approaches, they are usually biased by the source training dataset and suffer from a performance drop when applied on new domains. Pixelwise image annotation for each new encountered context is tedious and expensive. So how can HAS become more robust to new contexts without new annotations? In this first study of Unsupervised Domain Adaptation (UDA) for HAS, we present UDA-HPTR, a new method based on HPTR [1] (Human Parsing with TRansformers) combined with self-supervised and semi-supervised learning paradigms to deal with UDA. UDA-HPTR improves performance on both source (labeled) and target (unlabeled) datasets compared to the fully supervised version (HPTR). It also outperforms HRDA, a state-of-the-art UDA method in autonomous driving benchmarks, by +6.7 p.p. on the source and +8.8 p.p. on the target, when applied to HAS while using only half the number of parameters.
Parameter-Efficient Vision Transformer with Linear Attention

Recent advances in vision transformers (ViTs) have achieved outstanding performance in visual recognition tasks, including image classification and detection. ViTs can learn global representations with their self-attention mechanism, but they are usually heavy-weight and unsuitable for resource-constrained devices. In this paper, we propose a novel linear feature attention (LFA) module to reduce computation costs for vision transformers and combine efficient mobile CNN modules to form a parameter-efficient and high-performance CNN-ViT hybrid model, called LightFormer, which can serve as a general-purpose backbone to learn both global and local representation. Comprehensive experiments demonstrate that LightFormer achieves competitive performance across different visual recognition tasks. On the ImageNet-1K dataset, LightFormer achieves top-1 accuracy of 78.5% with 5.5 million parameters. Our model also performs well when transferred to object detection and semantic segmentation tasks. On the MS COCO dataset, LightFormer attains mAP of 33.2 within the YOLOv3 framework, and on the Cityscapes dataset, with only a simple all-MLP decoder, LightFormer achieves mIoU of 78.5 and FPS of 15.3, surpassing state-of-the-art lightweight segmentation networks.
Hybrid Contrastive Prototypical Network for Few-Shot Scene Classification

Few-shot learning has received widespread attention in remote sensing image scene classification. Many existing methods address this challenge by utilizing meta-learning and metric learning, which focus on developing feature extractors that can quickly adapt to novel few-shot scene classification (FSSC) tasks. However, these methods are often insufficient for real-world datasets with class confusion, where there is high inter-class compactness and intra-class diversity. To overcome this issue, we investigate efficient strategies, i.e., meta-learning-based transferable feature representation and contrastive-based prototypical regularization for learning task-adaptive class boundaries for FSSC. Specifically, we designed a combination of Query-vs-Prototype contrastive loss and Prototype-vs-Prototype contrastive loss to normalize the prototypical representation to be more discriminative in a novel FSSC task. Our proposed model is named the Hybrid Contrastive Prototypical Network (HCP-Net). Experiment results on three popular datasets under two standard benchmarks, i.e., general few-shot classification and few-shot domain generalization, indicate the effectiveness of the proposed method.
Dynamic Unilateral Dual Learning for Text to Image Synthesis

Dual learning trains two inverse processes tasks dually to further improve the selected tasks’ performance. There are currently two training paradigms in dual learning. One is to directly utilize two existing models for training in a dual manner to improve the selected models’ performance. However, it cannot effectively guarantee the improvement of the selected models. Another is that the networks of both parties are manually designed. Nevertheless, the network performance of both parties will be poor in the initial stage of training, which will easily lead to unsatisfactory training. Besides, most of dual learning researches can only be used for the conversion between the same data types, but it is powerless for the conversion of different data types. To address the above issues, a paradigm called unilateral dual learning (UDL) is proposed and verified in the text-to-image (T2I) synthesis field. UDL allows one party to design the network manually, and the other party calls the pre-trained model to promote the training of the manually designed network to achieve satisfactory training. Experimental results on the Oxford-102 flower and Caltech-UCSD Birds datasets demonstrate the feasibility of our proposed UDL paradigm in the T2I field, and it achieves excellent performance qualitatively and quantitatively.
SGSR: A Saliency-Guided Image Super-Resolution Network

The human visual system finds salient regions in images and allows the cognitive ability to focus on them. Hence, such salient regions play substantial roles in determining the quality of images. However, the existing super-resolution (SR) methods restore all regions of low-resolution images in the same manner. In this paper, we first propose a saliency-guided image super-resolution (SGSR) network where its restoration ability concentrates on the salient regions in the natural images. For this, we propose a saliency learning scheme using newly computed saliency scores for object regions. Then, by providing the saliency features to the saliency-guided attention (SGA) module and using a novel saliency-weighted loss function, the SGSR maximizes the image quality of salient regions and suppresses the excessive generation of unnecessary structures in backgrounds. To the best of our knowledge, this SGSR is the first attempt to induce discriminatory results guided by saliency in the field of natural image SR.
Self-Enhanced Training Framework for Referring Expression Grounding

Weakly-supervised referring expression grounding (REG) aims at locating the image region described by a query sentence, where the mapping between the referential region and query is not available during the training stage. Noticing the significant gap between the fully- and weakly-supervised approaches, we develop a Self-Enhanced Training(SET) framework in this paper. Specifically, we first train the network under a weakly-supervised setting. Then, the model outputs are collected and filtered according to the confidence score and serve as pseudo-labels. Finally, with the help of these pseudo-labels, we tune the model under a fully-supervised setting. The SET framework provides a simple way of generating pseudo-labels that build a bridge between weak and full supervision. Experimental results demonstrate that model trained through our SET framework outperforms existing traditional methods on RefCOCO, RefCOCO+, and RefCOCOg datasets. The code is available at https://github.com/HTDL98/SET-framework.
Deep CNN-Based Pre-Encoding Perceptual Quality Control and Prediction

Inevitable utilization of lossy compression methods results in distortion and degrading of video perceptual quality. Predicting perceptual quality before compression is essential to optimize compression parameters, e.g., quantization parameter (QP) and assigning optimized bandwidth. This paper presents three Intra frame (I-frame) perceptual quality prediction methods. The proposed methods work based on deep CNN structure. The proposed methods are integrated with high efficiency video coding (HEVC, H.265) reference codec. The VMAF index has been utilized to measure the perceptual quality of video samples. An end-to-end CNN network performs spatial feature extraction for perceptual quality prediction. The proposed methods are designed based on our experimental observations. The proposed methods are evaluated with 17 video samples, and the results show a reliable, accurate performance of approaches.
Efficient any-Target Backdoor Attack with Pseudo Poisoned Samples

Deep neural networks present their potential vulnerabilities to backdoor attacks. They have a satisfactory performance for benign users with clean samples but will get malicious outputs when inputs are attached with the backdoor trigger. Current backdoor attacks on image classifiers usually target only one single class, making them not robust to defenses against this characteristic. In this work, we propose a new any-target attack that targets all the labels simultaneously with the triggers being invisible and input-dependent. Specifically, we train the classifier together with the image steganography model by encoding the one-hot encodings into the input images. The novel pseudo poisoned samples are then introduced to improve the effectiveness of our backdoor attack. Experimental results show that our method is both effective and efficient on several datasets and is robust to existing defenses.
Noise-Avoidance Sampling for Annotation Missing Object Detection

Excellent results can be achieved using object detection with fully supervised training on large well-annotated datasets. However, the problem of missing annotations in real-world datasets can considerably reduce the performance of object detectors. In this study, we thoroughly analyze the effect of missing annotations on both positive and negative samples in object detector training. To mitigate the negative impact caused by annotation missing problem, we propose a simple yet effective method, noise-avoidance sampling, to distinguish noisy training samples and subsequently reduce their negative impact. Experiments are conducted on the PASCAL VOC 07+12 dataset with varying levels of missing annotations. The results reveal that the proposed method achieves comparable or superior performance with state-of-the-art methods.
Leveraging Visual Prompts To Guide Language Modeling for Referring Video Object Segmentation

Referring Video Object Segmentation (R-VOS) aims to segment object masks in a target video given a language query describing the object. It is a challenging task that requires modeling the semantics of a natural language query and its correspondence to the target video. Previous works directly use visual-agnostic language features from uni-modal language models, and only interact with visual features in late decoding stages. We propose to encode visual-enriched language features by using visual prompts as guidance in the early encoding stage. The proposed visual prompt is constructed by modulating visual features of key frames with alignment scores to text inputs. The alignment score is computed with a pre-trained visual-language contrastive model. We concatenate visual prompts with text inputs to encode visual-enriched language features, which serve as queries for target object segmentation in a Transformer-based decoder. Our method outperforms the previous state-of-the-art method (+2.3) on Refer-Youtube-VOS benchmark.
Deep Unsupervised Hashing with Semantic Consistency Learning

Hashing method has attracted more attention in recent years because of its low storage consumption and high retrieval performance. Most unsupervised hashing methods first construct local similarity structure in high-dimensional feature space, and then learn binary hash codes which maintain similarity structure information. However, this local structure based on pairwise distance will bring false guidance and misguide the hashing model. Besides, previous methods rarely consider the robustness of the hashing model, resulting in the unstable hash codes generated under perturbation. Toward these issues, we propose a novel Semantic Consistency Hashing (SCH). Specifically, to avoid misguidance caused by local similarity structure, SCH converts the similarity structure into the probability distribution and preserves semantic information from the perspective of global data distribution. In addition, to improve the robustness of hash codes, we introduce transformation consistency learning to maximize the similarity of hash codes under different transformations of the same image. Experiments on three popular datasets show that SCH outperforms the state-of-the-art methods.
Cross-Layer Patch Alignment and Intra-and-Inter Patch Relations for Knowledge Distillation

To date, most existing distillation methods exploit coarse-grained instance-level information as valuable knowledge to transfer, such as instance logits, instance features and instance relations. However, the fine-grained knowledge of internal regions and relationships between semantic entities within a single instance are overlooked and not fully explored. To address above limitations, we propose a novel fine-grained patch-level distillation method, dubbed as Patch Aware Knowledge Distillation. PAKD rethink knowledge distillation from a new perspective regarding the significance of cross-layer patch alignment and patch relations within and across instances. Specifically, we first devise a novel cross-layer architecture to fuse patches across stages, which is capable of utilizing multi-level information of the teacher to guide one-level learning of the student. Then, we propose cross-layer patch alignment, allowing the student to be aware of patches discriminatively and find the best way to learn from the teacher. Besides, patch relations within and across instances are leveraged to supervise the structural knowledge distillation in the manifold space. We apply our method to image classification and object detection tasks. Consistent improvements over state-of-the-art approaches on different datasets and diverse teacher-student combinations manifest the great potential of our proposed PAKD.
A Decoupled Spatial-Channel Inverted Bottleneck For Image Compression

Residual block has achieved great success in deep networks to eliminate accuracy degradation, and there emerges a large number of variants with more competitive performance. However, these blocks are introduced for high-level tasks that only encode the input image into semantic and struc¬tural features but do not need to reconstruct. So for the low-level task like image compression where reconstruction quality contributes significantly to the rate-distortion perfor¬mance, the structure of the residual block needs modification for more suitable implementation. In this paper, we revisit the existing residual blocks and discover two key principles summarized as two decouplings: spatial-channel decoupling and linear-nonlinear decoupling. We propose an efficient nonlinear transform based on the principles dubbed decou¬pled spatial-channe inverted bottleneck(DSCIB), which has a linear-spatial branch for rough reconstruction and a nonlinear¬channel branch to provide detailed featrues. We employ the DSCIB module in the joint auto regression model to build an overall network. Experimental results show that our method achieves comparable performance with the existing learning-based image compression methods at high bitrate while re¬ducing 38% FLOPs.
Base Layer Efficiency in Scalable Human-Machine Coding

A basic premise in scalable human-machine coding is that the base layer is intended for AUTOMATED machine analysis and is therefore more compressible than the same content would be for human viewing. Use cases for such coding include video surveillance and traffic monitoring, where the majority of the content will never be seen by humans. Therefore, base layer efficiency is of paramount importance because the system would most frequently operate at the base-layer rate. In this paper, we analyze the coding efficiency of the base layer in a state-of-the-art scalable human-machine image codec, and show that it can be improved. In particular, we demonstrate that gains of 20-40% in BD-Rate compared to the currently best results on object detection and instance segmentation are possible.