Reviewerly/exHarmony huggingface dataset:
Far Away in the Deep Space: Dense Nearest-Neighbor-Based Out-of-Distribution Detection

The key to out-of-distribution detection is density estimation of the in-distribution data or of its feature representations. This is particularly challenging for dense anomaly detection in domains where the in-distribution data has a complex underlying structure. Nearest-Neighbors approaches have been shown to work well in object-centric data domains, such as industrial inspection and image classification. In this paper, we show that nearest-neighbor approaches also yield state-of-the-art results on dense novelty detection in complex driving scenes when working with an appropriate feature representation. In particular, we find that transformer-based architectures produce representations that yield much better similarity metrics for the task. We identify the multi-head structure of these models as one of the reasons, and demonstrate a way to transfer some of the improvements to CNNs. Ultimately, the approach is simple and non-invasive, i.e., it does not affect the primary segmentation performance, refrains from training on examples of anomalies, and achieves state-of-the-art results on RoadAnomaly, StreetHazards, and SegmentMeIfYouCan-Anomaly.
Image Generation Based Intra-class Variance Smoothing for Fine-Grained Visual Classification

Fine-grained visual classification (FGVC) is challenging because of the unsmooth intra-class data distribution caused by the combination of relatively significant intra-class variation and scarce training data. To this end, most works in FGVC focused on explicitly/implicitly enhancing the model representation ability. In this paper, however, we take a different stance – alleviating the unsmooth intra-class data distribution in FGVC datasets via data generation. In particular, we propose the following components for data augmentation: (i) SmoothGAN: an information-theoretic extension to the Generative Adversarial Network (GAN) that can generate high-quality fine-grained images with continuously varying intra-class differences. (ii) Dual-threshold-filtering: the generated data are selected according to both their reality and discriminability via SmoothGAN’s discriminator and a basic FGVC model. Experiments on popular FGVC datasets demonstrate that training with augmented data can significantly boost model performance in the FGVC task. The code is available at https://github.com/PRIS-CV/SmoothGAN .
Enhancing Feature Representation for Anomaly Detection via Local-and-Global Temporal Relations and a Multi-stage Memory

Weakly supervised video anomaly detection is a challenging task because frame-level labels are not accessible at the training time. Effectively tackling this task necessitates models to learn discriminative feature representation. To address this challenge, we propose a multi-stage memory-augmented feature discrimination learning (MMFDL) method. The first stage obtains the preliminary abnormal probabilities of clip features. In the second stage, an easy normal pattern memory (ENPM) are proposed to store normal patterns with low abnormal probabilities. In the last stage, we bring clip features with high abnormal probabilities in normal videos close to ENPM and away from the clip features with high probabilities of being abnormal in abnormal videos to make models learn more discriminative features for anomaly detection. Furthermore, we propose a local-and-global temporal relations modeling (LGTRM) module to enhance clip features by aggregating local and global contexts. Our LGTRM module can be divided into two subnetworks: DW-Net and TF-Net. DW-Net integrates the current clip feature with its adjacent clip features to capture local-range temporal dependencies. TF-Net utilizes the multi-head self-attention mechanism of the transformer to capture global-range temporal dependencies. Experiments on two datasets demonstrate that our method outperforms state-of-the-art approaches. The code is available at https://github.com/xuanli01/PRCV347 .
Lecture 6—Input Devices and Tracking

Input devices are an essential component of every Virtual Reality system. Research into possible ways of interacting with Virtual environments has spawned numerous innovations regarding such devices. We begin this lecture by talking about the properties of input devices in general and move our attention to some of the input devices designed specifically for interaction with virtual environments, such as data gloves.
ContextNet: Learning Context Information for Texture-Less Light Field Depth Estimation

Depth estimation in texture-less regions of the light field is an important research direction. However, there are few existing methods dedicated to this issue. We find that context information is significantly crucial for depth estimation in texture-less regions. In this paper, we propose a simple yet effective method called ContextNet for texture-less light field depth estimation by learning context information. Specifically, we aim to enlarge the receptive field of feature extraction by using dilated convolutions and increasing the training patch size. Moreover, we design the Augment SPP (AugSPP) module to aggregate features of multiple-scale and multiple-level. Extensive experiments demonstrate the effectiveness of our method, significantly improving depth estimation results in texture-less regions. The performance of our method outperforms the current state-of-the-art methods (e.g., LFattNet, DistgDisp, OACC-Net, and SubFocal) on the UrbanLF-Syn dataset in terms of MSE $$\times $$ 100, BadPix 0.07, BadPix 0.03, and BadPix 0.01. Our method also ranks third place of comprehensive results in the competition about LFNAT Light Field Depth Estimation Challenge at CVPR 2023 Workshop without any post-processing steps (The code and model are available at https://github.com/chaowentao/ContextNet .).
Lecture 9—Augmented Reality

In this lecture, we will discuss Augmented Reality, a concept closely related to VR with which it shares many ideas and technological solutions. We begin our lecture by examining the relationship between these two concepts, their similarities, and conceptual differences.
SSHRF-GAN: Spatial-Spectral Joint High Receptive Field GAN for Old Photo Restoration

Old photo restoration is a challenging problem since it suffers from mixed degradation like noise, blurriness, scratch, dots, or color fading during its preserving or digitalizing process. The common practice of old photo restoration is to apply single degradation restoring methods sequentially, but it cannot restore the old images at one time. The existing mixed degradation restoring methods are designed in a complex architecture or training paradigm. To tackle this problem in a simple way, we propose SSHRF-GAN: Spatial-Spectral joint High Receptive Field GAN. It contains three independent modules for the mixed degradation that appeared in old photos: (1) High Receptive Field Fast Convolution block for noise and blurriness restoration; (2) Partial convolution block for scratch, dots, and cracks inpainting and (3) Image-wise histogram attention for color tone mapping. We evaluate our methods on synthesized datasets and gather real old data. The result shows that our simple network has a comparative performance with those sequential applied single degradation restoration models and previously proposed complex models.
FSCD-Net: A Few-Shot Stego Cross-Domain Net for Image Steganalysis

Image steganalysis plays a critical role in various applications, aiming to detect hidden data within images. However, its effectiveness is significantly impeded by the cover source mismatch, where training and testing images come from different sources. This challenge is particularly pronounced in few-shot scenarios with limited training samples. In this paper, we propose a novel steganalysis model explicitly tailored to address the cover source mismatch in few-shot settings. By leveraging the pair constraint and incorporating features from accessible frequency domain images, our model enhances transferability and generalization, effectively improving steganalysis performance. To capture intricate relationships among embedded image regions across diverse domains, we introduce the FixedEMD module, which focuses on high-dimensional channels. Extensive experiments validate the efficacy of our model, demonstrating its superiority in image steganalysis under cover source mismatch conditions. Our findings underscore the potential of our model to overcome cover source mismatch challenges, making significant contributions to the advancement of image steganalysis.
Multimodal Local Feature Enhancement Network for Video Summarization

Multimodal information processing has garnered considerable attention in recent years. Due to the inherent multimodal information in videos, multimodal learning has been introduced in the domain of video summarization, leading to a significant improvement in summary accuracy. However, previous multimodal video summarization methods have overlooked the local fine-grained features of the fused multimodal information, constraining the representational capacity of the models. Therefore, we propose a Multimodal Local Feature Enhancement Network (MLFEN) to address this issue. Firstly, we utilize multi-head attention to encode single-modal information, capturing their mutual dependencies, and then fuse multiple modalities. Secondly, we further process the multimodal information by employing local attention to explore local fine-grained interdependencies and global attention to capture global dependencies. Finally, a regression network is employed to predict the frame-level importance scores used in generating video summaries. In addition, we explored different fusion methods for integrating different modalities of information. Our method demonstrates superior performance on both evaluation metrics, achieving F-measure of 0.528, Kendall’s $$\tau $$ of 0.076, and Spearman’s $$\rho $$ of 0.099 on the SumMe dataset, as well as F-measure of 0.612, Kendall’s $$\tau $$ of 0.155, and Spearman’s $$\rho $$ of 0.204 on the TVSum dataset. These results surpass the performance of most existing video summarization methods.
Graphormer-Based Contextual Reasoning Network for Small Object Detection

Different from common objects, it is usually difficult to obtain fine enough visual information for small objects. Existing context-based learning methods only focus on whether the local contextual information of the object can be better extracted, but these methods easily lead to the loss of other effective information in the scene. To address this issue, this study proposes a Graphomer-based Contextual Reasoning Network (GCRN), which improves the feature expression of objects by modeling and reasoning about contextual information. Specifically, the method first establishes contextual information encoding using a context relation module, which is used to capture the local contextual features of objects as well as the dependencies among objects. Then, all the contextual information is aggregated by Graphormer in the context reasoning module to enrich the visual features of small objects. We conduct extensive experiments on two public datasets (i.e. MSCOCO and TinyPerson). The results show that this method can enhance the feature expression of small objects to a certain extent, and has good performance in feature information transmission.
SSCL: Semi-supervised Contrastive Learning for Industrial Anomaly Detection

Anomaly detection is an important machine learning task that aims to identify data points that are inconsistent with normal data patterns. In real-world scenarios, it is common to have access to some labeled and unlabeled samples that are known to be either normal or anomalous. To make full use of both types of data, we propose a semi-supervised contrastive learning method that combines self-supervised contrastive learning and supervised contrastive learning, forming a new framework: SSCL. Our method can learn a data representation that can distinguish between normal and anomalous data patterns, based on limited labeled data and abundant unlabeled data. We evaluate our method on multiple benchmark datasets, including MNIST, CIFAR-10 and industrial anomaly detection MVtec, STC. The experimental results show that our method achieves superior performance on all datasets compared to existing state-of-the-art methods.
RCV2023 Challenges: Benchmarking Model Training and Inference for Resource-Constrained Deep Learning

This paper delves into the results of two resource-constrained deep learning challenges, part of the workshop on Resource-Efficient Deep Learning for Computer Vision (RCV) at ICCV 2023, focusing on memory and time limitations. The challenges garnered significant global participation and showcased a range of intriguing solutions. The paper outlines the problem statements for both tracks, summarizes baseline and top-performing approaches, and provides a detailed analysis of the methods used. While the presented solutions constitute promising initial progress, they represent the beginning of efforts needed to address this complex issue. We conclude by emphasizing the importance of sustained research efforts to fully address the challenges of resource-constrained deep learning.
A Novel Cross Frequency-Domain Interaction Learning for Aerial Oriented Object Detection

Aerial oriented object detection is a vital task in computer vision, receiving significant attention for its role in remote image understanding. However, most Convolutional Neural Networks (CNNs) methods easily ignore the frequency domain because they only focus on the spatial/channel interaction. To address these limitations, we propose a novel approach called Cross Frequency-domain Interaction Learning (CFIL) for aerial oriented object detection. Our method consists of two modules: the Extraction of Frequency-domain Features (EFF) module and the Interaction of Frequency-domain Features (IFF) module. The EFF module extracts frequency-domain information from the feature maps, enhancing the richness of feature information across different frequencies. The IFF module facilitates efficient interaction and fusion of the frequency-domain feature maps obtained from the EFF module across channels. Finally, these frequency-domain weights are combined with the time-domain feature maps. By enabling full and efficient interaction and fusion of EFF feature weights across channels, the IFF module ensures effective utilization of frequency-domain information. Extensive experiments are conducted on the DOTA V1.0, DOTA V1.5, and HRSC2016 datasets to demonstrate the competitive performance of the proposed CFIL in the aerial oriented object detection. Our code and models will be publicly released.
Mask-Guided Joint Single Image Specular Highlight Detection and Removal

Detecting and removing specular highlights is a highly challenging task that can effectively enhance the performance of other computer vision tasks in real-world scenarios. Traditional highlight removal algorithms struggle to accurately distinguish the differences between pure white or nearly white materials and highlights, while recent deep learning-based highlight removal algorithms suffer from complex network architectures, lack of flexibility, and limited object adaptability. To address these issues, we propose an end-to-end framework for single-image highlight detection and removal that utilizes mask guidance. Our framework adopts an encoder-decoder structure, with EfficientNet serving as the backbone network for feature extraction in the encoder. The decoder gradually restores the feature map to its original size through upsampling. In the highlight detection module, the network layers are deepened, and residual modules are introduced to extract more feature information and improve detection accuracy. In the highlight removal module, we introduce the Convolutional Block Attention Module, which dynamically learns the importance of each channel and spatial position in the input feature map. This helps the model better distinguish foreground and background and improve adaptability and accuracy in complex scenes.Experimental results demonstrate that our proposed method outperforms existing methods, as evaluated on the public SHIQ dataset through comparative experiments. Our method achieves superior performance in highlight detection and removal.
SVDML: Semantic and Visual Space Deep Mutual Learning for Zero-Shot Learning

The key challenge of zero-shot learning (ZSL) is how to identify novel objects for which no samples are available during the training process. Current approaches either align the global features of images to the corresponding class semantic vectors or use unidirectional attentions to locate the local visual features of images via semantic attributes to avoid interference from other noise in the image. However, they still have not found a way to establish a robust correlation between the semantic and visual representation. To solve the issue, we propose a Semantic and Visual space Deep Mutual Learning (SVDML), which consists of three modules: class representation learning, attribute embedding, and mutual learning, to establish the intrinsic semantic relations between visual features and attribute features. SVDML uses two kinds of prototype generators to separately guide the learning of global and local features of images and achieves interaction between two learning pipelines by mutual learning, so that promotes the recognition of the fine-grained features and strengthens the knowledge generalization ability in zero-shot learning. The proposed SVDML yields significant improvements over the strong baselines, leading to the new state-of the-art performances on three popular challenging benchmarks.
Temporal-Semantic Context Fusion for Robust Weakly Supervised Video Anomaly Detection

Video anomaly detection (VAD) has emerged as a vital and challenging task in computer vision, driven by the rapid advancement of surveillance videos. Weakly supervised learning is a critical branch in this field, with Multiple Instance Learning (MIL) standing out as the prevalent approach. In the weakly supervised VAD task, video context information provides crucial cues. However, the majority of studies in this field have primarily focused on the temporal context, neglecting the emphasis of semantic context in videos, which provides a deeper understanding of the observed video content. Therefore, this paper proposes a MIL-based framework for efficiently capturing video context information in weakly supervised VAD task. A GCN-based Temporal-Semantic context fusion module is employed to comprehensively capture both temporal and semantic context. Additionally, in order to enhance feature discrimination and learning robustness, our approach integrates highly effective anomaly classification learning and feature magnitude learning loss functions. Extensive experiments on two challenging datasets (i.e., ShanghaiTech and UCF-Crime) outperform some recent methods and demonstrate our approach’s effectiveness.
Dual-Memory Feature Aggregation for Video Object Detection

Recent studies on video object detection have shown the advantages of aggregating features across frames to capture temporal information, which can mitigate appearance degradation, such as occlusion, motion blur, and defocus. However, these methods often employ a sliding window or memory queue to store temporal information frame by frame, leading to discarding features of earlier frames over time. To address this, we propose a dual-memory feature aggregation framework (DMFA). DMFA simultaneously constructs a local feature cache and a global feature memory in a feature-wise updating way at different granularities, i.e., pixel level and proposal level. This approach can partially preserve key features across frames. The local feature cache stores the spatio-temporal contexts from nearby frames to boost the localization capacity, while the global feature memory enhances semantic feature representation by capturing temporal information from all previous frames. Moreover, we introduce contrastive learning to improve the discriminability of temporal features, resulting in more accurate proposal-level feature aggregation. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the ImageNet VID benchmark.
Relation-Guided Multi-stage Feature Aggregation Network for Video Object Detection

Video object detection task has received extensive research attention and various methods have been proposed. The quality of single frame in the original video is usually deteriorated by motion blur and object occlusion, which leads to the failure of detection. Although some methods have attempted to enhance the feature representation of each frame by aggregating temporal context information from other frames, the existing methods are usually sensitive to the change of object appearance and scale, which lead to false or missing detection. Therefore, in this paper, we propose a Relation-guided Multi-stage Feature Aggregation (RMFA) network for video object detection. First, a Multi-Stage Feature Aggregation (MSFA) framework is devised to aggregate the feature representation of global and local support frames in each stage. In this way, both global semantic information and local motion information could be better captured. Furthermore, a Multi-sources Feature Aggregation (MFA) module is proposed to enhance the quality of support frames, hence the feature representation of current frame could be improved. Finally, a Temporal Relation-Guided (TRG) module is proposed to improve the feature aggregation perception by supervising the semantic similarity relationships between different object proposals. Therefore, the model adaptability to selectively store valuable features could be enhanced. Qualitative and quantitative experimental results on the ImageNet VID dataset demonstrate that our model could achieve superior video object detection results against a number of the state-of-the-art ones. Especially, when object is occluded or under fast motion, our model shows outstanding performances.
Cross-Domain Soft Adaptive Teacher for Syn2Real Object Detection

Current state-of-the-art object detectors are constructed using supervised deep-learning approaches. These approaches require a large amount of annotated training data. Although synthetic image-generation methods can provide a large amount of annotated data, unsupervised transfer of object-recognition models from synthetic to real domains is a complicated problem given the large gap between the domains. To mitigate this problem, in this paper, we propose a general synthetic-to-real cross-domain object-detection framework. In this framework, we establish a simple mean teacher model for most detectors and propose a teacher–student framework named soft adaptive teacher (SAT). This leverages domain adversarial learning and domain-adaption augmentation to address the domain gap. Specifically, we alleviate bias by augmenting training samples with image-level adaptations for the student model. Moreover, we employ feature-level adversarial training in the student model, allowing features derived from the source and target domains to share similar distributions. Finally, we introduce the soft teacher mechanism to select reliable pseudo-labels for the teacher model. By tackling the model-bias issue using these strategies, our SAT model was found to achieve average precision values of 57.2% (55.7%) on the Sim10k to Cityscape (Sim10k to BDD100k) benchmarks, 3.1 (10.4) percentage points higher than the previous state-of-the-art methods. Furthermore, we achieved an average precision of 66.2% on the dataset for object detection in aerial images (DOTA), and this is 31.2% points higher than the results from the Faster RCNN model without domain adaptation trained only with labeled source domain images.