Reviewerly/exHarmony huggingface dataset:
DistgEPIT: Enhanced Disparity Learning for Light Field Image Super-Resolution

Light Field (LF) cameras capture rich information in 4D LF images by recording both intensity and angular directions, making it crucial to learn the inherent spatial-angular correlation in low-resolution (LR) images for superior results. Despite impressive progress made by several CNN-based deep methods and pioneering Transformer-based methods for LF image super resolution (SR), most of them fail to fully leverage the LF spatial-angular correlation and tend to perform poorly in scenes with varying disparities. In this paper, we propose a hybrid method called DistgEPIT that implements an enhanced disparity learning mechanism with both convolution-based and transformer-based modules. It enables the capture of angular correlation, refinement of adjacent disparities, and extraction of essential spatial features. Additionally, we introduce a Position-Sensitive Windowing (PSW) strategy to maintain consistency of disparity between the training and inference stages, which yields an average PSNR gain of 0.2 dB by replacing the traditional padding and windowing method. Extensive experiments with ablation studies demonstrate the effectiveness of our proposed method, which ranked 1st place in the NITRE2023 LF image SR challenge. The code is available at https://github.com/OpenMeow/NTIRE23_LFSR_DistgEPIT.
TEVAD: Improved video anomaly detection with captions

Video surveillance systems are used to enhance the public safety and private assets. Automatic anomaly detection is vital in such surveillance systems to reduce the human labor and its associated costs. Previous works only consider spatial-temporal features. In many complex real-world scenarios, such visual features are unable to capture the semantic meanings required to further improve accuracy. To deal with such issues, we propose a novel framework: Text Empowered Video Anomaly Detection (TEVAD) which utilizes both visual and text features. Text features complements the visual features as they are semantically rich. Specifically, we compute text features based on the captions of the videos to capture the semantic meanings of abnormal events and thus improve the overall performance of video anomaly detection. Extensive experiments demonstrate that our proposed framework achieves state-of-the-art results on four benchmark datasets (i.e. ShanghaiTech, UCF-Crime, XD-Violence, and UCSD-Pedestrians) and achieves improved robustness. We further analyze the captions to provide additional explainability for the anomalous videos identified by our proposed algorithm. Our codes are available at https://github.com/coranholmes/TEVAD.
Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion

In this work, we investigate the problem of Model-Agnostic Zero-Shot Classification (MA-ZSC), which refers to training non-specific classification architectures (downstream models) to classify real images without using any real images during training. Recent research has demonstrated that generating synthetic training images using diffusion models provides a potential solution to address MA-ZSC. However, the performance of this approach currently falls short of that achieved by large-scale vision-language models. One possible explanation is a potential significant domain gap between synthetic and real images. Our work offers a fresh perspective on the problem by providing initial insights that MA-ZSC performance can be improved by improving the diversity of images in the generated dataset. We propose a set of modifications to the text-to-image generation process using a pre-trained diffusion model to enhance diversity, which we refer to as our bag of tricks. Our approach shows notable improvements in various classification architectures, with results comparable to state-of-the-art models such as CLIP. To validate our approach, we conduct experiments on CIFAR10, CIFAR100, and EuroSAT, which is particularly difficult for zero-shot classification due to its satellite image domain. We evaluate our approach with five classification architectures, including ResNet and ViT. Our findings provide initial insights into the problem of MA-ZSC using diffusion models. All code is available at https://github.com/Jordan-HS/Diversity_is_Definitely_Needed
RL-CAM: Visual Explanations for Convolutional Networks using Reinforcement Learning

Convolutional Neural Networks (CNNs) are state-of-the-art models for computer vision tasks such as image classification, object detection, and segmentation. However, these models suffer from their inability to explain decisions, particularly in fields like healthcare and security, where interpretability is critical. Previous research has developed various methods for interpreting CNNs, including visualization-based approaches (e.g., saliency maps) that aim to reveal the underlying features used by the model to make predictions. In this work, we propose a novel approach that uses reinforcement learning to generate a visual explanation for CNNs. Our method considers the black-box CNN model and relies solely on the probability distribution of the model’s output to localize the features contributing to a particular prediction. The proposed reinforcement learning algorithm has an agent with two actions, a forward action that explores the input image and identifies the most sensitive region to generate a localization mask, and a reverse action that fine-tunes the localization mask. We evaluate the performance of our approach using multiple image segmentation metrics and compare it with existing visualization-based methods. The experimental results demonstrate that our proposed method outperforms the existing techniques, producing more accurate localization masks of regions of interest in the input images.
TFRGAN: Leveraging Text Information for Blind Face Restoration with Extreme Degradation

Blind face restoration aims to recover high-quality face images from unknown degraded low-quality images. Previous works that are based on geometric or generative priors have achieved impressive performance, but the task remains challenging, particularly when it comes to restoring severely degraded faces. To address this issue, we propose a novel approach TFRGAN, that leverages textual information to improve the restoration of extremely degraded face images. Specifically, we propose to generate a better and more accurate latent code for StyleGAN2 prior via fusing the text and image information in the latent code space. Besides, extracted textual features are used to modulate the decoding features to obtain more realistic and natural facial images with more reasonable details. Experimental results demonstrate the superiority of the proposed method for restoring severely degraded face images.
Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations

The rapid development and application of foundation models have revolutionized the field of artificial intelligence. Large diffusion models have gained significant attention for their ability to generate photorealistic images and support various tasks. On-device deployment of these models provides benefits such as lower server costs, offline functionality, and improved user privacy. However, common large diffusion models have over 1 billion parameters and pose challenges due to restricted computational and memory resources on devices. We present a series of implementation optimizations for large diffusion models that achieve the fastest reported inference latency to-date(under 12 seconds for Stable Diffusion 1.4 without INT8 quantization for a 512 × 512 image with 20 iterations) on GPU-equipped mobile devices. These enhancements broaden the applicability of generative AI and improve the overall user experience across a wide range of devices.
Adaptive RoI with pretrained models for Automated Retail Checkout

In this paper, we present a solution for automatic checkout in a retail store as a part of AI City Challenge 2023 Track 4. We propose a methodology which involves usage of pretrained Yolov5 models to detect person and media pipe models to detect hands of the person. This information is utilized to compute the Region of Interest (RoI) which is adaptive in nature. Afterwards, a custom trained object detection model is used detect products in the frame. We then use a tracker to track the products across video frames to avoid duplicated counting. The method is evaluated on the AI City challenge 2023 – Track 4 and gets the F1 score 0.6571 on the test A set, which places us on 6th place on the public leader board. The code is made public and available on GitHub. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Thermal Image Super-Resolution Challenge Results - PBVS 2023

This paper presents the results of two tracks from the fourth Thermal Image Super-Resolution (TISR) challenge, held at the Perception Beyond the Visible Spectrum (PBVS) 2023 workshop. Track-1 uses the same thermal image dataset as previous challenges, with 951 training images and 50 validation images at each resolution. In this track, two evaluations were conducted: the first consists of generating a SR image from a HR thermal noisy image downsampled by four, and the second consists of generating a SR image from a mid-resolution image and compare it with its semi-registered HR image (acquired with another camera). The results of Track-1 outperformed those from last year’s challenge. On the other hand, Track-2 uses a new acquired dataset consisting of 160 registered visible and thermal images of the same scenario for training and 30 validation images. This year, more than 150 teams participated in the challenge tracks, demonstrating the community’s ongoing interest in this topic.
Exploring Anomaly Detection Techniques for Enhancing VANET Availability

In VANETs the quicker an anomaly can be detected and properly classified, the faster the issue it causes can be dealt with. In this paper, we propose deploying Vehicular Edge Computing (VEC) to detect anomalies characterized by the absence of message exchange between vehicles. The detection of anomalies spread across an urban area can possibly benefit from the processing and storage capacity of the VAC. VANETs are dynamic networks, whose vehicle density varies considerably over time. VANETs components do not usually store much information, making it difficult to efficiently detect the loss of messages from multiple vehicles across the neighborhoods of a city. The article aims to investigate whether the use of VEC and conventional anomaly detection techniques benefits loss of messages detection by increasing its fault coverage capacity. To measure fault coverage, fault injection experiments were conducted using simulation.
Multimodal Object Detection by Channel Switching and Spatial Attention

Multimodal object detection has attracted great attention in recent years since the information specific to different modalities can complement each other and effectively improve the accuracy and stability of the detection model. However, compared to processing the inputs from a single modality, fusing information from multiple modalities can significantly increase the computational complexity of the model, thus impairing its efficiency. Therefore the multi-modal fusion module needs to be carefully designed to enhance the performance of the detection model while keeping the computational consumption low. In this paper, we propose a novel lightweight fusion module that can efficiently fuse the inputs from different modalities using channel switching and spatial attention (CSSA). The effectiveness and generalizability of the module are tested using two public multimodal datasets LLVIP and FLIR, both of which comprise paired infrared (IR) and visible (RGB) images. The experiments demonstrate that the proposed CSSA module can substantially improve the accuracy of multimodal object detection without consuming excessive computing resources.
A Data-Centric Solution to NonHomogeneous Dehazing via Vision Transformer

Recent years have witnessed an increased interest in image dehazing. Many deep learning methods have been proposed to tackle this challenge, and have made significant accomplishments dealing with homogeneous haze. However, these solutions cannot maintain comparable performance when they are applied to images with non-homogeneous haze, e.g., NH-HAZE23 dataset introduced by NTIRE challenge. One of the reasons for such failures is that non-homogeneous haze does not obey one of the assumptions that is required for modeling homogeneous haze. In addition, a large number of pairs of non-homogeneous hazy image and the clean counterpart is required using traditional end-to-end training approaches, while NH-HAZE23 dataset is of limited quantities. Although it is possible to augment the NH-HAZE23 dataset by leveraging other non-homogeneous dehazing datasets, we observe that it is necessary to design a proper data-preprocessing technique that reduces the distribution gaps between the target dataset and the augmented one. This finding indeed aligns with the essence of data-centric AI. With a novel network architecture and a principled data-preprocessing approach that systematically enhances data quality, we present an innovative dehazing method. Specifically, we apply RGB-channel-wise transformations on the augmented datasets, and incorporate the state-of-the-art transformers as the backbone in the two-branch framework. We conduct extensive experiments and ablation studies to demonstrate the effectiveness of our proposed method. The source code is available at https://github.com/yangyiliu21/ntire2023_ITBdehaze.
BokehOrNot: Transforming Bokeh Effect with Image Transformer and Lens Metadata Embedding

Bokeh effect is an optical phenomenon that offers a pleasant visual experience, typically generated by high-end cameras with wide aperture lenses. The task of bokeh effect transformation aims to produce a desired effect in one set of lenses and apertures based on another combination. Current models are limited in their ability to render a specific set of bokeh effects, primarily transformations from sharp to blur. In this paper, we propose a novel universal method for embedding lens metadata into the model and introducing a loss calculation method using alpha masks from the newly released Bokeh Effect Transformation Dataset (BETD) [3]. Based on the above techniques, we propose the BokehOrNot model, which is capable of producing both blur-to-sharp and sharp-to-blur bokeh effect with various combinations of lenses and aperture sizes. Our proposed model outperforms current leading bokeh rendering and image restoration models and renders visually natural bokeh effects. Our code is available at: https://github.com/indicator0/bokehornot.
The mediating role of security anxiety in internet threat avoidance behavior

In the prior technology threat avoidance literature, inconsistent results are found in examining the relationships between individuals’ perceived susceptibility, perceived severity, and perceived threat, as well as the intention to avoid security threats. One possible explanation for this paradox is lacking consideration of the role of individuals’ negative emotions while facing threats, which is rarely studied before. Therefore, the current study aims to extend the landscape of technology threat avoidance theory by exploring the role of security anxiety in individuals’ responses to security threats from the Internet. Our results demonstrate that Internet users’ risk aversion is strongly influenced by their perception of security threats, response efficacy, and security self-efficacy within the online environment. Also, our results indicate that individuals’ security anxiety can partially mediate the relationship between perceived threat and Internet threat avoidance behavior.
EPI-Guided Cost Construction Network for Light Field Disparity Estimation

Recent learning-based light field (LF) disparity estimation methods construct cost volume by sequentially shifting each sub-aperture image (SAI) with a series of predefined offsets. They only use the visual information of SAIs and lose the geometry of LF. In this paper, we design a simple network that can cleverly integrate EPI features with cost volume to estimate the disparity. Firstly, we propose an efficient EPI extraction module to use abundant line characteristics. Secondly, we offer an EPI-Cost volume construction module that can create volume guided by the EPI line and the color consistency of images. Finally, after completing it, we adopt an inter-volume fusion module to considerably correlate the validity of EPI lines in both directions. Experimental results show the proposed method achieves state-of-the-art performance in the quantitative and qualitative evaluation of the UrbanLF-Syn dataset.
AsConvSR: Fast and Lightweight Super-Resolution Network with Assembled Convolutions

In recent years, videos and images in 720p (HD), 1080p (FHD) and 4K (UHD) resolution have become more popular for display devices such as TVs, mobile phones and VR. However, these high resolution images cannot achieve the expected visual effect due to the limitation of the internet bandwidth, and bring a great challenge for super-resolution networks to achieve real-time performance. Following this challenge, we explore multiple efficient network designs, such as pixel-unshuffle, repeat upscaling, and local skip connection removal, and propose a fast and lightweight super-resolution network. Furthermore, by analyzing the applications of the idea of divide-and-conquer in super-resolution, we propose assembled convolutions which can adapt convolution kernels according to the input features. Experiments suggest that our method outperforms all the state-of-the-art efficient super-resolution models, and achieves optimal results in terms of runtime and quality. In addition, our method also wins the first place in NTIRE 2023 Real-Time Super-Resolution - Track 1 (×2). The code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/AsConvSR
SportsPose - A Dynamic 3D sports pose dataset

Accurate 3D human pose estimation is essential for sports analytics, coaching, and injury prevention. However, existing datasets for monocular pose estimation do not adequately capture the challenging and dynamic nature of sports movements. In response, we introduce SportsPose, a large-scale 3D human pose dataset consisting of highly dynamic sports movements. With more than 176,000 3D poses from 24 different subjects performing 5 different sports activities, SportsPose provides a diverse and comprehensive set of 3D poses that reflect the complex and dynamic nature of sports movements. Contrary to other markerless datasets we have quantitatively evaluated the precision of SportsPose by comparing our poses with a commercial marker-based system and achieve a mean error of 34.5 mm across all evaluation sequences. This is comparable to the error reported on the commonly used 3DPW dataset. We further introduce a new metric, local movement, which describes the movement of the wrist and ankle joints in relation to the body. With this, we show that SportsPose contains more movement than the Human3.6M and 3DPW datasets in these extremum joints, indicating that our movements are more dynamic. The dataset with accompanying code can be downloaded from our website <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . We hope that SportsPose will allow researchers and practitioners to develop and evaluate more effective models for the analysis of sports performance and injury prevention. With its realistic and diverse dataset, SportsPose provides a valuable resource for advancing the state-of-the-art in pose estimation in sports.
Sign Language Translation from Instructional Videos

The advances in automatic sign language translation (SLT) to spoken languages have been mostly benchmarked with datasets of limited size and restricted domains. Our work advances the state of the art by providing the first baseline results on How2Sign, a large and broad dataset.We train a Transformer over I3D video features, using the reduced BLEU as a reference metric for validation, instead of the widely used BLEU score. We report a result of 8.03 on the BLEU score, and publish the first open-source implementation of its kind to promote further advances.
CLVOS23: A Long Video Object Segmentation Dataset for Continual Learning

Continual learning in real-world scenarios is a major challenge. A general continual learning model should have a constant memory size and no predefined task boundaries, as is the case in semi-supervised Video Object Segmentation (VOS), where continual learning challenges particularly present themselves in working on long video sequences. In this article, we first formulate the problem of semi-supervised VOS, specifically online VOS, as a continual learning problem, and then secondly provide a public VOS dataset, CLVOS23, focusing on continual learning. Finally, we propose and implement a regularization-based continual learning approach on LWL, an existing online VOS baseline, to demonstrate the efficacy of continual learning when applied to online VOS and to establish a CLVOS23 baseline. We apply the proposed baseline to the Long Videos dataset as well as to two short video VOS datasets, DAVIS16 and DAVIS17. To the best of our knowledge, this is the first time that VOS has been defined and addressed as a continual learning problem. The proposed CLVOS23 dataset has been released at https://github.com/Amir4g/CLVOS23.
PanopticVis: Integrated Panoptic Segmentation for Visibility Estimation at Twilight and Night

Visibility affects traffic flow and control on city roads, highways, and runways. Visibility distance or level is an important measure for predicting the risk on the road. Particularly, it is known that traffic accidents can be raised at foggy twilight and night. Cameras monitor visual conditions like fog. However, only a few papers have tackled such nighttime vision with visibility estimation. This paper proposes a Panoptic Segmentation-based foggy night visibility estimation integrating multiple Deep Learning models: DeepReject/Depth/ Scene/Vis/Fog using single images. We call PanopticVis. DeepFog is trained for no-fog and heavy fog. DeepVis for medium fog is trained by annotated visibility physical scales in a regression manner. DeepDepth is improved to be robust to strong local illumination. DeepScene panoptic-segments scenes with stuff and things, booted by Deep-Depth. DeepReject conducts adversarial visual conditions: strong illumination and darkness. Notably, the proposed multiple Deep Learning framework provides high efficiency in memory, cost, and easy-tomaintenance. Unlike previous synthetic test images, experimental results show the effectiveness of the proposed integrated multiple Deep Learning approaches for estimating visibility distances on real foggy night roads. The superiority of PanopticVis is demonstrated over state-of-the-art panoptic-based Deep Learning models in terms of stability, robustness, and accuracy.