Reviewerly/exHarmony huggingface dataset:
TempT: Temporal consistency for Test-time adaptation

We introduce Temporal consistency for Test-time adaptation (TempT), a novel method for test-time adaptation on videos through the use of temporal coherence of predictions across sequential frames as a self-supervision signal. TempT is an approach with broad potential applications in computer vision tasks, including facial expression recognition (FER) in videos. We evaluate TempT’s performance on the AffWild2 dataset. Our approach focuses solely on the unimodal visual aspect of the data and utilizes a popular 2D CNN backbone, in contrast to larger sequential or attention-based models used in other approaches. Our preliminary experimental results demonstrate that TempT has competitive performance compared to the previous years’ reported performances, and its efficacy provides a compelling proof-of-concept for its use in various real-world applications.
DACNet: A Deep Automated Checkout Network with Selective Deblurring

Automated checkout systems have become increasingly popular as the state-of-the-art deep learning models are efficient and accurate enough for this to become a reality. However, challenges still exist due to the differences between synthetic training data and real-life products, the blurred product images captured during the checkout process, and discontinuous detections due to product similarities or tracking misses. This paper presents a robust deep learning YOLO-based pipeline, DACNet, that counters the above challenges. During training, data augmentation involving overlaying training images onto expected backgrounds creates a more diverse and accurate training dataset. When inferencing, selective deblurring is also incorporated to enhance the clarity of the items to be recognized while maintaining efficiency. And to improve accuracy further, we introduced a retrospective checking algorithm that analyzes previous detections and corrects any inaccuracies due to flickering detections or incorrect tracking results. Together, this pipeline ensures a network that produces reliable training results and high prediction accuracies even in complex retail environments with multiple items present. The proposed method has been submitted to 2023 AI City Challenge by NVIDIA and achieved a top-3 finish on the test set A with an F1-score of 0.8254. Our code is open sourced here: https://github.com/cycv5/AICityChallenge.
Face Transformer: Towards High Fidelity and Accurate Face Swapping

Face swapping aims to generate swapped images that fuse the identity of source faces and the attributes of target faces. Most existing works address this challenging task through 3D modelling or generation using generative adversarial networks (GANs), but 3D modelling suffers from limited reconstruction accuracy and GANs often struggle in preserving subtle yet important identity details of source faces (e.g., skin colors, face features) and structural attributes of target faces (e.g., face shapes, facial expressions). This paper presents Face Transformer, a novel face swapping network that can accurately preserve source identities and target attributes simultaneously in the swapped face images. We introduce a transformer network for the face swapping task, which learns high-quality semantic-aware correspondence between source and target faces and maps identity features of source faces to the corresponding region in target faces. The high-quality semantic-aware correspondence enables smooth and accurate transfer of source identity information with minimal modification of target shapes and expressions. In addition, our Face Transformer incorporates a multi-scale transformation mechanism for preserving the rich fine facial details. Extensive experiments show that our Face Transformer achieves superior face swapping performance qualitatively and quantitatively.
Comprehensive Visual Features and Pseudo Labeling for Robust Natural Language-based Vehicle Retrieval

Vehicle retrieval has become crucial for public transportation and intelligent transportation systems due to the exponential development of large-scale transportation videos. Vehicle re-identification and vehicle tracking are the main components of most vision-based vehicle recovery systems. Unfortunately, the limited amount of information provided by traffic video feeds limits the vision-based vehicle retrieval algorithm’s efficacy. Therefore, this article proposes a contrastive cross-modal vehicle retrieval approach to maximize the complementarity of natural language and visual representations. An efficient method to fuse multiple input image features is also proposed to extract comprehensive information from various vehicles along with pseudo labeling and efficient post-processing techniques to enhance retrieval accuracy. The proposed method achieved the 3rd ranking of Mean Reciprocal Rank (MRR) score of 0.4795 on the test set for the Challenge Track 2: Tracked-Vehicle Retrieval by Natural Language Descriptions 2023. Source code for the proposed approaches is openly accessible at https://github.com/anminhhung/AICity-2023-Track2.
Hierarchical Clustering and Refinement for Generalized Multi-Camera Person Tracking

Multi-camera person tracking has gained significant attention in recent times, owing to its widespread application in surveillance scenarios. However, this task is challenging due to the variance viewpoints, heavy occlusion, and illumination changes. In order to tackle these challenges, we propose a novel Hierarchical Clustering and Refinement framework for Generalized Multi-Camera Person Tracking. Specifically, our framework comprises two main components: hierarchical clustering and hierarchical refinement. Compared with directly clustering tracklets among multiple cameras, our hierarchical clustering strategy can progressively assign tracklets to correct targets. Nevertheless, the clustering and tracking process would inevitably produce incorrect matchings. Therefore, a hierarchical refinement strategy is proposed to reduce these incorrect matches which includes: intra-camera tracklet level refinement, appearance refinement, spatial-temporal refinement, and face refinement. Extensive experiments show the effectiveness of our method, which achieves 92% IDF1 in 2023 AI CITY CHALLENGE track1, ranking 5th on the leaderboard.
An Improved Association Pipeline for Multi-Person Tracking

The association task of assigning detections to tracks in multi-person tracking has recently been improved by integration of a second matching stage for low-confident detections that are usually discarded in the tracking process. Despite its success, we find that this two stage matching has some weaknesses. For example, high-confident detections are preferred over low-confident detections in any case, even if the low-confident ones are more accurate. Therefore, a Combined Matching (CM) is proposed which considers all possible assignments simultaneously in a single matching stage and thus improves the association accuracy. Moreover, shortcomings of existing motion and appearance distance combinations are identified and a novel Combined Distance (CD) for motion and appearance information is introduced that significantly outperforms previous fusion approaches. Furthermore, we propose an Occlusion Aware Initialization (OAI) which prevents the start of ghost tracks from duplicate detections under occlusion. The effectiveness of our components is shown with extensive ablative experiments and the competitiveness of our tracker is demonstrated on the MOT17 and MOT20 benchmarks, where the current state-of-the-art is notably surpassed.
Unlimited-Size Diffusion Restoration

Recently, using diffusion models for zero-shot image restoration (IR) has become a new hot paradigm. This type of method only needs to use the pre-trained off-the-shelf diffusion models, without any finetuning, and can directly handle various IR tasks. The upper limit of the restoration performance depends on the pre-trained diffusion models, which are in rapid evolution. However, current methods only discuss how to deal with fixed-size images, but dealing with images of arbitrary sizes is very important for practical applications. This paper focuses on how to use those diffusion-based zero-shot IR methods to deal with any size while maintaining the excellent characteristics of zero-shot. A simple way to solve arbitrary size is to divide it into fixed-size patches and solve each patch independently. But this may yield significant artifacts since it neither considers the global semantics ofall patches nor the local information of adjacent patches. Inspired by the Range-Null space Decomposition, we propose the Mask-Shift Restoration to address local incoherence and propose the Hierarchical Restoration to alleviate out-of-domain issues. Our simple, parameterfree approaches can be used not only for image restoration but also for image generation of unlimited sizes, with the potential to be a general tool for diffusion models. Code: https://github.com/wyhuai/DDNM/tree/main/hq_demo.
SCONE-GAN: Semantic Contrastive learning-based Generative Adversarial Network for an end-to-end image translation

SCONE-GAN presents an end-to-end image translation, which is shown to be effective for learning to generate realistic and diverse scenery images. Most current image-to-image translation approaches are devised as two mappings: a translation from the source to target domain and another to represent its inverse. While successful in many applications, these approaches may suffer from generating trivial solutions with limited diversity. That is because these methods learn more frequent associations rather than the scene structures. To mitigate the problem, we propose SCONE-GAN that utilises graph convolutional networks to learn the objects dependencies, maintain the image structure and preserve its semantics while transferring images into the target domain. For more realistic and diverse image generation we introduce style reference image. We enforce the model to maximize the mutual information between the style image and output. The proposed method explicitly maximizes the mutual information between the related patches, thus encouraging the generator to produce more diverse images. We validate the proposed algorithm for image-to-image translation and stylizing outdoor images. Both qualitative and quantitative results demonstrate the effectiveness of our approach on four dataset.
GAN-based Vision Transformer for High-Quality Thermal Image Enhancement

Generative Adversarial Networks (GANs) have shown an outstanding ability to generate high-quality images with visual realism and similarity to real images. This paper presents a new architecture for thermal image enhancement. Precisely, the strengths of architecture-based vision transformers and generative adversarial networks are exploited. The thermal loss feature introduced in our approach is specifically used to produce high-quality images. Thermal image enhancement also relies on fine-tuning based on visible images, resulting in an overall improvement in image quality. A visual quality metric was used to evaluate the performance of the proposed architecture. Significant improvements were found over the original thermal images and other enhancement methods established on a subset of the KAIST dataset. The performance of the proposed enhancement architecture is also verified on the detection results by obtaining better performance with a considerable margin regarding different versions of the YOLO detector.
A Single Residual Network with ESA Modules and Distillation

Although there are many methods based on deep learning that have superior performance on single image super-resolution (SISR), it is difficult to run in real time on devices with limited computing power. Some recent studies have found that simply relying on reducing parameters or reducing the theoretical FLOPs of the model does not speed up the inference time of the network in a practical sense. Actual speed on the device is probably a better measure than FLOPs. In this work, we propose a new single residual network (SRN). On the one hand, we try to introduce and optimize an attention mechanism module to improve the performance of the network with a relatively small speed loss. On the other hand, we find that residuals in residual blocks do not have a positive impact on networks with adjusted ESA. Therefore, the residual of the network residual block is removed, which not only improves the speed of the network, but also improves the performance of the network. Finally, we reduced the number of channels and the number of residual blocks of the classic model EDSR, and removed the last convolution before the long residual. We set this tuned EDSR as the teacher model and our newly proposed SRN as the student model. Under the joint effect of the original loss and the distillation loss, the performance of the network can be improved without losing the inference time. Combining the above strategies, our proposed model runs much faster than similarly performing models. As an example, we built a Fast and Efficient Network (SRN) and its small version SRN-S, which run 30%-37% faster than the state-of-the-art EISR model: a paper champion RLFN. Furthermore, the shallow version of SRN-S achieves the second-shortest inference time as well as the second-smallest number of activations in the NTIRE2023 challenge. Code will be available at https://github.com/wnxbwyc/SRN.
Spectral Transfer Guided Active Domain Adaptation For Thermal Imagery

The exploitation of visible spectrum datasets has led deep networks to show remarkable success. However, real-world tasks include low-lighting conditions which arise performance bottlenecks for models trained on large-scale RGB image datasets. Thermal IR cameras are more robust against such conditions. Therefore, the usage of thermal imagery in real-world applications can be useful. Unsupervised domain adaptation (UDA) allows transferring information from a source domain to a fully unlabeled target domain. Despite substantial improvements in UDA, the performance gap between UDA and its supervised learning counterpart remains significant. By picking a small number of target samples to annotate and using them in training, active domain adaptation tries to mitigate this gap with minimum annotation expense. We propose an active domain adaptation method in order to examine the efficiency of combining the visible spectrum and thermal imagery modalities. When the domain gap is considerably large as in the visible-to-thermal task, we may conclude that the methods without explicit domain alignment cannot achieve their full potential. To this end, we propose a spectral transfer guided active domain adaptation method to select the most informative unlabeled target samples while aligning source and target domains. We used the large-scale visible spectrum dataset MS-COCO as the source domain and the thermal dataset FLIR ADAS as the target domain to present the results of our method. Extensive experimental evaluation demonstrates that our proposed method outperforms the state-of-the-art active domain adaptation methods. The code and models are publicly available. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Making the V in Text-VQA Matter

Text-based VQA aims at answering questions by reading the text present in the images. It requires a large amount of scene-text relationship understanding compared to the VQA task. Recent studies have shown that the question-answer pairs in the dataset are more focused on the text present in the image but less importance is given to visual features and some questions do not require understanding the image. The models trained on this dataset predict biased answers due to the lack of understanding of visual context. For example, in questions like "What is written on the signboard?", the answer predicted by the model is always "STOP" which makes the model to ignore the image. To address these issues, we propose a method to learn visual features (making V matter in TextVQA) along with the OCR features and question features using VQA dataset as external knowledge for Text-based VQA. Specifically, we combine the TextVQA dataset and VQA dataset and train the model on this combined dataset. Such a simple, yet effective approach increases the understanding and correlation between the image features and text present in the image, which helps in the better answering of questions. We further test the model on different datasets and compare their qualitative and quantitative results.
Image Inpainting with Hypergraphs for Resolution Improvement in Scanning Acoustic Microscopy

Scanning Acoustic Microscopy (SAM) uses high-frequency acoustic waves to generate non-ionizing, label-free images of the surface and internal structures of industrial objects and biological specimens. The resolution of SAM images is limited by several factors such as the frequency of excitation signals, the signal-to-noise ratio, and the pixel size. We propose to use a hypergraphs image inpainting technique for SAM that fills in missing information to improve the resolution of the SAM image. We compared the performance of our technique with four other different techniques based on generative adversarial networks (GANs), including AOTGAN, DeepFill v2, Edge-Connect and DMFN. Our results show that the hypergraphs image inpainting model provides the SOTA average SSIM of 0.82 with a PSNR of 27.96 for 4× image size enhancement over the raw SAM image. We emphasize the importance of hypergraphs' interpretability to bridge the gap between human and machine perception, particularly for robust image recovery tools for acoustic scan imaging. We show that combining SAM with hypergraphs can yield more noise-robust explanations.
On Advantages of Mask-level Recognition for Outlier-aware Segmentation

Most dense recognition approaches bring a separate decision in each particular pixel. These approaches deliver competitive performance in usual closed-set setups. However, important applications in the wild typically require strong performance in presence of outliers. We show that this demanding setup greatly benefits from mask-level predictions, even in the case of non-finetuned baseline models. Moreover, we propose an alternative formulation of dense recognition uncertainty that effectively reduces false positive responses at semantic borders. The proposed formulation produces a further improvement over a very strong baseline and sets the new state of the art in outlier-aware semantic segmentation with and without training on negative data. Our contributions also lead to performance improvement in a recent panoptic setup. In-depth experiments confirm that our approach succeeds due to implicit aggregation of pixel-level cues into mask-level predictions.
A Closer Look at Rehearsal-Free Continual Learning *

Continual learning is a setting where machine learning models learn novel concepts from continuously shifting training data, while simultaneously avoiding degradation of knowledge on previously seen classes which may disappear from the training data for extended periods of time (a phenomenon known as the catastrophic forgetting problem). Current approaches for continual learning of a single expanding task (aka class-incremental continual learning) require extensive rehearsal of previously seen data to avoid this degradation of knowledge. Unfortunately, rehearsal comes at a cost to memory, and it may also violate data-privacy. Instead, we explore combining knowledge distillation and parameter regularization in new ways to achieve strong continual learning performance without rehearsal. Specifically, we take a deep dive into common continual learning techniques: prediction distillation, feature distillation, L2 parameter regularization, and EWC parameter regularization. We first disprove the common assumption that parameter regularization techniques fail for rehearsal-free continual learning of a single, expanding task. Next, we explore how to leverage knowledge from a pre-trained model in rehearsal-free continual learning and find that vanilla L2 parameter regularization outperforms EWC parameter regularization and feature distillation. Finally, we explore the recently popular ImageNet-R benchmark, and show that L2 parameter regularization implemented in self-attention blocks of a ViT transformer outperforms recent popular prompting for continual learning methods.
Learning Epipolar-Spatial Relationship for Light Field Image Super-Resolution

Light field (LF) imaging has become increasingly popular in recent years for capturing and processing visual information. A significant challenge in LF processing is super-resolution (SR), which aims to enhance the resolution of low-resolution LF images. This article proposes a new LF image super-resolution (LFSR) approach that leverages the epipolar-spatial relationship within the LF. To train a deep neural network for LFSR, the proposed method involves extracting three types of information from the LF: spatial, horizontal epipolar, and vertical epipolar. Experimental results demonstrate the effectiveness of the proposed approach compared with state-of-the-art (SOTA) performance, as evidenced by quantitative metrics and visual quality. In addition, we conducted ablation studies to assess the effectiveness of each type of information and gain insights into the underlying mechanisms of the proposed method. Our approach achieved competitive results on the NTIRE 2023 Light Field Image Super-Resolution Challenge: our proposed model was ranked 10th on the test set and 6th on the validation set among 148 participants. Paper’s code is available at: https://github.com/ahmeddiefy/EpiS_LFSR.
MIPI 2023 Challenge on RGBW Remosaic: Methods and Results

Developing and integrating advanced image sensors with novel algorithms in camera systems are prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for an in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). With the success of the 1st MIPI Workshop@ECCV 2022, we introduce the second MIPI challenge, including four tracks focusing on novel image sensors and imaging algorithms. This paper summarizes and reviews the RGBW Joint Remosaic and Denoise track on MIPI 2023. In total, 81 participants were successfully registered, and 4 teams submitted results in the final testing phase. The final results are evaluated using objective metrics, including PSNR, SSIM, LPIPS, and KLD. A detailed description of the top three models developed in this challenge is provided in this paper. More details of this challenge and the link to the dataset can be found at https://mipi-challenge.org/MIPI2023/.
FLIGHT Mode On: A Feather-Light Network for Low-Light Image Enhancement

Low-light image enhancement (LLIE) is an ill-posed inverse problem due to the lack of knowledge of the desired image which is obtained under ideal illumination conditions. Low-light conditions give rise to two main issues: a suppressed image histogram and inconsistent relative color distributions with low signal-to-noise ratio. In order to address these problems, we propose a novel approach named FLIGHT-Net using a sequence of neural architecture blocks. The first block regulates illumination conditions through pixel-wise scene dependent illumination adjustment. The output image is produced in the output of the second block, which includes channel attention and denoising sub-blocks. Our highly efficient neural network architecture delivers state-of-the-art performance with only 25K parameters. The method’s code, pretrained models and resulting images will be publicly available.