Reviewerly/exHarmony huggingface dataset:
Mitigating Catastrophic Interference using Unsupervised Multi-Part Attention for RGB-IR Face Recognition

Modern algorithms for RGB-IR facial recognition— a challenging problem where infrared probe images are matched with visible gallery images—leverage precise and accurate guidance from curated (i.e., labeled) data to bridge large spectral differences. However, supervised cross-spectral face recognition methods are often extremely sensitive due to over-fitting to labels, performing well in some settings but not in others. Moreover, when fine-tuning on data from additional settings, supervised cross-spectral face recognition are prone to catastrophic forgetting. Therefore, we propose a novel unsupervised framework for RGB-IR face recognition to minimize the cost and time inefficiencies pertaining to labeling large-scale, multispectral data required to train supervised cross-spectral recognition methods and to alleviate the effect of forgetting by removing over dependence on hard labels to bridge such large spectral differences. The proposed framework integrates an efficient backbone network architecture with part-based attention models, which collectively enhances common information between visible and infrared faces. Then, the framework is optimized using pseudo-labels and a new cross-spectral memory bank loss. This framework is evaluated on the ARL-VTF and TUFTS datasets, achieving 98.55% and 43.28% true accept rate, respectively. Additionally, we analyze effects of forgetting and show that our framework is less prone to these effects.
Exploring Video Frame Redundancies for Efficient Data Sampling and Annotation in Instance Segmentation

In recent years, deep neural network architectures and learning algorithms have greatly improved the performance of computer vision tasks. However, acquiring and annotating large-scale datasets for training such models can be expensive. In this work, we explore the potential of reducing dataset sizes by leveraging redundancies in video frames, specifically for instance segmentation. To accomplish this, we investigate two sampling strategies for extracting keyframes, uniform frame sampling with adjusted stride (UFS) and adaptive frame sampling (AFS), which employs visual (Optical flow, SSIM) or semantic (feature representations) dissimilarities measured by learning free methods. In addition, we show that a simple copy-paste augmentation can bridge the big mAP gap caused by frame reduction. We train and evaluate Mask R-CNN with the BDD100K MOTS dataset and verify the potential of reducing training data by extracting keyframes in the video. With only 20% of the data, we achieve similar performance to the full dataset mAP; with only 33% of the data, we surpass it. Lastly, based on our findings, we offer practical solutions for developing effective sampling methods and data annotation strategies for instance segmentation models. Supplementary on https://github.com/jihun-yoon/EVFR.
Temporally Averaged Regression for Semi-Supervised Low-Light Image Enhancement

Constructing annotated paired datasets for low-light image enhancement is complex and time-consuming, and existing deep learning models often generate noisy outputs or misinterpret shadows. To effectively learn intricate relationships between features in image space with limited labels, we introduce a deep learning model with a backbone structure that incorporates both spatial and layer-wise dependencies. The proposed model features a baseline image-enhancing network with spatial dependencies and an optimized layer attention mechanism to learn feature sparsity and importance. We present a progressive supervised loss function for improvement. Furthermore, we propose a novel Multi-Consistency Regularization (MCR) loss and integrate it within a Multi-Consistency Mean Teacher (MCMT) framework, which enforces agreement on high-level features and incorporates intermediate features for better understanding of the entire image. By combining the MCR loss with the progressive supervised loss, student network parameters can be updated in a single step. Our approach achieves significant performance improvements using fewer labeled data and unlabeled low-light images within our semi-supervised framework. Qualitative evaluations demonstrate the effectiveness of our method in leveraging comprehensive dependencies and unlabeled data for low-light image enhancement.
Unsupervised Style-based Explicit 3D Face Reconstruction from Single Image

Inferring 3D object structures from a single image is an ill-posed task due to depth ambiguity and occlusion. Typical resolutions in the literature include leveraging 2D or 3D ground truth for supervised learning, as well as imposing hand-crafted symmetry priors or using an implicit representation to hallucinate novel viewpoints for unsupervised methods. In this work, we propose a general adversarial learning framework for solving Unsupervised 2D to Explicit 3D Style Transfer (UE3DST). Specifically, we merge two architectures: the unsupervised explicit 3D reconstruction network of Wu et al. and the Generative Adversarial Network (GAN) named StarGAN-v2. We experiment across three facial datasets (Basel Face Model, 3DFAW and CelebA-HQ) and show that our solution is able to outperform well established solutions such as DepthNet in 3D reconstruction and Pix2NeRF in conditional style transfer, while we also justify the individual contributions of our model components via ablation. In contrast to the aforementioned baselines, our scheme produces features for explicit 3D rendering, which can be manipulated and utilized in downstream tasks.
Kappa Angle Regression with Ocular Counter-Rolling Awareness for Gaze Estimation

Conventional appearance-based 3D gaze estimation methods generally use the roll of the head pose to represent the eyeball’s roll status by default. To reduce degrees of freedom of head poses, a normalization step was proposed to apply global transformations to images to make heads upright and eyelids horizontal. However, due to the ocular countering-rolling (OCR) response, the eyeball will rotate in the opposite direction when the head tilts to the side. After normalization, the eyeball will have an extra roll compared to the roll status of the eyeball when the head is not tilted. This roll from the OCR response causes a changed orientation of the eyeball in normalized eye images, which represents the roll status of the anatomical structure inside the eyeball and consequently affects gaze directions. Thus in this work, we propose a pipeline to regress the person-dependent anatomical variation as a calibration process with considering the OCR response, which can work with our proposed eye-image-based person-independent gaze estimator trained with real and synthetic eye images. The proposed method firstly brings the OCR response into the gaze estimation task, achieving better performances on the two benchmark datasets with fewer parameters under the real-time scenarios. With a replacement of a deeper network, compared to state-of-the-art methods, the proposed method is more efficient, achieving a). better average estimate (3.9% and 2.5% improvement), b). much better standard deviation (lower by 59.0% and 44.2%) and c). a much lower number of parameters (reduced by 88.0%).
Multi-view Semantic Information Guidance for Light Field Image Segmentation

One of the great important fields of computer vision is semantic segmentation. As for single image semantic segmentation, due to limited available information, it appears poor performance when the occlusion and similar color interference occur, and has difficulty exploiting the rich scene information. In comparison, the special micro-len array structure of light field camera can record multi-view information of the scene, which provides us with a new solution to solve this issue. In this paper, we propose a multi-view semantic information guidance network (MSIGNet) for light field semantic segmentation. It can effectively utilize semantic information from multi-view images to guide pixel feature of center view image. First, we extract feature of each view image and further obtain semantic probability. Then all probabilities are aggregated through a self-adaptive multi-view probability fusion module. Last, the resulting coarse fusion representation interacts with center view feature to obtain the refined segmentation result. The proposed method shows excellent performance on both real-world and synthetic light field datasets.
Beyond AUROC &amp; co. for evaluating out-of-distribution detection performance

While there has been a growing research interest in developing out-of-distribution (OOD) detection methods, there has been comparably little discussion around how these methods should be evaluated. Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. In this work, we take a closer look at the go-to metrics for evaluating OOD detection, and question the approach of exclusively reducing OOD detection to a binary classification task with little consideration for the detection threshold. We illustrate the limitations of current metrics (AUROC & its friends) and propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples. Scripts and data are available at https://github.com/glhr/beyond-auroc
Image Reference-guided Fashion Design with Structure-aware Transfer by Diffusion Models

Image-based fashion design with AI techniques has attracted increasing attention in recent years. We focus on a new fashion design task, where we aim to transfer a reference appearance image onto a clothing image while preserving the structure of the clothing image. It is a challenging task since there are no reference images available for the newly designed output fashion images. Although diffusion-based image translation or neural style transfer (NST) has enabled flexible style transfer, it is often difficult to maintain the original structure of the image realistically during the reverse diffusion, especially when the referenced appearance image greatly differs from the common clothing appearance. To tackle this issue, we present a novel diffusion model-based unsupervised structure-aware transfer method to semantically generate new clothes from a given clothing image and a reference appearance image. In specific, we decouple the foreground clothing with automatically generated semantic masks by conditioned labels. And the mask is further used as guidance in the denoising process to preserve the structure information. Moreover, we use the pre-trained Vision Transformer (ViT) for both appearance and structure guidance. Our experimental results show that the proposed method outperforms state-of-the-art baseline models, generating more realistic images in the fashion design task. Code and demo are released at https://github.com/Rem105-210/DiffFashion.
PEDRo: an Event-based Dataset for Person Detection in Robotics

Event-based cameras are devices based on neuromorphic sensors that are gaining popularity in different fields, including robotics. They are suitable for tasks where high-speed, low-latency, low-power operations are required. Person detection is one of these, to allow mobile robots to monitor areas and navigate in crowded environments. Most of the available event-based datasets that contain annotated human figures and collected with a moving camera are designed for autonomous driving tasks. Yet, robotic tasks are certainly not limited to the recognition of pedestrians walking on sidewalks, which makes the above datasets of limited utility. To address this impasse, we introduce a new dataset called PEDRo, which is fully manually labeled. This dataset has been specifically developed for person detection and it counts a total number of 43259 bounding boxes included in 119 recordings. A moving DAVIS346 event-based camera has been used to collect events in a large variety of indoor and outdoor scenarios with various lighting and meteorological conditions (such as sunny, rainy and snowy). To the best of our knowledge, this is now the largest available dataset for event-based person detection, which has been recorded with a moving camera and manually labeled.
DeepSim-Nets: Deep Similarity Networks for Stereo Image Matching

We present three multi-scale similarity learning architectures, or DeepSim networks. These models learn pixel-level matching with a contrastive loss and are agnostic to the geometry of the considered scene. We establish a middle ground between hybrid and end-to-end approaches by learning to densely allocate all corresponding pixels of an epipolar pair at once. Our features are learnt on large image tiles to be expressive and capture the scene’s wider context. We also demonstrate that curated sample mining can enhance the overall robustness of the predicted similarities and improve the performance on radiometri-cally homogeneous areas. We run experiments on aerial and satellite datasets. Our DeepSim-Nets outperform the baseline hybrid approaches and generalize better to unseen scene geometries than end-to-end methods. Our flexible architecture can be readily adopted in standard multi-resolution image matching pipelines. The code is available at https://github.com/DaliCHEBBI/DeepSimNets.
Exploring the Potential of Neural Dataset Search

Although we have witnessed Neural Architecture Search (NAS), which automatically explores architecture for best performance, the discussion has not advanced considering a dataset. We discuss the potential of Neural Dataset Search (NDS), which explores the appropriate configuration in a pre-training dataset to achieve a better pre-training effect. The NDS is designed to train in order to find the optimal parameters in the pre-training dataset for a given network architecture and downstream tasks. This allows for predicting the optimal pre-training parameters for a new unseen task in one shot. Thus, the NDS has the potential to bottom up the effectiveness of the pre-training. Therefore, this paper focuses on formula-driven supervised learning, and as a first consideration, we verify the appropriate configuration in Residual Network (ResNet) and Fractal DataBase (FractalDB). From the experimental results, we confirmed that the FractalDB generation parameters that provide the best pre-training effect are different for each ResNet-{18, 50, 152}. These observations reveal that there is an adapted image representation or dataset structure (e.g., input size, parameter, category) for a particular architecture. We hope these results will encourage further research on NDS that fully exploits the pre-training of synthetic images.
Robust Hierarchical Symbolic Explanations in Hyperbolic Space for Image Classification

Explanations for black-box models help us to understand model decisions as well as provide information on model biases and inconsistencies. Most of the current post-hoc explainability techniques provide a single level of explanation, often in terms of feature importance scores or feature attention maps in the input space. The explanations provided by these methods are also sensitive to perturbations in the input space. Our focus is on explaining deep discriminative models for images at multiple levels of abstraction, from fine-grained to fully abstract explanations. We use the natural properties of hyperbolic geometry to more efficiently model a hierarchical relationship of symbolic features with decreased distortion to generate robust hierarchical explanations. Specifically, we distill the underpinning knowledge in an image classifier by quantising the continuous latent space to form hyperbolic symbols and learn the relations between these symbols in a hierarchical manner to induce a knowledge tree. We traverse the tree to extract hierarchical explanations in terms of chains of symbols and their corresponding visual semantics. Code is available at https://github.com/AinkaranSanthi/HyperbolicReasoning
Zero-shot Classification at Different Levels of Granularity

Zero-shot classification (ZSC) is the task of learning predictors for classes not seen during training. The different methods proposed in literature are evaluated over specific datasets with their specific class partitions, but little attention has been paid to the impact of the dataset granularity when ZSC is performed. The novelty of this work is to generate synthetic datasets by controlling their granularity level to analyze the ZSC performance afterwards. Moreover, it presents an approach that allows us to preserve the visual and semantic structures. The experiments show that ZSC performance exhibits strong differences depending on the data granularity and it reveals the relevance of both visual and semantic spaces when performing ZSC.
Hardware-aware NAS by Genetic Optimisation with a Design Space Exploration Simulator

Neural Architecture Search (NAS) has shown its potential in aiding in the development of more efficient neural networks. In regard to hardware, efficiency often equates to power usage or latency. Over the years many researchers have incorporated hardware performance into their NAS experiments. However, accurately modelling hardware performance is a challenge in itself. We look to the field of design space exploration (DSE) for more precise performance metrics on neural network accelerators and incorporate the results into our NAS search. Our experiments show that doing so achieves a significant reduction in latency and energy consumption. The approach we propose also enables detailed insight in the breakdown of the energy consumption and latency of the optimised model.
A New Dataset and Approach for Timestamp Supervised Action Segmentation Using Human Object Interaction

This paper focuses on leveraging Human Object Interaction (HOI) information to improve temporal action segmentation under timestamp supervision, where only one frame is annotated for each action segment. This information is obtained from an off-the-shelf pre-trained HOI detector, that requires no additional HOI-related annotations in our experimental datasets. Our approach generates pseudo labels by expanding the annotated timestamps into intervals and allows the system to exploit the spatio-temporal continuity of human interaction with an object to segment the video. We also propose the (3+1)Real-time Cooking (ReC) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> dataset as a realistic collection of videos from 30 participants cooking 15 breakfast items. Our dataset has three main properties: 1) to our knowledge, the first to offer synchronized third and first person videos, 2) it incorporates diverse actions and tasks, and 3) it consists of high resolution frames to detect fine-grained information. In our experiments we benchmark state-of-the-art segmentation methods under different levels of supervision on our dataset. We also quantitatively show the advantages of using HOI information, as our framework improves its baseline segmentation method on several challenging datasets with varying viewpoints, providing improvements of up to 10.9% and 5.3% in F1 score and frame-wise accuracy respectively.
Training Strategies for Vision Transformers for Object Detection

Vision-based Transformer have shown huge application in the perception module of autonomous driving in terms of predicting accurate 3D bounding boxes, owing to their strong capability in modeling long-range dependencies between the visual features. However Transformers, initially designed for language models, have mostly focused on the performance accuracy, and not so much on the inference-time budget. For a safety critical system like autonomous driving, real-time inference at the on-board compute is an absolute necessity. This keeps our object detection algorithm under a very tight run-time budget. In this paper, we evaluated a variety of strategies to optimize on the inference-time of vision transformers based object detection methods keeping a close-watch on any performance variations. Our chosen metric for these strategies is accuracy-runtime joint optimization. Moreover, for actual inference-time analysis we profile our strategies with float32 and float16 precision with TensorRT module. This is the most common format used by the industry for deployment of their Machine Learning networks on the edge devices. We showed that our strategies are able to improve inference-time by 63% at the cost of performance drop of mere 3% for our problem-statement defined in Sec. 3. These strategies brings down Vision Transformers detectors [3], [15], [18], [19], [36] inference-time even less than traditional single-image based CNN detectors like FCOS [17], [25], [33]. We recommend practitioners use these techniques to deploy Transformers based hefty multi-view networks on a budge-constrained robotic platform.
A Scale-Invariant Trajectory Simplification Method for Efficient Data Collection in Videos

Training data is a critical requirement for machine learning tasks, and labeled training data can be expensive to acquire, often requiring manual or semi-automated data collection pipelines. For tracking applications, the data collection involves drawing bounding boxes around the classes of interest on each frame, and associate detections of the same "instance" over frames. In a semi-automated data collection pipeline, this can be achieved by running a baseline detection and tracking algorithm, and relying on manual correction to add/remove/change bounding boxes on each frame, as well as resolving errors in the associations over frames (track switches). In this paper, we propose a data correction pipeline to generate ground-truth data more efficiently in this semi-automated scenario. Our method simplifies the trajectories from the tracking systems and let the annotator verify and correct the objects in the sampled keyframes. Once the objects in the keyframes are corrected, the bounding boxes in the other frames are obtained by interpolation. Our method achieves substantial reduction in the number of frames requiring manual correction. In the MOT dataset, it reduces the number of frames by 30x while maintaining a HOTA score of 89.61% . Moreover, it reduces the number of frames by a factor of 10x while achieving a HOTA score of 79.24% in the SoccerNet dataset, and 85.79% in the DanceTrack dataset. The project code and data are publicly released at github/foreverYoungGitHub.
Pre-training Auto-generated Volumetric Shapes for 3D Medical Image Segmentation

In 3D medical image segmentation, data collection and annotation costs require significant human efforts. Moreover, obtaining training data is challenging due to privacy constraints. Consequently, achieving efficient learning with limited data is an urgent 3D medical image segmentation issue. One approach to address this problem is using pre-trained models, which have been widely researched. Recently, self-supervised learning for 3D medical images has gained popularity, but the data available for such learning is also scarce, limiting the number of pre-training datasets. In recent years, formula-driven supervised learning has garnered attention. It can achieve high pre-training effects using only easily accessible synthetic data, making it a promising alternative for pre-training datasets. Inspired by this approach, we propose the Auto-generated Volumetric Shapes Database (AVS-DB) for data-scarce 3D medical image segmentation tasks. AVS-DB is automatically generated from a combination of dozens of 3D models based on polygons and shape similarity ratio variations. Our experiments show that AVS-DB pre-trained models significantly outperform models trained from scratch and achieve comparable or better performance than existing self-supervised learning methods we compared. AVS-DB can potentially enhance 3D medical image segmentation models and address limited data availability challenges.