Reviewerly/exHarmony huggingface dataset:
Stereo Cross Global Learnable Attention Module for Stereo Image Super-Resolution

Stereo super-resolution is a technique that utilizes corresponding information from multiple viewpoints to enhance the texture of low-resolution images. In recent years, numerous impressive works have advocated attention mechanisms based on epipolar constraints to boost the performance of stereo super-resolution. However, techniques that exclusively depend on epipolar constraint attention are insufficient to recover realistic and natural textures for heavily corrupted low-resolution images. We noticed that global self-similarity features within the image and across the views can proficiently fix the texture details of low-resolution images that are severely damaged. Therefore, in the current paper, we propose a stereo cross global learnable attention module (SCGLAM), aiming to improve the performance of stereo super-resolution. The experimental outcomes show that our approach outperforms others when dealing with heavily damaged low-resolution images. The relevant code is made available on this link as open source.
CheckSORT: Refined Synthetic Data Combination and Optimized SORT for Automatic Retail Checkout

In this paper, we propose a method called CheckSORT for automatic retail checkout. We demonstrate CheckSORT on the multi-class product counting and recognition task in Track 4 of AI CITY CHALLENGE 2023. This task aims to count and identify products as they move along a retail checkout white tray, which is challenging due to occlusion, similar appearance, or blur. Based on the constraints and training data provided by the sponsor, we propose two new ideas to solve this task. The first idea is to design a controllable synthetic training data generation paradigm to bridge the gap between training data and real test videos as much as possible. The second innovation is to improve the efficiency of existing SORT tracking algorithms by proposing decomposed Kalman filter and dynamic tracklet feature sequence. Our experiments resulted in state-of-the-art (when compared with DeepSORT and StrongSORT) F1-scores of 70.3% and 62.1% on the TestA data of AI CITY CHALLENGE 2022 and 2023 respectively in the estimation of the time (in seconds) for the product to appear on the tray. Training and testing code will be available soon on github.
Many-Task Federated Learning: A New Problem Setting and A Simple Baseline

Federated Learning (FL) was originally proposed to effectively exploit more data that are distributed at local clients even though the local data follow non-i.i.d. distributions. The fundamental intuition is that, the more data we can use the better model we are likely to obtain in spite of the increased difficulty of learning due to the non-i.i.d. data distribution, or data heterogeneity. With this intuition, we strive to further scale up FL to cover more clients to participate and increase the effective coverage of more user data, by enabling FL to handle collaboration between clients that perform different yet related task types, i.e., enabling a new level of heterogeneity: task heterogeneity, which can be entangled with data heterogeneity and lead to more intractable clients. However, solving such compound heterogeneities from both data and task levels raises major challenges, against the current global, static, and identical federated aggregation ways across clients. To tackle this new and challenging FL setting, we propose an intuitive clustering-based training baseline to tackle the significant data and task heterogeneities. Specifically, each agent dynamically infers its "proximity" with others by comparing their layer-wise weight updates sent to the server, and then flexibly determines how to aggregate weights with selected similar clients. We construct new testbeds to examine our novel problem setting and algorithm on two benchmark datasets in multi-task learning: NYU Depth and PASCAL-Context datasets. Extensive experiments demonstrate that our proposed method shows superiority over plain FL algorithms such as FedAvg and FedProx in the 5-task setting on Pascal-Context and even enables jointly federated learning over the combined set of PASCAL-Context and NYU Depth (9 tasks, 2 data domains). Codes are available at: https://github.com/VITA-Group/MaT-FL.
DETR-based Layered Clothing Segmentation and Fine-Grained Attribute Recognition

Clothing segmentation and fine-grained attribute recognition are challenging tasks at the crossing of computer vision and fashion, which segment the entire ensemble clothing instances as well as recognize detailed attributes of the clothing products from any input human images. Many new models have been developed for the tasks in recent years, nevertheless the segmentation accuracy is less than satisfactory in case of layered clothing or fashion products in different scales. In this paper, a new DEtection TRansformer (DETR) based method is proposed to segment and recognize fine-grained attributes of ensemble clothing instances with high accuracy. In this model, we propose a multi-layered attention module by aggregating features of different scales, determining the various scale components of a single instance, and merging them together. We train our model on the Fashionpedia dataset and demonstrate our method surpasses SOTA models in tasks of layered clothing segmentation and fine-grained attribute recognition.
Video Analytics for Detecting Motorcyclist Helmet Rule Violations

The use of helmets is essential for motorcyclists' safety, but non-compliance with helmet rules remains a common issue. In this study, we extend the frontier of AI video analytic technologies for detecting violations of helmet rules among motorcyclists. Our method can handle highly challenging conditions for traditional methods, including occlusions, fast vehicle movement, shadows, large viewing angles, poor illumination and weather conditions. We adopt the widely used YOLOv7 object detector and develop a first baseline using YOLOv7-E6E. We further develop two improved versions, namely YOLOv7-CBAM and YOLOv7-SimAM that better address the challenges. Experiments are performed on the 2023 AI City Challenge Track 5 contest benchmark. Evaluation on the 100 test videos of the contest demonstrates the effectiveness of our approach. The baseline YOLOv7-E6E model trained with image size 1920 achieves 0.6112 mAP. The YOLOv7-CBAM achieves 0.6389 mAP, and YOLOv7-SimAM achieves 0.6422 mAP, where both are trained with image size 1280. These models rank sixth, fifth, and fourth on the public leaderboard, respectively, which outperforms over 36 global participating teams. The code for our models is available at: https://github.com/cmtsai2023/AICITY2023_Track5_DVHRM.
Sniffer Faster R-CNN: A Joint Camera-LiDAR Object Detection Framework with Proposal Refinement

In this paper we present Sniffer Faster R-CNN (SFR-CNN), a novel camera-LiDAR sensor fusion framework for fast and accurate object detection in autonomous driving scenarios. The proposed detection framework architecture uses both LiDAR point clouds and Camera RGB images to generate region proposals. Current implementation of the regional proposal network (RPN) requires the generation of a large number of region proposals, majority of which are unproductive. As such, we devise a novel proposal refinement algorithm, to jointly optimize and filter a number of proposals in RPN through the combined application of both sets of LiDAR and image-based proposals thereby accelerating the LiDAR-Camera fusion algorithm without sacrificing detection precision and accuracy. Our experiments show that number of proposals is a complementary factor in determining the computational overhead in a detection network. Our proposed architecture is shown to produce state of art results on the KITTI joint object detection benchmark with the comparison being based on the execution time. While maintaining efficient detection accuracy we decrease the computational overhead by more than 20 % on the KITTI dataset.
Robust and Scalable Vehicle Re-Identification via Self-Supervision

Many state-of-the-art solutions for vehicle re-identification (re-id) mostly focus on improving the accuracy on existing re-id benchmarks using additional annotated data. To balance the demands of accuracy, availability of annotated data, and computational efficiency, we propose a simple yet effective hybrid solution empowered by self-supervised learning which is free of intricate and computationally-demanding add-on attention modules often seen in state-of-the-art approaches. Through extensive experimentation, we show our approach, termed Self-Supervised and Boosted VEhicle Re-Identification (SSBVER), is on par with state-of-the-art alternatives in terms of accuracy without introducing any additional overhead during deployment. Additionally, we show that our approach, generalizes to different backbone architectures which accommodates various resource constraints and consistently results in a significant accuracy boost. Our code is available at https://github.com/Pirazh/SSBVER.
Back to the future: a night photography rendering ISP without deep learning

Rendering night photography pictures is a challenging task that requires advanced processing techniques. Although deep learning-based Image Signal Processing (ISP) pipelines have shown promising results, current limitations are set by the lack of proper nighttime image datasets, their high computational requirements, and low explainability. In this paper, we propose a traditional ISP pipeline for rendering visually pleasing photographs of night scenes. Our pipeline is comprised of various algorithms addressing the different challenges presented by night images, and it is characterized by a shallow structure, explainable steps, and a low parameter count, resulting in computationally efficient processing. Moreover, it does not require training data. Experiments show that our pipeline can produce more pleasing results compared to other deep learning-based ISP pipelines, as it won first place in people’s choice track and third place in photographer’s choice track in the NTIRE 2023 Night Photography Rendering Challenge.
Face Animation with an Attribute-Guided Diffusion Model

Face animation has achieved much progress in computer vision. However, prevailing GAN-based methods suffer from unnatural distortions and artifacts due to sophisticated motion deformation. In this paper, we propose a Face Animation framework with an attribute-guided Diffusion Model (FADM), which is the first work to exploit the superior modeling capacity of diffusion models for photorealistic talking-head generation. To mitigate the uncontrollable synthesis effect of the diffusion model, we design an Attribute-Guided Conditioning Network (AGCN) to adaptively combine the coarse animation features and 3D face reconstruction results, which can incorporate appearance and motion conditions into the diffusion process. These specific designs help FADM rectify unnatural artifacts and distortions, and also enrich high-fidelity facial details through iterative diffusion refinements with accurate animation attributes. FADM can flexibly and effectively improve existing animation videos. Extensive experiments on widely used talking-head benchmarks validate the effectiveness of FADM over prior arts. The source code is available in https://github.com/zengbohan0217/FADM.
Are we certain it’s anomalous?

The progress in modelling time series and, more generally, sequences of structured data has recently revamped research in anomaly detection. The task stands for identifying abnormal behaviors in financial series, IT systems, aerospace measurements, and the medical domain, where anomaly detection may aid in isolating cases of depression and attend the elderly. Anomaly detection in time series is a complex task since anomalies are rare due to highly non-linear temporal correlations and since the definition of anomalous is sometimes subjective.Here we propose the novel use of Hyperbolic uncertainty for Anomaly Detection (HypAD). HypAD learns self-supervisedly to reconstruct the input signal. We adopt best practices from the state-of-the-art to encode the sequence by an LSTM, jointly learned with a decoder to reconstruct the signal, with the aid of GAN critics. Uncertainty is estimated end-to-end by means of a hyperbolic neural network. By using uncertainty, HypAD may assess whether it is certain about the input signal but it fails to reconstruct it because this is anomalous; or whether the reconstruction error does not necessarily imply anomaly, as the model is uncertain, e.g. a complex but regular input signal. The novel key idea is that a detectable anomaly is one where the model is certain but it predicts wrongly. HypAD outperforms the current state-of-the-art for univariate anomaly detection on established benchmarks based on data from NASA, Yahoo, Numenta, Amazon, and Twitter. It also yields state-of-the-art performance on a multivariate dataset of anomaly activities in elderly home residences, and it outperforms the baseline on SWaT. Overall, HypAD yields the lowest false alarms at the best performance rate, thanks to successfully identifying detectable anomalies.
Nordic Vehicle Dataset (NVD): Performance of vehicle detectors using newly captured NVD from UAV in different snowy weather conditions

Vehicle detection and recognition in drone images is a complex problem that has been used for different safety purposes. The main challenge of these images is captured at oblique angles and poses several challenges like non-uniform illumination effect, degradations, blur, occlusion, loss of visibility, etc. Additionally, weather conditions play a crucial role in causing safety concerns and add another high level of challenge to the collected data. Over the past few decades, various techniques have been employed to detect and track vehicles in different weather conditions. However, detecting vehicles in heavy snow is still in the early stages because of a lack of available data. Furthermore, there has been no research on detecting vehicles in snowy weather using real images captured by unmanned aerial vehicles (UAVs). This study aims to address this gap by providing the scientific community with data on vehicles captured by UAVs in different settings and under various snow cover conditions in the Nordic region. The data covers different adverse weather conditions like overcast with snowfall, low light and low contrast conditions with patchy snow cover, high brightness, sunlight, fresh snow, and the temperature reaching far below – 0 degrees Celsius. The study also evaluates the performance of commonly used object detection methods such as YOLOv8s, YOLOv5s, and Faster RCNN. Additionally, data augmentation techniques are explored, and those that enhance the detectors' performance in such scenarios are proposed. The code and the dataset will be available at https://nvd.ltu-ai.dev
Improving Normalizing Flows with the Approximate Mass for Out-of-Distribution Detection

Normalizing flows are generative models that show poor performance on out-of-distribution (OOD) detection tasks with a likelihood-based test. In this study we focus on the "approximate mass" metric. We show that while it improves OOD detection performance, it has limitations under a maximum likelihood training. To solve this limitation we modify the training objective by incorporating the approximate mass. It smooths the learnt distribution in the vicinity of training in-distribution data. We measure an average of 97.6% AUROC in our experiments on different benchmarks, showing an improvement of 16% with respect to the best baseline we tested against.
SimDE: A Simple Domain Expansion Approach for Single-source Domain Generalization

Single domain generalization challenges model generalizability to unseen target domains when only one source domain is provided for training. To tackle this problem, domain expansion is adopted to learn domain-invariant information by exposing the model to more domain variations, which is still under-explored in previous work. In this paper, we propose a new and simplified objective for learning the desirable domain expansions by generating unconfident samples through the combination of entropy maximization and cross-entropy minimization. We further devise a novel framework that trains a pair of generators from different views by switching the guidance from the dual classifiers. In this way, the resulting method called Simple Domain Expansion (SimDE) can learn diverse domain expansions effectively and efficiently. Extensive experiments on prevalent single domain generalization benchmarks demonstrate the superiority of our method by offering improved results over the state-of-the-arts methods.
DeepSegmenter: Temporal Action Localization for Detecting Anomalies in Untrimmed Naturalistic Driving Videos

Identifying unusual driving behaviors exhibited by drivers during driving is essential for understanding driver behavior and the underlying causes of crashes. Previous studies have primarily approached this problem as a classification task, assuming that naturalistic driving videos come discretized. However, both activity segmentation and classification are required for this task due to the continuous nature of naturalistic driving videos. The current study therefore departs from conventional approaches and introduces a novel methodological framework, DeepSegmenter, that simultaneously performs activity segmentation and classification in a single framework. The proposed framework consists of four major modules namely Data Module, Activity Segmentation Module, Classification Module and Postprocessing Module. Our proposed method won 8th place in the 2023 AI City Challenge, Track 3, with an activity overlap score of 0.5426 on experimental validation data. The experimental results demonstrate the effectiveness, efficiency, and robustness of the proposed system. The code is available at https://github.com/aboah1994/DeepSegment.git.
Spatial-Angular Multi-Scale Mechanism for Light Field Spatial Super-Resolution

Light Field (LF) cameras are promising due to their ability to capture both spatial and angular information of scenes. However, the trade-off between spatial and angular resolution significantly limits the real-world applications. In this paper, we propose a spatial-angular multi-scale decoupling network to reconstruct high-resolution LF images. Considering the epipolar geometry, we propose a spatial-angular multi-scale processing approach to explore the correspondence of sub-pixel information with different disparity ranges between sub-aperture images in LFs. We extract sub-pixel information from various dimensions and fuse it to generate global high-frequency details. Finally, we combine upsampled low-frequency and high-frequency details to generate high resolution results. To further filter the correct interpolation information, we use the shear operation to change the disparity range of the LF images and fine-tune the results. Experimental results on synthetic and real-world datasets demonstrate that our method outperforms other state-of-the-art methods in visual and numerical evaluations, especially on datasets with small disparity ranges. Furthermore, our approach fully considers the epipolar geometry of the LF image, enabling us to recover information that better maintains the imaging consistency of the LF.
A<sup>2</sup>-Aug: Adaptive Automated Data Augmentation

Data augmentation is a promising way to enhance the generalization ability of deep learning models. Many proxy-free and proxy-based automated augmentation methods are proposed to search for the best augmentation for target datasets. However, the proxy-free methods require lots of searching overhead, while the proxy-based methods introduce optimization gaps with the actual task. In this paper, we explore a new proxy-free approach that only needs a small number of searches (~ 5 vs 100 of RandAugment) to alleviate these issues. Specifically, we propose Adaptive Automated Augmentation (A <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Aug), a simple and effective proxy-free framework, which seeks to mine the adaptive ensemble knowledge of multiple augmentations to further improve the adaptability of each candidate augmentation. Firstly, A <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Aug automatically learns the ensemble logit from multiple candidate augmentations, which is jointly optimized and adaptive to target tasks. Secondly, the adaptive ensemble logit is used to distill each logit of input augmentation via KL divergence. In this way, these a few candidate augmentations can implicitly learn strong adaptability for the target datasets, which enjoy similar effects with many searches of RandAugment. Finally, equipped with joint training via separate BatchNorm and normalized distillation, A <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Aug obtains state-of-the-art performance with less training budget. In experiments, our A <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Aug achieves 4% performance gain on CIFAR-100, which substantially outperforms other methods. On ImageNet, we obtain a top-1 accuracy of 79.2% for ResNet-50, a 1.6% boosting over the AutoAugment with at least 25× faster training speed.
EKILA: Synthetic Media Provenance and Attribution for Generative Art

We present EKILA; a decentralized framework that enables creatives to receive recognition and reward for their contributions to generative AI (GenAI). EKILA proposes a robust visual attribution technique and combines this with an emerging content provenance standard (C2PA) to address the problem of synthetic image provenance – determining the generative model and training data responsible for an AI-generated image. Furthermore, EKILA extends the non-fungible token (NFT) ecosystem to introduce a tokenized representation for rights, enabling a triangular relationship between the asset’s Ownership, Rights, and Attribution (ORA). Leveraging the ORA relationship enables creators to express agency over training consent and, through our attribution model, to receive apportioned credit, including royalty payments for the use of their assets in GenAI.
SPECTRE: Visual Speech-Informed Perceptual 3D Facial Expression Reconstruction from Videos

The recent state of the art on monocular 3D face reconstruction from image data has made some impressive advancements, thanks to the advent of Deep Learning. However, it has mostly focused on input coming from a single RGB image, overlooking the following important factors: a) Nowadays, the vast majority of facial image data of interest do not originate from single images but rather from videos, which contain rich dynamic information. b) Furthermore, these videos typically capture individuals in some form of verbal communication (public talks, teleconferences, audiovisual human-computer interactions, interviews, monologues/dialogues in movies, etc). When existing 3D face reconstruction methods are applied in such videos, the artifacts in the reconstruction of the shape and motion of the mouth area are often severe, since they do not match well with the speech audio.To overcome the aforementioned limitations, we present the first method for visual speech-informed perceptual reconstruction of 3D mouth expressions. We do this by proposing a "lipreading" loss, which guides the fitting process so that the elicited perception from the 3D reconstructed talking head resembles that of the original video footage. We demonstrate that, interestingly, the lipreading loss is better suited for 3D reconstruction of mouth movements compared to traditional landmark losses, and even direct 3D supervision. Furthermore, the devised method does not rely on any text transcriptions or corresponding audio, rendering it ideal for training in unlabeled datasets. We verify the efficiency of our method through objective evaluations on three large-scale datasets, as well as subjective evaluation with two web-based user studies. Project webpage: https://filby89.github.io/spectre/