Reviewerly/exHarmony huggingface dataset:
PointMotionNet: Point-Wise Motion Learning for Large-Scale LiDAR Point Clouds Sequences

We propose a point-based spatiotemporal pyramid architecture, called PointMotionNet, to learn motion information from a sequence of large-scale 3D LiDAR point clouds. A core component of PointMotionNet is a novel technique for point-based spatiotemporal convolution, which finds the point correspondences across time by leveraging a time-invariant spatial neighboring space and extracts spatiotemporal features. To validate PointMotionNet, we consider two motion-related tasks: point-based motion prediction and multisweep semantic segmentation. For each task, we design an end-to-end system where PointMotionNet is the core module that learns motion information. We conduct extensive experiments and show that i) for point-based motion prediction, PointMotionNet achieves less than 0.5m mean squared error on Argoverse dataset, which is a significant improvement over existing methods; and ii) for multisweep semantic segmentation, PointMotionNet with a pretrained segmentation backbone outperforms previous SOTA by over 3.3 % mIoU on SemanticKITTI dataset with 25 classes including 6 moving objects.
Person Re-identification Method Based on Color Attack and Joint Defence

The main challenges of ReID is the intra-class variations caused by color deviation under different camera conditions. Simultaneously, we find that most of the existing adversarial metric attacks are realized by interfering with the color characteristics of the sample. Based on this observation, we first propose a local transformation attack (LTA) based on color variation. It uses more obvious color variation to randomly disturb the color of the retrieved image, rather than adding random noise. Experiments show that the performance of the proposed LTA method is better than the advanced attack methods. Furthermore, considering that the contour feature is the main factor of the robustness of adversarial training, and the color feature will directly affect the success rate of attack. Therefore, we further propose joint adversarial defense (JAD) method, which includes proactive defense and passive defense. Proactive defense fuse multi-modality images to enhance the contour feature and color feature, and considers local homomorphic transformation to solve the over-fitting problem. Passive defense exploits the invariance of contour feature during image scaling to mitigate the adversarial disturbance on contour feature. Finally, a series of experimental results show that the proposed joint adversarial defense method is more competitive than a state-of-the-art method.
GenISP: Neural ISP for Low-Light Machine Cognition

Object detection in low-light conditions remains a challenging but important problem with many practical implications. Some recent works show that, in low-light conditions, object detectors using raw image data are more robust than detectors using image data processed by a traditional ISP pipeline. To improve detection performance in low-light conditions, one can fine-tune the detector to use raw image data or use a dedicated low-light neural pipeline trained with paired low- and normal-light data to restore and enhance the image. However, different camera sensors have different spectral sensitivity and learning-based models using raw images process data in the sensor-specific color space. Thus, once trained, they do not guarantee generalization to other camera sensors. We propose to improve generalization to unseen camera sensors by implementing a minimal neural ISP pipeline for machine cognition, named GenISP, that explicitly incorporates Color Space Transformation to a device-independent color space. We also propose a two-stage color processing implemented by two image-to-parameter modules that take down-sized image as input and regress global color correction parameters. Moreover, we propose to train our proposed GenISP under the guidance of a pre-trained object detector and avoid making assumptions about perceptual quality of the image, but rather optimize the image representation for machine cognition. At the inference stage, GenISP can be paired with any object detector. We perform extensive experiments to compare our proposed method to other low-light image restoration and enhancement methods in an extrinsic task-based evaluation and validate that GenISP can generalize to unseen sensors and object detectors. Finally, we contribute a low-light dataset of 7K raw images annotated with 46K bounding boxes for task-based benchmarking of future low-light image restoration and low-light object detection.
PyTorch-OOD: A Library for Out-of-Distribution Detection based on PyTorch

Machine Learning models based on Deep Neural Networks behave unpredictably when presented with inputs that do not stem from the training distribution and sometimes make egregiously wrong predictions with high confidence. This property undermines the trustworthiness of systems depending on such models and potentially threatens the safety of their users. Out-of-Distribution (OOD) detection mechanisms can be used to prevent errors by detecting inputs that are so dissimilar from the training set that the model can not be expected to make reliable predictions. In this paper, we present PyTorch-OOD, a Python library for OOD detection based on PyTorch. Its primary goals are to accelerate OOD detection research and improve the reproducibility and comparability of experiments. PyTorch-OOD provides well-tested and documented implementations of OOD detection methods with a unified interface, as well as training and benchmark datasets, architectures, pre-trained models, and utility functions. The library is available online <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> under the permissive Apache 2.0 license and can be installed via Python Package Index (PyPI).
An Effective Temporal Localization Method with Multi-View 3D Action Recognition for Untrimmed Naturalistic Driving Videos

Naturalistic driving studies with computer vision techniques have become an emergent research issue. The objective is to classify the distracted behavior actions by drivers. Specifically, this issue is regarded as temporal action localization (TAL) of untrimmed videos, which is a challenging task in the research field of video analysis. Particularly, TAL remains as one of the most challenging unsolved problems in computer vision that requires not only the recognition of action but the localization of the start and end times of each action. Most state-of-the-art approaches adopt complex architectures, which are expensive training and inefficient inference time. In this study, we propose a new framework for untrimmed naturalistic driving videos by utilizing the results from 3D action recognition with video clip classification for short temporal and spatial correlation. Then, simple post-processing based on data-driven is presented for long temporal correlation in untrimmed videos. The proposed method is evaluated on the AI City Challenge 2022 dataset for Naturalistic Driving Action Recognition. Accordingly, our method achieves the top 1 on the public leaderboard of the challenge.
RestoreX-AI: A Contrastive Approach towards Guiding Image Restoration via Explainable AI Systems

Modern applications such as self-driving cars and drones rely heavily upon robust object detection techniques. However, weather corruptions can hinder the object detectability and pose a serious threat to their navigation and reliability. Thus, there is a need for efficient denoising, deraining, and restoration techniques. Generative adversarial networks and transformers have been widely adopted for image restoration. However, the training of these methods is often unstable and time-consuming. Furthermore, when used for object detection (OD), the output images generated by these methods may provide unsatisfactory results despite image clarity. In this work, we propose a contrastive approach towards mitigating this problem, by evaluating images generated by restoration models during and post training. This approach leverages OD scores combined with attention maps for predicting the usefulness of restored images for the OD task. We conduct experiments using two novel use-cases of conditional GANs and two transformer methods that probe the robustness of the proposed approach on multi-weather corruptions in the OD task. Our approach achieves an averaged 178 increase in mAP between the input and restored images under adverse weather conditions like dust tornadoes and snowfall. We report unique cases where greater denoising does not improve OD performance and conversely where noisy generated images demonstrate good results. We conclude the need for explainability frameworks to bridge the gap between human and machine perception, especially in the context of robust object detection for autonomous vehicles.
Tracked-Vehicle Retrieval by Natural Language Descriptions With Domain Adaptive Knowledge

This paper introduces our solution for Track 2 in AI City Challenge 2022. The task is Tracked-Vehicle Retrieval by Natural Language Descriptions with a real-world dataset of various scenarios and cameras. We mainly focus on developing a robust natural language-based vehicle retrieval system to address the domain bias problem due to unseen scenarios and multi-view multi-camera vehicle tracks. Specifically, we apply CLIP [16] to effectively extract both visual and textual representations for contrastive representation learning. Furthermore, for new scenarios in the test set, we pro-pose a novel Domain Adaptive Training method that utilizes information from labeled data and transfers it to the unseen domain by generating pseudo labels. By using this simple and effective strategy, we not only bridge the domain gap between the training set and test set, but also require less computational cost and data compared to previous top performance methods. Finally, we employ a context-sensitive post-processing method to address model’s uncertainty and eliminate the wrong retrieved vehicle track. Taking one step further, we also investigate the impact of different text formats and the number of pseudo labels data for the fine-tuning process. Our proposed method has achieved 3rd place in the AI City Challenge 2022, yielding a competitive performance of 47.73% MRR accuracy on the private test set, which verified the effectiveness of the proposed solution.
How do requirements evolve during elicitation? An empirical study combining interviews and app store analysis

Abstract Requirements are elicited from the customer and other stakeholders through an iterative process of interviews, prototyping, and other interactive sessions. Then, requirements can be further extended, based on the analysis of the features of competing products available on the market. Understanding how this process takes place can help to identify the contribution of the different elicitation phases, thereby allowing requirements analysts to better distribute their resources. In this work, we empirically study in which way requirements get transformed from initial ideas into documented needs, and then evolve based on the inspiration coming from similar products. To this end, we select 30 subjects that act as requirements analysts, and we perform interview-based elicitation sessions with a fictional customer. After the sessions, the analysts produce a first set of requirements for the system. Then, they are required to search similar products in the app stores and extend the requirements, inspired by the identified apps. The requirements documented at each step are evaluated, to assess to which extent and in which way the initial idea evolved throughout the process. Our results show that only between 30% and 38% of the requirements produced after the interviews include content that can be fully traced to initial customer’s ideas. The rest of the content is dedicated to new requirements, and up to 21% of it belongs to completely novel topics. Furthermore, up to 42% of the requirements inspired by the app stores cover additional features compared to the ones identified after the interviews. The results empirically show that requirements are not elicited in strict sense, but actually co-created through interviews, with analysts playing a crucial role in the process. In addition, we show evidence that app store-inspired elicitation can be particularly beneficial to complete the requirements.
A Hybrid-Integrated Photonic Spiking Neural Network Framework Based on an MZI Array and VCSELs-SA

We propose an inferencing framework of a hybrid-integrated photonic spiking neural network (PSNN) to perform pattern recognition tasks, where the linear computation is realized based on a 4 × 4 silicon photonic Mach-Zehnder interferometer (MZI) array, and the nonlinear computation is performed by an InP-based spiking neuron array based on vertical-cavity surface-emitting lasers with an embedded saturable absorber (VCSELs-SA). With the modified Tempotron-like remote supervised method (ReSuMe) training algorithm, we realize two pattern recognition tasks, the recognition of numbers “0-3” and optical character recognition (OCR). The phase shifts in the MZI array are accurately configured to represent the weight matrix according to the decomposition procedure of a 4 × 4 triangular MZI mesh. Besides, the effects of the phase shift error and quantization precision of phase shifters (PSs) on the recognition performance are analyzed. For the OCR task, the 400 × 10 PSNN is realized by multiplexing the 4 × 4 MZI array based on the matrix blocking and the reconfigurability of the MZI array. This work provides a systematic computational model of the hybrid-integrated PSNN based on the silicon photonics and InP platforms, enabling the co-design and optimization of hardware architecture and algorithm, which contributes one step forward toward the construction of a hybrid-integrated PSNN hardware system.
RDONet: Rate-Distortion Optimized Learned Image Compression with Variable Depth

Rate-distortion optimization (RDO) is responsible for large gains in image and video compression. While RDO is a standard tool in traditional image and video coding, it is not yet widely used in novel end-to-end trained neural methods. The major reason is that the decoding function is trained once and does not have free parameters. In this paper, we present RDONet, a network containing state-of-the-art components, which is perceptually optimized and capable of rate-distortion optimization. With this network, we are able to outperform VVC Intra on MS-SSIM and two different perceptual LPIPS metrics. This paper is part of the CLIC challenge, where we participate under the team name RDONet FAU.
Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization

In this paper, we investigate the continual learning of Vision Transformers (ViT) for the challenging exemplar-free scenario, with special focus on how to efficiently distill the knowledge of its crucial self-attention mechanism (SAM). Our work takes an initial step towards a surgical investigation of SAM for designing coherent continual learning methods in ViTs. We first carry out an evaluation of established continual learning regularization techniques. We then examine the effect of regularization when applied to two key enablers of SAM: (a) the contextualized embedding layers, for their ability to capture well-scaled representations with respect to the values, and (b) the prescaled attention maps, for carrying value-independent global contextual information. We depict the perks of each distilling strategy on two image recognition benchmarks (CIFAR100 and ImageNet-32) – while (a) leads to a better overall accuracy, (b) helps enhance the rigidity by maintaining competitive performances. Furthermore, we identify the limitation imposed by the symmetric nature of regularization losses. To alleviate this, we propose an asymmetric variant and apply it to the pooled output distillation (POD) loss adapted for ViTs. Our experiments confirm that introducing asymmetry to POD boosts its plasticity while retaining stability across (a) and (b). Moreover, we acknowledge low forgetting measures for all the compared methods, indicating that ViTs might be naturally inclined continual learners. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Multi-Task Learning for Video Surveillance with Limited Data

Learning from limited data in video surveillance is important for sustainable performance while adapting to new information in a scene over time or adapting to a different scene. In a real-world scene, for an anomaly detection algorithm, all possible nominal patterns and behaviors are not typically available immediately for a single training session. In contrast, labeled nominal data patterns may become available irregularly over a long time horizon, and the anomaly detection algorithm needs to quickly learn such new patterns from limited samples for acceptable performance. Otherwise, it would suffer from frequent false alarms. Additionally, the anomaly detection algorithm needs to continually learn new nominal patterns in multiple training sessions without forgetting the previous knowledge and losing performance. Cross-domain adaptability (i.e., transfer learning to another surveillance scene) is another task where the anomaly detection algorithm has to quickly learn from limited nominal training data to achieve acceptable performance. To overcome these challenges, we design a modular framework and use it to extract semantic embeddings, which we then train on by using deep metric learning. Particularly, we study these three problems (few-shot learning, continual learning, cross-domain adaptability) in a multi-task learning setting. We also compare our proposed framework to existing state-of-the-art approaches using various evaluation metrics. The empirical results indicate that the proposed approach is able to outperform the existing approaches on all three tasks for three benchmark datasets.
NL-FFC: Non-Local Fast Fourier Convolution for Image Super Resolution

Deep neural networks have shown promising results in image super-resolution by learning a complex mapping from low resolution to high resolution image. However, most of the approaches learns to upsample by using convolution in spatial domain and are confined to local features. This results into restricting the receptive field of the network and therefore deteriorates the overall quality of the high-resolution image. To alleviate this issue, we propose an architecture that learns both local and global features, and fuses them together to generate high quality images. The network uses a non-local attention aided Fast Fourier Convolutions (NL-FFC) to widen the receptive field and learn long-range dependencies. The analyses further show that these Fourier features implicitly provide faster convergence on low frequency components only to learn prior for unobserved high frequency components. The model generalizes well to different datasets. We further investigate the role of non-local attention, and the ratio of local and global features to maximize the performance gain in the ablation study.
The Reliability of Forensic Body-Shape Identification

Photo-based forensic identification can be critical in the prosecution of, and defense against, criminal charges. Identification techniques range from the specific biometric-based to the more generic, based on height, weight, gender, and race. Although fairly basic, accurate height and weight estimation remains challenging due to physiological factors, concealing clothing, body pose, and the scale ambiguity inherent to the photographic process. We describe an extension to 3D body-pose estimation that more accurately estimates body shape across a broader range of body sizes. We evaluate the reliability of this technique in making metric estimates of height and weight, and in making non-metric categorization of people based on a scale-agnostic measure of body shape. Although this approach improves on previous efforts, we find that accurate body-shape identification from a single, reference-free image remains challenging.
Rethinking Illumination for Person Re-Identification: A Unified View

As a fundamental problem in video surveillance, person re-identification (re-ID) contributes a lot to the development of modern metro city. Recently, learning from synthetic data on re-ID task, which benefits from the popularity of synthetic data engine, has achieved remarkable performance in both supervised and unsupervised manner. However, previous researches mainly lay emphasis on employing synthetic data to achieve the state-of-the-art performance with a strong backbone, while neglects to perform quantitative studies on how visual factors affect re-ID system. To facilitate the research in this field, firstly, we manually construct a large-scale synthetic dataset named SynPerson, which has diversified human characters and distinguished attributes with accurate annotations. Secondly, we quantitatively analyze the influence of illumination on re-ID system. To our best knowledge, this is the first attempt to explicitly dissect person re-ID from the aspect of illumination on synthetic dataset. Comprehensive experiments help us have a deeper understanding of the fundamental problems in person re-ID. Furthermore, we will release SynPerson to the community, as part of efforts to alleviate the shortage of large-scale pedestrian dataset of future works <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Augmentation Invariance and Adaptive Sampling in Semantic Segmentation of Agricultural Aerial Images

In this paper, we investigate the problem of Semantic Segmentation for agricultural aerial imagery. We observe that the existing methods used for this task are designed without considering two characteristics of the aerial data: (i) the top-down perspective implies that the model cannot rely on a fixed semantic structure of the scene, because the same scene may be experienced with different rotations of the sensor; (ii) there can be a strong imbalance in the distribution of semantic classes because the relevant objects of the scene may appear at extremely different scales (e.g., a field of crops and a small vehicle). We propose a solution to these problems based on two ideas: (i) we use together a set of suitable augmentation and a consistency loss to guide the model to learn semantic representations that are invariant to the photometric and geometric shifts typical of the top-down perspective (Augmentation Invariance); (ii) we use a sampling method (Adaptive Sampling) that selects the training images based on a measure of pixel-wise distribution of classes and actual network confidence. With an extensive set of experiments conducted on the Agriculture-Vision dataset, we demonstrate that our proposed strategies improve the performance of the current state-of-the-art method. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .