Reviewerly/exHarmony huggingface dataset:
Language Guided Local Infiltration for Interactive Image Retrieval

Interactive Image Retrieval (IIR) aims to retrieve images that are generally similar to the reference image but under the requested text modification. The existing methods usually concatenate or sum the features of image and text simply and roughly, which, however, is difficult to precisely change the local semantics of the image that the text intends to modify. To solve this problem, we propose a Language Guided Local Infiltration (LGLI) system, which fully utilizes the text information and penetrates text features into image features as much as possible. Specifically, we first propose a Language Prompt Visual Localization (LPVL) module to generate a localization mask which explicitly locates the region (semantics) intended to be modified. Then we introduce a Text Infiltration with Local Awareness (TILA) module, which is deployed in the network to precisely modify the reference image and generate image-text infiltrated representation. Extensive experiments on various benchmark databases validate that our method outperforms most state-of-the-art IIR approaches.
Shape of You: Precise 3D shape estimations for diverse body types

This paper presents Shape of You (SoY), an approach to improve the accuracy of 3D body shape estimation for vision-based clothing recommendation systems. While existing methods have successfully estimated 3D poses, there remains a lack of work in precise shape estimation, particularly for diverse human bodies. To address this gap, we propose two loss functions that can be readily integrated into parametric 3D human reconstruction pipelines. Additionally, we propose a test-time optimization routine that further improves quality. Our method improves over the recent SHAPY [7] method by 17.7% on the challenging SSP-3D dataset [16]. We consider our work to be a step towards a more accurate 3D shape estimation system that works reliably on diverse body types and holds promise for practical applications in the fashion industry.
Pareto-aware Neural Architecture Generation for Diverse Computational Budgets

Designing feasible and effective architectures under diverse computational budgets, incurred by different applications/devices, is essential for deploying deep models in real-world applications. To achieve this goal, existing methods often perform an independent architecture search process for each target budget, which is very inefficient yet unnecessary. More critically, these independent search processes cannot share their learned knowledge (i.e., the distribution of good architectures) with each other and thus often result in limited search results. To address these issues, we propose a Pareto-aware Neural Architecture Generator (PNAG) which only needs to be trained once and dynamically produces the Pareto optimal architecture for any given budget via inference. To train our PNAG, we learn the whole Pareto frontier by jointly finding multiple Pareto optimal architectures under diverse budgets. Such a joint search algorithm not only greatly reduces the overall search cost but also improves the search results. Extensive experiments on three hardware platforms (i.e., mobile device, CPU, and GPU) show the superiority of our method over existing methods.
Strong Detector with Simple Tracker

Unmanned aerial vehicle (UAV) tracking is a research direction with practical application value and has received sufficient attention in recent years. Challenges such as complex backgrounds, small targets, and motion blur in UAV tracking make it difficult to directly apply existing tracking or detection methods. For example, some state-of-the-art (SOTA) single-object tracking methods such as Ostrack perform poorly when encountering target disappearance or camera offset. Existing detection methods are also difficult to apply directly to this task. This paper proposes a detection-based method with cascading post-processing modules to solve this task. Our entire process includes generating detection candidate boxes, adjusting candidate box scores through video classification, connecting candidate boxes between different frames through a simple tracker, and determining moving targets in the video through background modeling, followed by single-object tracking as post-processing to adjust the results. We finally achieved first place in the 3rd Anti-UAV challenge track1 and top three in track2.
BSA Tool: An Experience Report of Software Automation to Perform Sanity Tests in a Global Software Development Environment

Software testing is essential to ensure the quality of a software product since it is possible to determine if the product performs its purpose correctly. However, given the speed at which new software products are developed, manual testing becomes unfeasible when the workload is increasing, and the test cycle is shorter. In addition, current technologies make it possible to create automated scripts which can support manual test approaches. The goal of this study is to evaluate the benefits of the introduction of an automated tool to perform sanity testing on Android smartphones and tablets, in a global software development context. Through data collected from surveys and observational studies, it was possible to understand the impact of using software automation as a testing strategy. The analysis concluded that there are time gains of up to 28% for beginner testers and 53% for experienced testers, showing that the determining factor for the time gain is the use of automation as a support to deal with high testing demands, allowing the number of manual activities to be reduced and effort to be concentrated only on what cannot be automated.
Continual Learning for LiDAR Semantic Segmentation: Class-Incremental and Coarse-to-Fine strategies on Sparse Data

During the last few years, Continual Learning (CL) strategies for image classification and segmentation have been widely investigated designing innovative solutions to tackle catastrophic forgetting, like knowledge distillation and selfinpainting. However, the application of continual learning paradigms to point clouds is still unexplored and investigation is required, especially using architectures that capture the sparsity and uneven distribution of LiDAR data. The current paper analyzes the problem of class incremental learning applied to point cloud semantic segmentation, comparing approaches and state-of-the-art architectures. To the best of our knowledge, this is the first example of class- incremental continual learning for LiDAR point cloud semantic segmentation. Different CL strategies were adapted to LiDAR point clouds and tested, tackling both classic fine-tuning scenarios and the Coarse-to-Fine learning paradigm. The framework has been evaluated through two different architectures on SemanticKITTI [2], [16], obtaining results in line with state-of-the-art CL strategies and standard offline learning.
Envisioning a Next Generation Extended Reality Conferencing System with Efficient Photorealistic Human Rendering

Meeting online is becoming the new normal. Creating an immersive experience for online meetings is a necessity towards more diverse and seamless environments. Efficient photorealistic rendering of human 3D dynamics is the core of immersive meetings. Current popular applications achieve real-time conferencing but fall short in delivering photorealistic human dynamics, either due to limited 2D space or the use of avatars that lack realistic interactions between participants. Recent advances in neural rendering, such as the Neural Radiance Field (NeRF), offer the potential for greater realism in metaverse meetings. However, the slow rendering speed of NeRF poses challenges for real-time conferencing. We envision a pipeline for a future extended reality metaverse conferencing system that leverages monocular video acquisition and free-viewpoint synthesis to enhance data and hardware efficiency. Towards an immersive conferencing experience, we explore an accelerated NeRF-based free-viewpoint synthesis algorithm for rendering photorealistic human dynamics more efficiently. We show that our algorithm achieves comparable rendering quality while performing training and inference 44.5% and 213% faster than state-of-the-art methods, respectively. Our exploration provides a design basis for constructing metaverse conferencing systems that can handle complex application scenarios, including dynamic scene relighting with customized themes and multi-user conferencing that harmonizes real-world people into an extended world.
Perceive, Excavate and Purify: A Novel Object Mining Framework for Instance Segmentation

Recently, instance segmentation has made great progress with the rapid development of deep neural networks. However, there still exist two main challenges including discovering indistinguishable objects and modeling the relationship between instances. To deal with these difficulties, we propose a novel object mining framework for instance segmentation. In this framework, we first introduce the semantics perceiving subnetwork to capture pixels that may belong to an obvious instance from the bottom up. Then, we propose an object excavating mechanism to discover indistinguishable objects. In the mechanism, preliminary perceived semantics are regarded as original instances with classifications and locations, and then indistinguishable objects around these original instances are mined, which ensures that hard objects are fully excavated. Next, an instance purifying strategy is put forward to model the relationship between instances, which pulls the similar instances close and pushes away different instances to keep intra-instance similarity and inter-instance discrimination. In this manner, the same objects are combined as the one instance and different objects are distinguished as independent instances. Extensive experiments on the COCO dataset show that the proposed approach outperforms state-of-the-art methods, which validates the effectiveness of the proposed object mining framework.
High-Speed Long-Range Physical-Layer Key Distribution Assisted by Neural Networks

We propose and demonstrate the unidirectional physical-layer secure key generation and distribution in fiber networks using neural networks, with a Gb/s key generation rate demonstrated over 100-km standard single-mode fiber.
Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval

Cross-modal retrieval methods are the preferred tool to search databases for the text that best matches a query image and vice versa. However, image-text retrieval models commonly learn to memorize spurious correlations in the training data, such as frequent object co-occurrence, instead of looking at the actual underlying reasons for the prediction in the image. For image-text retrieval, this manifests in retrieved sentences that mention objects that are not present in the query image. In this work, we introduce ODmAP@k, an object decorrelation metric that measures a model’s robustness to spurious correlations in the training data. We use automatic image and text manipulations to control the presence of such object correlations in designated test data. Additionally, our data synthesis technique is used to tackle model biases due to spurious correlations of semantically unrelated objects in the training data. We apply our proposed pipeline, which involves the finetuning of image-text retrieval frameworks on carefully designed synthetic data, to three state-of-the-art models for image-text retrieval. This results in significant improvements for all three models, both in terms of the standard retrieval performance and in terms of our object decorrelation metric. The code is available at https://github.com/ExplainableML/Spurious_CM_Retrieval.
Co-Phase Over-the-Air Aggregation for Multi-Server Federated Learning with Randomized Transmissions

While federated learning is widely studied as a distributed machine learning technique that effectively uses distributed datasets for training, there are still a number of challenges. In particular, for datasets that reside on mobile devices (or mobile phone users), the limited bandwidth becomes a bottleneck as the number of devices increases, because the local gradient vectors are be transmitted over the scarce wireless channel. Thus, the notion of over-the-air (OTA) computation has been considered to avoid bandwidth expansion due to the increase in devices by exploiting the superposition property of radio channels. In this paper, we consider a generalization of OTA to the case of multiple servers, where each server has own model. Using co-phase OTA aggregation, it is shown that a shared channel can be used for all different servers/models. We also propose a digital OTA approach for multi-server federated learning with randomized transmissions.
Reliable Student: Addressing Noise in Semi-Supervised 3D Object Detection

Semi-supervised 3D object detection can benefit from the promising pseudo-labeling technique when labeled data is limited. However, recent approaches have overlooked the impact of noisy pseudo-labels during training, despite efforts to enhance pseudo-label quality through confidence-based filtering. In this paper, we examine the impact of noisy pseudo-labels on IoU-based target assignment and propose the Reliable Student framework, which incorporates two complementary approaches to mitigate errors. First, it involves a class-aware target assignment strategy that reduces false negative assignments in difficult classes. Second, it includes a reliability weighting strategy that suppresses false positive assignment errors while also addressing remaining false negatives from the first step. The reliability weights are determined by querying the teacher network for confidence scores of the student-generated proposals. Our work surpasses the previous state-of-the-art on KITTI 3D object detection benchmark on point clouds in the semi-supervised setting. On 1% labeled data, our approach achieves a 6.2% AP improvement for the pedestrian class, despite having only 37 labeled samples available. The improvements become significant for the 2% setting, achieving 6.0% AP and 5.7% AP improvements for the pedestrian and cyclist classes, respectively. Our code will be released at https://github.com/fnozarian/ReliableStudent
Revisiting Class Imbalance for End-to-end Semi-Supervised Object Detection

Semi-supervised object detection (SSOD) has made significant progress with the development of pseudo-label-based end-to-end methods. However, many of these methods face challenges due to class imbalance, which hinders the effectiveness of the pseudo-label generator. Furthermore, in the literature, it has been observed that low-quality pseudo-labels severely limit the performance of SSOD. In this paper, we examine the root causes of low-quality pseudo-labels and present novel learning mechanisms to improve the label generation quality. To cope with high false-negative and low precision rates, we introduce an adaptive thresholding mechanism that helps the proposed network to filter out optimal bounding boxes. We further introduce a Jitter-Bagging module to provide accurate information on localization to help refine the bounding boxes. Additionally, two new losses are introduced using the background and foreground scores predicted by the teacher and student networks to improvise the pseudo-label recall rate. Furthermore, our method applies strict supervision to the teacher network by feeding strong & weak augmented data to generate robust pseudo-labels so that it can detect small and complex objects. Finally, the extensive experiments show that the proposed network outperforms state-of-the-art methods on MS-COCO and Pascal VOC datasets and allows the baseline network to achieve 100% supervised performance with much less (i.e., 20%) labeled data.
Dual-Weight Particle Filter for Radar-Based Dynamic Bayesian Grid Maps

Through constant improvements in recent years radar sensors have become a viable alternative to lidar as the main distancing sensor of an autonomous vehicle. Although robust and with the possibility to directly measure the radial velocity, it brings it’s own set of challenges, for which existing algorithms need to be adapted. One core algorithm of a perception system is dynamic occupancy grid mapping, which has traditionally relied on lidar. In this paper we present a dual-weight particle filter as an extension for a bayesian occupancy grid mapping framework to allow to operate it with radar as its main sensors. It uses two separate particle weights that are computed differently to compensate that a radial velocity measurement in many situations is not able to capture the actual velocity of an object. We evaluate the method extensively with simulated data and show the advantages over existing single weight solutions.
Investigating CLIP Performance for Meta-data Generation in AD Datasets

Using Machine Learning (ML) models for safety-critical perception tasks in Autonomous Driving (AD) or other domains requires a thorough evaluation of the model performance and the data coverage w.r.t. the intended Operational Design Domain (ODD). However, obtaining the needed per-image semantic meta-data along the relevant dimensions of the ODD for real-world image datasets is non-trivial. Recent advances in self-supervised foundation models, specifically CLIP, suggest that such meta-data could be obtained for real-world images in an automated fashion using zero-shot classification. While CLIP was already reported to achieve promising performance on tasks such as the recognition of gender or age on facial images, we investigate to which extent less prominent and more fine-grained observables, e.g., presence of accessories such as spectacles or the shirt- or hair-color, can be determined. We provide an analysis of CLIP for generating fine-grained meta-data on three datasets from the AD domain, one of synthetic origin including ground truth, the others being Cityscapes and Railsem19. We also compare with a standard facial dataset where more elaborate attribute annotations are present. To improve the quality of generated meta-data, we additionally extend the ensemble approach of CLIP by a simple noise-suppressing technique.
Learning unbiased classifiers from biased data with meta-learning

It is well known that large deep architectures are powerful models when adequately trained, but may exhibit undesirable behavior leading to confident incorrect predictions, even when evaluated on slightly different test examples. Test data characterized by distribution shifts (from training data distribution), outliers, and adversarial samples are among the types of data affected by this problem. This situation worsens whenever data are biased, meaning that predictions are mostly based on spurious correlations present in the data. Unfortunately, since such correlations occur in the most of data, a model is prevented from correctly generalizing the considered classes. In this work, we tackle this problem from a meta-learning perspective. Considering the dataset as composed of unknown biased and unbiased samples, we first identify these two subsets by a pseudo-labeling algorithm, even if coarsely. Subsequently, we apply a bi-level optimization algorithm in which, in the inner loop, we look for the best parameters guiding the training of the two subsets, while in the outer loop, we train the final model taking benefit from augmented data generated using Mixup. Properly tuning the contributions of biased and unbiased data, together with the regularization introduced by the mixed data has proved to be an effective training strategy to learn unbiased models, showing superior generalization capabilities. Experimental results on synthetically and realistically biased datasets surpass state-of-the-art performance, as compared to existing methods.
A Strategy for Boundary Adherence and Exploration in Black-Box Testing of Autonomous Vehicles

The validation of artificial intelligence (AI) controlled vehicles is a vexing challenge. Black box testing of decision making in these vehicles has been used to abstract out the inner complexity of the AI. In scenario-based black box testing, the AI is placed within a scenario, and the input space for that scenario is explored. The fundamental metric of “how well tested the system is for that scenario” is based on the input state space coverage. Since most of these spaces have a high number of dimensions, it is critical to sample the state space efficiently and identify performance boundaries for the vehicle under test. In this paper, we propose a boundary adherence approach for autonomous vehicle validation that can explore the boundary between targeted and non-targeted behavior. This paper significantly improves and extends our previous approach that focused on generic black-box testing of AI systems by optimizing the algorithm itself, adding new tools for exploration, and applying the strategy to scenario-based AV testing. We provide an example regression of a scenario which illustrates the ability to model boundaries after they have been explored. Further results on higher dimensions show differing adherence strategies can improve exploration efficiency and how boundary exploration focuses on more “interesting” scenarios. Upon exploring the boundary, we found that predictions can be made about whether or not the system will result in targeted behavior.
Adversarial Reprogramming as Natural Multitask and Compression Enabler

Deep learning models that exceed human capability are also pruned to error when facing adversarial attacks. Recent works on adversarial reprogramming have unleashed the capability of repurposing a machine-learning model for another task without changing the model parameters. In this paper, we prove the possibility of repurposing a single machine-learning model to solve multiple tasks by adding adversarial-reprogramming functions. In addition, adversarial reprogramming can also be a solution in a memory-limited device that saves more space by storing only one model instead of multiple task-specific models in a single device. In our experiment, adversarial reprogramming can achieve 95% accuracy prediction from a model trained on a different domain. Even in the cross-model type, adversarial reprogramming can perform 83% accuracy on sentiment analysis of a sentence with a model that receives an image as its input type. We also did the experiment on a low-resource language, Bahasa Indonesia, and achieve 75% accuracy without changing the model parameter
Self-supervised 3D Human Pose Estimation from a Single Image

We propose a new self-supervised method for predicting 3D human body pose from a single image. The prediction network is trained from a dataset of unlabelled images depicting people in typical poses and a set of unpaired 2D poses. By minimising the need for annotated data, the method has the potential for rapid application to pose estimation of other articulated structures (e.g. animals). The self-supervision comes from an earlier idea exploiting consistency between predicted pose under 3D rotation. Our method is a substantial advance on state-of-the-art self-supervised methods in training a mapping directly from images, without limb articulation constraints or any 3D empirical pose prior. We compare performance with state-of-the-art self-supervised methods using benchmark datasets that provide images and ground-truth 3D pose (Human3.6M, MPI-INF-3DHP). Despite the reduced requirement for annotated data, we show that the method outperforms on Human3.6M and matches performance on MPI-INF-3DHP. Qualitative results on a dataset of human hands show the potential for rapidly learning to predict 3D pose for articulated structures other than the human body. Project page: https://josesosajs.github.io/imagepose/