Reviewerly/exHarmony huggingface dataset:
Accidental Turntables: Learning 3D Pose by Watching Objects Turn

We propose a technique for learning single-view 3D object pose estimation models by utilizing a new source of data — in-the-wild videos where objects turn. Such videos are prevalent in practice (e.g. cars in roundabouts, airplanes near runways) and easy to collect. We show that classical structure-from-motion algorithms, coupled with the recent advances in instance detection and feature matching, provide surprisingly accurate relative 3D pose estimation on such videos. We propose a multi-stage training scheme that first learns a canonical pose across a collection of videos and then supervises a model for single-view pose estimation. The proposed technique achieves competitive performance with respect to the existing state-of-the-art on standard benchmarks for 3D pose estimation without requiring any pose labels during training. We also contribute an Accidental Turntables Dataset, containing a challenging set of 41,212 images of cars in cluttered backgrounds, motion blur, and illumination changes that serve as a benchmark for 3D pose estimation.
Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?

Self-supervised representation learning (SSL) in computer vision aims to leverage the inherent structure and relationships within data to learn meaningful representations without explicit human annotation, enabling a holistic understanding of visual scenes. Robustness in vision machine learning ensures reliable and consistent performance, enhancing generalization, adaptability, and resistance to noise, variations, and adversarial attacks. Self-supervised representation learning paradigms, namely contrastive learning, knowledge distillation, mutual information maximization, and clustering, have been considered to have shown advances in invariant learning representations. This work investigates the robustness of learned representations of SSL approaches focusing on distribution shifts and image corruptions in computer vision. Detailed experiments have been conducted to study the robustness of SSL methods on distribution shifts and image corruptions. The empirical analysis demonstrates a clear relationship between the performance of learned representations within SSL paradigms and the severity of distribution shifts and corruptions. Notably, higher levels of shifts and corruptions are found to significantly diminish the robustness of the learned representations. These findings highlight the critical impact of distribution shifts and image corruptions on the performance and resilience of SSL methods, emphasizing the need for effective strategies to mitigate their adverse effects. The study strongly advocates for future research in the field of self-supervised representation learning to prioritize the key aspects of safety and robustness in order to ensure practical applicability. The source code and results are available on GitHub. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
ACTIS: Improving data efficiency by leveraging semi-supervised Augmentation Consistency Training for Instance Segmentation

Segmenting objects like cells or nuclei in biomedical microscopy data is a standard task required for many downstream analyses. However, existing pre-trained models are continuously challenged by ever-evolving experimental setups and imaging platforms. On the other hand, training new models still requires a considerable number of annotated samples, rendering it infeasible for small to midsized experiments. To address this challenge, we propose a semi-supervised learning approach for instance segmentation that leverages a small number of annotated samples together with a larger number of unannotated samples. Our pipeline, Augmentation Consistency Training for Instance Segmentation (ACTIS), incorporates methods from consistency regularization and entropy minimization. In addition, we introduce a robust confidence-based loss masking scheme which we find empirically to work well on highly imbalanced class frequencies. We show that our model can surpass the performance of supervised models trained on more than twice as much annotated data. It achieves state-of-the-art results on three benchmark datasets in the biomedical domain, demonstrating its effectiveness for semi-supervised instance segmentation. Code: https://github.com/Kainmueller-Lab/ACTIS
AntiNODE: Evaluating Efficiency Robustness of Neural ODEs

Recently, Neural ODE (Ordinary Differential Equation) models have been proposed, which use ordinary differential equation solving to predict the output of neural networks. Due to Neural ODE models’ noticeably lower parameter usage compared to traditional Deep Neural Networks (DNN) and higher robustness against gradient-based attacks, they are being adopted in many type of real-time applications. For real-time applications, response-time (latency) has paramount importance due to the convenience of the user. Through our observation, we find that the latency during Neural ODE inference can be highly dynamic and sometimes detrimental to the system due to the adaptive nature of the ODE solvers. Because of that reason, understanding and evaluating efficiency robustness of Neural ODE models is needed, which has not received much attention yet. However, evaluating efficiency robustness of any model is dependent on the relationship between input and latency, which has not been defined yet for Neural ODE models. In this work, we first formulate the relationship between input and dynamic latency consumption of Neural ODEs. Based on the formulation, we propose AntiNODE, which generates latency-surging adversarial inputs for Neural ODEs by increasing the computations in Neural ODEs. We evaluate AntiNODE on two popular datasets and three ODE solvers on both hardware dependent and independent metrics. Results show that the adversarial inputs generated by AntiNODE can decrease up to 335% efficiency during inference. Our evaluation also shows that the generated adversarial inputs are transferable across multiple solvers and multiple architectures, which indicates the feasibility of black-box attack.
COSE: A Consistency-Sensitivity Metric for Saliency on Image Classification

We present a set of metrics that utilize vision priors to effectively assess the performance of saliency methods on image classification tasks. To understand behavior in deep learning models, many methods provide visual saliency maps emphasizing image regions that most contribute to a model prediction. However, there is limited work on analyzing the reliability of saliency methods in explaining model decisions. We propose the metric COnsistency-SEnsitivity (COSE) that quantifies the equivariant and invariant properties of visual model explanations using simple data augmentations. Through our metrics, we show that although saliency methods are thought to be architecture-independent, most methods could better explain transformer-based models over convolutional-based models. In addition, GradCAM was found to outperform other methods in terms of COSE but was shown to have limitations such as lack of variability for fine-grained datasets. The duality between consistency and sensitivity allow the analysis of saliency methods from different angles. Ultimately, we find that it is important to balance these two metrics for a saliency map to faithfully show model behavior.
SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis

Text-conditioned image generation has made significant progress in recent years with generative adversarial networks and more recently, diffusion models. While diffusion models conditioned on text prompts have produced impressive and high-quality images, accurately representing complex text prompts such as the number of instances of a specific object remains challenging.To address this limitation, we propose a novel guidance approach for the sampling process in the diffusion model that leverages bounding box and segmentation map information at inference time without additional training data. Through a novel loss in the sampling process, our approach guides the model with semantic features from CLIP embeddings and enforces geometric constraints, leading to high-resolution images that accurately represent the scene. To obtain bounding box and segmentation map information, we structure the text prompt as a scene graph and enrich the nodes with CLIP embeddings. Our proposed model achieves state-of-the-art performance on two public benchmarks for image generation from scene graphs, surpassing both scene graph to image and text-based diffusion models in various metrics. Our results demonstrate the effectiveness of incorporating bounding box and segmentation map guidance in the diffusion model sampling process for more accurate text-to-image generation. Project Page: scenegenie.github.io/SceneGenie/
Unsupervised Camouflaged Object Segmentation as Domain Adaptation

Deep learning for unsupervised image segmentation remains challenging due to the absence of human labels. The common idea is to train a segmentation head, with the supervision of pixel-wise pseudo-labels generated based on the representation of self-supervised backbones. By doing so, the model performance depends much on the distance between the distribution of target datasets, and the one of backbones’ pre-training dataset (e.g., ImageNet). In this work, we investigate a new task, namely unsupervised camouflaged object segmentation (UCOS), where the target objects own a common rarely-seen attribute, i.e., camouflage. Unsurprisingly, we find that the state-of-the-art unsupervised models struggle in adapting UCOS, due to the domain gap between the properties of generic and camouflaged objects. To this end, we formulate the UCOS as a source-free unsupervised domain adaptation task (UCOS-DA), where both source labels and target labels are absent during the whole model training process. Specifically, we define a source model consisting of self-supervised vision transformers pre-trained on ImageNet. On the other hand, the target domain includes a simple linear layer (i.e., our target model) and unlabeled camouflaged objects. We then design a pipeline for foreground-background-contrastive self-adversarial domain adaptation, to achieve robust UCOS. As a result, our baseline model achieves superior segmentation performance when compared with competing unsupervised models on the UCOS benchmark, with the training set which’s scale is only one tenth of the supervised COS counterpart. The UCOS benchmark and our baseline model are now publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Towards Hierarchical Regional Transformer-based Multiple Instance Learning

The classification of gigapixel histopathology images with deep multiple instance learning models has become a critical task in digital pathology and precision medicine. In this work, we propose a Transformer-based multiple instance learning method that replaces the traditional learned attention mechanism with a regional, Vision Transformer inspired self-attention mechanism. We additionally propose a method that fuses regional patch information to derive slide-level predictions. We then show how this regional aggregation can be stacked to hierarchically process features on different distance levels. To increase predictive accuracy, especially for datasets with small, local morphological features, we also suggest a method to focus the image processing on high attention regions during inference. Our approach is able to significantly improve performance over the baseline on two histopathology datasets and points towards promising directions for further research.
E-Voting System Using Solana Blockchain

Electronic voting methods are gaining popularity in contemporary culture. It has the potential to reduce administrative expenses and enhance voter participation. Electronic voting systems must be lawful, accurate, secure, and convenient when utilized in elections. This paper aims to establish a project to make electoral systems more decentralized and scalable. As functionality increases, it significantly enhances the Proof-of-Work idea embraced by Bitcoin and Ethereum. Through Solana, it will be simple to implement business logic and ensure that it assists in resolving the world's common problems. We used Solana libraries and the Anchor framework to implement our system to provide convenient tools for developing secure Solana programs and rust programming language.
Research on Radar Emitter Identification Algorithm Based on Multi-Sensor Data Fusion

To improve the reliability and anti-jamming ability of the radar countermeasure reconnaissance system in the complex electromagnetic environment, we propose a multi-sensor data fusion algorithm for parameter estimation. Firstly, we eliminate abnormal data based on the quartile distribution plot method and obtain the mean and variance by batch estimation. Secondly, the consistency test is used to remove the sensor data with low support. After that, the inherent accuracy weights are determined based on the systematic errors of the sensors, the reliability weights are calculated by the adaptive fusion, and finally, the fusion results are obtained by weighting the batch estimation values with the combined weights of the two. Simulation experiments show that the proposed algorithm achieves excellent fusion adaptability, possesses better accuracy than the existing algorithms, and effectively improves the performance of multi-sensor radiation emitter identification.
VSCHH 2023: A Benchmark for the View Synthesis Challenge of Human Heads

This manuscript presents the results of the "A View Synthesis Challenge for Humans Heads (VSCHH)", which was part of the ICCV 2023 workshops. This paper describes the competition setup and provides details on replicating our initial baseline, TensoRF. Additionally, we provide a summary of the participants’ methods and their results in our benchmark table. The challenge aimed to synthesize novel camera views of human heads using a given set of sparse training view images. The proposed solutions of the participants were evaluated and ranked based on objective fidelity metrics, such as PSNR and SSIM, computed against unseen validation and test sets. In the supplementary material, we detailed the methods used by all participants in the VSCHH challenge, which opened on May 15th, 2023, and concluded on July 24th, 2023.
PARTICLE: Part Discovery and Contrastive Learning for Fine-grained Recognition

We develop techniques for refining representations for fine-grained classification and segmentation tasks in a self-supervised manner. We find that fine-tuning methods based on instance-discriminative contrastive learning are not as effective, and posit that recognizing part-specific variations is crucial for fine-grained categorization. We present an iterative learning approach that incorporates part-centric equivariance and invariance objectives. First, pixel representations are clustered to discover parts. We analyze the representations from convolutional and vision transformer networks that are best suited for this task. Then, a part-centric learning step aggregates and contrasts representations of parts within an image. We show that this improves the performance on image classification and part segmentation tasks across datasets. For example, under a linear-evaluation scheme, the classification accuracy of a ResNet50 trained on ImageNet using DetCon [17], a self-supervised learning approach, improves from 35.4% to 42.0% on the Caltech-UCSD Birds, from 35.5% to 44.1% on the FGVC Aircraft, and from 29.7% to 37.4% on the Stanford Cars. We also observe significant gains in few-shot part segmentation tasks using the proposed technique, while instance-discriminative learning was not as effective. Smaller, yet consistent, improvements are also observed for stronger networks based on transformers.
Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features

The rapid development of 3D object detection systems for self-driving cars has significantly improved accuracy. However, these systems struggle to generalize across diverse driving environments, which can lead to safety-critical failures in detecting traffic participants. To address this, we propose a method that utilizes unlabeled repeated traversals of multiple locations to adapt object detectors to new driving environments. By incorporating statistics computed from repeated LiDAR scans, we guide the adaptation process effectively. Our approach enhances LiDAR-based detection models using spatial quantized historical features and introduces a lightweight regression head to leverage the statistics for feature regularization. Additionally, we leverage the statistics for a novel self-training process to stabilize the training. The framework is detector model-agnostic and experiments on real-world datasets demonstrate significant improvements, achieving up to a 20-point performance gain, especially in detecting pedestrians and distant objects. Code is available at https://github.com/zhangtravis/Hist-DA.
Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation

Within the context of autonomous driving, encountering unknown objects becomes inevitable during deployment in the open world. Therefore, it is crucial to equip standard semantic segmentation models with anomaly awareness. Many previous approaches have utilized synthetic out-of-distribution (OoD) data augmentation to tackle this problem. In this work, we advance the OoD synthesis process by reducing the domain gap between the OoD data and driving scenes, effectively mitigating the style difference that might otherwise act as an obvious shortcut during training. Additionally, we propose a simple fine-tuning loss that effectively induces a pre-trained semantic segmentation model to generate a "none of the given classes" prediction, leveraging per-pixel OoD scores for anomaly segmentation. With minimal fine-tuning effort, our pipeline enables the use of pre-trained models for anomaly segmentation while maintaining the performance on the original task.
Which Tokens to Use? Investigating Token Reduction in Vision Transformers

Since the introduction of the Vision Transformer (ViT), researchers have sought to make ViTs more efficient by removing redundant information in the processed tokens. While different methods have been explored to achieve this goal, we still lack understanding of the resulting reduction patterns and how those patterns differ across token reduction methods and datasets. To close this gap, we set out to understand the reduction patterns of 10 different token reduction methods using four image classification datasets. By systematically comparing these methods on the different classification tasks, we find that the Top-K pruning method is a surprisingly strong baseline. Through in-depth analysis of the different methods, we determine that: the reduction patterns are generally not consistent when varying the capacity of the backbone model, the reduction patterns of pruning-based methods significantly differ from fixed radial patterns, and the reduction patterns of pruning-based methods are correlated across classification datasets. Finally we report that the similarity of reduction patterns is a moderate-to-strong proxy for model performance. Project page at https://vap.aau.dk/tokens.
SortedAP: Rethinking evaluation metrics for instance segmentation

Designing metrics for evaluating instance segmentation revolves around comprehensively considering object detection and segmentation accuracy. However, other important properties, such as sensitivity, continuity, and equality, are overlooked in the current study. In this paper, we reveal that most existing metrics have a limited resolution of segmentation quality. They are only conditionally sensitive to the change of masks or false predictions. For certain metrics, the score can change drastically in a narrow range which could provide a misleading indication of the quality gap between results. Therefore, we propose a new metric called sortedAP, which strictly decreases with both object- and pixel-level imperfections and has an uninterrupted penalization scale over the entire domain. We provide the evaluation toolkit and experiment code at https://www.github.com/looooongChen/sortedAP.
OpenIncrement: A Unified Framework for Open Set Recognition and Deep Class-Incremental Learning

In most works on deep incremental learning research, it is assumed that novel samples are pre-identified for neural network retraining. However, practical deep classifiers often misidentify these samples, leading to erroneous predictions. Such misclassifications can degrade model performance. Techniques like open set recognition offer a means to detect these novel samples, representing a significant area in the machine learning domain.In this paper, we introduce a deep class-incremental learning framework integrated with open set recognition. Our approach refines class-incrementally learned features to adapt them for distance-based open set recognition. Experimental results validate that our method outperforms state-of-the-art incremental learning techniques and exhibits superior performance in open set recognition compared to baseline methods.
Confidence-Guided Open-World Semi-supervised Learning

Open-world semi-supervised learning differs from traditional semi-supervised learning by allowing labeled and unlabeled data from different data distributions, i.e., unlabeled data may contain novel classes. The goal is to identify samples of known classes while simultaneously detecting and clustering samples of novel classes found in the unlabeled data. To address this challenging task, we propose a novel open-world semi-supervised learning model, which integrates two new confidence-guided techniques to better leverage unlabeled data, especially those from novel classes. Specifically, we first approximate the confidence of model on unlabeled samples by inspecting the prediction consistency through the training process, which guides the temperature scaling for cross-entropy loss. Then, we introduce an adaptive class-specific reweighting contrastive learning method, which further synchronizes the learning speed of known and unknown classes, while reducing the influence of noisy labels. Extensive empirical results show our approach achieves significant performance improvement in both seen and unseen classes compared with previous studies, especially in challenging scenarios with limited labeled data.
FlashViT: A Flash Vision Transformer with Large-Scale Token Merging for Congenital Heart Disease Detection

Congenital heart disease (CHD) is the most common congenital malformation and imaging examination is an important means to diagnose it. Currently, deep learning-based methods have achieved remarkable results in various types of imaging examinations. However, the issues of large parameter size and low throughput limit their clinical applications. In this paper, we design an efficient, light-weight hybrid model named FlashViT, to assist cardiovascular radiologists in early screening and diagnosis of CHD. Specifically, we propose the Large-scale Token Merging Module (LTM) for more aggressive similar token merging without sacrificing accuracy, which alleviate the problem of high computational complexity and resource consumption of self-attention mechanism. In addition, we propose an unsupervised homogenous pre-training strategy to tackle the issue of insufficient medical image data and poor generalization ability. Compared with conventional pre-training strategy that use ImageNet1K, our strategy only utilizes less than 1 $$\%$$ of the class-agnostic medical images from ImageNet1K, resulting in faster convergence speed and advanced performance of the model. We conduct extensive validation on the collected CHD dataset and the results indicate that our proposed FlashViT-S achieves accuracy of 92.2 $$\%$$ and throughput of 3753 fps with about 3.8 million parameters. We hope that this work can provide some assistance in designing laboratory models for future application in clinical practice.