OpenRaiser/Intern-Atlas huggingface dataset:
Demo: Testing AI-driven MAC Learning in Autonomic Networks
Venue: arXiv cs.NI 2024
Year: 2024

6G networks will be highly dynamic, re-configurable, and resilient. To enable and support such features, employing AI has been suggested. Integrating AIin networks will likely require distributed AI deployments with resilient connectivity, e.g., for communication between RL agents and environment. Such approaches need to be validated in realistic network environments. In this demo, we use ContainerNet to emulate AI-capable and autonomic networks that employ the routing protocol KIRA to provide resilient connectivity and service discovery. As an example AI application, we train and infer deep RL agents learning medium access control (MAC) policies for a wireless network environment in the emulated network.
LCD-Net: A Lightweight Remote Sensing Change Detection Network Combining Feature Fusion and Gating Mechanism
Venue: arXiv cs.CV 2024
Year: 2024

Remote sensing image change detection (RSCD) is crucial for monitoring dynamic surface changes, with applications ranging from environmental monitoring to disaster assessment. While traditional CNN-based methods have improved detection accuracy, they often suffer from high computational complexity and large parameter counts, limiting their use in resource-constrained environments. To address these challenges, we propose a Lightweight remote sensing Change Detection Network (LCD-Net in short) that reduces model size and computational cost while maintaining high detection performance. LCD-Net employs MobileNetV2 as the encoder to efficiently extract features from bitemporal images. A Temporal Interaction and Fusion Module (TIF) enhances the interaction between bitemporal features, improving temporal context awareness. Additionally, the Feature Fusion Module (FFM) aggregates multiscale features to better capture subtle changes while suppressing background noise. The Gated Mechanism Module (GMM) in the decoder further enhances feature learning by dynamically adjusting channel weights, emphasizing key change regions. Experiments on LEVIR-CD+, SYSU, and S2Looking datasets show that LCD-Net achieves competitive performance with just 2.56M parameters and 4.45G FLOPs, making it well-suited for real-time applications in resource-limited settings. The code is available at https://github.com/WenyuLiu6/LCD-Net.
Black-box Uncertainty Quantification Method for LLM-as-a-Judge
Venue: arXiv cs.LG 2024
Year: 2024

LLM-as-a-Judge is a widely used method for evaluating the performance of Large Language Models (LLMs) across various tasks. We address the challenge of quantifying the uncertainty of LLM-as-a-Judge evaluations. While uncertainty quantification has been well-studied in other domains, applying it effectively to LLMs poses unique challenges due to their complex decision-making capabilities and computational demands. In this paper, we introduce a novel method for quantifying uncertainty designed to enhance the trustworthiness of LLM-as-a-Judge evaluations. The method quantifies uncertainty by analyzing the relationships between generated assessments and possible ratings. By cross-evaluating these relationships and constructing a confusion matrix based on token probabilities, the method derives labels of high or low uncertainty. We evaluate our method across multiple benchmarks, demonstrating a strong correlation between the accuracy of LLM evaluations and the derived uncertainty scores. Our findings suggest that this method can significantly improve the reliability and consistency of LLM-as-a-Judge evaluations.
Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models
Venue: arXiv cs.CV 2024
Year: 2024

Vision-language pre-training (VLP) models, trained on large-scale image-text pairs, have become widely used across a variety of downstream vision-and-language (V+L) tasks. This widespread adoption raises concerns about their vulnerability to adversarial attacks. Non-universal adversarial attacks, while effective, are often impractical for real-time online applications due to their high computational demands per data instance. Recently, universal adversarial perturbations (UAPs) have been introduced as a solution, but existing generator-based UAP methods are significantly time-consuming. To overcome the limitation, we propose a direct optimization-based UAP approach, termed DO-UAP, which significantly reduces resource consumption while maintaining high attack performance. Specifically, we explore the necessity of multimodal loss design and introduce a useful data augmentation strategy. Extensive experiments conducted on three benchmark VLP datasets, six popular VLP models, and three classical downstream tasks demonstrate the efficiency and effectiveness of DO-UAP. Specifically, our approach drastically decreases the time consumption by 23-fold while achieving a better attack performance.
RS-MOCO: A deep learning-based topology-preserving image registration method for cardiac T1 mapping
Venue: arXiv cs.CV 2024
Year: 2024

Cardiac T1 mapping can evaluate various clinical symptoms of myocardial tissue. However, there is currently a lack of effective, robust, and efficient methods for motion correction in cardiac T1 mapping. In this paper, we propose a deep learning-based and topology-preserving image registration framework for motion correction in cardiac T1 mapping. Notably, our proposed implicit consistency constraint dubbed BLOC, to some extent preserves the image topology in registration by bidirectional consistency constraint and local anti-folding constraint. To address the contrast variation issue, we introduce a weighted image similarity metric for multimodal registration of cardiac T1-weighted images. Besides, a semi-supervised myocardium segmentation network and a dual-domain attention module are integrated into the framework to further improve the performance of the registration. Numerous comparative experiments, as well as ablation studies, demonstrated the effectiveness and high robustness of our method. The results also indicate that the proposed weighted image similarity metric, specifically crafted for our network, contributes a lot to the enhancement of the motion correction efficacy, while the bidirectional consistency constraint combined with the local anti-folding constraint ensures a more desirable topology-preserving registration mapping.
VisualRWKV-HD and UHD: Advancing High-Resolution Processing for Visual Language Models
Venue: arXiv cs.CV 2024
Year: 2024

Accurately understanding complex visual information is crucial for visual language models (VLMs). Enhancing image resolution can improve visual perception capabilities, not only reducing hallucinations but also boosting performance in tasks that demand high resolution, such as text-rich or document analysis. In this paper, we present VisualRWKV-HD and VisualRWKV-UHD, two advancements in the VisualRWKV model family, specifically designed to process high-resolution visual inputs. For VisualRWKV-HD, we developed a lossless downsampling method to effectively integrate a high-resolution vision encoder with low-resolution encoders, without extending the input sequence length. For the VisualRWKV-UHD model, we enhanced image representation by dividing the image into four segments, which are then recombined with the original image. This technique allows the model to incorporate both high-resolution and low-resolution features, effectively balancing coarse and fine-grained information. As a result, the model supports resolutions up to 4096 x 4096 pixels, offering a more detailed and comprehensive visual processing capability. Both VisualRWKV-HD and VisualRWKV-UHD not only achieve strong results on VLM benchmarks but also show marked improvements in performance for text-rich tasks.
It's Just Another Day: Unique Video Captioning by Discriminative Prompting
Venue: arXiv cs.CV 2024
Year: 2024

Long videos contain many repeating actions, events and shots. These repetitions are frequently given identical captions, which makes it difficult to retrieve the exact desired clip using a text search. In this paper, we formulate the problem of unique captioning: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We propose Captioning by Discriminative Prompting (CDP), which predicts a property that can separate identically captioned clips, and use it to generate unique captions. We introduce two benchmarks for unique captioning, based on egocentric footage and timeloop movies - where repeating actions are common. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15% for egocentric videos and 10% in timeloop movies.
The Age of DDoScovery: An Empirical Comparison of Industry and Academic DDoS Assessments
Venue: arXiv cs.CR 2024
Year: 2024

Motivated by the impressive but diffuse scope of DDoS research and reporting, we undertake a multistakeholder (joint industry-academic) analysis to seek convergence across the best available macroscopic views of the relative trends in two dominant classes of attacks - direct-path attacks and reflection-amplification attacks. We first analyze 24 industry reports to extract trends and (in)consistencies across observations by commercial stakeholders in 2022. We then analyze ten data sets spanning industry and academic sources, across four years (2019-2023), to find and explain discrepancies based on data sources, vantage points, methods, and parameters. Our method includes a new approach: we share an aggregated list of DDoS targets with industry players who return the results of joining this list with their proprietary data sources to reveal gaps in visibility of the academic data sources. We use academic data sources to explore an industry-reported relative drop in spoofed reflection-amplification attacks in 2021-2022. Our study illustrates the value, but also the challenge, in independent validation of security-related properties of Internet infrastructure. Finally, we reflect on opportunities to facilitate greater common understanding of the DDoS landscape. We hope our results inform not only future academic and industry pursuits but also emerging policy efforts to reduce systemic Internet security vulnerabilities.
Personas with Attitudes: Controlling LLMs for Diverse Data Annotation
Venue: arXiv cs.CL 2024
Year: 2024

We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results show that persona-prompted LLMs produce more diverse annotations than LLMs prompted without personas and that these effects are both controllable and repeatable, making our approach a suitable tool for improving data annotation in subjective NLP tasks like toxicity detection.
Evaluating Software Contribution Quality: Time-to-Modification Theory
Venue: arXiv cs.SE 2024
Year: 2024

The durability and quality of software contributions are critical factors in the long-term maintainability of a codebase. This paper introduces the Time to Modification (TTM) Theory, a novel approach for quantifying code quality by measuring the time interval between a code segment's introduction and its first modification. TTM serves as a proxy for code durability, with longer intervals suggesting higher-quality, more stable contributions. This work builds on previous research, including the "Time-Delta Method for Measuring Software Development Contribution Rates" dissertation, from which it heavily borrows concepts and methodologies. By leveraging version control systems such as Git, TTM provides granular insights into the temporal stability of code at various levels ranging from individual lines to entire repositories. TTM Theory contributes to the software engineering field by offering a dynamic metric that captures the evolution of a codebase over time, complementing traditional metrics like code churn and cyclomatic complexity. This metric is particularly useful for predicting maintenance needs, optimizing developer performance assessments, and improving the sustainability of software systems. Integrating TTM into continuous integration pipelines enables real-time monitoring of code stability, helping teams identify areas of instability and reduce technical debt.
Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?
Venue: arXiv cs.SE 2024
Year: 2024

Search-based software testing (SBST) is a widely adopted technique for testing complex systems with large input spaces, such as Deep Learning-enabled (DL-enabled) systems. Many SBST techniques focus on Pareto-based optimization, where multiple objectives are optimized in parallel to reveal failures. However, it is important to ensure that identified failures are spread throughout the entire failure-inducing area of a search domain and not clustered in a sub-region. This ensures that identified failures are semantically diverse and reveal a wide range of underlying causes. In this paper, we present a theoretical argument explaining why testing based on Pareto optimization is inadequate for covering failure-inducing areas within a search domain. We support our argument with empirical results obtained by applying two widely used types of Pareto-based optimization techniques, namely NSGA-II (an evolutionary algorithm) and OMOPSO (a swarm-based Pareto-optimization algorithm), to two DL-enabled systems: an industrial Automated Valet Parking (AVP) system and a system for classifying handwritten digits. We measure the coverage of failure-revealing test inputs in the input space using a metric that we refer to as the Coverage Inverted Distance quality indicator. Our results show that NSGA-II-based search and OMOPSO are not more effective than a naïve random search baseline in covering test inputs that reveal failures. The replication package for this study is available in a GitHub repository.
End-to-End Mathematical Modeling of Stress Communication Between Plants
Venue: arXiv cs.ET 2024
Year: 2024

Molecular Communication (MC) is an important communication paradigm found in nature. Odor-based Molecular Communication (OMC) is a specific type of MC with promising potential and a wide range of applications. In this paper, we examine OMC communication between plants in the context of stress communication. Specifically, we explore how plants use Biological Volatile Organic Compounds (BVOCs) to convey information about the stresses they are experiencing to neighboring plants. We constructed an end-to-end mathematical model that discovers the underlying physical and biological phenomena affecting stress communication. To the best of our knowledge, this is the first study to model this end-to-end stress communication. We numerically analyzed our system under different scenarios using MATLAB. Using experimental data from the literature, we demonstrated that continuous gene regulation can approximate BVOC emissions in plants under different stress conditions. Consequently, we applied this model to these stressors and plants to accurately approximate BVOC emissions. We also investigated a modulation method that plants use to send their messages, namely Ratio Shift Keying. Upon analyzing this method, we found that it benefits plants by both enabling a multiple access channel and preventing competitor plants from obtaining the information.
OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
Venue: arXiv cs.RO 2024
Year: 2024

We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately. Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, outperforming the state-of-the-art baseline on open-world imitation from observation. Furthermore, OKAMI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve an average success rate of 79.2% without the need for labor-intensive teleoperation. More videos can be found on our website https://ut-austin-rpl.github.io/OKAMI/.
SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing
Venue: arXiv cs.CV 2024
Year: 2024

Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics.
CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos
Venue: arXiv cs.CV 2024
Year: 2024

Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task. However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The new model eliminates or simplifies components from previous trackers, resulting in a simpler and often smaller architecture. This training scheme is much simpler than prior work and achieves better results using 1,000 times less data. We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking. The model is available in online and offline variants and reliably tracks visible and occluded points.
Comparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task
Venue: arXiv cs.HC 2024
Year: 2024

When designing an AI-assisted decision-making system, there is often a tradeoff between precision and recall in the AI's recommendations. We argue that careful exploitation of this tradeoff can harness the complementary strengths in the human-AI collaboration to significantly improve team performance. We investigate a real-world video anonymization task for which recall is paramount and more costly to improve. We analyze the performance of 78 professional annotators working with a) no AI assistance, b) a high-precision "restrained" AI, and c) a high-recall "zealous" AI in over 3,466 person-hours of annotation work. In comparison, the zealous AI helps human teammates achieve significantly shorter task completion time and higher recall. In a follow-up study, we remove AI assistance for everyone and find negative training effects on annotators trained with the restrained AI. These findings and our analysis point to important implications for the design of AI assistance in recall-demanding scenarios.
Post-Userist Recommender Systems : A Manifesto
Venue: arXiv cs.IR 2024
Year: 2024

We define userist recommendation as an approach to recommender systems framed solely in terms of the relation between the user and system. Post-userist recommendation posits a larger field of relations in which stakeholders are embedded and distinguishes the recommendation function (which can potentially connect creators with audiences) from generative media. We argue that in the era of generative media, userist recommendation becomes indistinguishable from personalized media generation, and therefore post-userist recommendation is the only path forward for recommender systems research.
GazeGenie: Enhancing Multi-Line Reading Research with an Innovative User-Friendly Tool
Venue: arXiv cs.HC 2024
Year: 2024

In the study of reading, eye-tracking technology offers unique insights into the time-course of how individuals extract information from text. A significant hurdle in using multi-line paragraph stimuli is the need to align eye gaze position with the correct line. This is made more difficult by positional noise in the eye-tracking data, primarily due to vertical drift, and often necessitates manual intervention. Such manual correction is labor-intensive, subjective, and limits the scalability of research efforts. As a result, automated solutions are desirable, especially those that do not require extensive technical skills and still allow close control over the outcome. To address this, we introduce GazeGenie: a comprehensive software solution designed specifically for researchers in eye-tracking studies on multi-line reading. Accessible via an intuitive web browser-based user interface and easily installed using Docker, GazeGenie streamlines the entire data processing pipeline from parsing fixations from raw data to calculation of word and sentence-based measures based on cleaned and drift-corrected fixations. The software's core features include the recently introduced Dual Input Stream Transformer (DIST) model and various classical algorithms all of which can be combined within a Wisdom of the Crowds (WOC) approach to enhance accuracy in fixation line-assignment. By providing an all-in-one solution for researchers, we hope to make automated fixation alignment more accessible, reducing researchers' reliance on manual intervention in vertical fixation alignment. This should lead to more accurate, efficient, and reproducible analyses of multi-line eye-movement data and pave the way to enabling larger scale studies to be carried out.