OpenRaiser/Intern-Atlas huggingface dataset:
3D Multimodal Image Registration for Plant Phenotyping
Venue: arXiv cs.CV 2024
Year: 2024

The use of multiple camera technologies in a combined multimodal monitoring system for plant phenotyping offers promising benefits. Compared to configurations that only utilize a single camera technology, cross-modal patterns can be recorded that allow a more comprehensive assessment of plant phenotypes. However, the effective utilization of cross-modal patterns is dependent on precise image registration to achieve pixel-accurate alignment, a challenge often complicated by parallax and occlusion effects inherent in plant canopy imaging. In this study, we propose a novel multimodal 3D image registration method that addresses these challenges by integrating depth information from a time-of-flight camera into the registration process. By leveraging depth data, our method mitigates parallax effects and thus facilitates more accurate pixel alignment across camera modalities. Additionally, we introduce an automated mechanism to identify and differentiate different types of occlusions, thereby minimizing the introduction of registration errors. To evaluate the efficacy of our approach, we conduct experiments on a diverse image dataset comprising six distinct plant species with varying leaf geometries. Our results demonstrate the robustness of the proposed registration algorithm, showcasing its ability to achieve accurate alignment across different plant types and camera compositions. Compared to previous methods it is not reliant on detecting plant specific image features and can thereby be utilized for a wide variety of applications in plant sciences. The registration approach principally scales to arbitrary numbers of cameras with different resolutions and wavelengths. Overall, our study contributes to advancing the field of plant phenotyping by offering a robust and reliable solution for multimodal image registration.
Affine Frequency Division Multiplexing for Compressed Sensing of Time-Varying Channels
Venue: arXiv cs.IT 2024
Year: 2024

This paper addresses compressed sensing of linear time-varying (LTV) wireless propagation links under the assumption of double sparsity i.e., sparsity in both the delay and Doppler domains, using Affine Frequency Division Multiplexing (AFDM) measurements. By rigorously linking the double sparsity model to the hierarchical sparsity paradigm, a compressed sensing algorithm with recovery guarantees is proposed for extracting delay-Doppler profiles of LTV channels using AFDM. Through mathematical analysis and numerical results, the superiority of AFDM over other waveforms in terms of channel estimation overhead and minimal sampling rate requirements in sub-Nyquist radar applications is demonstrated.
What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks
Venue: arXiv cs.CL 2024
Year: 2024

Tool learning methods have enhanced the ability of large language models (LLMs) to interact with real-world applications. Many existing works fine-tune LLMs or design prompts to enable LLMs to select appropriate tools and correctly invoke them to meet user requirements. However, it is observed in previous works that the performance of tool learning varies from tasks, datasets, training settings, and algorithms. Without understanding the impact of these factors, it can lead to inconsistent results, inefficient model deployment, and suboptimal tool utilization, ultimately hindering the practical integration and scalability of LLMs in real-world scenarios. Therefore, in this paper, we explore the impact of both internal and external factors on the performance of tool learning frameworks. Through extensive experiments on two benchmark datasets, we find several insightful conclusions for future work, including the observation that LLMs can benefit significantly from increased trial and exploration. We believe our empirical study provides a new perspective for future tool learning research.
Exploiting Dialect Identification in Automatic Dialectal Text Normalization
Venue: arXiv cs.CL 2024
Year: 2024

Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, and pretrained models publicly available.
xApp Distillation: AI-based Conflict Mitigation in B5G O-RAN
Venue: arXiv cs.NI 2024
Year: 2024

The advancements of machine learning-based (ML) decision-making algorithms created various research and industrial opportunities. One of these areas is ML-based near-real-time network management applications (xApps) in Open-Radio Access Network (O-RAN). Normally, xApps are designed solely for the desired objectives, and fine-tuned for deployment. However, telecommunication companies can employ multiple xApps and deploy them in overlapping areas. Consider the different design objectives of xApps, the deployment might cause conflicts. To prevent such conflicts, we proposed the xApp distillation method that distills knowledge from multiple xApps, then uses this knowledge to train a single model that has retained the capabilities of Previous xApps. Performance evaluations show that compared conflict mitigation schemes can cause up to six times more network outages than xApp distillation in some cases.
Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning
Venue: arXiv cs.RO 2024
Year: 2024

Teleoperation is a crucial tool for collecting human demonstrations, but controlling robots with bimanual dexterous hands remains a challenge. Existing teleoperation systems struggle to handle the complexity of coordinating two hands for intricate manipulations. We introduce Bunny-VisionPro, a real-time bimanual dexterous teleoperation system that leverages a VR headset. Unlike previous vision-based teleoperation systems, we design novel low-cost devices to provide haptic feedback to the operator, enhancing immersion. Our system prioritizes safety by incorporating collision and singularity avoidance while maintaining real-time performance through innovative designs. Bunny-VisionPro outperforms prior systems on a standard task suite, achieving higher success rates and reduced task completion times. Moreover, the high-quality teleoperation demonstrations improve downstream imitation learning performance, leading to better generalizability. Notably, Bunny-VisionPro enables imitation learning with challenging multi-stage, long-horizon dexterous manipulation tasks, which have rarely been addressed in previous work. Our system's ability to handle bimanual manipulations while prioritizing safety and real-time performance makes it a powerful tool for advancing dexterous manipulation and imitation learning.
Consistent Point Orientation for Manifold Surfaces via Boundary Integration
Venue: arXiv cs.CV 2024
Year: 2024

This paper introduces a new approach for generating globally consistent normals for point clouds sampled from manifold surfaces. Given that the generalized winding number (GWN) field generated by a point cloud with globally consistent normals is a solution to a PDE with jump boundary conditions and possesses harmonic properties, and the Dirichlet energy of the GWN field can be defined as an integral over the boundary surface, we formulate a boundary energy derived from the Dirichlet energy of the GWN. Taking as input a point cloud with randomly oriented normals, we optimize this energy to restore the global harmonicity of the GWN field, thereby recovering the globally consistent normals. Experiments show that our method outperforms state-of-the-art approaches, exhibiting enhanced robustness to noise, outliers, complex topologies, and thin structures. Our code can be found at \url{https://github.com/liuweizhou319/BIM}.
Localization in Dynamic Planar Environments Using Few Distance Measurements
Venue: arXiv cs.RO 2024
Year: 2024

We present a method for determining the unknown location of a sensor placed in a known 2D environment in the presence of unknown dynamic obstacles, using only few distance measurements. We present guarantees on the quality of the localization, which are robust under mild assumptions on the density of the unknown/dynamic obstacles in the known environment. We demonstrate the effectiveness of our method in simulated experiments for different environments and varying dynamic-obstacle density. Our open source software is available at https://github.com/TAU-CGL/vb-fdml2-public.
PPO-based Dynamic Control of Uncertain Floating Platforms in the Zero-G Environment
Venue: arXiv cs.RO 2024
Year: 2024

In the field of space exploration, floating platforms play a crucial role in scientific investigations and technological advancements. However, controlling these platforms in zero-gravity environments presents unique challenges, including uncertainties and disturbances. This paper introduces an innovative approach that combines Proximal Policy Optimization (PPO) with Model Predictive Control (MPC) in the zero-gravity laboratory (Zero-G Lab) at the University of Luxembourg. This approach leverages PPO's reinforcement learning power and MPC's precision to navigate the complex control dynamics of floating platforms. Unlike traditional control methods, this PPO-MPC approach learns from MPC predictions, adapting to unmodeled dynamics and disturbances, resulting in a resilient control framework tailored to the zero-gravity environment. Simulations and experiments in the Zero-G Lab validate this approach, showcasing the adaptability of the PPO agent. This research opens new possibilities for controlling floating platforms in zero-gravity settings, promising advancements in space exploration.
Single Character Perturbations Break LLM Alignment
Venue: arXiv cs.LG 2024
Year: 2024

When LLMs are deployed in sensitive, human-facing settings, it is crucial that they do not output unsafe, biased, or privacy-violating outputs. For this reason, models are both trained and instructed to refuse to answer unsafe prompts such as "Tell me how to build a bomb." We find that, despite these safeguards, it is possible to break model defenses simply by appending a space to the end of a model's input. In a study of eight open-source models, we demonstrate that this acts as a strong enough attack to cause the majority of models to generate harmful outputs with very high success rates. We examine the causes of this behavior, finding that the contexts in which single spaces occur in tokenized training data encourage models to generate lists when prompted, overriding training signals to refuse to answer unsafe requests. Our findings underscore the fragile state of current model alignment and promote the importance of developing more robust alignment methods. Code and data will be available at https://github.com/hannah-aught/space_attack.
How Similar Are Elected Politicians and Their Constituents? Quantitative Evidence From Online Social Networks
Venue: arXiv cs.SI 2024
Year: 2024

How similar are politicians to those who vote for them? This is a critical question at the heart of democratic representation and particularly relevant at times when political dissatisfaction and populism are on the rise. To answer this question we compare the online discourse of elected politicians and their constituents. We collect a two and a half years (September 2020 - February 2023) constituency-level dataset for USA and UK that includes: (i) the Twitter timelines (5.6 Million tweets) of elected political representatives (595 UK Members of Parliament and 433 USA Representatives), (ii) the Nextdoor posts (21.8 Million posts) of the constituency (98.4% USA and 91.5% UK constituencies). We find that elected politicians tend to be equally similar to their constituents in terms of content and style regardless of whether a constituency elects a right or left-wing politician. The size of the electoral victory and the level of income of a constituency shows a nuanced picture. The narrower the electoral victory, the more similar the style and the more dissimilar the content is. The lower the income of a constituency, the more similar the content is. In terms of style, poorer constituencies tend to have a more similar sentiment and more dissimilar psychological text traits (i.e. measured with LIWC categories).
For a semiotic AI: Bridging computer vision and visual semiotics for computational observation of large scale facial image archives
Venue: arXiv cs.CV 2024
Year: 2024

Social networks are creating a digital world in which the cognitive, emotional, and pragmatic value of the imagery of human faces and bodies is arguably changing. However, researchers in the digital humanities are often ill-equipped to study these phenomena at scale. This work presents FRESCO (Face Representation in E-Societies through Computational Observation), a framework designed to explore the socio-cultural implications of images on social media platforms at scale. FRESCO deconstructs images into numerical and categorical variables using state-of-the-art computer vision techniques, aligning with the principles of visual semiotics. The framework analyzes images across three levels: the plastic level, encompassing fundamental visual features like lines and colors; the figurative level, representing specific entities or concepts; and the enunciation level, which focuses particularly on constructing the point of view of the spectator and observer. These levels are analyzed to discern deeper narrative layers within the imagery. Experimental validation confirms the reliability and utility of FRESCO, and we assess its consistency and precision across two public datasets. Subsequently, we introduce the FRESCO score, a metric derived from the framework's output that serves as a reliable measure of similarity in image content.
VCHAR:Variance-Driven Complex Human Activity Recognition framework with Generative Representation
Venue: arXiv cs.AI 2024
Year: 2024

Complex human activity recognition (CHAR) remains a pivotal challenge within ubiquitous computing, especially in the context of smart environments. Existing studies typically require meticulous labeling of both atomic and complex activities, a task that is labor-intensive and prone to errors due to the scarcity and inaccuracies of available datasets. Most prior research has focused on datasets that either precisely label atomic activities or, at minimum, their sequence approaches that are often impractical in real world settings.In response, we introduce VCHAR (Variance-Driven Complex Human Activity Recognition), a novel framework that treats the outputs of atomic activities as a distribution over specified intervals. Leveraging generative methodologies, VCHAR elucidates the reasoning behind complex activity classifications through video-based explanations, accessible to users without prior machine learning expertise. Our evaluation across three publicly available datasets demonstrates that VCHAR enhances the accuracy of complex activity recognition without necessitating precise temporal or sequential labeling of atomic activities. Furthermore, user studies confirm that VCHAR's explanations are more intelligible compared to existing methods, facilitating a broader understanding of complex activity recognition among non-experts.
Advanced Smart City Monitoring: Real-Time Identification of Indian Citizen Attributes
Venue: arXiv cs.CV 2024
Year: 2024

This project focuses on creating a smart surveillance system for Indian cities that can identify and analyze people's attributes in real time. Using advanced technologies like artificial intelligence and machine learning, the system can recognize attributes such as upper body color, what the person is wearing, accessories they are wearing, headgear, etc., and analyze behavior through cameras installed around the city.
Constant-Factor EFX Exists for Chores
Venue: arXiv cs.GT 2024
Year: 2024

We study the problem of fair allocation of chores to agents with additive preferences. In the discrete setting, envy-freeness up to any chore (EFX) has emerged as a compelling fairness criterion. However, establishing its (non-)existence or achieving a meaningful approximation remains a major open question. The current best guarantee is the existence of $O(n^2)$-EFX allocations for $n$ agents, obtained through a sophisticated algorithm (Zhou and Wu, 2022). In this paper, we show the existence of $4$-EFX allocations, providing the first constant-factor approximation of EFX. We also investigate the existence of allocations that are both fair and efficient, using Pareto optimality (PO) as our efficiency criterion. For the special case of bivalued instances, we establish the existence of allocations that are both $3$-EFX and PO, thus improving the current best factor of $O(n)$-EFX without any efficiency guarantees. For general additive instances, the existence of allocations that are $α$-EF$k$ and PO has remained open for any constant values of $α$ and $k$, where EF$k$ denotes envy-freeness up to $k$ chores. We provide the first positive result in this direction by showing the existence of allocations that are $2$-EF$2$ and PO. Our results are obtained via a novel economic framework called earning restricted (ER) competitive equilibrium for fractional allocations, which limits agents' earnings from each chore. We show the existence of ER equilibria by formulating it as an linear complementarity problem (LCP) and proving that the classic complementary pivot algorithm on the LCP terminates at an ER equilibrium. We design algorithms that carefully round fractional ER equilibria, and perform bundle swaps and merges to meet the desired fairness and efficiency criteria. We expect that the concept of ER equilibrium will be useful in deriving further results on related problems.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
Venue: arXiv cs.CV 2024
Year: 2024

We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks. The InternLM-XComposer-2.5 is publicly available at https://github.com/InternLM/InternLM-XComposer.
Efficient Visibility Approximation for Game AI using Neural Omnidirectional Distance Fields
Venue: arXiv cs.CV 2024
Year: 2024

Visibility information is critical in game AI applications, but the computational cost of raycasting-based methods poses a challenge for real-time systems. To address this challenge, we propose a novel method that represents a partitioned game scene as neural Omnidirectional Distance Fields (ODFs), allowing scalable and efficient visibility approximation between positions without raycasting. For each position of interest, we map its omnidirectional distance data from the spherical surface onto a UV plane. We then use multi-resolution grids and bilinearly interpolated features to encode directions. This allows us to use a compact multi-layer perceptron (MLP) to reconstruct the high-frequency directional distance data at these positions, ensuring fast inference speed. We demonstrate the effectiveness of our method through offline experiments and in-game evaluation. For in-game evaluation, we conduct a side-by-side comparison with raycasting-based visibility tests in three different scenes. Using a compact MLP (128 neurons and 2 layers), our method achieves an average cold start speedup of 9.35 times and warm start speedup of 4.8 times across these scenes. In addition, unlike the raycasting-based method, whose evaluation time is affected by the characteristics of the scenes, our method's evaluation time remains constant.
ESBMC-Python: A Bounded Model Checker for Python Programs
Venue: arXiv cs.SE 2024
Year: 2024

This paper introduces a tool for verifying Python programs, which, using type annotation and front-end processing, can harness the capabilities of a bounded model-checking (BMC) pipeline. It transforms an input program into an abstract syntax tree to infer and add type information. Then, it translates Python expressions and statements into an intermediate representation. Finally, it converts this description into formulae evaluated with satisfiability modulo theories (SMT) solvers. The proposed approach was realized with the efficient SMT-based bounded model checker (ESBMC), which resulted in a tool called ESBMC-Python, the first BMC-based Python-code verifier. Experimental results, with a test suite specifically developed for this purpose, showed its effectiveness, where successful and failed tests were correctly evaluated. Moreover, it found a real problem in the Ethereum Consensus Specification.
AntibotV: A Multilevel Behaviour-based Framework for Botnets Detection in Vehicular Networks
Venue: arXiv cs.CR 2024
Year: 2024

Connected cars offer safety and efficiency for both individuals and fleets of private vehicles and public transportation companies. However, equipping vehicles with information and communication technologies raises privacy and security concerns, which significantly threaten the user's data and life. Using bot malware, a hacker may compromise a vehicle and control it remotely, for instance, he can disable breaks or start the engine remotely. In this paper, besides in-vehicle attacks existing in the literature, we consider new zeroday bot malware attacks specific to the vehicular context, WSMP-Flood, and Geo-WSMP Flood. Then, we propose AntibotV, a multilevel behaviour-based framework for vehicular botnets detection in vehicular networks. The proposed framework combines two main modules for attack detection, the first one monitors the vehicle's activity at the network level, whereas the second one monitors the in-vehicle activity. The two intrusion detection modules have been trained on a historical network and in-vehicle communication using decision tree algorithms. The experimental results showed that the proposed framework outperforms existing solutions, it achieves a detection rate higher than 97% and a false positive rate lower than 0.14%.