OpenRaiser/Intern-Atlas huggingface dataset:
Embodiment-Agnostic Navigation Policy Trained with Visual Demonstrations
Venue: arXiv cs.RO 2024
Year: 2024

Learning to navigate in unstructured environments is a challenging task for robots. While reinforcement learning can be effective, it often requires extensive data collection and can pose risk. Learning from expert demonstrations, on the other hand, offers a more efficient approach. However, many existing methods rely on specific robot embodiments, pre-specified target images and require large datasets. We propose the Visual Demonstration-based Embodiment-agnostic Navigation (ViDEN) framework, a novel framework that leverages visual demonstrations to train embodiment-agnostic navigation policies. ViDEN utilizes depth images to reduce input dimensionality and relies on relative target positions, making it more adaptable to diverse environments. By training a diffusion-based policy on task-centric and embodiment-agnostic demonstrations, ViDEN can generate collision-free and adaptive trajectories in real-time. Our experiments on human reaching and tracking demonstrate that ViDEN outperforms existing methods, requiring a small amount of data and achieving superior performance in various indoor and outdoor navigation scenarios. Project website: https://nimicurtis.github.io/ViDEN/.
An Optimization Driven Link SINR Assurance in RIS-assisted Indoor Networks
Venue: arXiv cs.NI 2024
Year: 2024

Future smart factories are expected to deploy applications over high-performance indoor wireless channels in the millimeter-wave (mmWave) bands, which on the other hand are susceptible to high path losses and Line-of Sight (LoS) blockages. Low-cost Reconfigurable Intelligent Surfaces (RISs) can provide great opportunities in such scenarios, due to its ability to alleviate LoS link blockages. In this paper, we formulate a combinatorial optimization problem, solved with Integer Linear Programming (ILP) to optimally maintain connectivity by solving the problem of allocating RIS to robots in a wireless indoor network. Our model exploits the characteristic of nulling interference from RISs by tuning RIS reflection coefficients. We further consider Quality-of-Service (QoS) at receivers in terms of Signal-to-Interference-plus-Noise Ratio (SINR) and connection outages due to insufficient transmission quality service. Numerical results for optimal solutions and heuristics show the benefits of optimally deploying RISs by providing continuous connectivity through SINR, which significantly reduces outages due to link quality.
Towards Ideal Temporal Graph Neural Networks: Evaluations and Conclusions after 10,000 GPU Hours
Venue: arXiv cs.LG 2024
Year: 2024

Temporal Graph Neural Networks (TGNNs) have emerged as powerful tools for modeling dynamic interactions across various domains. The design space of TGNNs is notably complex, given the unique challenges in runtime efficiency and scalability raised by the evolving nature of temporal graphs. We contend that many of the existing works on TGNN modeling inadequately explore the design space, leading to suboptimal designs. Viewing TGNN models through a performance-focused lens often obstructs a deeper understanding of the advantages and disadvantages of each technique. Specifically, benchmarking efforts inherently evaluate models in their original designs and implementations, resulting in unclear accuracy comparisons and misleading runtime. To address these shortcomings, we propose a practical comparative evaluation framework that performs a design space search across well-known TGNN modules based on a unified, optimized code implementation. Using our framework, we make the first efforts towards addressing three critical questions in TGNN design, spending over 10,000 GPU hours: (1) investigating the efficiency of TGNN module designs, (2) analyzing how the effectiveness of these modules correlates with dataset patterns, and (3) exploring the interplay between multiple modules. Key outcomes of this directed investigative approach include demonstrating that the most recent neighbor sampling and attention aggregator outperform uniform neighbor sampling and MLP-Mixer aggregator; Assessing static node memory as an effective node memory alternative, and showing that the choice between static or dynamic node memory should be based on the repetition patterns in the dataset. Our in-depth analysis of the interplay between TGNN modules and dataset patterns should provide a deeper insight into TGNN performance along with potential research directions for designing more general and effective TGNNs.
BikeNodePlanner: a data-driven decision support tool for bicycle node network planning
Venue: arXiv cs.CY 2024
Year: 2024

A bicycle node network is a wayfinding system targeted at recreational cyclists, consisting of numbered signposts placed alongside already existing infrastructure. Bicycle node networks are becoming increasingly popular as they encourage sustainable tourism and rural cycling, while also being flexible and cost-effective to implement. However, the lack of a formalized methodology and data-driven tools for the planning of such networks is a hindrance to their adaptation on a larger scale. To address this need, we present the BikeNodePlanner: a fully open-source decision support tool, consisting of modular Python scripts to be run in the free and open-source geographic information system QGIS. The BikeNodePlanner allows the user to evaluate and compare bicycle node network plans through a wide range of metrics, such as land use, proximity to points of interest, and elevation across the network. The BikeNodePlanner provides data-driven decision support for bicycle node network planning, and can hence be of great use for regional planning, cycling tourism, and the promotion of rural cycling.
FaGeL: Fabric LLMs Agent empowered Embodied Intelligence Evolution with Autonomous Human-Machine Collaboration
Venue: arXiv cs.HC 2024
Year: 2024

Recent advancements in Large Language Models (LLMs) have enhanced the reasoning capabilities of embodied agents, driving progress toward AGI-powered robotics. While LLMs have been applied to tasks like semantic reasoning and task generalization, their potential in open physical space exploration remains underexplored. This paper introduces FaGeL (Fabric aGent empowered by embodied intelligence with LLMs), an embodied agent integrating smart fabric technology for seamless, non-intrusive human-agent interaction. FaGeL autonomously generates tasks using multimodal data from wearable and ambient sensors, refining its behavior based on implicit human feedback in generated text, without explicit ratings or preferences. We also introduce a token-level saliency map to visualize LLM fine-tuning, enhancing the interpretability of token-level alignment. The system leverages dual feedback mechanisms to improve token-level alignment and addresses challenges in non-intrusive human-machine interaction and cognition evolution. Our contributions include FaGeL's development, the DualCUT algorithm for AI alignment, and experimental validation in cooperative tasks, demonstrating FaGeL's ability to adapt and evolve autonomously through implicit feedback. In the future, we plan to explore FaGeL's scalability in dynamic environments and its integration with other AI systems to develop AGI agents that adapt seamlessly to diverse human needs.
Protein Structure Prediction in the 3D HP Model Using Deep Reinforcement Learning
Venue: arXiv cs.LG 2024
Year: 2024

We address protein structure prediction in the 3D Hydrophobic-Polar lattice model through two novel deep learning architectures. For proteins under 36 residues, our hybrid reservoir-based model combines fixed random projections with trainable deep layers, achieving optimal conformations with 25% fewer training episodes. For longer sequences, we employ a long short-term memory network with multi-headed attention, matching best-known energy values. Both architectures leverage a stabilized Deep Q-Learning framework with experience replay and target networks, demonstrating consistent achievement of optimal conformations while significantly improving training efficiency compared to existing methods.
Exploring the Magnitude-Shape Plot Framework for Anomaly Detection in Crowded Video Scenes
Venue: arXiv cs.CV 2024
Year: 2024

Detecting anomalies in crowded video scenes is critical for public safety, enabling timely identification of potential threats. This study explores video anomaly detection within a Functional Data Analysis framework, focusing on the application of the Magnitude-Shape (MS) Plot. Autoencoders are used to learn and reconstruct normal behavioral patterns from anomaly-free training data, resulting in low reconstruction errors for normal frames and higher errors for frames with potential anomalies. The reconstruction error matrix for each frame is treated as multivariate functional data, with the MS-Plot applied to analyze both magnitude and shape deviations, enhancing the accuracy of anomaly detection. Using its capacity to evaluate the magnitude and shape of deviations, the MS-Plot offers a statistically principled and interpretable framework for anomaly detection. The proposed methodology is evaluated on two widely used benchmark datasets, UCSD Ped2 and CUHK Avenue, demonstrating promising performance. It performs better than traditional univariate functional detectors (e.g., FBPlot, TVDMSS, Extremal Depth, and Outliergram) and several state-of-the-art methods. These results highlight the potential of the MS-Plot-based framework for effective anomaly detection in crowded video scenes.
A Multidisciplinary Approach to Telegram Data Analysis
Venue: arXiv cs.CR 2024
Year: 2024

This paper presents a multidisciplinary approach to analyzing data from Telegram for early warning information regarding cyber threats. With the proliferation of hacktivist groups utilizing Telegram to disseminate information regarding future cyberattacks or to boast about successful ones, the need for effective data analysis methods is paramount. The primary challenge lies in the vast number of channels and the overwhelming volume of data, necessitating advanced techniques for discerning pertinent risks amidst the noise. To address this challenge, we employ a combination of neural network architectures and traditional machine learning algorithms. These methods are utilized to classify and identify potential cyber threats within the Telegram data. Additionally, sentiment analysis and entity recognition techniques are incorporated to provide deeper insights into the nature and context of the communicated information. The study evaluates the effectiveness of each method in detecting and categorizing cyber threats, comparing their performance and identifying areas for improvement. By leveraging these diverse analytical tools, we aim to enhance early warning systems for cyber threats, enabling more proactive responses to potential security breaches. This research contributes to the ongoing efforts to bolster cybersecurity measures in an increasingly interconnected digital landscape.
Comparative Performance of Advanced NLP Models and LLMs in Multilingual Geo-Entity Detection
Venue: arXiv cs.CL 2024
Year: 2024

The integration of advanced Natural Language Processing (NLP) methodologies and Large Language Models (LLMs) has significantly enhanced the extraction and analysis of geospatial data from multilingual texts, impacting sectors such as national and international security. This paper presents a comprehensive evaluation of leading NLP models -- SpaCy, XLM-RoBERTa, mLUKE, GeoLM -- and LLMs, specifically OpenAI's GPT 3.5 and GPT 4, within the context of multilingual geo-entity detection. Utilizing datasets from Telegram channels in English, Russian, and Arabic, we examine the performance of these models through metrics such as accuracy, precision, recall, and F1 scores, to assess their effectiveness in accurately identifying geospatial references. The analysis exposes each model's distinct advantages and challenges, underscoring the complexities involved in achieving precise geo-entity identification across varied linguistic landscapes. The conclusions drawn from this experiment aim to direct the enhancement and creation of more advanced and inclusive NLP tools, thus advancing the field of geospatial analysis and its application to global security.
Multimodal Variational Autoencoder: a Barycentric View
Venue: arXiv cs.LG 2024
Year: 2024

Multiple signal modalities, such as vision and sounds, are naturally present in real-world phenomena. Recently, there has been growing interest in learning generative models, in particular variational autoencoder (VAE), to for multimodal representation learning especially in the case of missing modalities. The primary goal of these models is to learn a modality-invariant and modality-specific representation that characterizes information across multiple modalities. Previous attempts at multimodal VAEs approach this mainly through the lens of experts, aggregating unimodal inference distributions with a product of experts (PoE), a mixture of experts (MoE), or a combination of both. In this paper, we provide an alternative generic and theoretical formulation of multimodal VAE through the lens of barycenter. We first show that PoE and MoE are specific instances of barycenters, derived by minimizing the asymmetric weighted KL divergence to unimodal inference distributions. Our novel formulation extends these two barycenters to a more flexible choice by considering different types of divergences. In particular, we explore the Wasserstein barycenter defined by the 2-Wasserstein distance, which better preserves the geometry of unimodal distributions by capturing both modality-specific and modality-invariant representations compared to KL divergence. Empirical studies on three multimodal benchmarks demonstrated the effectiveness of the proposed method.
TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication
Venue: arXiv cs.DC 2024
Year: 2024

Efficient parallelization of Large Language Models (LLMs) with long sequences is essential but challenging due to their significant computational and memory demands, particularly stemming from communication bottlenecks in attention mechanisms. While sequence parallelism (SP) has been introduced as a potential solution, existing methods often suffer from limited scalability or inefficiency, rendering their effectiveness. Ring-Attention demonstrates the potential for scaling sequence processing but faces significant limitations due to its reliance on peer-to-peer (P2P) communication and inefficient utilization of network resources. As the degree of SP increases, the quadratic decrease in computation time per step contrasts sharply with the linear reduction in communication volume, exacerbating communication bottlenecks. To address these challenges, we propose TokenRing, a fine-grained parallel framework that leverages bidirectional P2P communication to effectively overlap computation and data transmission. By partitioning the attention block and concurrently transmitting Query and block outputs (i.e., $block\_out$ and $block\_lse$) within a fully connected mesh topology, TokenRing achieves significant reductions in communication overhead and better load balancing. These innovations improve the scalability and efficiency of distributed Transformer models, particularly for long-context sequences. Experimental results demonstrate that TokenRing enhances throughput and reduces communication latency. Moreover, its design adapts seamlessly to various multi-GPU interconnect solutions, such as Huawei Ascend, ensuring broad compatibility and cost-effectiveness for distributed LLM inference and training. The code is available at: \url{https://github.com/ACA-Lab-SJTU/token-ring}.
Stratify: Unifying Multi-Step Forecasting Strategies
Venue: arXiv cs.LG 2024
Year: 2024

A key aspect of temporal domains is the ability to make predictions multiple time steps into the future, a process known as multi-step forecasting (MSF). At the core of this process is selecting a forecasting strategy, however, with no existing frameworks to map out the space of strategies, practitioners are left with ad-hoc methods for strategy selection. In this work, we propose Stratify, a parameterised framework that addresses multi-step forecasting, unifying existing strategies and introducing novel, improved strategies. We evaluate Stratify on 18 benchmark datasets, five function classes, and short to long forecast horizons (10, 20, 40, 80). In over 84% of 1080 experiments, novel strategies in Stratify improved performance compared to all existing ones. Importantly, we find that no single strategy consistently outperforms others in all task settings, highlighting the need for practitioners explore the Stratify space to carefully search and select forecasting strategies based on task-specific requirements. Our results are the most comprehensive benchmarking of known and novel forecasting strategies. We make code available to reproduce our results.
Ns3 meets Sionna: Using Realistic Channels in Network Simulation
Venue: arXiv cs.NI 2024
Year: 2024

Network simulators are indispensable tools for the advancement of wireless network technologies, offering a cost-effective and controlled environment to simulate real-world network behavior. However, traditional simulators, such as the widely used ns-3, exhibit limitations in accurately modeling indoor and outdoor scenarios due to their reliance on simplified statistical and stochastic channel propagation models, which often fail to accurately capture physical phenomena like multipath signal propagation and shadowing by obstacles in the line-of-sight path. We present Ns3Sionna, which integrates a ray tracing-based channel model, implemented using the Sionna RT framework, within the ns-3 network simulator. It allows to simulate environment-specific and physically accurate channel realizations for a given 3D scene and wireless device positions. Additionally, a mobility model based on ray tracing was developed to accurately represent device movements within the simulated 3D space. Ns3Sionna provides more realistic path and delay loss estimates for both indoor and outdoor environments than existing ns-3 propagation models, particularly in terms of spatial and temporal correlation. Moreover, fine-grained channel state information is provided, which could be used for the development of sensing applications. Due to the significant computational demands of ray tracing, Ns3Sionna takes advantage of the parallel execution capabilities of modern GPUs and multi-core CPUs by incorporating intelligent pre-caching mechanisms that leverage the channel's coherence time to optimize runtime performance. This enables the efficient simulation of scenarios with a small to medium number of mobile nodes.
Attacks on the neural network and defense methods
Venue: arXiv cs.CR 2024
Year: 2024

This article will discuss the use of attacks on a neural network trained on audio data, as well as possible methods of protection against these attacks. FGSM, PGD and CW attacks, as well as data poisoning, will be considered. Within the framework of protection, Art-IBM and advertorch libraries will be considered. The obtained accuracy metrics within the framework of attack applications are presented
The intrinsic motivation of reinforcement and imitation learning for sequential tasks
Venue: arXiv cs.AI 2024
Year: 2024

This work in the field of developmental cognitive robotics aims to devise a new domain bridging between reinforcement learning and imitation learning, with a model of the intrinsic motivation for learning agents to learn with guidance from tutors multiple tasks, including sequential tasks. The main contribution has been to propose a common formulation of intrinsic motivation based on empirical progress for a learning agent to choose automatically its learning curriculum by actively choosing its learning strategy for simple or sequential tasks: which task to learn, between autonomous exploration or imitation learning, between low-level actions or task decomposition, between several tutors. The originality is to design a learner that benefits not only passively from data provided by tutors, but to actively choose when to request tutoring and what and whom to ask. The learner is thus more robust to the quality of the tutoring and learns faster with fewer demonstrations. We developed the framework of socially guided intrinsic motivation with machine learning algorithms to learn multiple tasks by taking advantage of the generalisability properties of human demonstrations in a passive manner or in an active manner through requests of demonstrations from the best tutor for simple and composing subtasks. The latter relies on a representation of subtask composition proposed for a construction process, which should be refined by representations used for observational processes of analysing human movements and activities of daily living. With the outlook of a language-like communication with the tutor, we investigated the emergence of a symbolic representation of the continuous sensorimotor space and of tasks using intrinsic motivation. We proposed within the reinforcement learning framework, a reward function for interacting with tutors for automatic curriculum learning in multi-task learning.
"The Prophet said so!": On Exploring Hadith Presence on Arabic Social Media
Venue: arXiv cs.SI 2024
Year: 2024

Hadith, the recorded words and actions of the prophet Muhammad, is a key source of the instructions and foundations of Islam, alongside the Quran. Interpreting individual hadiths and verifying their authenticity can be difficult, even controversial, and the subject has attracted the attention of many scholars who have established an entire science of Hadith criticism. Recent quantitative studies of hadiths focus on developing systems for automatic classification, authentication, and information retrieval that operate over existing hadith compilations. Qualitative studies on the other hand try to discuss different social and political issues from the perspective of hadiths, or they inspect how hadiths are used in specific contexts in official communications and press releases for argumentation and propaganda. However, there are no studies that attempt to understand the actual presence of hadiths among Muslims in their daily lives and interactions. In this study, we try to fill this gap by exploring the presence of hadiths on Twitter from January 2019 to January 2023. We highlight the challenges that quantitative methods should consider while processing texts that include hadiths and we provide a methodology for Islamic scholars to validate their hypotheses about hadiths on big data that better represent the position of the society and Hadith influence on it.
Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection
Venue: arXiv cs.CL 2024
Year: 2024

This study demonstrates that the modern generation of Large Language Models (LLMs, such as GPT-4) suffers from the same out-of-domain (OOD) performance gap observed in prior research on pre-trained Language Models (PLMs, such as BERT). We demonstrate this across two non-topical classification tasks: 1) genre classification and 2) generated text detection. Our results show that when demonstration examples for In-Context Learning (ICL) come from one domain (e.g., travel) and the system is tested on another domain (e.g., history), classification performance declines significantly. To address this, we introduce a method that controls which predictive indicators are used and which are excluded during classification. For the two tasks studied here, this ensures that topical features are omitted, while the model is guided to focus on stylistic rather than content-based attributes. This approach reduces the OOD gap by up to 20 percentage points in a few-shot setup. Straightforward Chain-of-Thought (CoT) methods, used as the baseline, prove insufficient, while our approach consistently enhances domain transfer performance.