OpenRaiser/Intern-Atlas huggingface dataset:
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Venue: arXiv cs.CL 2024
Year: 2024

AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves reproducing the results of a study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language-only and vision-language tasks. We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to a sequential implementation. We evaluated two baseline agents: the general-purpose AutoGPT and a task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 21% on the hardest task, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is a necessary step towards building agents that can conduct novel research and could verify and improve the performance of other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.
Training Datasets Generation for Machine Learning: Application to Vision Based Navigation
Venue: arXiv cs.CV 2024
Year: 2024

Vision Based Navigation consists in utilizing cameras as precision sensors for GNC after extracting information from images. To enable the adoption of machine learning for space applications, one of obstacles is the demonstration that available training datasets are adequate to validate the algorithms. The objective of the study is to generate datasets of images and metadata suitable for training machine learning algorithms. Two use cases were selected and a robust methodology was developed to validate the datasets including the ground truth. The first use case is in-orbit rendezvous with a man-made object: a mockup of satellite ENVISAT. The second use case is a Lunar landing scenario. Datasets were produced from archival datasets (Chang'e 3), from the laboratory at DLR TRON facility and at Airbus Robotic laboratory, from SurRender software high fidelity image simulator using Model Capture and from Generative Adversarial Networks. The use case definition included the selection of algorithms as benchmark: an AI-based pose estimation algorithm and a dense optical flow algorithm were selected. Eventually it is demonstrated that datasets produced with SurRender and selected laboratory facilities are adequate to train machine learning algorithms.
UniLCD: Unified Local-Cloud Decision-Making via Reinforcement Learning
Venue: arXiv cs.RO 2024
Year: 2024

Embodied vision-based real-world systems, such as mobile robots, require a careful balance between energy consumption, compute latency, and safety constraints to optimize operation across dynamic tasks and contexts. As local computation tends to be restricted, offloading the computation, ie, to a remote server, can save local resources while providing access to high-quality predictions from powerful and large models. However, the resulting communication and latency overhead has led to limited usability of cloud models in dynamic, safety-critical, real-time settings. To effectively address this trade-off, we introduce UniLCD, a novel hybrid inference framework for enabling flexible local-cloud collaboration. By efficiently optimizing a flexible routing module via reinforcement learning and a suitable multi-task objective, UniLCD is specifically designed to support the multiple constraints of safety-critical end-to-end mobile systems. We validate the proposed approach using a challenging, crowded navigation task requiring frequent and timely switching between local and cloud operations. UniLCD demonstrates improved overall performance and efficiency, by over 35% compared to state-of-the-art baselines based on various split computing and early exit strategies.
AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs
Venue: arXiv cs.CL 2024
Year: 2024

Arabic, with its rich diversity of dialects, remains significantly underrepresented in Large Language Models, particularly in dialectal variations. We address this gap by introducing seven synthetic datasets in dialects alongside Modern Standard Arabic (MSA), created using Machine Translation (MT) combined with human post-editing. We present AraDiCE, a benchmark for Arabic Dialect and Cultural Evaluation. We evaluate LLMs on dialect comprehension and generation, focusing specifically on low-resource Arabic dialects. Additionally, we introduce the first-ever fine-grained benchmark designed to evaluate cultural awareness across the Gulf, Egypt, and Levant regions, providing a novel dimension to LLM evaluation. Our findings demonstrate that while Arabic-specific models like Jais and AceGPT outperform multilingual models on dialectal tasks, significant challenges persist in dialect identification, generation, and translation. This work contributes $\approx$45K post-edited samples, a cultural benchmark, and highlights the importance of tailored training to improve LLM performance in capturing the nuances of diverse Arabic dialects and cultural contexts. We have released the dialectal translation models and benchmarks developed in this study (https://huggingface.co/datasets/QCRI/AraDiCE).
Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion
Venue: arXiv cs.CV 2024
Year: 2024

In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.
CyberNFTs: Conceptualizing a decentralized and reward-driven intrusion detection system with ML
Venue: arXiv cs.CR 2024
Year: 2024

The rapid evolution of the Internet, particularly the emergence of Web3, has transformed the ways people interact and share data. Web3, although still not well defined, is thought to be a return to the decentralization of corporations' power over user data. Despite the obsolescence of the idea of building systems to detect and prevent cyber intrusions, this is still a topic of interest. This paper proposes a novel conceptual approach for implementing decentralized collaborative intrusion detection networks (CIDN) through a proof-of-concept. The study employs an analytical and comparative methodology, examining the synergy between cutting-edge Web3 technologies and information security. The proposed model incorporates blockchain concepts, cyber non-fungible token (cyberNFT) rewards, machine learning algorithms, and publish/subscribe architectures. Finally, the paper discusses the strengths and limitations of the proposed system, offering insights into the potential of decentralized cybersecurity models.
AIvril: AI-Driven RTL Generation With Verification In-The-Loop
Venue: arXiv cs.AI 2024
Year: 2024

Large Language Models (LLMs) are computational models capable of performing complex natural language processing tasks. Leveraging these capabilities, LLMs hold the potential to transform the entire hardware design stack, with predictions suggesting that front-end and back-end tasks could be fully automated in the near future. Currently, LLMs show great promise in streamlining Register Transfer Level (RTL) generation, enhancing efficiency, and accelerating innovation. However, their probabilistic nature makes them prone to inaccuracies - a significant drawback in RTL design, where reliability and precision are essential. To address these challenges, this paper introduces AIvril, an advanced framework designed to enhance the accuracy and reliability of RTL-aware LLMs. AIvril employs a multi-agent, LLM-agnostic system for automatic syntax correction and functional verification, significantly reducing - and in many cases, completely eliminating - instances of erroneous code generation. Experimental results conducted on the VerilogEval-Human dataset show that our framework improves code quality by nearly 2x when compared to previous works, while achieving an 88.46% success rate in meeting verification objectives. This represents a critical step toward automating and optimizing hardware design workflows, offering a more dependable methodology for AI-driven RTL design.
Three Pillars Towards Next-Generation Routing System
Venue: arXiv cs.NI 2024
Year: 2024

The routing results are playing an increasingly important role in transportation efficiency, but they could generate traffic congestion unintentionally. This is because the traffic condition and routing system are disconnected components in the current routing paradigm. In this paper, we propose a next-generation routing paradigm that could reduce traffic congestion by considering the influence of the routing results in real-time. Specifically, we regard the routing results as the root cause of the future traffic flow, which at the same time is identified as the root cause of traffic conditions. To implement such a system, we identify three essential components: 1) the traffic condition simulation that establishes the relation between traffic flow and traffic condition with guaranteed accuracy; 2) the future route management that supports efficient simulation with dynamic route update; 3) the global routing optimization that improves the overall transportation system efficiency. Preliminary design and experimental results will be presented, and the corresponding challenges and research directions will also be discussed.
Next-generation Probabilistic Computing Hardware with 3D MOSAICs, Illusion Scale-up, and Co-design
Venue: arXiv cs.DC 2024
Year: 2024

The vast majority of 21st century AI workloads are based on gradient-based deterministic algorithms such as backpropagation. One of the key reasons for the dominance of deterministic ML algorithms is the emergence of powerful hardware accelerators (GPU and TPU) that have enabled the wide-scale adoption and implementation of these algorithms. Meanwhile, discrete and probabilistic Monte Carlo algorithms have long been recognized as one of the most successful algorithms in all of computing with a wide range of applications. Specifically, Markov Chain Monte Carlo (MCMC) algorithm families have emerged as the most widely used and effective method for discrete combinatorial optimization and probabilistic sampling problems. We adopt a hardware-centric perspective on probabilistic computing, outlining the challenges and potential future directions to advance this field. We identify two critical research areas: 3D integration using MOSAICs (Monolithic/Stacked/Assembled ICs) and the concept of Illusion, a hardware-agnostic distributed computing framework designed to scale probabilistic accelerators.
Learning a Terrain- and Robot-Aware Dynamics Model for Autonomous Mobile Robot Navigation
Venue: arXiv cs.RO 2024
Year: 2024

Mobile robots should be capable of planning cost-efficient paths for autonomous navigation. Typically, the terrain and robot properties are subject to variations. For instance, properties of the terrain such as friction may vary across different locations. Also, properties of the robot may change such as payloads or wear and tear, e.g., causing changing actuator gains or joint friction. Autonomous navigation approaches should thus be able to adapt to such variations. In this article, we propose a novel approach for learning a probabilistic, terrain- and robot-aware forward dynamics model (TRADYN) which can adapt to such variations and demonstrate its use for navigation. Our learning approach extends recent advances in meta-learning forward dynamics models based on Neural Processes for mobile robot navigation. We evaluate our method in simulation for 2D navigation of a robot with uni-cycle dynamics with varying properties on terrain with spatially varying friction coefficients. In our experiments, we demonstrate that TRADYN has lower prediction error over long time horizons than model ablations which do not adapt to robot or terrain variations. We also evaluate our model for navigation planning in a model-predictive control framework and under various sources of noise. We demonstrate that our approach yields improved performance in planning control-efficient paths by taking robot and terrain properties into account.
Golden Ratio Search: A Low-Power Adversarial Attack for Deep Learning based Modulation Classification
Venue: arXiv cs.CR 2024
Year: 2024

We propose a minimal power white box adversarial attack for Deep Learning based Automatic Modulation Classification (AMC). The proposed attack uses the Golden Ratio Search (GRS) method to find powerful attacks with minimal power. We evaluate the efficacy of the proposed method by comparing it with existing adversarial attack approaches. Additionally, we test the robustness of the proposed attack against various state-of-the-art architectures, including defense mechanisms such as adversarial training, binarization, and ensemble methods. Experimental results demonstrate that the proposed attack is powerful, requires minimal power, and can be generated in less time, significantly challenging the resilience of current AMC methods.
A generalized non-hourglass updated Lagrangian formulation for SPH solid dynamics
Venue: arXiv cs.CE 2024
Year: 2024

Hourglass modes, characterized by zigzag particle and stress distributions, are a common numerical instability encountered when simulating solid materials with updated Lagrangian smoother particle hydrodynamics (ULSPH). While recent solutions have effectively addressed this issue in elastic materials using an essentially non-hourglass formulation, extending these solutions to plastic materials with more complex constitutive equations has proven challenging due to the need to express shear forces in the form of a velocity Laplacian. To address this, a generalized non-hourglass formulation is proposed within the ULSPH framework, suitable for both elastic and plastic materials. Specifically, a penalty force is introduced into the momentum equation to resolve the disparity between the linearly predicted and actual velocities of neighboring particle pairs, thereby mitigating the hourglass issue. The stability, convergence, and accuracy of the proposed method are validated through a series of classical elastic and plastic cases, with a dual-criterion time-stepping scheme to improve computational efficiency. The results show that the present method not only matches or even surpasses the performance of the recent essentially non-hourglass formulation in elastic cases but also performs well in plastic scenarios.
Good Grasps Only: A data engine for self-supervised fine-tuning of pose estimation using grasp poses for verification
Venue: arXiv cs.RO 2024
Year: 2024

In this paper, we present a novel method for self-supervised fine-tuning of pose estimation. Leveraging zero-shot pose estimation, our approach enables the robot to automatically obtain training data without manual labeling. After pose estimation the object is grasped, and in-hand pose estimation is used for data validation. Our pipeline allows the system to fine-tune while the process is running, removing the need for a learning phase. The motivation behind our work lies in the need for rapid setup of pose estimation solutions. Specifically, we address the challenging task of bin picking, which plays a pivotal role in flexible robotic setups. Our method is implemented on a robotics work-cell, and tested with four different objects. For all objects, our method increases the performance and outperforms a state-of-the-art method trained on the CAD model of the objects. Project page available at gogoengine.github.io
Learning-Augmented Frequency Estimation in Sliding Windows
Venue: arXiv cs.DS 2024
Year: 2024

We show how to utilize machine learning approaches to improve sliding window algorithms for approximate frequency estimation problems, under the ``algorithms with predictions'' framework. In this dynamic environment, previous learning-augmented algorithms are less effective, since properties in sliding window resolution can differ significantly from the properties of the entire stream. Our focus is on the benefits of predicting and filtering out items with large next arrival times -- that is, there is a large gap until their next appearance -- from the stream, which we show improves the memory-accuracy tradeoffs significantly. We provide theorems that provide insight into how and by how much our technique can improve the sliding window algorithm, as well as experimental results using real-world data sets. Our work demonstrates that predictors can be useful in the challenging sliding window setting.
Denotational semantics driven simplicial homology?
Venue: arXiv cs.LO 2024
Year: 2024

We look at the proofs of a fragment of Linear Logic as a whole: in fact, Linear Logic's coherent semantics interprets the proofs of a given formula $A$ as faces of an abstract simplicial complex, thus allowing us to see the set of the (interpretations of the) proofs of $A$ as a geometrical space, not just a set. This point of view has never been really investigated. For a ``webbed'' denotational semantics -- say the relational one --, it suffices to down-close the set of (the interpretations of the) proofs of $A$ in order to give rise to an abstract simplicial complex whose faces do correspond to proofs of $A$. Since this space comes triangulated by construction, a natural geometrical property to consider is its homology. However, we immediately stumble on a problem: if we want the homology to be invariant w.r.t. to some notion of type-isomorphism, we are naturally led to consider the homology functor acting, at the level of morphisms, on ``simplicial relations'' rather than simplicial maps as one does in topology. The task of defining the homology functor on this modified category can be achieved by considering a very simple monad, which is almost the same as the power-set monad; but, doing so, we end up considering not anymore the homology of the original space, but rather of its transformation under the action of the monad. Does this transformation keep the homology invariant ? Is this transformation meaningful from a geometrical or logical/computational point of view ?
Stability Property for the Call-by-Value $λ$-calculus through Taylor Expansion
Venue: arXiv cs.LO 2024
Year: 2024

We prove the Stability Property for the call-by-value $λ$-calculus (CbV in the following). This result states necessary conditions under which the contexts of the CbV $λ$-calculus commute with intersections of approximants. This is an important non-trivial result, which implies the sequentiality of the calculus. We prove it via the tool of Taylor-resource approximation, whose power has been shown in several recent papers. This technique is usually conceived for the ordinary $λ$-calculus, but it can be easily defined for the CbV setting. Our proof is the adaptation of the one for the ordinary calculus using the same technique, with some minimal technical modification due to the fact that in the CbV setting one linearises terms in a slightly different way than usual (cfr. $!(A\multimap B)$ vs $!A\multimap B$). The content of this article is taken from the PhD thesis of the author.
HEARTS: A Holistic Framework for Explainable, Sustainable and Robust Text Stereotype Detection
Venue: arXiv cs.CL 2024
Year: 2024

Stereotypes are generalised assumptions about societal groups, and even state-of-the-art LLMs using in-context learning struggle to identify them accurately. Due to the subjective nature of stereotypes, where what constitutes a stereotype can vary widely depending on cultural, social, and individual perspectives, robust explainability is crucial. Explainable models ensure that these nuanced judgments can be understood and validated by human users, promoting trust and accountability. We address these challenges by introducing HEARTS (Holistic Framework for Explainable, Sustainable, and Robust Text Stereotype Detection), a framework that enhances model performance, minimises carbon footprint, and provides transparent, interpretable explanations. We establish the Expanded Multi-Grain Stereotype Dataset (EMGSD), comprising 57,201 labelled texts across six groups, including under-represented demographics like LGBTQ+ and regional stereotypes. Ablation studies confirm that BERT models fine-tuned on EMGSD outperform those trained on individual components. We then analyse a fine-tuned, carbon-efficient ALBERT-V2 model using SHAP to generate token-level importance values, ensuring alignment with human understanding, and calculate explainability confidence scores by comparing SHAP and LIME outputs...
DAF-Net: A Dual-Branch Feature Decomposition Fusion Network with Domain Adaptive for Infrared and Visible Image Fusion
Venue: arXiv cs.CV 2024
Year: 2024

Infrared and visible image fusion aims to combine complementary information from both modalities to provide a more comprehensive scene understanding. However, due to the significant differences between the two modalities, preserving key features during the fusion process remains a challenge. To address this issue, we propose a dual-branch feature decomposition fusion network (DAF-Net) with domain adaptive, which introduces Multi-Kernel Maximum Mean Discrepancy (MK-MMD) into the base encoder and designs a hybrid kernel function suitable for infrared and visible image fusion. The base encoder built on the Restormer network captures global structural information while the detail encoder based on Invertible Neural Networks (INN) focuses on extracting detail texture information. By incorporating MK-MMD, the DAF-Net effectively aligns the latent feature spaces of visible and infrared images, thereby improving the quality of the fused images. Experimental results demonstrate that the proposed method outperforms existing techniques across multiple datasets, significantly enhancing both visual quality and fusion performance. The related Python code is available at https://github.com/xujian000/DAF-Net.