OpenRaiser/Intern-Atlas huggingface dataset:
Learning Self-Supervised Audio-Visual Representations for Sound Recommendations
Venue: arXiv cs.CV 2024
Year: 2024

We propose a novel self-supervised approach for learning audio and visual representations from unlabeled videos, based on their correspondence. The approach uses an attention mechanism to learn the relative importance of convolutional features extracted at different resolutions from the audio and visual streams and uses the attention features to encode the audio and visual input based on their correspondence. We evaluated the representations learned by the model to classify audio-visual correlation as well as to recommend sound effects for visual scenes. Our results show that the representations generated by the attention model improves the correlation accuracy compared to the baseline, by 18% and the recommendation accuracy by 10% for VGG-Sound, which is a public video dataset. Additionally, audio-visual representations learned by training the attention model with cross-modal contrastive learning further improves the recommendation performance, based on our evaluation using VGG-Sound and a more challenging dataset consisting of gameplay video recordings.
DSFEC: Efficient and Deployable Deep Radar Object Detection
Venue: arXiv cs.CV 2024
Year: 2024

Deploying radar object detection models on resource-constrained edge devices like the Raspberry Pi poses significant challenges due to the large size of the model and the limited computational power and the memory of the Pi. In this work, we explore the efficiency of Depthwise Separable Convolutions in radar object detection networks and integrate them into our model. Additionally, we introduce a novel Feature Enhancement and Compression (FEC) module to the PointPillars feature encoder to further improve the model performance. With these innovations, we propose the DSFEC-L model and its two versions, which outperform the baseline (23.9 mAP of Car class, 20.72 GFLOPs) on nuScenes dataset: 1). An efficient DSFEC-M model with a 14.6% performance improvement and a 60% reduction in GFLOPs. 2). A deployable DSFEC-S model with a 3.76% performance improvement and a remarkable 78.5% reduction in GFLOPs. Despite marginal performance gains, our deployable model achieves an impressive 74.5% reduction in runtime on the Raspberry Pi compared to the baseline.
RAG-based Question Answering over Heterogeneous Data and Text
Venue: arXiv cs.CL 2024
Year: 2024

This article presents the QUASAR system for question answering over unstructured text, structured tables, and knowledge graphs, with unified treatment of all sources. The system adopts a RAG-based architecture, with a pipeline of evidence retrieval followed by answer generation, with the latter powered by a moderate-sized language model. Additionally and uniquely, QUASAR has components for question understanding, to derive crisper input for evidence retrieval, and for re-ranking and filtering the retrieved evidence before feeding the most informative pieces into the answer generation. Experiments with three different benchmarks demonstrate the high answering quality of our approach, being on par with or better than large GPT models, while keeping the computational cost and energy consumption orders of magnitude lower.
A Robust Sustainability Assessment Methodology for Aircraft Parts: Application to a Fuselage Panel
Venue: arXiv cs.CE 2024
Year: 2024

The paper presents a cradle-to-gate sustainability assessment methodology specifically designed to evaluate aircraft components in a robust and systematic manner. This methodology integrates multi-criteria decision-making (MCDM) analysis across ten criteria, categorized under environmental impact, cost, and performance. Environmental impact is analyzed through life cycle assessment and cost through life cycle costing, with both analyses facilitated by SimaPro software. Performance is measured in terms of component mass and specific stiffness. The robustness of this methodology is tested through various MCDM techniques, normalization approaches, and objective weighting methods. To demonstrate the methodology, the paper assesses the sustainability of a fuselage panel, comparing nine variants that differ in materials, joining techniques, and part thicknesses. All approaches consistently identify thermoplastic CFRP panels as the most sustainable option, with the geometric mean aggregation of weights providing balanced criteria consideration across environmental, cost, and performance aspects. The adaptability of this proposed methodology is illustrated, showing its applicability to any aircraft component with the requisite data. This structured approach offers critical insights to support sustainable decision-making in aircraft component design and procurement.
Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent
Venue: arXiv cs.LG 2024
Year: 2024

Natural gradient descent (NGD) is a powerful optimization technique for machine learning, but the computational complexity of the inverse Fisher information matrix limits its application in training deep neural networks. To overcome this challenge, we propose a novel optimization method for training deep neural networks called structured natural gradient descent (SNGD). Theoretically, we demonstrate that optimizing the original network using NGD is equivalent to using fast gradient descent (GD) to optimize the reconstructed network with a structural transformation of the parameter matrix. Thereby, we decompose the calculation of the global Fisher information matrix into the efficient computation of local Fisher matrices via constructing local Fisher layers in the reconstructed network to speed up the training. Experimental results on various deep networks and datasets demonstrate that SNGD achieves faster convergence speed than NGD while retaining comparable solutions. Furthermore, our method outperforms traditional GDs in terms of efficiency and effectiveness. Thus, our proposed method has the potential to significantly improve the scalability and efficiency of NGD in deep learning applications. Our source code is available at https://github.com/Chaochao-Lin/SNGD.
Stealthy and Robust Backdoor Attack against 3D Point Clouds through Additional Point Features
Venue: arXiv cs.CV 2024
Year: 2024

Recently, 3D backdoor attacks have posed a substantial threat to 3D Deep Neural Networks (3D DNNs) designed for 3D point clouds, which are extensively deployed in various security-critical applications. Although the existing 3D backdoor attacks achieved high attack performance, they remain vulnerable to preprocessing-based defenses (e.g., outlier removal and rotation augmentation) and are prone to detection by human inspection. In pursuit of a more challenging-to-defend and stealthy 3D backdoor attack, this paper introduces the Stealthy and Robust Backdoor Attack (SRBA), which ensures robustness and stealthiness through intentional design considerations. The key insight of our attack involves applying a uniform shift to the additional point features of point clouds (e.g., reflection intensity) widely utilized as part of inputs for 3D DNNs as the trigger. Without altering the geometric information of the point clouds, our attack ensures visual consistency between poisoned and benign samples, and demonstrate robustness against preprocessing-based defenses. In addition, to automate our attack, we employ Bayesian Optimization (BO) to identify the suitable trigger. Extensive experiments suggest that SRBA achieves an attack success rate (ASR) exceeding 94% in all cases, and significantly outperforms previous SOTA methods when multiple preprocessing operations are applied during training.
Modeling Speculative Trading Patterns in Token Markets: An Agent-Based Analysis with TokenLab
Venue: arXiv cs.MA 2024
Year: 2024

This paper presents the application of Tokenlab, an agent-based modeling framework designed to analyze price dynamics and speculative behavior within token-based economies. By decomposing complex token systems into discrete agent interactions governed by fundamental behavioral rules, Tokenlab simplifies the simulation of otherwise intricate market scenarios. Its core innovation lies in its ability to model a range of speculative strategies and assess their collective influence on token price evolution. Through a novel controller mechanism, Tokenlab facilitates the simulation of multiple speculator archetypes and their interactions, thereby providing valuable insights into market sentiment and price formation. This method enables a systematic exploration of how varying degrees of speculative activity and evolving strategies across different market stages shape token price trajectories. Our findings enhance the understanding of speculation in token markets and present a quantitative framework for measuring and interpreting market heat indicators.
Statistical Precoder Design in Multi-User Systems via Graph Neural Networks and Generative Modeling
Venue: arXiv cs.IT 2024
Year: 2024

This letter proposes a graph neural network (GNN)-based framework for statistical precoder design that leverages model-based insights to compactly represent statistical knowledge, resulting in efficient, lightweight architectures. The framework also supports approximate statistical information in frequency division duplex (FDD) systems obtained through a Gaussian mixture model (GMM)-based limited feedback scheme in massive multiple-input multiple-output (MIMO) systems with low pilot overhead. Simulations using a spatial channel model and measurement data demonstrate the effectiveness of the proposed framework. It outperforms baseline methods, including stochastic iterative algorithms and Discrete Fourier transform (DFT) codebook-based approaches, particularly in low pilot overhead systems.
Optimization-Driven Design of Monolithic Soft-Rigid Grippers
Venue: arXiv cs.RO 2024
Year: 2024

Sim-to-real transfer remains a significant challenge in soft robotics due to the unpredictability introduced by common manufacturing processes such as 3D printing and molding. These processes often result in deviations from simulated designs, requiring multiple prototypes before achieving a functional system. In this study, we propose a novel methodology to address these limitations by combining advanced rapid prototyping techniques and an efficient optimization strategy. Firstly, we employ rapid prototyping methods typically used for rigid structures, leveraging their precision to fabricate compliant components with reduced manufacturing errors. Secondly, our optimization framework minimizes the need for extensive prototyping, significantly reducing the iterative design process. The methodology enables the identification of stiffness parameters that are more practical and achievable within current manufacturing capabilities. The proposed approach demonstrates a substantial improvement in the efficiency of prototype development while maintaining the desired performance characteristics. This work represents a step forward in bridging the sim-to-real gap in soft robotics, paving the way towards a faster and more reliable deployment of soft robotic systems.
PVP: Polar Representation Boost for 3D Semantic Occupancy Prediction
Venue: arXiv cs.CV 2024
Year: 2024

Recently, polar coordinate-based representations have shown promise for 3D perceptual tasks. Compared to Cartesian methods, polar grids provide a viable alternative, offering better detail preservation in nearby spaces while covering larger areas. However, they face feature distortion due to non-uniform division. To address these issues, we introduce the Polar Voxel Occupancy Predictor (PVP), a novel 3D multi-modal predictor that operates in polar coordinates. PVP features two key design elements to overcome distortion: a Global Represent Propagation (GRP) module that integrates global spatial data into 3D volumes, and a Plane Decomposed Convolution (PD-Conv) that simplifies 3D distortions into 2D convolutions. These innovations enable PVP to outperform existing methods, achieving significant improvements in mIoU and IoU metrics on the OpenOccupancy dataset.
Adapting to Non-Stationary Environments: Multi-Armed Bandit Enhanced Retrieval-Augmented Generation on Knowledge Graphs
Venue: arXiv cs.AI 2024
Year: 2024

Despite the superior performance of Large language models on many NLP tasks, they still face significant limitations in memorizing extensive world knowledge. Recent studies have demonstrated that leveraging the Retrieval-Augmented Generation (RAG) framework, combined with Knowledge Graphs that encapsulate extensive factual data in a structured format, robustly enhances the reasoning capabilities of LLMs. However, deploying such systems in real-world scenarios presents challenges: the continuous evolution of non-stationary environments may lead to performance degradation and user satisfaction requires a careful balance of performance and responsiveness. To address these challenges, we introduce a Multi-objective Multi-Armed Bandit enhanced RAG framework, supported by multiple retrieval methods with diverse capabilities under rich and evolving retrieval contexts in practice. Within this framework, each retrieval method is treated as a distinct ``arm''. The system utilizes real-time user feedback to adapt to dynamic environments, by selecting the appropriate retrieval method based on input queries and the historical multi-objective performance of each arm. Extensive experiments conducted on two benchmark KGQA datasets demonstrate that our method significantly outperforms baseline methods in non-stationary settings while achieving state-of-the-art performance in stationary environments. Code and data are available at https://github.com/FUTUREEEEEE/Dynamic-RAG.git
Searching for Structure: Investigating Emergent Communication with Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Human languages have evolved to be structured through repeated language learning and use. These processes introduce biases that operate during language acquisition and shape linguistic systems toward communicative efficiency. In this paper, we investigate whether the same happens if artificial languages are optimised for implicit biases of Large Language Models (LLMs). To this end, we simulate a classical referential game in which LLMs learn and use artificial languages. Our results show that initially unstructured holistic languages are indeed shaped to have some structural properties that allow two LLM agents to communicate successfully. Similar to observations in human experiments, generational transmission increases the learnability of languages, but can at the same time result in non-humanlike degenerate vocabularies. Taken together, this work extends experimental findings, shows that LLMs can be used as tools in simulations of language evolution, and opens possibilities for future human-machine experiments in this field.
RRT-GPMP2: A Motion Planner for Mobile Robots in Complex Maze Environments
Venue: arXiv cs.RO 2024
Year: 2024

With the development of science and technology, mobile robots are playing a significant important role in the new round of world revolution. Further, mobile robots might assist or replace human beings in a great number of areas. To increase the degree of automation for mobile robots, advanced motion planners need to be integrated into them to cope with various environments. Complex maze environments are common in the potential application scenarios of different mobile robots. This article proposes a novel motion planner named the rapidly exploring random tree based Gaussian process motion planner 2, which aims to tackle the motion planning problem for mobile robots in complex maze environments. To be more specific, the proposed motion planner successfully combines the advantages of a trajectory optimisation motion planning algorithm named the Gaussian process motion planner 2 and a sampling-based motion planning algorithm named the rapidly exploring random tree. To validate the performance and practicability of the proposed motion planner, we have tested it in several simulations in the Matrix laboratory and applied it on a marine mobile robot in a virtual scenario in the Robotic operating system.
SimVS: Simulating World Inconsistencies for Robust View Synthesis
Venue: arXiv cs.CV 2024
Year: 2024

Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: https://alextrevithick.github.io/simvs
SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
Venue: arXiv cs.CV 2024
Year: 2024

Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: https://jianhongbai.github.io/SynCamMaster/.
Towards Agentic Schema Refinement
Venue: arXiv cs.DB 2024
Year: 2024

Large enterprise databases can be complex and messy, obscuring the data semantics needed for analytical tasks. We propose a semantic layer in-between the database and the user as a set of small and easy-to-interpret database views, effectively acting as a refined version of the schema. To discover these views, we introduce a multi-agent Large Language Model (LLM) simulation where LLM agents collaborate to iteratively define and refine views with minimal input. Our approach paves the way for LLM-powered exploration of unwieldy databases.
Time, bits, and nickel: Managing digital and analog continuity
Venue: arXiv cs.DL 2024
Year: 2024

In 1998, the Getty Center hosted the ''Time and Bits: Managing Digital Continuity'' conference, gathering the founders and thinkers of two San Francisco non-profit organizations interested in long-term thinking and archiving: the Internet Archive and the Long Now Foundation. This chapter proposes to discuss two different ways of archiving through time, in digital and analog formats, for virtual web contents and physical paper-based ones. It explores various types of archiving methods and tools and the management challenges they raise, in terms of time and space, but also innovation, maintenance, and ''continuity''. It depicts two distinct visions of the future of archiving which nonetheless converge in their mission of safeguarding, sharing, and giving access to information and knowledge for the decades and centuries to come.
Multi-Response Preference Optimization with Augmented Ranking Dataset
Venue: arXiv cs.CL 2024
Year: 2024

Recent advancements in Large Language Models (LLMs) have been remarkable, with new models consistently surpassing their predecessors. These advancements are underpinned by extensive research on various training mechanisms. Among these, Preference Optimization has played a significant role in improving the performance of LLMs by incorporating human preferences into the training process. However, constructing preference optimization datasets is challenging and the optimization process is highly sensitive to the dataset quality. In this study, we propose a novel approach to augment Preference Optimization datasets. Additionally, we introduce a Multi-response-based Preference Optimization training method that enables the simultaneous learning of multiple responses.
Evaluating the Potential of Federated Learning for Maize Leaf Disease Prediction
Venue: arXiv cs.LG 2024
Year: 2024

The diagnosis of diseases in food crops based on machine learning seemed satisfactory and suitable for use on a large scale. The Convolutional Neural Networks (CNNs) perform accurately in the disease prediction considering the image capture of the crop leaf, being extensively enhanced in the literature. These machine learning techniques fall short in data privacy, as they require sharing the data in the training process with a central server, disregarding competitive or regulatory concerns. Thus, Federated Learning (FL) aims to support distributed training to address recognized gaps in centralized training. As far as we know, this paper inaugurates the use and evaluation of FL applied in maize leaf diseases. We evaluated the performance of five CNNs trained under the distributed paradigm and measured their training time compared to the classification performance. In addition, we consider the suitability of distributed training considering the volume of network traffic and the number of parameters of each CNN. Our results indicate that FL potentially enhances data privacy in heterogeneous domains.
How Should We Represent History in Interpretable Models of Clinical Policies?
Venue: arXiv cs.LG 2024
Year: 2024

Modeling policies for sequential clinical decision-making based on observational data is useful for describing treatment practices, standardizing frequent patterns in treatment, and evaluating alternative policies. For each task, it is essential that the policy model is interpretable. Learning accurate models requires effectively capturing the state of a patient, either through sequence representation learning or carefully crafted summaries of their medical history. While recent work has favored the former, it remains a question as to how histories should best be represented for interpretable policy modeling. Focused on model fit, we systematically compare diverse approaches to summarizing patient history for interpretable modeling of clinical policies across four sequential decision-making tasks. We illustrate differences in the policies learned using various representations by breaking down evaluations by patient subgroups, critical states, and stages of treatment, highlighting challenges specific to common use cases. We find that interpretable sequence models using learned representations perform on par with black-box models across all tasks. Interpretable models using hand-crafted representations perform substantially worse when ignoring history entirely, but are made competitive by incorporating only a few aggregated and recent elements of patient history. The added benefits of using a richer representation are pronounced for subgroups and in specific use cases. This underscores the importance of evaluating policy models in the context of their intended use.