OpenRaiser/Intern-Atlas huggingface dataset:
Adaptive Decoding via Latent Preference Optimization
Venue: arXiv cs.CL 2024
Year: 2024

During language model decoding, it is known that using higher temperature sampling gives more creative responses, while lower temperatures are more factually accurate. However, such models are commonly applied to general instruction following, which involves both creative and fact seeking tasks, using a single fixed temperature across all examples and tokens. In this work, we introduce Adaptive Decoding, a layer added to the model to select the sampling temperature dynamically at inference time, at either the token or example level, in order to optimize performance. To learn its parameters we introduce Latent Preference Optimization (LPO) a general approach to train discrete latent variables such as choices of temperature. Our method outperforms all fixed decoding temperatures across a range of tasks that require different temperatures, including UltraFeedback, Creative Story Writing, and GSM8K.
Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
Venue: arXiv cs.CV 2024
Year: 2024

Multi-modal large language models (MLLMs) have achieved remarkable success in fine-grained visual understanding across a range of tasks. However, they often encounter significant challenges due to inadequate alignment for fine-grained knowledge, which restricts their ability to accurately capture local details and attain a comprehensive global perception. While recent advancements have focused on aligning object expressions with grounding information, they typically lack explicit integration of object images, which contain affluent information beyond mere texts or coordinates. To bridge this gap, we introduce a novel fine-grained visual knowledge alignment method that effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images. This innovative method is underpinned by our multi-scale fine-grained enhancement data synthesis pipeline, which provides over 300K essential training data to enhance alignment and improve overall performance. Furthermore, we present TinyGroundingGPT, a series of compact models optimized for high-level alignments. With a scale of approximately 3B parameters, TinyGroundingGPT achieves outstanding results in grounding tasks while delivering performance comparable to larger MLLMs in complex visual scenarios.
The role of broadband connectivity in achieving Sustainable Development Goals (SDGs)
Venue: arXiv cs.CY 2024
Year: 2024

Broadband connectivity is a tool for catalyzing socio-economic development and reducing the societal inequalities. Recent studies have investigated the supporting role of broadband in addressing Sustainable Development Goals (SDGs). Relationally, emerging ultra-dense broadband networks such as 5/6G have been linked to increased power consumption and more carbon footprint. With SDGs recognized as interdependent and addressing one should not jeopardize the achievement of the other, there is need for sustainability research. Despite the need to narrow the digital divide and address the SDGs by 2030, it is surprising that limited comprehensive studies exist on broadband sustainability. To this end, we review 113 peer reviewed journals focusing on six key areas (SDGs addressed, application areas, country income, technology, methodology and spatial focus). We further discuss our findings before making four key recommendations on broadband sustainability research to fast-track SDG achievement by 2030 especially for developing economies.
An IoT Based Smart Waste Management System for the Municipality or City Corporations
Venue: arXiv cs.CY 2024
Year: 2024

The population of the urban areas is increasing daily, and this migration is causing serious environmental pollution. A larger population is creating pressure on the municipality's waste management and the city corporations of developing countries such as Bangladesh, further threatening human health. New generation technologies, such as the Internet of Things (IoT)-based waste management systems, can help improve this serious issue. IoT-enabled smart dustbins and mobile applications-based interactive management can effectively solve this problem. In this article, we combine these two technologies to offer an acceptable solution to this problem. The proposed waste management model enables smart dustbins to communicate with waste collectors or waste control centers whenever it is necessary. Additionally, city dwellers can use mobile applications to report their observations in their neighborhoods. As a result, both sensors and humans are involved directly in the development loop. We have conducted a detailed survey to study the acceptance of such a system in the community and received some encouraging results.
Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search
Venue: arXiv cs.LG 2024
Year: 2024

Batch reinforcement learning enables policy learning without direct interaction with the environment during training, relying exclusively on previously collected sets of interactions. This approach is, therefore, well-suited for high-risk and cost-intensive applications, such as industrial control. Learned policies are commonly restricted to act in a similar fashion as observed in the batch. In a real-world scenario, learned policies are deployed in the industrial system, inevitably leading to the collection of new data that can subsequently be added to the existing recording. The process of learning and deployment can thus take place multiple times throughout the lifespan of a system. In this work, we propose to exploit this iterative nature of applying offline reinforcement learning to guide learned policies towards efficient and informative data collection during deployment, leading to continuous improvement of learned policies while remaining within the support of collected data. We present an algorithmic methodology for iterative batch reinforcement learning based on ensemble-based model-based policy search, augmented with safety and, importantly, a diversity criterion.
Analyzing the AI Nudification Application Ecosystem
Venue: arXiv cs.HC 2024
Year: 2024

Given a source image of a clothed person (an image subject), AI-based nudification applications can produce nude (undressed) images of that person. Moreover, not only do such applications exist, but there is ample evidence of the use of such applications in the real world and without the consent of an image subject. Still, despite the growing awareness of the existence of such applications and their potential to violate the rights of image subjects and cause downstream harms, there has been no systematic study of the nudification application ecosystem across multiple applications. We conduct such a study here, focusing on 20 popular and easy-to-find nudification websites. We study the positioning of these web applications (e.g., finding that most sites explicitly target the nudification of women, not all people), the features that they advertise (e.g., ranging from undressing-in-place to the rendering of image subjects in sexual positions, as well as differing user-privacy options), and their underlying monetization infrastructure (e.g., credit cards and cryptocurrencies). We believe this work will empower future, data-informed conversations -- within the scientific, technical, and policy communities -- on how to better protect individuals' rights and minimize harm in the face of modern (and future) AI-based nudification applications. Content warning: This paper includes descriptions of web applications that can be used to create synthetic non-consensual explicit AI-created imagery (SNEACI). This paper also includes an artistic rendering of a user interface for such an application.
SpecPCM: A Low-power PCM-based In-Memory Computing Accelerator for Full-stack Mass Spectrometry Analysis
Venue: arXiv cs.AR 2024
Year: 2024

Mass spectrometry (MS) is essential for proteomics and metabolomics but faces impending challenges in efficiently processing the vast volumes of data. This paper introduces SpecPCM, an in-memory computing (IMC) accelerator designed to achieve substantial improvements in energy and delay efficiency for both MS spectral clustering and database (DB) search. SpecPCM employs analog processing with low-voltage swing and utilizes recently introduced phase change memory (PCM) devices based on superlattice materials, optimized for low-voltage and low-power programming. Our approach integrates contributions across multiple levels: application, algorithm, circuit, device, and instruction sets. We leverage a robust hyperdimensional computing (HD) algorithm with a novel dimension-packing method and develop specialized hardware for the end-to-end MS pipeline to overcome the non-ideal behavior of PCM devices. We further optimize multi-level PCM devices for different tasks by using different materials. We also perform a comprehensive design exploration to improve energy and delay efficiency while maintaining accuracy, exploring various combinations of hardware and software parameters controlled by the instruction set architecture (ISA). SpecPCM, with up to three bits per cell, achieves speedups of up to 82x and 143x for MS clustering and DB search tasks, respectively, along with a four-orders-of-magnitude improvement in energy efficiency compared with state-of-the-art CPU/GPU tools.
Scalable Readability Evaluation for Graph Layouts: 2D Geometric Distributed Algorithms
Venue: arXiv cs.DC 2024
Year: 2024

Graphs, consisting of vertices and edges, are vital for representing complex relationships in fields like social networks, finance, and blockchain. Visualizing these graphs helps analysts identify structural patterns, with readability metrics-such as node occlusion and edge crossing-assessing layout clarity. However, calculating these metrics is computationally intensive, making scalability a challenge for large graphs. Without efficient readability metrics, layout generation processes-despite numerous studies focused on accelerating them-face bottleneck, making it challenging to select or produce optimized layouts swiftly. Previous approaches attempted to accelerate this process through machine learning models. Machine learning approaches aimed to predict readability scores from rendered images of graphs. While these models offered some improvement, they struggled with scalability and accuracy, especially for graphs with thousands of nodes. For instance, this approach requires substantial memory to process large images, as it relies on rendered images of the graph; graphs with more than 600 nodes cannot be inputted into the model, and errors can exceed 55% in some readability metrics due to difficulties in generalizing across diverse graph layouts. This study addresses these limitations by introducing scalable algorithms for readability evaluation in distributed environments, utilizing Spark's DataFrame and GraphFrame frameworks to efficiently manage large data volumes across multiple machines. Experimental results show that these distributed algorithms significantly reduce computation time, achieving up to a 17x speedup for node occlusion and a 146x improvement for edge crossing on large datasets. These enhancements make scalable graph readability evaluation practical and efficient, overcoming the limitations of previous machine-learning approaches.
The Good, The Efficient and the Inductive Biases: Exploring Efficiency in Deep Learning Through the Use of Inductive Biases
Venue: arXiv cs.LG 2024
Year: 2024

The emergence of Deep Learning has marked a profound shift in machine learning, driven by numerous breakthroughs achieved in recent years. However, as Deep Learning becomes increasingly present in everyday tools and applications, there is a growing need to address unresolved challenges related to its efficiency and sustainability. This dissertation delves into the role of inductive biases -- particularly, continuous modeling and symmetry preservation -- as strategies to enhance the efficiency of Deep Learning. It is structured in two main parts. The first part investigates continuous modeling as a tool to improve the efficiency of Deep Learning algorithms. Continuous modeling involves the idea of parameterizing neural operations in a continuous space. The research presented here demonstrates substantial benefits for the (i) computational efficiency -- in time and memory, (ii) the parameter efficiency, and (iii) design efficiency -- the complexity of designing neural architectures for new datasets and tasks. The second focuses on the role of symmetry preservation on Deep Learning efficiency. Symmetry preservation involves designing neural operations that align with the inherent symmetries of data. The research presented in this part highlights significant gains both in data and parameter efficiency through the use of symmetry preservation. However, it also acknowledges a resulting trade-off of increased computational costs. The dissertation concludes with a critical evaluation of these findings, openly discussing their limitations and proposing strategies to address them, informed by literature and the author insights. It ends by identifying promising future research avenues in the exploration of inductive biases for efficiency, and their wider implications for Deep Learning.
Deep Autoencoders for Unsupervised Anomaly Detection in Wildfire Prediction
Venue: arXiv cs.LG 2024
Year: 2024

Wildfires pose a significantly increasing hazard to global ecosystems due to the climate crisis. Due to its complex nature, there is an urgent need for innovative approaches to wildfire prediction, such as machine learning. This research took a unique approach, differentiating from classical supervised learning, and addressed the gap in unsupervised wildfire prediction using autoencoders and clustering techniques for anomaly detection. Historical weather and normalised difference vegetation index datasets of Australia for 2005 - 2021 were utilised. Two main unsupervised approaches were analysed. The first used a deep autoencoder to obtain latent features, which were then fed into clustering models, isolation forest, local outlier factor and one-class SVM for anomaly detection. The second approach used a deep autoencoder to reconstruct the input data and use reconstruction errors to identify anomalies. Long Short-Term Memory (LSTM) autoencoders and fully connected (FC) autoencoders were employed in this part, both in an unsupervised way learning only from nominal data. The FC autoencoder outperformed its counterparts, achieving an accuracy of 0.71, an F1-score of 0.74, and an MCC of 0.42. These findings highlight the practicality of this method, as it effectively predicts wildfires in the absence of ground truth, utilising an unsupervised learning technique.
Power Allocation for Compute-and-Forward over Fading Channels
Venue: arXiv cs.IT 2024
Year: 2024

Compute-and-forward (CF) is a relaying strategy which allows the relay to decode a linear combination of the transmitted messages. This work studies the optimal power allocation problem for the CF scheme in fast fading channels for maximizing the symmetric computation rate, which is a non-convex optimization problem with no simple analytical or numerical solutions. In the first part of the paper, we investigate the problem when there are finitely many channel states (discrete case). We establish several important properties of the optimal solutions and show that if all users share the same power allocation policy (symmetric policy), the optimal solution takes the form of a water-filling type when the power constraint exceeds a certain threshold. However, if asymmetric policies are allowed, the optimal solution does not take this form for any power constraint. We propose a low-complexity order-based algorithm for both scenarios and compare its performance with baseline algorithms. In the second part of the paper, we state relevant results when the channel coefficients are modelled as continuous random variables (continuous case) and propose a similar low-complexity iterative algorithm for the symmetric policy scenario. Numerical results are provided for both discrete and continuous cases. It is shown that in general our proposed algorithm finds good suboptimal solutions with low complexity, and for some examples considered, finds an exact optimal solution.
Impact-Aware Control using Time-Invariant Reference Spreading
Venue: arXiv cs.RO 2024
Year: 2024

With the goal of increasing the speed and efficiency in robotic manipulation, a control approach is presented that aims to utilize intentional simultaneous impacts to its advantage. This approach exploits the concept of the time-invariant reference spreading framework, in which partly-overlapping ante- and post-impact reference vector fields are used. These vector fields are coupled via an impact model in proximity of the expected impact area, minimizing the otherwise large impact-induced velocity errors and control efforts. We show how a nonsmooth physics engine can be used to construct this impact model for complex scenarios, which warrants applicability to a large range of possible impact states without requiring contact stiffness and damping parameters. In addition, a novel interim-impact control mode provides robustness in the execution against the inevitable lack of exact impact simultaneity and the corresponding unreliable velocity error during the time when contact is only partially established. This interim mode uses a position feedback signal that is derived from the ante-impact velocity reference to promote contact completion, and smoothly transitions into the post-impact mode. An experimental validation of time-invariant reference spreading control is presented for the first time through a set of 600 robotic hit-and-push and dual-arm grabbing experiments.
Memory Proxy Maps for Visual Navigation
Venue: arXiv cs.CV 2024
Year: 2024

Visual navigation takes inspiration from humans, who navigate in previously unseen environments using vision without detailed environment maps. Inspired by this, we introduce a novel no-RL, no-graph, no-odometry approach to visual navigation using feudal learning to build a three tiered agent. Key to our approach is a memory proxy map (MPM), an intermediate representation of the environment learned in a self-supervised manner by the high-level manager agent that serves as a simplified memory, approximating what the agent has seen. We demonstrate that recording observations in this learned latent space is an effective and efficient memory proxy that can remove the need for graphs and odometry in visual navigation tasks. For the mid-level manager agent, we develop a waypoint network (WayNet) that outputs intermediate subgoals, or waypoints, imitating human waypoint selection during local navigation. For the low-level worker agent, we learn a classifier over a discrete action space that avoids local obstacles and moves the agent towards the WayNet waypoint. The resulting feudal navigation network offers a novel approach with no RL, no graph, no odometry, and no metric map; all while achieving SOTA results on the image goal navigation task.
Exploiting Cross-Layer Vulnerabilities: Off-Path Attacks on the TCP/IP Protocol Suite
Venue: arXiv cs.CR 2024
Year: 2024

After more than 40 years of development, the fundamental TCP/IP protocol suite, serving as the backbone of the Internet, is widely recognized for having achieved an elevated level of robustness and security. Distinctively, we take a new perspective to investigate the security implications of cross-layer interactions within the TCP/IP protocol suite caused by ICMP error messages. Through a comprehensive analysis of interactions among Wi-Fi, IP, ICMP, UDP, and TCP due to ICMP errors, we uncover several significant vulnerabilities, including information leakage, desynchronization, semantic gaps, and identity spoofing. These vulnerabilities can be exploited by off-path attackers to manipulate network traffic stealthily, affecting over 20% of popular websites and more than 89% of public Wi-Fi networks, thus posing risks to the Internet. By responsibly disclosing these vulnerabilities to affected vendors and proposing effective countermeasures, we enhance the robustness of the TCP/IP protocol suite, receiving acknowledgments from well-known organizations such as the Linux community, the OpenWrt community, the FreeBSD community, Wi-Fi Alliance, Qualcomm, HUAWEI, China Telecom, Alibaba, and H3C.
mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual Reality
Venue: arXiv cs.CR 2024
Year: 2024

Virtual reality (VR), while enhancing user experiences, introduces significant privacy risks. This paper reveals a novel vulnerability in VR systems that allows attackers to capture VR privacy through obstacles utilizing millimeter-wave (mmWave) signals without physical intrusion and virtual connection with the VR devices. We propose mmSpyVR, a novel attack on VR user's privacy via mmWave radar. The mmSpyVR framework encompasses two main parts: (i) A transfer learning-based feature extraction model to achieve VR feature extraction from mmWave signal. (ii) An attention-based VR privacy spying module to spy VR privacy information from the extracted feature. The mmSpyVR demonstrates the capability to extract critical VR privacy from the mmWave signals that have penetrated through obstacles. We evaluate mmSpyVR through IRB-approved user studies. Across 22 participants engaged in four experimental scenes utilizing VR devices from three different manufacturers, our system achieves an application recognition accuracy of 98.5\% and keystroke recognition accuracy of 92.6\%. This newly discovered vulnerability has implications across various domains, such as cybersecurity, privacy protection, and VR technology development. We also engage with VR manufacturer Meta to discuss and explore potential mitigation strategies. Data and code are publicly available for scrutiny and research at https://github.com/luoyumei1-a/mmSpyVR/
Zero-shot Voice Conversion with Diffusion Transformers
Venue: arXiv cs.SD 2024
Year: 2024

Zero-shot voice conversion aims to transform a source speech utterance to match the timbre of a reference speech from an unseen speaker. Traditional approaches struggle with timbre leakage, insufficient timbre representation, and mismatches between training and inference tasks. We propose Seed-VC, a novel framework that addresses these issues by introducing an external timbre shifter during training to perturb the source speech timbre, mitigating leakage and aligning training with inference. Additionally, we employ a diffusion transformer that leverages the entire reference speech context, capturing fine-grained timbre features through in-context learning. Experiments demonstrate that Seed-VC outperforms strong baselines like OpenVoice and CosyVoice, achieving higher speaker similarity and lower word error rates in zero-shot voice conversion tasks. We further extend our approach to zero-shot singing voice conversion by incorporating fundamental frequency (F0) conditioning, resulting in comparative performance to current state-of-the-art methods. Our findings highlight the effectiveness of Seed-VC in overcoming core challenges, paving the way for more accurate and versatile voice conversion systems.
Explanation for Trajectory Planning using Multi-modal Large Language Model for Autonomous Driving
Venue: arXiv cs.CV 2024
Year: 2024

End-to-end style autonomous driving models have been developed recently. These models lack interpretability of decision-making process from perception to control of the ego vehicle, resulting in anxiety for passengers. To alleviate it, it is effective to build a model which outputs captions describing future behaviors of the ego vehicle and their reason. However, the existing approaches generate reasoning text that inadequately reflects the future plans of the ego vehicle, because they train models to output captions using momentary control signals as inputs. In this study, we propose a reasoning model that takes future planning trajectories of the ego vehicle as inputs to solve this limitation with the dataset newly collected.