OpenRaiser/Intern-Atlas huggingface dataset:
The Compressor-Retriever Architecture for Language Model OS
Venue: arXiv cs.CL 2024
Year: 2024

Recent advancements in large language models (LLMs) have significantly enhanced their capacity to aggregate and process information across multiple modalities, enabling them to perform a wide range of tasks such as multimodal data querying, tool usage, web interactions, and handling long documents. These capabilities pave the way for transforming LLMs from mere chatbots into general-purpose agents capable of interacting with the real world. This paper explores the concept of using a language model as the core component of an operating system (OS), effectively acting as a CPU that processes data stored in a context window, which functions as RAM. A key challenge in realizing such an LM OS is managing the life-long context and ensuring statefulness across sessions, a feature limited by the current session-based interaction paradigm due to context window size limit. To address this, we introduce compressor-retriever, a model-agnostic architecture designed for life-long context management. Unlike other long-context solutions such as retrieval-augmented generation, our approach exclusively uses the base model's forward function to compress and retrieve context, ensuring end-to-end differentiability. Preliminary experiments demonstrate the effectiveness of this architecture in in-context learning tasks, marking a step towards the development of a fully stateful LLM OS. Project repo available at: https://github.com/gblackout/LM-OS
AMG: Avatar Motion Guided Video Generation
Venue: arXiv cs.CV 2024
Year: 2024

Human video generation task has gained significant attention with the advancement of deep generative models. Generating realistic videos with human movements is challenging in nature, due to the intricacies of human body topology and sensitivity to visual artifacts. The extensively studied 2D media generation methods take advantage of massive human media datasets, but struggle with 3D-aware control; whereas 3D avatar-based approaches, while offering more freedom in control, lack photorealism and cannot be harmonized seamlessly with background scene. We propose AMG, a method that combines the 2D photorealism and 3D controllability by conditioning video diffusion models on controlled rendering of 3D avatars. We additionally introduce a novel data processing pipeline that reconstructs and renders human avatar movements from dynamic camera videos. AMG is the first method that enables multi-person diffusion video generation with precise control over camera positions, human motions, and background style. We also demonstrate through extensive evaluation that it outperforms existing human video generation methods conditioned on pose sequences or driving videos in terms of realism and adaptability.
The category of well-filtered dcpos is not $Γ$-faithful
Venue: arXiv cs.LO 2024
Year: 2024

The Ho-Zhao problem asks whether any two dcpo's with isomorphic Scott closed set lattices are themselves isomorphic, that is, whether the category $\mathbf{DCPO}$ of dcpo's and Scott-continuous maps is $Γ$-faithful. In 2018, Ho, Goubault-Larrecq, Jung and Xi answered this question in the negative, and they introduced the category $\mathbf{DOMI}$ of dominated dcpo's and proved that it is {$Γ$-faithful}. Dominated dcpo's subsume many familiar families of dcpo's in domain theory, such as the category of bounded-complete dcpo's and that of sober dcpo's, among others. However, it is unknown whether the category of dominated dcpo's dominates all well-filtered dcpo's, a class strictly larger than that of bounded-complete lattices and that of sober dcpo's. In this paper, we address this very natural question and show that the category $\mathbf{WF}$ of well-filtered dcpo's is not $Γ$-faithful, and as a result of it, well-filtered dcpo's need not be dominated in general. Since not all dcpo's are well-filtered, our work refines the results of Ho, Goubault-Larrecq, Jung and Xi. As a second contribution, we confirm that the Lawson's category of $Ω^{*}$-compact dcpo's is $Γ$-faithful. Moreover, we locate a class of dcpo's which we call weakly dominated dcpo's, and show that this class is $Γ$-faithful and strictly larger than $\mathbf{DOMI}$.
An Implementation of Werewolf Agent That does not Truly Trust LLMs
Venue: arXiv cs.CL 2024
Year: 2024

Werewolf is an incomplete information game, which has several challenges when creating a computer agent as a player given the lack of understanding of the situation and individuality of utterance (e.g., computer agents are not capable of characterful utterance or situational lying). We propose a werewolf agent that solves some of those difficulties by combining a Large Language Model (LLM) and a rule-based algorithm. In particular, our agent uses a rule-based algorithm to select an output either from an LLM or a template prepared beforehand based on the results of analyzing conversation history using an LLM. It allows the agent to refute in specific situations, identify when to end the conversation, and behave with persona. This approach mitigated conversational inconsistencies and facilitated logical utterance as a result. We also conducted a qualitative evaluation, which resulted in our agent being perceived as more human-like compared to an unmodified LLM. The agent is freely available for contributing to advance the research in the field of Werewolf game.
An Array Intermediate Language for Mixed Cryptography
Venue: arXiv cs.CR 2024
Year: 2024

We introduce AIRduct, a new array-based intermediate representation designed to support generating efficient code for interactive programs employing multiple cryptographic mechanisms. AIRduct is intended as an IR for the Viaduct compiler, which can synthesize secure, distributed programs with an extensible suite of cryptography. Therefore, AIRduct supports an extensible variety of cryptographic mechanisms, including MPC and ZKP.
Dynamic Motion Synthesis: Masked Audio-Text Conditioned Spatio-Temporal Transformers
Venue: arXiv cs.CV 2024
Year: 2024

Our research presents a novel motion generation framework designed to produce whole-body motion sequences conditioned on multiple modalities simultaneously, specifically text and audio inputs. Leveraging Vector Quantized Variational Autoencoders (VQVAEs) for motion discretization and a bidirectional Masked Language Modeling (MLM) strategy for efficient token prediction, our approach achieves improved processing efficiency and coherence in the generated motions. By integrating spatial attention mechanisms and a token critic we ensure consistency and naturalness in the generated motions. This framework expands the possibilities of motion generation, addressing the limitations of existing approaches and opening avenues for multimodal motion synthesis.
Data-driven topology design based on principal component analysis for 3D structural design problems
Venue: arXiv cs.LG 2024
Year: 2024

Topology optimization is a structural design methodology widely utilized to address engineering challenges. However, sensitivity-based topology optimization methods struggle to solve optimization problems characterized by strong non-linearity. Leveraging the sensitivity-free nature and high capacity of deep generative models, data-driven topology design (DDTD) methodology is considered an effective solution to this problem. Despite this, the training effectiveness of deep generative models diminishes when input size exceeds a threshold while maintaining high degrees of freedom is crucial for accurately characterizing complex structures. To resolve the conflict between the both, we propose DDTD based on principal component analysis (PCA). Its core idea is to replace the direct training of deep generative models with material distributions by using a principal component score matrix obtained from PCA computation and to obtain the generated material distributions with new features through the restoration process. We apply the proposed PCA-based DDTD to the problem of minimizing the maximum stress in 3D structural mechanics and demonstrate it can effectively address the current challenges faced by DDTD that fail to handle 3D structural design problems. Various experiments are conducted to demonstrate the effectiveness and practicability of the proposed PCA-based DDTD.
PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applications
Venue: arXiv cs.LG 2024
Year: 2024

In practice, we are often faced with small-sized tabular data. However, current tabular benchmarks are not geared towards data-scarce applications, making it very difficult to derive meaningful conclusions from empirical comparisons. We introduce PMLBmini, a tabular benchmark suite of 44 binary classification datasets with sample sizes $\leq$ 500. We use our suite to thoroughly evaluate current automated machine learning (AutoML) frameworks, off-the-shelf tabular deep neural networks, as well as classical linear models in the low-data regime. Our analysis reveals that state-of-the-art AutoML and deep learning approaches often fail to appreciably outperform even a simple logistic regression baseline, but we also identify scenarios where AutoML and deep learning methods are indeed reasonable to apply. Our benchmark suite, available on https://github.com/RicardoKnauer/TabMini , allows researchers and practitioners to analyze their own methods and challenge their data efficiency.
Acoustic Levitation for Environmental Remediation: An Effective Approach for Containment and Forecasting of Oil Spills
Venue: arXiv cs.ET 2024
Year: 2024

The ocean ecology is badly impacted by large-scale oil spills, plastic waste, and chemical pollution, which destroy ecosystems and endanger marine life. Acknowledging the detrimental effects of oil spills on ecosystems, our research aims to establish the foundation for creative methods to lessen their impact. With an emphasis on the containment and prediction of oil spills, this research investigates the potential of acoustic levitation as a cutting-edge technique for environmental cleanup. Effectively separating and eliminating pollutants without causing additional ecological harm is a major issue for traditional oil spill cleanup techniques. Acoustic levitation provides a non-invasive, accurate, and effective alternative by using sound waves to precisely and subtly separate oil droplets from water in controlled environments. This proposed approach can reduce the negative effects on the environment and increase the efficacy of cleanup efforts. The findings have been examined and assessed by proof of concept experiments with oil droplets, identifying the relationship between the intensity of ultrasonic pressure and the proportion of oil droplets collected.
$S^2$NeRF: Privacy-preserving Training Framework for NeRF
Venue: arXiv cs.CR 2024
Year: 2024

Neural Radiance Fields (NeRF) have revolutionized 3D computer vision and graphics, facilitating novel view synthesis and influencing sectors like extended reality and e-commerce. However, NeRF's dependence on extensive data collection, including sensitive scene image data, introduces significant privacy risks when users upload this data for model training. To address this concern, we first propose SplitNeRF, a training framework that incorporates split learning (SL) techniques to enable privacy-preserving collaborative model training between clients and servers without sharing local data. Despite its benefits, we identify vulnerabilities in SplitNeRF by developing two attack methods, Surrogate Model Attack and Scene-aided Surrogate Model Attack, which exploit the shared gradient data and a few leaked scene images to reconstruct private scene information. To counter these threats, we introduce $S^2$NeRF, secure SplitNeRF that integrates effective defense mechanisms. By introducing decaying noise related to the gradient norm into the shared gradient information, $S^2$NeRF preserves privacy while maintaining a high utility of the NeRF model. Our extensive evaluations across multiple datasets demonstrate the effectiveness of $S^2$NeRF against privacy breaches, confirming its viability for secure NeRF training in sensitive applications.
USTC-KXDIGIT System Description for ASVspoof5 Challenge
Venue: arXiv cs.SD 2024
Year: 2024

This paper describes the USTC-KXDIGIT system submitted to the ASVspoof5 Challenge for Track 1 (speech deepfake detection) and Track 2 (spoofing-robust automatic speaker verification, SASV). Track 1 showcases a diverse range of technical qualities from potential processing algorithms and includes both open and closed conditions. For these conditions, our system consists of a cascade of a frontend feature extractor and a back-end classifier. We focus on extensive embedding engineering and enhancing the generalization of the back-end classifier model. Specifically, the embedding engineering is based on hand-crafted features and speech representations from a self-supervised model, used for closed and open conditions, respectively. To detect spoof attacks under various adversarial conditions, we trained multiple systems on an augmented training set. Additionally, we used voice conversion technology to synthesize fake audio from genuine audio in the training set to enrich the synthesis algorithms. To leverage the complementary information learned by different model architectures, we employed activation ensemble and fused scores from different systems to obtain the final decision score for spoof detection. During the evaluation phase, the proposed methods achieved 0.3948 minDCF and 14.33% EER in the close condition, and 0.0750 minDCF and 2.59% EER in the open condition, demonstrating the robustness of our submitted systems under adversarial conditions. In Track 2, we continued using the CM system from Track 1 and fused it with a CNN-based ASV system. This approach achieved 0.2814 min-aDCF in the closed condition and 0.0756 min-aDCF in the open condition, showcasing superior performance in the SASV system.
Accented Character Entry Using Physical Keyboards in Virtual Reality
Venue: arXiv cs.HC 2024
Year: 2024

Research on text entry in Virtual Reality (VR) has gained popularity but the efficient entry of accented characters, characters with diacritical marks, in VR remains underexplored. Entering accented characters is supported on most capacitive touch keyboards through a long press on a base character and a subsequent selection of the accented character. However, entering those characters on physical keyboards is still challenging, as they require a recall and an entry of respective numeric codes. To address this issue this paper investigates three techniques to support accented character entry on physical keyboards in VR. Specifically, we compare a context-aware numeric code technique that does not require users to recall a code, a key-press-only condition in which the accented characters are dynamically remapped to physical keys next to a base character, and a multimodal technique, in which eye gaze is used to select the accented version of a base character previously selected by key-press on the keyboard. The results from our user study (n=18) reveal that both the key-press-only and the multimodal technique outperform the baseline technique in terms of text entry speed.
4D-CAT: Synthesis of 4D Coronary Artery Trees from Systole and Diastole
Venue: arXiv cs.CV 2024
Year: 2024

The three-dimensional vascular model reconstructed from CT images is widely used in medical diagnosis. At different phases, the beating of the heart can cause deformation of vessels, resulting in different vascular imaging states and false positive diagnostic results. The 4D model can simulate a complete cardiac cycle. Due to the dose limitation of contrast agent injection in patients, it is valuable to synthesize a 4D coronary artery trees through finite phases imaging. In this paper, we propose a method for generating a 4D coronary artery trees, which maps the systole to the diastole through deformation field prediction, interpolates on the timeline, and the motion trajectory of points are obtained. Specifically, the centerline is used to represent vessels and to infer deformation fields using cube-based sorting and neural networks. Adjacent vessel points are aggregated and interpolated based on the deformation field of the centerline point to obtain displacement vectors of different phases. Finally, the proposed method is validated through experiments to achieve the registration of non-rigid vascular points and the generation of 4D coronary trees.
Federated Prediction-Powered Inference from Decentralized Data
Venue: arXiv cs.LG 2024
Year: 2024

In various domains, the increasing application of machine learning allows researchers to access inexpensive predictive data, which can be utilized as auxiliary data for statistical inference. Although such data are often unreliable compared to gold-standard datasets, Prediction-Powered Inference (PPI) has been proposed to ensure statistical validity despite the unreliability. However, the challenge of `data silos' arises when the private gold-standard datasets are non-shareable for model training, leading to less accurate predictive models and invalid inferences. In this paper, we introduces the Federated Prediction-Powered Inference (Fed-PPI) framework, which addresses this challenge by enabling decentralized experimental data to contribute to statistically valid conclusions without sharing private information. The Fed-PPI framework involves training local models on private data, aggregating them through Federated Learning (FL), and deriving confidence intervals using PPI computation. The proposed framework is evaluated through experiments, demonstrating its effectiveness in producing valid confidence intervals.
Multi-Branch Attention Convolutional Neural Network for Online RIS Configuration with Discrete Responses: A Neuroevolution Approach
Venue: arXiv cs.NI 2024
Year: 2024

In this paper, we consider the problem of jointly controlling the configuration of a Reconfigurable Intelligent Surface (RIS) with unit elements of discrete responses and a codebook-based transmit precoder in RIS-empowered Multiple-Input Single-Output (MISO) communication systems. The adjustable elements of the RIS and the precoding vector need to be jointly modified in real time to account for rapid changes in the wireless channels, making the application of complicated discrete optimization algorithms impractical. We present a novel Multi-Branch Attention Convolutional Neural Network (MBACNN) architecture for this design objective which is optimized using NeuroEvolution (NE), leveraging its capability to effectively tackle the non-differentiable problem arising from the discrete phase states of the RIS elements. The channel matrices of all involved links are first passed to separate self-attention layers to obtain initial embeddings, which are then concatenated and passed to a convolutional network for spatial feature extraction, before being fed to a per-element multi-layered perceptron for the final RIS phase configuration calculation. Our MBACNN architecture is then extended to multi-RIS-empowered MISO communication systems, and a novel NE-based optimization approach for the online distributed configuration of multiple RISs is presented. The superiority of the proposed single-RIS approach over both learning-based and classical discrete optimization benchmarks is showcased via extensive numerical evaluations over both stochastic and geometrical channel models. It is also demonstrated that the proposed distributed multi-RIS approach outperforms both distributed controllers with feedforward neural networks and fully centralized ones.
UWStereo: A Large Synthetic Dataset for Underwater Stereo Matching
Venue: arXiv cs.CV 2024
Year: 2024

Despite recent advances in stereo matching, the extension to intricate underwater settings remains unexplored, primarily owing to: 1) the reduced visibility, low contrast, and other adverse effects of underwater images; 2) the difficulty in obtaining ground truth data for training deep learning models, i.e. simultaneously capturing an image and estimating its corresponding pixel-wise depth information in underwater environments. To enable further advance in underwater stereo matching, we introduce a large synthetic dataset called UWStereo. Our dataset includes 29,568 synthetic stereo image pairs with dense and accurate disparity annotations for left view. We design four distinct underwater scenes filled with diverse objects such as corals, ships and robots. We also induce additional variations in camera model, lighting, and environmental effects. In comparison with existing underwater datasets, UWStereo is superior in terms of scale, variation, annotation, and photo-realistic image quality. To substantiate the efficacy of the UWStereo dataset, we undertake a comprehensive evaluation compared with nine state-of-the-art algorithms as benchmarks. The results indicate that current models still struggle to generalize to new domains. Hence, we design a new strategy that learns to reconstruct cross domain masked images before stereo matching training and integrate a cross view attention enhancement module that aggregates long-range content information to enhance the generalization ability.
LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planning
Venue: arXiv cs.AI 2024
Year: 2024

Effective planning is essential for the success of any task, from organizing a vacation to routing autonomous vehicles and developing corporate strategies. It involves setting goals, formulating plans, and allocating resources to achieve them. LLMs are particularly well-suited for automated planning due to their strong capabilities in commonsense reasoning. They can deduce a sequence of actions needed to achieve a goal from a given state and identify an effective course of action. However, it is frequently observed that plans generated through direct prompting often fail upon execution. Our survey aims to highlight the existing challenges in planning with language models, focusing on key areas such as embodied environments, optimal scheduling, competitive and cooperative games, task decomposition, reasoning, and planning. Through this study, we explore how LLMs transform AI planning and provide unique insights into the future of LM-assisted planning.
Dialogue You Can Trust: Human and AI Perspectives on Generated Conversations
Venue: arXiv cs.CL 2024
Year: 2024

As dialogue systems and chatbots increasingly integrate into everyday interactions, the need for efficient and accurate evaluation methods becomes paramount. This study explores the comparative performance of human and AI assessments across a range of dialogue scenarios, focusing on seven key performance indicators (KPIs): Coherence, Innovation, Concreteness, Goal Contribution, Commonsense Contradiction, Incorrect Fact, and Redundancy. Utilizing the GPT-4o API, we generated a diverse dataset of conversations and conducted a two-part experimental analysis. In Experiment 1, we evaluated multi-party conversations on Coherence, Innovation, Concreteness, and Goal Contribution, revealing that GPT models align closely with human judgments. Notably, both human and AI evaluators exhibited a tendency towards binary judgment rather than linear scaling, highlighting a shared challenge in these assessments. Experiment 2 extended the work of Finch et al. (2023) by focusing on dyadic dialogues and assessing Commonsense Contradiction, Incorrect Fact, and Redundancy. The results indicate that while GPT-4o demonstrates strong performance in maintaining factual accuracy and commonsense reasoning, it still struggles with reducing redundancy and self-contradiction. Our findings underscore the potential of GPT models to closely replicate human evaluation in dialogue systems, while also pointing to areas for improvement. This research offers valuable insights for advancing the development and implementation of more refined dialogue evaluation methodologies, contributing to the evolution of more effective and human-like AI communication tools.