OpenRaiser/Intern-Atlas huggingface dataset:
Synthetic Test Data Generation Using Recurrent Neural Networks: A Position Paper
Venue: arXiv cs.SE 2024
Year: 2024

Testing in production-like test environments is an essential part of quality assurance processes in many industries. Provisioning of such test environments, for information-intensive services, involves setting up databases that are rich-enough to enable simulating a wide variety of user scenarios. While production data is perhaps the gold-standard here, many organizations, particularly within the public sectors, are not allowed to use production data for testing purposes due to privacy concerns. The alternatives are to use anonymized data, or synthetically generated data. In this paper, we elaborate on these alternatives and compare them in an industrial context. Further we focus on synthetic data generation and investigate the use of recurrent neural networks for this purpose. In our preliminary experiments, we were able to generate representative and highly accurate data using a recurrent neural network. These results open new research questions that we discuss here, and plan to investigate in our future research.
SBoRA: Low-Rank Adaptation with Regional Weight Updates
Venue: arXiv cs.AI 2024
Year: 2024

This paper introduces Standard Basis LoRA (SBoRA), a novel parameter-efficient fine-tuning approach for Large Language Models that builds upon the pioneering works of Low-Rank Adaptation (LoRA) and Orthogonal Adaptation. SBoRA reduces the number of trainable parameters by half or doubles the rank with the similar number of trainable parameters as LoRA, while improving learning performance. By utilizing orthogonal standard basis vectors to initialize one of the low-rank matrices (either $\mathbf{A}$ or $\mathbf{B}$), SBoRA facilitates regional weight updates and memory-efficient fine-tuning. This results in two variants, SBoRA-FA and SBoRA-FB, where only one of the matrices is updated, leading to a sparse update matrix $\mathrmΔ \mathbf{W}$ with predominantly zero rows or columns. Consequently, most of the fine-tuned model's weights $(\mathbf{W}_0+\mathrmΔ \mathbf{W})$ remain unchanged from the pre-trained weights, akin to the modular organization of the human brain, which efficiently adapts to new tasks. Our empirical results demonstrate the superiority of SBoRA-FA over LoRA in various fine-tuning tasks, including commonsense reasoning and arithmetic reasoning. Furthermore, we evaluate the effectiveness of QSBoRA on quantized LLaMA models of varying scales, highlighting its potential for efficient adaptation to new tasks. Code is available at https://github.com/cityuhkai/SBoRA
See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition
Venue: arXiv cs.LG 2024
Year: 2024

The rapid expansion of large foundation models within the pre-training and fine-tuning framework has underscored that larger models often yield better results. However, the scaling up of large foundation models has led to soaring costs in fine-tuning and parameter storage, rendering extensive adaptations impractical. This challenge has sparked the development of parameter-efficient fine-tuning (PEFT), which focuses on optimizing a select subset of parameters while keeping the rest fixed, significantly lowering computational and storage overheads. While recent years have witnessed a significant success in PEFT, a deep understanding of the fundamental principles behind these methods remains unexplored. To this end, here we take the first step to unify all approaches by dissecting them from a decomposition perspective. We initiate a comprehensive mathematical analysis of these methods, allowing us to delve deeply into their underlying mechanisms, and we explore the reasons behind the variations in performance among different techniques. Furthermore, inspired by our theoretical analysis, we introduce two novel PEFT methods alongside a simple yet effective framework designed to enhance the performance of PEFT techniques across various applications. Our empirical validations, conducted across multiple datasets, demonstrate the efficacy of these methods, showcasing both theoretical validity and practical performance improvements under the guidance of our analytical findings. We believe our work will deepen researchers' understanding of PEFT and other techniques, prompting further contemplation and advancing the research across the whole community.
ClutterGen: A Cluttered Scene Generator for Robot Learning
Venue: arXiv cs.RO 2024
Year: 2024

We introduce ClutterGen, a physically compliant simulation scene generator capable of producing highly diverse, cluttered, and stable scenes for robot learning. Generating such scenes is challenging as each object must adhere to physical laws like gravity and collision. As the number of objects increases, finding valid poses becomes more difficult, necessitating significant human engineering effort, which limits the diversity of the scenes. To overcome these challenges, we propose a reinforcement learning method that can be trained with physics-based reward signals provided by the simulator. Our experiments demonstrate that ClutterGen can generate cluttered object layouts with up to ten objects on confined table surfaces. Additionally, our policy design explicitly encourages the diversity of the generated scenes for open-ended generation. Our real-world robot results show that ClutterGen can be directly used for clutter rearrangement and stable placement policy training.
Understanding Professional Needs to Create Privacy-Preserving and Secure Emergent Digital Artworks
Venue: arXiv cs.HC 2024
Year: 2024

In recent years, immersive art installations featuring interactive artworks have been on the rise. These installations are an integral part of museums and art centers like selfie museums, teamLab Borderless, ARTECHOUSE, and Meow Wolf. Moreover, immersive art have also been increasingly incorporated into traditional museums as well. However, immersive art requires active user participation and often captures information from viewers and participants through cameras, sensors, microphones, embodied interaction devices, surveillance, and kinetic mirrors. Therefore, we propose a new line of research to examine the security and privacy postures of immersive artworks. In our pilot study, we conducted a semi-structured interview with five experienced practitioners from either the art (2) or cybersecurity (3) fields. Our aim was to understand their current security and privacy practices, along with their needs when it comes to immersive art. From their responses, we created a list of security and privacy parameters, such as, providing opt-in mechanics for data collection, knowledge of data collection tools such as proximity sensors, and creating security awareness amongst participants by communicating security protocols and threat models. These parameters allow us to build privacy-preserving, secure, and accessible software for individuals working in media arts, who often have no background on security and privacy. In the future, we plan to utilize these parameters to develop software in response to those needs and then host an art exhibition of immersive artworks utilizing the platform.
Hiring for An Uncertain Task: Joint Design of Information and Contracts
Venue: arXiv cs.GT 2024
Year: 2024

In this paper, we initiate the computational problem of jointly designing information and contracts. We consider three possible classes of contracts with decreasing flexibility and increasing simplicity: ambiguous contracts, menus of explicit contracts and explicit single contract. Ambiguous contracts allow the principal to conceal the applied payment schemes through a contract that depends on the unknown state of nature, while explicit contracts reveal the contract prior to the agent's decision. Our results show a trade-off between the simplicity of the contracts and the computational complexity of the joint design. Indeed, we show that an approximately-optimal mechanism with ambiguous contracts can be computed in polynomial time. However, they are convoluted mechanisms and not well-suited for some real-world scenarios. Conversely, explicit menus of contracts and single contracts are simpler mechanisms, but they cannot be computed efficiently. In particular, we show that computing the optimal mechanism with explicit menus of contracts and single contracts is APX-Hard. We also characterize the structure of optimal mechanisms. Interestingly, direct mechanisms are optimal for both the most flexible ambiguous contracts and the least flexible explicit single contract, but they are suboptimal for that with menus of contracts. Finally, motivated by our hardness results, we turn our attention to menus of linear contracts and single linear contracts. We show that both the problem of computing the optimal mechanism with an explicit menu of linear contracts and an explicit single linear contract admits an FPTAS.
Experiments with truth using Machine Learning: Spectral analysis and explainable classification of synthetic, false, and genuine information
Venue: arXiv cs.AI 2024
Year: 2024

Misinformation is still a major societal problem and the arrival of Large Language Models (LLMs) only added to it. This paper analyzes synthetic, false, and genuine information in the form of text from spectral analysis, visualization, and explainability perspectives to find the answer to why the problem is still unsolved despite multiple years of research and a plethora of solutions in the literature. Various embedding techniques on multiple datasets are used to represent information for the purpose. The diverse spectral and non-spectral methods used on these embeddings include t-distributed Stochastic Neighbor Embedding (t-SNE), Principal Component Analysis (PCA), and Variational Autoencoders (VAEs). Classification is done using multiple machine learning algorithms. Local Interpretable Model-Agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), and Integrated Gradients are used for the explanation of the classification. The analysis and the explanations generated show that misinformation is quite closely intertwined with genuine information and the machine learning algorithms are not as effective in separating the two despite the claims in the literature.
Sequential Gaussian Variational Inference for Nonlinear State Estimation and Its Application in Robot Navigation
Venue: arXiv cs.RO 2024
Year: 2024

Probabilistic state estimation is essential for robots navigating uncertain environments. Accurately and efficiently managing uncertainty in estimated states is key to robust robotic operation. However, nonlinearities in robotic platforms pose significant challenges that require advanced estimation techniques. Gaussian variational inference (GVI) offers an optimization perspective on the estimation problem, providing analytically tractable solutions and efficiencies derived from the geometry of Gaussian space. We propose a Sequential Gaussian Variational Inference (S-GVI) method to address nonlinearity and provide efficient sequential inference processes. Our approach integrates sequential Bayesian principles into the GVI framework, which are addressed using statistical approximations and gradient updates on the information geometry. Validations through simulations and real-world experiments demonstrate significant improvements in state estimation over the Maximum A Posteriori (MAP) estimation method.
How Effective are State Space Models for Machine Translation?
Venue: arXiv cs.CL 2024
Year: 2024

Transformers are the current architecture of choice for NLP, but their attention layers do not scale well to long contexts. Recent works propose to replace attention with linear recurrent layers -- this is the case for state space models, which enjoy efficient training and inference. However, it remains unclear whether these models are competitive with transformers in machine translation (MT). In this paper, we provide a rigorous and comprehensive experimental comparison between transformers and linear recurrent models for MT. Concretely, we experiment with RetNet, Mamba, and hybrid versions of Mamba which incorporate attention mechanisms. Our findings demonstrate that Mamba is highly competitive with transformers on sentence and paragraph-level datasets, where in the latter both models benefit from shifting the training distribution towards longer sequences. Further analysis show that integrating attention into Mamba improves translation quality, robustness to sequence length extrapolation, and the ability to recall named entities.
Addressing single object tracking in satellite imagery through prompt-engineered solutions
Venue: arXiv cs.CV 2024
Year: 2024

Object tracking in satellite videos remains a complex endeavor in remote sensing due to the intricate and dynamic nature of satellite imagery. Existing state-of-the-art trackers in computer vision integrate sophisticated architectures, attention mechanisms, and multi-modal fusion to enhance tracking accuracy across diverse environments. However, the challenges posed by satellite imagery, such as background variations, atmospheric disturbances, and low-resolution object delineation, significantly impede the precision and reliability of traditional Single Object Tracking (SOT) techniques. Our study delves into these challenges and proposes prompt engineering methodologies, leveraging the Segment Anything Model (SAM) and TAPIR (Tracking Any Point with per-frame Initialization and temporal Refinement), to create a training-free point-based tracking method for small-scale objects on satellite videos. Experiments on the VISO dataset validate our strategy, marking a significant advancement in robust tracking solutions tailored for satellite imagery in remote sensing applications.
SSPARE: Space Solar Power Autonomously Reconfigurable Elements
Venue: arXiv cs.RO 2024
Year: 2024

GEO communication satellites generate significant revenue but can only function reliably for approximately 10 years on orbit. One of the main drivers that limits the reliability of a GEO satellite is the electric power system, and in particular, anomalies related to batteries and degradation of the solar arrays. Given the high cost and relatively short lifespan of GEO satellites, there has been increased research activity towards developing on-orbit servicing systems. However, most of the existing servicing systems are expensive, highly customized, and focus on refueling tasks. On-orbit refueling can be very useful, however, it does not improve satellite reliability which is crucial for long-term missions. Therefore, we propose SSPARE (Space Solar Power Autonomously Reconfigurable Elements), a cost-effective, self-servicing power system. Aside from improving satellite reliability, SSPARE enables to generate up to 6 times more power per launch compared to a traditional GEO communication satellite. This study explores why GEO satellites fail and elaborates on the SSPARE concept. A comparison of SSPARE against a traditional on-orbit servicing mission highlights the benefits of the proposed concept. With humanity striving to become more and more Earth-independent, this work aims to build a foundation for future systems such as large solar power farms on-orbit.
Read, Watch and Scream! Sound Generation from Text and Video
Venue: arXiv cs.CV 2024
Year: 2024

Despite the impressive progress of multimodal generative models, video-to-audio generation still suffers from limited performance and limits the flexibility to prioritize sound synthesis for specific objects within the scene. Conversely, text-to-audio generation methods generate high-quality audio but pose challenges in ensuring comprehensive scene depiction and time-varying control. To tackle these challenges, we propose a novel video-and-text-to-audio generation method, called \ours, where video serves as a conditional control for a text-to-audio generation model. Especially, our method estimates the structural information of sound (namely, energy) from the video while receiving key content cues from a user prompt. We employ a well-performing text-to-audio model to consolidate the video control, which is much more efficient for training multimodal diffusion models with massive triplet-paired (audio-video-text) data. In addition, by separating the generative components of audio, it becomes a more flexible system that allows users to freely adjust the energy, surrounding environment, and primary sound source according to their preferences. Experimental results demonstrate that our method shows superiority in terms of quality, controllability, and training efficiency. Code and demo are available at https://naver-ai.github.io/rewas.
Cost-Efficient Computation Offloading in SAGIN: A Deep Reinforcement Learning and Perception-Aided Approach
Venue: arXiv cs.NI 2024
Year: 2024

The Space-Air-Ground Integrated Network (SAGIN), crucial to the advancement of sixth-generation (6G) technology, plays a key role in ensuring universal connectivity, particularly by addressing the communication needs of remote areas lacking cellular network infrastructure. This paper delves into the role of unmanned aerial vehicles (UAVs) within SAGIN, where they act as a control layer owing to their adaptable deployment capabilities and their intermediary role. Equipped with millimeter-wave (mmWave) radar and vision sensors, these UAVs are capable of acquiring multi-source data, which helps to diminish uncertainty and enhance the accuracy of decision-making. Concurrently, UAVs collect tasks requiring computing resources from their coverage areas, originating from a variety of mobile devices moving at different speeds. These tasks are then allocated to ground base stations (BSs), low-earth-orbit (LEO) satellite, and local processing units to improve processing efficiency. Amidst this framework, our study concentrates on devising dynamic strategies for facilitating task hosting between mobile devices and UAVs, offloading computations, managing associations between UAVs and BSs, and allocating computing resources. The objective is to minimize the time-averaged network cost, considering the uncertainty of device locations, speeds, and even types. To tackle these complexities, we propose a deep reinforcement learning and perception-aided online approach (DRL-and-Perception-aided Approach) for this joint optimization in SAGIN, tailored for an environment filled with uncertainties. The effectiveness of our proposed approach is validated through extensive numerical simulations, which quantify its performance relative to various network parameters.
Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imaging
Venue: arXiv cs.CV 2024
Year: 2024

Recently, transfer learning and self-supervised learning have gained significant attention within the medical field due to their ability to mitigate the challenges posed by limited data availability, improve model generalisation, and reduce computational expenses. Transfer learning and self-supervised learning hold immense potential for advancing medical research. However, it is crucial to recognise that transfer learning and self-supervised learning architectures exhibit distinct advantages and limitations, manifesting variations in accuracy, training speed, and robustness. This paper compares the performance and robustness of transfer learning and self-supervised learning in the medical field. Specifically, we pre-trained two models using the same source domain datasets with different pre-training methods and evaluated them on small-sized medical datasets to identify the factors influencing their final performance. We tested data with several common issues in medical domains, such as data imbalance, data scarcity, and domain mismatch, through comparison experiments to understand their impact on specific pre-trained models. Finally, we provide recommendations to help users apply transfer learning and self-supervised learning methods in medical areas, and build more convenient and efficient deployment strategies.
Described Spatial-Temporal Video Detection
Venue: arXiv cs.CV 2024
Year: 2024

Detecting visual content on language expression has become an emerging topic in the community. However, in the video domain, the existing setting, i.e., spatial-temporal video grounding (STVG), is formulated to only detect one pre-existing object in each frame, ignoring the fact that language descriptions can involve none or multiple entities within a video. In this work, we advance the STVG to a more practical setting called described spatial-temporal video detection (DSTVD) by overcoming the above limitation. To facilitate the exploration of DSTVD, we first introduce a new benchmark, namely DVD-ST. Notably, DVD-ST supports grounding from none to many objects onto the video in response to queries and encompasses a diverse range of over 150 entities, including appearance, actions, locations, and interactions. The extensive breadth and diversity of the DVD-ST dataset make it an exemplary testbed for the investigation of DSTVD. In addition to the new benchmark, we further present two baseline methods for our proposed DSTVD task by extending two representative STVG models, i.e., TubeDETR, and STCAT. These extended models capitalize on tubelet queries to localize and track referred objects across the video sequence. Besides, we adjust the training objectives of these models to optimize spatial and temporal localization accuracy and multi-class classification capabilities. Furthermore, we benchmark the baselines on the introduced DVD-ST dataset and conduct extensive experimental analysis to guide future investigation. Our code and benchmark will be publicly available.
New Directions in Text Classification Research: Maximizing The Performance of Sentiment Classification from Limited Data
Venue: arXiv cs.CL 2024
Year: 2024

The stakeholders' needs in sentiment analysis for various issues, whether positive or negative, are speed and accuracy. One new challenge in sentiment analysis tasks is the limited training data, which often leads to suboptimal machine learning models and poor performance on test data. This paper discusses the problem of text classification based on limited training data (300 to 600 samples) into three classes: positive, negative, and neutral. A benchmark dataset is provided for training and testing data on the issue of Kaesang Pangarep's appointment as Chairman of PSI. External data for aggregation and augmentation purposes are provided, consisting of two datasets: the topic of Covid Vaccination sentiment and an open topic. The official score used is the F1-score, which balances precision and recall among the three classes, positive, negative, and neutral. A baseline score is provided as a reference for researchers for unoptimized classification methods. The optimized score is provided as a reference for the target score to be achieved by any proposed method. Both scoring (baseline and optimized) use the SVM method, which is widely reported as the state-of-the-art in conventional machine learning methods. The F1-scores achieved by the baseline and optimized methods are 40.83% and 51.28%, respectively.