OpenRaiser/Intern-Atlas huggingface dataset:
Learned Random Label Predictions as a Neural Network Complexity Metric
Venue: arXiv cs.LG 2024
Year: 2024

We empirically investigate the impact of learning randomly generated labels in parallel to class labels in supervised learning on memorization, model complexity, and generalization in deep neural networks. To this end, we introduce a multi-head network architecture as an extension of standard CNN architectures. Inspired by methods used in fair AI, our approach allows for the unlearning of random labels, preventing the network from memorizing individual samples. Based on the concept of Rademacher complexity, we first use our proposed method as a complexity metric to analyze the effects of common regularization techniques and challenge the traditional understanding of feature extraction and classification in CNNs. Second, we propose a novel regularizer that effectively reduces sample memorization. However, contrary to the predictions of classical statistical learning theory, we do not observe improvements in generalization.
ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
Venue: arXiv cs.CL 2024
Year: 2024

During the development of large language models (LLMs), pre-training data play a critical role in shaping LLMs' capabilities. In recent years several large-scale and high-quality pre-training datasets have been released to accelerate the research of LLMs, including ChineseWebText1.0, C4, Pile, WanJuan, MAPCC and others. However, as LLMs continue to evolve, focus has increasingly shifted to domain-specific capabilities and safety concerns, making those previous coarse-grained texts insufficient for meeting training requirements. Furthermore, fine-grained information, such as quality, domain and toxicity, is becoming increasingly important in building powerful and reliable LLMs for various scenarios. To address these challenges, in this paper we propose a new tool-chain called MDFG-tool for constructing large-scale and high-quality Chinese datasets with multi-dimensional and fine-grained information. First, we employ manually crafted rules to discard explicit noisy texts from raw contents. Second, the quality evaluation model, domain classifier, and toxicity evaluation model are well-designed to assess the remaining cleaned data respectively. Finally, we integrate these three types of fine-grained information for each text. With this approach, we release the largest, high-quality and fine-grained Chinese text ChineseWebText2.0, which consists of 3.8TB and each text is associated with a quality score, domain labels, a toxicity label and a toxicity score, facilitating the LLM researchers to select data based on various types of fine-grained information. The data, codes and the tool-chain are available on this website https://github.com/CASIA-LM/ChineseWebText-2.0
State of the Art on Stacked Intelligent Metasurfaces: Communication, Sensing and Computing in the Wave Domain
Venue: arXiv cs.IT 2024
Year: 2024

Stacked intelligent metasurface (SIM) is an emerging technology that capitalizes on reconfigurable metasurfaces for several applications in wireless communications. SIM is considered an enabler for integrating communication, sensing and computing in a unique platform. In this paper, we offer a survey on the state of the art of SIM for wireless communications.
A PDD-Inspired Channel Estimation Scheme in NOMA Network
Venue: arXiv cs.NI 2024
Year: 2024

In 5G networks, non-orthogonal multiple access (NOMA) provides a number of benefits by providing uneven power distribution to multiple users at once. On the other hand, effective power allocation, successful successive interference cancellation (SIC), and user fairness all depend on precise channel state information (CSI). Because of dynamic channels, imperfect models, and feedback overhead, CSI prediction in NOMA is difficult. Our aim is to propose a CSI prediction technique based on an ML model that accounts for partially decoded data (PDD), a byproduct of the SIC process. Our proposed technique has been shown to be efficient in handover failure (HOF) prediction and reducing pilot overhead, which is particularly important in 5G. We have shown how machine learning (ML) models may be used to forecast CSI in NOMA handover.
Relative Representations of Latent Spaces enable Efficient Semantic Channel Equalization
Venue: arXiv cs.LG 2024
Year: 2024

In multi-user semantic communication, language mismatche poses a significant challenge when independently trained agents interact. We present a novel semantic equalization algorithm that enables communication between agents with different languages without additional retraining. Our algorithm is based on relative representations, a framework that enables different agents employing different neural network models to have unified representation. It proceeds by projecting the latent vectors of different models into a common space defined relative to a set of data samples called \textit{anchors}, whose number equals the dimension of the resulting space. A communication between different agents translates to a communication of semantic symbols sampled from this relative space. This approach, in addition to aligning the semantic representations of different agents, allows compressing the amount of information being exchanged, by appropriately selecting the number of anchors. Eventually, we introduce a novel anchor selection strategy, which advantageously determines prototypical anchors, capturing the most relevant information for the downstream task. Our numerical results show the effectiveness of the proposed approach allowing seamless communication between agents with radically different models, including differences in terms of neural network architecture and datasets used for initial training.
HVAC-DPT: A Decision Pretrained Transformer for HVAC Control
Venue: arXiv cs.LG 2024
Year: 2024

Building operations consume approximately 40% of global energy, with Heating, Ventilation, and Air Conditioning (HVAC) systems responsible for up to 50% of this consumption. As HVAC energy demands are expected to rise, optimising system efficiency is crucial for reducing future energy use and mitigating climate change. Existing control strategies lack generalisation and require extensive training and data, limiting their rapid deployment across diverse buildings. This paper introduces HVAC-DPT, a Decision-Pretrained Transformer using in-context Reinforcement Learning (RL) for multi-zone HVAC control. HVAC-DPT frames HVAC control as a sequential prediction task, training a causal transformer on interaction histories generated by diverse RL agents. This approach enables HVAC-DPT to refine its policy in-context, without modifying network parameters, allowing for deployment across different buildings without the need for additional training or data collection. HVAC-DPT reduces energy consumption in unseen buildings by 45% compared to the baseline controller, offering a scalable and effective approach to mitigating the increasing environmental impact of HVAC systems.
A Multi-Loss Strategy for Vehicle Trajectory Prediction: Combining Off-Road, Diversity, and Directional Consistency Losses
Venue: arXiv cs.CV 2024
Year: 2024

Trajectory prediction is essential for the safety and efficiency of planning in autonomous vehicles. However, current models often fail to fully capture complex traffic rules and the complete range of potential vehicle movements. Addressing these limitations, this study introduces three novel loss functions: Offroad Loss, Direction Consistency Error, and Diversity Loss. These functions are designed to keep predicted paths within driving area boundaries, aligned with traffic directions, and cover a wider variety of plausible driving scenarios. As all prediction modes should adhere to road rules and conditions, this work overcomes the shortcomings of traditional "winner takes all" training methods by applying the loss functions to all prediction modes. These loss functions not only improve model training but can also serve as metrics for evaluating the realism and diversity of trajectory predictions. Extensive validation on the nuScenes and Argoverse 2 datasets with leading baseline models demonstrates that our approach not only maintains accuracy but significantly improves safety and robustness, reducing offroad errors on average by 47% on original and by 37% on attacked scenes. This work sets a new benchmark for trajectory prediction in autonomous driving, offering substantial improvements in navigating complex environments. Our code is available at https://github.com/vita-epfl/stay-on-track .
Evidence-Based Threat Modeling for ICS
Venue: arXiv cs.CR 2024
Year: 2024

ICS environments are vital to the operation of critical infrastructure such as power grids, water treatment facilities, and manufacturing plants. However, these systems are vulnerable to cyber attacks due to their reliance on interconnected devices and networks, which could lead to catastrophic failures. Therefore, securing these systems from cyber threats becomes paramount. In this context, threat modeling plays an essential role. Despite the advances in threat modeling, the fundamental gap in the state-of-the art is the lack of a systematic methodology for identifying threats in ICS comprehensively. Most threat models in the literature (i) rely on expert knowledge, (ii) only include generic threats such as spoofing, tampering, etc., and (iii) these threats are not comprehensive enough for the systems in question. To overcome these limitations, we propose a novel evidence-based methodology to systematically identify threats based on existing CVE entries of components and their associated fundamental weaknesses in the form of CWE entries - namely, CVE-CWE pairs - and thereby generate a comprehensive threat list. Furthermore, we have implemented our methodology as a ready-to-use tool and have applied it to a typical SCADA system to demonstrate that our methodology is practical and applicable in real-world settings.
A Cross-Corpus Speech Emotion Recognition Method Based on Supervised Contrastive Learning
Venue: arXiv cs.SD 2024
Year: 2024

Research on Speech Emotion Recognition (SER) often faces challenges such as the lack of large-scale public datasets and limited generalization capability when dealing with data from different distributions. To solve this problem, this paper proposes a cross-corpus speech emotion recognition method based on supervised contrast learning. The method employs a two-stage fine-tuning process: first, the self-supervised speech representation model is fine-tuned using supervised contrastive learning on multiple speech emotion datasets; then, the classifier is fine-tuned on the target dataset. The experimental results show that the WavLM-based model achieved unweighted accuracy (UA) of 77.41% on the IEMOCAP dataset and 96.49% on the CASIA dataset, outperforming the state-of-the-art results on the two datasets.
Minimization I.I.D. Prophet Inequality via Extreme Value Theory: A Unified Approach
Venue: arXiv cs.GT 2024
Year: 2024

The I.I.D. Prophet Inequality is a fundamental problem where, given $n$ independent random variables $X_1,\dots,X_n$ drawn from a known distribution $\mathcal{D}$, one has to decide at every step $i$ whether to stop and accept $X_i$ or discard it forever and continue. The goal is to maximize or minimize the selected value and compete against the all-knowing prophet. For maximization, a tight constant-competitive guarantee of $\approx 0.745$ is well-known (Correa et al, 2019), whereas minimization is qualitatively different: the optimal constant is distribution-dependent and can be arbitrarily large (Livanos and Mehta, 2024). In this paper, we provide a novel framework via the lens of Extreme Value Theory to analyze optimal threshold algorithms. We show that the competitive ratio for the minimization setting has a closed form described by a function $Λ$, which depends only on the extreme value index $γ$; in particular, it corresponds to $Λ(γ)$ for $γ\leq 0$. Despite the contrast of maximization and minimization, our framework turns out to be universal and we recover the results of (Kennedy and Kertz, 1991) for maximization as well. Surprisingly, the optimal competitive ratio for maximization is given by the same function $Λ(γ)$, but for $γ\geq 0$. Along the way, we obtain several results on the algorithm and the prophet's objectives from the perspective of extreme value theory, which might be of independent interest. We next study single-threshold algorithms for minimization. Using extreme value theory, we generalize the results of (Livanos and Mehta, 2024) which hold only for special classes of distributions, and obtain poly-logarithmic in $n$ guarantees. Finally, we consider the $k$-multi-unit prophet inequality for minimization and show that there exist constant-competitive single-threshold algorithms when $k \geq \log{n}$.
Cross-Domain Recommendation Meets Large Language Models
Venue: arXiv cs.IR 2024
Year: 2024

Cross-domain recommendation (CDR) has emerged as a promising solution to the cold-start problem, faced by single-domain recommender systems. However, existing CDR models rely on complex neural architectures, large datasets, and significant computational resources, making them less effective in data-scarce scenarios or when simplicity is crucial. In this work, we leverage the reasoning capabilities of large language models (LLMs) and explore their performance in the CDR domain across multiple domain pairs. We introduce two novel prompt designs tailored for CDR and demonstrate that LLMs, when prompted effectively, outperform state-of-the-art CDR baselines across various metrics and domain combinations in the rating prediction and ranking tasks. This work bridges the gap between LLMs and recommendation systems, showcasing their potential as effective cross-domain recommenders.
PDDLFuse: A Tool for Generating Diverse Planning Domains
Venue: arXiv cs.AI 2024
Year: 2024

Various real-world challenges require planning algorithms that can adapt to a broad range of domains. Traditionally, the creation of planning domains has relied heavily on human implementation, which limits the scale and diversity of available domains. While recent advancements have leveraged generative AI technologies such as large language models (LLMs) for domain creation, these efforts have predominantly focused on translating existing domains from natural language descriptions rather than generating novel ones. In contrast, the concept of domain randomization, which has been highly effective in reinforcement learning, enhances performance and generalizability by training on a diverse array of randomized new domains. Inspired by this success, our tool, PDDLFuse, aims to bridge this gap in Planning Domain Definition Language (PDDL). PDDLFuse is designed to generate new, diverse planning domains that can be used to validate new planners or test foundational planning models. We have developed methods to adjust the domain generators parameters to modulate the difficulty of the domains it generates. This adaptability is crucial as existing domain-independent planners often struggle with more complex problems. Initial tests indicate that PDDLFuse efficiently creates intricate and varied domains, representing a significant advancement over traditional domain generation methods and making a contribution towards planning research.
Quantifying the synthetic and real domain gap in aerial scene understanding
Venue: arXiv cs.CV 2024
Year: 2024

Quantifying the gap between synthetic and real-world imagery is essential for improving both transformer-based models - that rely on large volumes of data - and datasets, especially in underexplored domains like aerial scene understanding where the potential impact is significant. This paper introduces a novel methodology for scene complexity assessment using Multi-Model Consensus Metric (MMCM) and depth-based structural metrics, enabling a robust evaluation of perceptual and structural disparities between domains. Our experimental analysis, utilizing real-world (Dronescapes) and synthetic (Skyscenes) datasets, demonstrates that real-world scenes generally exhibit higher consensus among state-of-the-art vision transformers, while synthetic scenes show greater variability and challenge model adaptability. The results underline the inherent complexities and domain gaps, emphasizing the need for enhanced simulation fidelity and model generalization. This work provides critical insights into the interplay between domain characteristics and model performance, offering a pathway for improved domain adaptation strategies in aerial scene understanding.
Creating Hierarchical Dispositions of Needs in an Agent
Venue: arXiv cs.LG 2024
Year: 2024

We present a novel method for learning hierarchical abstractions that prioritize competing objectives, leading to improved global expected rewards. Our approach employs a secondary rewarding agent with multiple scalar outputs, each associated with a distinct level of abstraction. The traditional agent then learns to maximize these outputs in a hierarchical manner, conditioning each level on the maximization of the preceding level. We derive an equation that orders these scalar values and the global reward by priority, inducing a hierarchy of needs that informs goal formation. Experimental results on the Pendulum v1 environment demonstrate superior performance compared to a baseline implementation.We achieved state of the art results.
LeMoLE: LLM-Enhanced Mixture of Linear Experts for Time Series Forecasting
Venue: arXiv cs.LG 2024
Year: 2024

Recent research has shown that large language models (LLMs) can be effectively used for real-world time series forecasting due to their strong natural language understanding capabilities. However, aligning time series into semantic spaces of LLMs comes with high computational costs and inference complexity, particularly for long-range time series generation. Building on recent advancements in using linear models for time series, this paper introduces an LLM-enhanced mixture of linear experts for precise and efficient time series forecasting. This approach involves developing a mixture of linear experts with multiple lookback lengths and a new multimodal fusion mechanism. The use of a mixture of linear experts is efficient due to its simplicity, while the multimodal fusion mechanism adaptively combines multiple linear experts based on the learned features of the text modality from pre-trained large language models. In experiments, we rethink the need to align time series to LLMs by existing time-series large language models and further discuss their efficiency and effectiveness in time series forecasting. Our experimental results show that the proposed LeMoLE model presents lower prediction errors and higher computational efficiency than existing LLM models.
Unpacking the Individual Components of Diffusion Policy
Venue: arXiv cs.LG 2024
Year: 2024

Imitation Learning presents a promising approach for learning generalizable and complex robotic skills. The recently proposed Diffusion Policy generates robot action sequences through a conditional denoising diffusion process, achieving state-of-the-art performance compared to other imitation learning methods. This paper summarizes five key components of Diffusion Policy: 1) observation sequence input; 2) action sequence execution; 3) receding horizon; 4) U-Net or Transformer network architecture; and 5) FiLM conditioning. By conducting experiments across ManiSkill and Adroit benchmarks, this study aims to elucidate the contribution of each component to the success of Diffusion Policy in various scenarios. We hope our findings will provide valuable insights for the application of Diffusion Policy in future research and industry.
Act Now: A Novel Online Forecasting Framework for Large-Scale Streaming Data
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we find that existing online forecasting methods have the following issues: 1) They do not consider the update frequency of streaming data and directly use labels (future signals) to update the model, leading to information leakage. 2) Eliminating information leakage can exacerbate concept drift and online parameter updates can damage prediction accuracy. 3) Leaving out a validation set cuts off the model's continued learning. 4) Existing GPU devices cannot support online learning of large-scale streaming data. To address the above issues, we propose a novel online learning framework, Act-Now, to improve the online prediction on large-scale streaming data. Firstly, we introduce a Random Subgraph Sampling (RSS) algorithm designed to enable efficient model training. Then, we design a Fast Stream Buffer (FSB) and a Slow Stream Buffer (SSB) to update the model online. FSB updates the model immediately with the consistent pseudo- and partial labels to avoid information leakage. SSB updates the model in parallel using complete labels from earlier times. Further, to address concept drift, we propose a Label Decomposition model (Lade) with statistical and normalization flows. Lade forecasts both the statistical variations and the normalized future values of the data, integrating them through a combiner to produce the final predictions. Finally, we propose to perform online updates on the validation set to ensure the consistency of model learning on streaming data. Extensive experiments demonstrate that the proposed Act-Now framework performs well on large-scale streaming data, with an average 28.4% and 19.5% performance improvement, respectively. Experiments can be reproduced via https://github.com/Anoise/Act-Now.
Video Set Distillation: Information Diversification and Temporal Densification
Venue: arXiv cs.CV 2024
Year: 2024

The rapid development of AI models has led to a growing emphasis on enhancing their capabilities for complex input data such as videos. While large-scale video datasets have been introduced to support this growth, the unique challenges of reducing redundancies in video \textbf{sets} have not been explored. Compared to image datasets or individual videos, video \textbf{sets} have a two-layer nested structure, where the outer layer is the collection of individual videos, and the inner layer contains the correlations among frame-level data points to provide temporal information. Video \textbf{sets} have two dimensions of redundancies: within-sample and inter-sample redundancies. Existing methods like key frame selection, dataset pruning or dataset distillation are not addressing the unique challenge of video sets since they aimed at reducing redundancies in only one of the dimensions. In this work, we are the first to study Video Set Distillation, which synthesizes optimized video data by jointly addressing within-sample and inter-sample redundancies. Our Information Diversification and Temporal Densification (IDTD) method jointly reduces redundancies across both dimensions. This is achieved through a Feature Pool and Feature Selectors mechanism to preserve inter-sample diversity, alongside a Temporal Fusor that maintains temporal information density within synthesized videos. Our method achieves state-of-the-art results in Video Dataset Distillation, paving the way for more effective redundancy reduction and efficient AI model training on video datasets.