OpenRaiser/Intern-Atlas huggingface dataset:
On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse
Venue: arXiv cs.LG 2025
Year: 2025

Specifying all desirable properties of a language model is challenging, but certain requirements seem essential. Given samples from an unknown language, the trained model should produce valid strings not seen in training and be expressive enough to capture the language's full richness. Otherwise, outputting invalid strings constitutes "hallucination," and failing to capture the full range leads to "mode collapse." We ask if a language model can meet both requirements. We investigate this within a statistical language generation setting building on Gold and Angluin. Here, the model receives random samples from a distribution over an unknown language K, which belongs to a possibly infinite collection of languages. The goal is to generate unseen strings from K. We say the model generates from K with consistency and breadth if, as training size increases, its output converges to all unseen strings in K. Kleinberg and Mullainathan [KM24] asked if consistency and breadth in language generation are possible. We answer this negatively: for a large class of language models, including next-token prediction models, this is impossible for most collections of candidate languages. This contrasts with [KM24]'s result, showing consistent generation without breadth is possible for any countable collection of languages. Our finding highlights that generation with breadth fundamentally differs from generation without breadth. As a byproduct, we establish near-tight bounds on the number of samples needed for generation with or without breadth. Finally, our results offer hope: consistent generation with breadth is achievable for any countable collection of languages when negative examples (strings outside K) are available alongside positive ones. This suggests that post-training feedback, which encodes negative examples, can be crucial in reducing hallucinations while limiting mode collapse.
How do Machine Learning Models Change?
Venue: arXiv cs.SE 2025
Year: 2025

The proliferation of Machine Learning (ML) models and their open-source implementations has transformed Artificial Intelligence research and applications. Platforms like Hugging Face (HF) enable this evolving ecosystem, yet a large-scale longitudinal study of how these models change is lacking. This study addresses this gap by analyzing over 680,000 commits from 100,000 models and 2,251 releases from 202 of these models on HF using repository mining and longitudinal methods. We apply an extended ML change taxonomy to classify commits and use Bayesian networks to model temporal patterns in commit and release activities. Our findings show that commit activities align with established data science methodologies, such as the Cross-Industry Standard Process for Data Mining (CRISP-DM), emphasizing iterative refinement. Release patterns tend to consolidate significant updates, particularly in model outputs, sharing, and documentation, distinguishing them from granular commits. Furthermore, projects with higher popularity exhibit distinct evolutionary paths, often starting from a more mature baseline with fewer foundational commits in their public history. In contrast, those with intensive collaboration show unique documentation and technical evolution patterns. These insights enhance the understanding of model changes on community platforms and provide valuable guidance for best practices in model maintenance.
Evaluating 5G Networks for U-Space Applications: Insights from Dense Urban Measurement Campaign
Venue: arXiv cs.IT 2025
Year: 2025

Following the burgeoning interest in unmanned aerial vehicles (UAVs) utilization within human-inhabited spaces, critical challenges arise in ensuring reliable, low-latency communication-particularly important given the safety-critical nature of such operations in densely populated urban environments. Therefore, adequate cellular communication capabilities are essential to enable safe and effective operations within the so-called U-Spaces. In this context, this paper investigates the communication performance of cellular-connected UAVs in dense urban environments. In particular, the analysis is based on a comprehensive measurement campaign conducted in the city of Benidorm, Spain-an urban area well known for its high concentration of tall buildings and overall urban density. More specifically, we evaluated key performance indicators (KPIs) related to received signal strength and quality, data rate, and latency across various altitudes, mobile network operators, access technologies, and frequency bands, using multiple types of measurement equipment. The results highlight significant challenges, primarily due to the lack of dedicated planning for aerial coverage and interference management, revealing that current cellular networks may fall short in supporting reliable and ubiquitous UAVs communication. Thus, this paper calls for improved network solutions to ensure the reliability of UAV operations in urban airspace, thereby contributing to the integration of UAVs into urban logistics and mobility.
Squeezed Attention: Accelerating Long Context Length LLM Inference
Venue: arXiv cs.CL 2025
Year: 2025

Emerging Large Language Model (LLM) applications require long input context in order to perform complex tasks like document analysis and code generation. For these long context length applications, the length of the input prompt poses a significant challenge in terms of inference efficiency since the inference costs increase linearly with sequence length. However, for many of these applications, much of the context in the prompt is fixed across different user inputs, thereby providing the opportunity to perform offline optimizations in order to process user inputs quickly, as they are received. We propose Squeezed Attention to accelerate LLM applications where a large portion of the input context is fixed. We first leverage K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. During inference, we compare query tokens from the user input with the centroids to predict which keys from the fixed context are semantically relevant, and then compute exact attention using only the important keys, thereby reducing bandwidth and computational costs. We also present a hierarchical version of our algorithm which can reduce the complexity of attention from linear to logarithmic with respect to the fixed context length. We evaluate our method on long-context benchmarks including LongBench, where it achieves a 3.1$\times$ reduction in KV budget with no noticeable accuracy loss and up to an 8$\times$ reduction with only a 0.5 point accuracy gap for the LLaMA-2-7B-32K, LWM-Text-Chat-1M, and Longchat-7B-v1.5-32K models. Futhermore, we implement kernels for centroid comparison and sparse FlashAttention with important keys, achieving more than 4$\times$ speedups during both the prefill and generation phases for long-context inference. Our code is available at https://github.com/SqueezeAILab/SqueezedAttention.
Misbinding Raw Public Keys to Identities in TLS
Venue: arXiv cs.CR 2025
Year: 2025

The adoption of security protocols such as Transport Layer Security (TLS) has significantly improved the state of traffic encryption and integrity protection on the Internet. Despite rigorous analysis, vulnerabilities continue to emerge, sometimes due to fundamental flaws in the protocol specification. This paper examines the security of TLS when using Raw Public Key (RPK) authentication. This mode has not been as extensively studied as X.509 certificates and Pre-Shared Keys (PSK). We develop a formal model of TLS RPK using applied pi calculus and the ProVerif verification tool, revealing that the RPK mode is susceptible to identity misbinding attacks. Our contributions include formal models of TLS RPK with several mechanisms for binding the endpoint identity to its public key, verification results, practical scenarios demonstrating the misbinding attack, and recommendations for mitigating such vulnerabilities. These findings highlight the need for improved security measures in TLS RPK.
Towards Scalable Newborn Screening: Automated General Movement Assessment in Uncontrolled Settings
Venue: arXiv cs.LG 2025
Year: 2025

General movements (GMs) are spontaneous, coordinated body movements in infants that offer valuable insights into the developing nervous system. Assessed through the Prechtl GM Assessment (GMA), GMs are reliable predictors for neurodevelopmental disorders. However, GMA requires specifically trained clinicians, who are limited in number. To scale up newborn screening, there is a need for an algorithm that can automatically classify GMs from infant video recordings. This data poses challenges, including variability in recording length, device type, and setting, with each video coarsely annotated for overall movement quality. In this work, we introduce a tool for extracting features from these recordings and explore various machine learning techniques for automated GM classification.
Skew-Symmetric Matrix Decompositions on Shared-Memory Architectures
Venue: arXiv cs.MS 2025
Year: 2025

The factorization of skew-symmetric matrices is a critically understudied area of dense linear algebra, particularly in comparison to that of general and symmetric matrices. While some algorithms can be adapted from the symmetric case, the cost of algorithms can be reduced by exploiting skew-symmetry. This work examines the factorization of a skew-symmetric matrix $X$ into its $LTL^\mathrm{T}$ decomposition, where $L$ is unit lower triangular and $T$ is tridiagonal. This is also known as a triangular tridiagonalization. This operation is a means for computing the determinant of $X$ as the square of the (cheaply-computed) Pfaffian of the skew-symmetric tridiagonal matrix $T$ as well as for solving systems of equations, across fields such as quantum electronic structure and machine learning. Its application also often requires pivoting in order to improve numerical stability. We compare and contrast previously-published algorithms with those systematically derived using the FLAME methodology. Performant parallel CPU implementations are achieved by fusing operations at multiple levels in order to reduce memory traffic overhead. A key factor is the employment of new capabilities of the BLAS-like Library Instantiation Software (BLIS) framework, which now supports casting level-2 and level-3 BLAS-like operations by leveraging its gemm and other kernels, hierarchical parallelism, and cache blocking. A prototype, concise C++ API facilitates the translation of correct-by-construction algorithms into correct code. Experiments verify that the resulting implementations greatly exceed the performance of previous work.
LoRA-LiteE: A Computationally Efficient Framework for Chatbot Preference-Tuning
Venue: arXiv cs.CL 2025
Year: 2025

Effective preference tuning is pivotal in aligning chatbot responses with human expectations, enhancing user satisfaction and engagement. Traditional approaches, notably Reinforcement Learning from Human Feedback (RLHF) as employed in advanced models like GPT-4, have demonstrated considerable success in this domain. However, RLHF methods are often computationally intensive and resource-demanding, limiting their scalability and accessibility for broader applications. To address these challenges, this study introduces LoRA-Lite Ensemble (LoRA-LiteE), an innovative framework that combines Supervised Fine-tuning (SFT) with Low-Rank Adaptation (LoRA) and Ensemble Learning techniques to effectively aggregate predictions of lightweight models, which aim to achieve a balance between the performance and computational cost. Utilizing the Chatbot Arena benchmark dataset, we conduct a comprehensive comparative analysis among our LoRA-LiteE model, corresponding base models at different scales, and GPT-4 trained with RLHF. Our empirical results demonstrate that the proposed LoRA-LiteE model achieves comparable performance to un-finetuned GPT-4 and outperforms the single larger-scale models under limited resource constraints. These findings highlight that our LoRA-LiteE provides a feasible and efficient methodology for human preference prediction in chatbot systems, enhancing scalability and accessibility, and thereby broadening the applicability of preference-tuned chatbots in resource-constrained environments.
Steering AI-Driven Personalization of Scientific Text for General Audiences
Venue: arXiv cs.HC 2025
Year: 2025

Digital media platforms (e.g., science blogs) offer opportunities to communicate scientific content to general audiences at scale. However, these audiences vary in their scientific expertise, literacy levels, and personal backgrounds, making effective science communication challenging. To address this challenge, we designed TranSlider, an AI-powered tool that generates personalized translations of scientific text based on individual user profiles (e.g., hobbies, location, and education). Our tool features an interactive slider that allows users to steer the degree of personalization from 0 (weakly relatable) to 100 (strongly relatable), leveraging LLMs to generate the translations with chosen degrees. Through an exploratory study with 15 participants, we investigated both the utility of these AI-personalized translations and how interactive reading features influenced users' understanding and reading experiences. We found that participants who preferred higher degrees of personalization appreciated the relatable and contextual translations, while those who preferred lower degrees valued concise translations with subtle contextualization. Furthermore, participants reported the compounding effect of multiple translations on their understanding of scientific content. Drawing on these findings, we discuss several implications for facilitating science communication and designing steerable interfaces to support human-AI alignment.
DuSEGO: Dual Second-order Equivariant Graph Ordinary Differential Equation
Venue: arXiv cs.LG 2025
Year: 2025

Graph Neural Networks (GNNs) with equivariant properties have achieved significant success in modeling complex dynamic systems and molecular properties. However, their expressiveness ability is limited by: (1) Existing methods often overlook the over-smoothing issue caused by traditional GNN models, as well as the gradient explosion or vanishing problems in deep GNNs. (2) Most models operate on first-order information, neglecting that the real world often consists of second-order systems, which further limits the model's representation capabilities. To address these issues, we propose the \textbf{Du}al \textbf{S}econd-order \textbf{E}quivariant \textbf{G}raph \textbf{O}rdinary Differential Equation (\method{}) for equivariant representation. Specifically, \method{} apply the dual second-order equivariant graph ordinary differential equations (Graph ODEs) on graph embeddings and node coordinates, simultaneously. Theoretically, we first prove that \method{} maintains the equivariant property. Furthermore, we provide theoretical insights showing that \method{} effectively alleviates the over-smoothing problem in both feature representation and coordinate update. Additionally, we demonstrate that the proposed \method{} mitigates the exploding and vanishing gradients problem, facilitating the training of deep multi-layer GNNs. Extensive experiments on benchmark datasets validate the superiority of the proposed \method{} compared to baselines.
DeepMedcast: A Deep Learning Method for Generating Intermediate Weather Forecasts among Multiple NWP Models
Venue: arXiv cs.LG 2025
Year: 2025

Numerical weather prediction (NWP) centers around the world operate a variety of NWP models. In addition, recent advances in AI-driven NWP models have further increased the availability of NWP outputs. While this expansion holds the potential to improve forecast accuracy, it raises a critical question: which prediction is the most plausible? If the NWP models have comparable accuracy, it is impossible to determine in advance which one is the best. Traditional approaches, such as ensemble or weighted averaging, combine multiple NWP outputs to produce a single forecast with improved accuracy. However, they often result in meteorologically unrealistic and uninterpretable outputs, such as the splitting of tropical cyclone centers or frontal boundaries into multiple distinct systems. To address this issue, we propose DeepMedcast, a deep learning method that generates intermediate forecasts between two or more NWP outputs. Unlike averaging, DeepMedcast provides predictions in which meteorologically significant features -- such as the locations of tropical cyclones, extratropical cyclones, fronts, and shear lines -- approximately align with the arithmetic mean of the corresponding features predicted by the input NWP models, without distorting meteorological structures. We demonstrate the capability of DeepMedcast through case studies and verification results, showing that it produces realistic and interpretable forecasts with higher accuracy than the input NWP models. By providing plausible intermediate forecasts, DeepMedcast can significantly contribute to the efficiency and standardization of operational forecasting tasks, including general, marine, and aviation forecasts.
SoK: DAG-based Consensus Protocols
Venue: arXiv cs.DC 2025
Year: 2025

This paper is a Systematization of Knowledge (SoK) on Directed Acyclic Graph (DAG)-based consensus protocols, analyzing their performance and trade-offs within the framework of consistency, availability, and partition tolerance inspired by the CAP theorem. We classify DAG-based consensus protocols into availability-focused and consistency-focused categories, exploring their design principles, core functionalities, and associated trade-offs. Furthermore, we examine key properties, attack vectors, and recent developments, providing insights into security, scalability, and fairness challenges. Finally, we identify research gaps and outline directions for advancing DAG-based consensus mechanisms.
Constant Workspace Algorithms for Computing Relative Hulls in the Plane
Venue: arXiv cs.CG 2025
Year: 2025

Constant workspace algorithms use a constant number of words in addition to the read-only input to the algorithm. In this paper, we devise algorithms to efficiently compute relative hulls in the plane using a constant workspace. Specifically, we devise algorithms for the following three problems: (i) Given two simple polygons P and Q with P \subset Q, compute a simple polygon P' with a perimeter of minimum length such that P \subseteq P' \subseteq Q. (ii) Given two simple polygons P and Q such that Q does not intersect the relative interior of P but it does intersect the relative interior of the convex hull of P, compute a weakly simple polygon P' with a perimeter of minimum length such that P \subseteq P', the convex hull of P contains P', and P' does not intersect the relative interior of Q. (iii) Given a set S of points located in a simple polygon P, compute a weakly simple polygon P' with a perimeter of minimum length such that P' \subseteq P and P' contains all the points in S. To our knowledge, no prior work devised algorithms to compute relative hulls using a constant workspace, and this work is the first such attempt.
PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapse
Venue: arXiv cs.LG 2025
Year: 2025

Self-supervised learning (SSL) is a data-driven learning approach that utilizes the innate structure of the data to guide the learning process. In contrast to supervised learning, which depends on external labels, SSL utilizes the inherent characteristics of the data to produce its own supervisory signal. However, one frequent issue with SSL methods is representation collapse, where the model outputs a constant input-invariant feature representation. This issue hinders the potential application of SSL methods to new data modalities, as trying to avoid representation collapse wastes researchers' time and effort. This paper introduces a novel SSL algorithm for time-series data called Prediction of Functionals from Masked Latents (PFML). Instead of predicting masked input signals or their latent representations directly, PFML operates by predicting statistical functionals of the input signal corresponding to masked embeddings, given a sequence of unmasked embeddings. The algorithm is designed to avoid representation collapse, rendering it straightforwardly applicable to different time-series data domains, such as novel sensor modalities in clinical data. We demonstrate the effectiveness of PFML through complex, real-life classification tasks across three different data modalities: infant posture and movement classification from multi-sensor inertial measurement unit data, emotion recognition from speech data, and sleep stage classification from EEG data. The results show that PFML is superior to a conceptually similar SSL method and a contrastive learning-based SSL method. Additionally, PFML is on par with the current state-of-the-art SSL method, while also being conceptually simpler and without suffering from representation collapse.
The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
Venue: arXiv cs.LG 2025
Year: 2025

Visual Reinforcement Learning (RL) methods often require extensive amounts of data. As opposed to model-free RL, model-based RL (MBRL) offers a potential solution with efficient data utilization through planning. Additionally, RL lacks generalization capabilities for real-world tasks. Prior work has shown that incorporating pre-trained visual representations (PVRs) enhances sample efficiency and generalization. While PVRs have been extensively studied in the context of model-free RL, their potential in MBRL remains largely unexplored. In this paper, we benchmark a set of PVRs on challenging control tasks in a model-based RL setting. We investigate the data efficiency, generalization capabilities, and the impact of different properties of PVRs on the performance of model-based agents. Our results, perhaps surprisingly, reveal that for MBRL current PVRs are not more sample efficient than learning representations from scratch, and that they do not generalize better to out-of-distribution (OOD) settings. To explain this, we analyze the quality of the trained dynamics model. Furthermore, we show that data diversity and network architecture are the most important contributors to OOD generalization performance.