OpenRaiser/Intern-Atlas huggingface dataset:
ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling
Venue: arXiv cs.CL 2025
Year: 2025

Optimization modeling plays a critical role in the application of Operations Research (OR) tools to address real-world problems, yet they pose challenges and require extensive expertise from OR experts. With the advent of large language models (LLMs), new opportunities have emerged to streamline and automate such task. However, current research predominantly relies on closed-source LLMs such as GPT-4, along with extensive prompt engineering techniques. This reliance stems from the scarcity of high-quality training datasets for optimization modeling, resulting in elevated costs, prolonged processing times, and privacy concerns. To address these challenges, our work is the first to propose a viable path for training open-source LLMs that are capable of optimization modeling and developing solver codes, eventually leading to a superior ability for automating optimization modeling and solving. Particularly, we design the {\sc OR-Instruct}, a semi-automated data synthesis framework for optimization modeling that enables customizable enhancements for specific scenarios or model types. This work also introduces IndustryOR, the first industrial benchmark for evaluating LLMs in solving practical OR problems. We train several 7B-scale open-source LLMs using synthesized data (dubbed ORLMs{https://github.com/Cardinal-Operations/ORLM}), which exhibit significantly enhanced optimization modeling capabilities, achieving competitive performance across the NL4OPT, MAMO, and IndustryOR benchmarks. Additionally, our experiments highlight the potential of scaling law and reinforcement learning to further enhance the performance of ORLMs. The workflows and human-machine interaction paradigms of ORLMs in practical industrial applications are also discussed in the paper.
Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization
Venue: arXiv cs.LG 2025
Year: 2025

Adaptive moment estimation (Adam), as a Stochastic Gradient Descent (SGD) variant, has gained widespread popularity in federated learning (FL) due to its fast convergence. However, federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead compared to federated SGD (FedSGD) algorithms, which arises from the necessity to transmit both local model updates and first and second moment estimates from distributed devices to the centralized server for aggregation. Driven by this issue, we propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates of local model parameters and moment estimates and subsequently upload the sparse representations to the centralized server. To further reduce the communication overhead, the updates of local model parameters and moment estimates incorporate a shared sparse mask (SSM) into the sparsification process, eliminating the need for three separate sparse masks. Theoretically, we develop an upper bound on the divergence between the local model trained by FedAdam-SSM and the desired model trained by centralized Adam, which is related to sparsification error and imbalanced data distribution. By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error. Additionally, we provide convergence bounds for FedAdam-SSM in both convex and non-convex objective function settings, and investigate the impact of local epoch, learning rate and sparsification ratio on the convergence rate of FedAdam-SSM. Experimental results show that FedAdam-SSM outperforms baselines in terms of convergence rate (over 1.1$\times$ faster than the sparse FedAdam baselines) and test accuracy (over 14.5\% ahead of the quantized FedAdam baselines).
Exploring the Escalation of Source Bias in User, Data, and Recommender System Feedback Loop
Venue: arXiv cs.IR 2025
Year: 2025

Recommender systems are essential for information access, allowing users to present their content for recommendation. With the rise of large language models (LLMs), AI-generated content (AIGC), primarily in the form of text, has become a central part of the content ecosystem. As AIGC becomes increasingly prevalent, it is important to understand how it affects the performance and dynamics of recommender systems. To this end, we construct an environment that incorporates AIGC to explore its short-term impact. The results from popular sequential recommendation models reveal that AIGC are ranked higher in the recommender system, reflecting the phenomenon of source bias. To further explore the long-term impact of AIGC, we introduce a feedback loop with realistic simulators. The results show that the model's preference for AIGC increases as the user clicks on AIGC rises and the model trains on simulated click data. This leads to two issues: In the short term, bias toward AIGC encourages LLM-based content creation, increasing AIGC content, and causing unfair traffic distribution. From a long-term perspective, our experiments also show that when AIGC dominates the content ecosystem after a feedback loop, it can lead to a decline in recommendation performance. To address these issues, we propose a debiasing method based on L1-loss optimization to maintain long-term content ecosystem balance. In a real-world environment with AIGC generated by mainstream LLMs, our method ensures a balance between AIGC and human-generated content in the ecosystem. The code and dataset are available at https://github.com/Yuqi-Zhou/Rec_SourceBias.
A Calibration Tool for Refractive Underwater Vision
Venue: arXiv cs.CV 2025
Year: 2025

Many underwater applications rely on vision sensors and require proper camera calibration, i.e. knowing the incoming light ray for each pixel in the image. While for the ideal pinhole camera model all viewing rays intersect in a single 3D point, underwater cameras suffer from - possibly multiple - refractions of light rays at the interfaces of water, glass and air. These changes of direction depend on the position and orientation of the camera inside the water-proof housing, as well as on the shape and properties of the optical window, the port, itself. In recent years explicit models for underwater vision behind common ports such as flat or dome port have been proposed, but the underwater community is still lacking a calibration tool which can determine port parameters through refractive calibration. With this work we provide the first open source implementation of an underwater refractive camera calibration toolbox. It allows end-to-end calibration of underwater vision systems, including camera, stereo and housing calibration for systems with dome or flat ports. The implementation is verified using rendered datasets and real-world experiments.
Rank-Refining Seed Selection Methods for Budget Constrained Influence Maximisation in Multilayer Networks under Linear Threshold Model
Venue: arXiv cs.SI 2025
Year: 2025

The problem of selecting an optimal seed set to maximise influence in networks has been a subject of intense research in recent years. However, despite numerous works addressing this area, it remains a topic that requires further elaboration. Most often, it is considered within the scope of classically defined graphs with a spreading model in the form of Independent Cascades. In this work, we focus on the problem of budget-constrained influence maximisation in multilayer networks using a Linear Threshold Model. Both the graph model and the spreading process we employ are less prevalent in the literature, even though their application allows for a more precise representation of the opinion dynamics in social networks. This paper aims to answer which of the sixteen evaluated seed selection methods is the most effective and how similar they are. Additionally, we focus our analysis on the impact of spreading model parameters, network characteristics, a budget, and the seed selection methods on the diffusion effectiveness in multilayer networks. Our contribution also includes extending several centrality measures and heuristics to the case of such graphs. The results indicate that all the factors mentioned above collectively contribute to the effectiveness of influence maximisation. Moreover, there is no seed selection method which always provides the best results. However, the seeds chosen with VoteRank-based methods (especially with the $v-rnk-m$ variant we propose) usually provide the most extensive diffusion.
Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding
Venue: arXiv cs.AI 2025
Year: 2025

Empowering safe exploration of reinforcement learning (RL) agents during training is a critical challenge towards their deployment in many real-world scenarios. When prior knowledge of the domain or task is unavailable, training RL agents in unknown, black-box environments presents an even greater safety risk. We introduce ADVICE (Adaptive Shielding with a Contrastive Autoencoder), a novel post-shielding technique that distinguishes safe and unsafe features of state-action pairs during training, and uses this knowledge to protect the RL agent from executing actions that yield likely hazardous outcomes. Our comprehensive experimental evaluation against state-of-the-art safe RL exploration techniques shows that ADVICE significantly reduces safety violations (approx 50%) during training, with a competitive outcome reward compared to other techniques.
Metaheuristics and Large Language Models Join Forces: Toward an Integrated Optimization Approach
Venue: arXiv cs.AI 2025
Year: 2025

Since the rise of Large Language Models (LLMs) a couple of years ago, researchers in metaheuristics (MHs) have wondered how to use their power in a beneficial way within their algorithms. This paper introduces a novel approach that leverages LLMs as pattern recognition tools to improve MHs. The resulting hybrid method, tested in the context of a social network-based combinatorial optimization problem, outperforms existing state-of-the-art approaches that combine machine learning with MHs regarding the obtained solution quality. By carefully designing prompts, we demonstrate that the output obtained from LLMs can be used as problem knowledge, leading to improved results. Lastly, we acknowledge LLMs' potential drawbacks and limitations and consider it essential to examine them to advance this type of research further. Our method can be reproduced using a tool available at: https://github.com/camilochs/optipattern.
SpecTra: Enhancing the Code Translation Ability of Language Models by Generating Multi-Modal Specifications
Venue: arXiv cs.SE 2025
Year: 2025

Large language models (LLMs) are increasingly being used for the task of automated code translation, which has important real-world applications. However, most existing approaches use only the source code of a program as an input to an LLM, and do not consider the different kinds of specifications that can be extracted from a program. In this paper, we propose SpecTra, a multi-stage approach that uses a novel self-consistency filter to first generate high-quality static specifications, test cases, and natural language descriptions from a given program, and then uses these along with the source code to improve the quality of LLM-generated translations. We evaluate SpecTra on three code translation tasks - C to Rust, C to Go, and JavaScript to TypeScript - and show that it can enhance the performance of six popular LLMs on these tasks by up to a relative improvement of 46%. We also present a case study on extending this approach to handle translation of a full C project to Rust. Our research suggests that generating high-quality specifications could be a promising and efficient way to improve the performance of LLMs for code translation.
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference
Venue: arXiv cs.LG 2025
Year: 2025

The auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance. While recent research has investigated various speculative decoding techniques for multi-token generation, these efforts have primarily focused on improving processing speed such as throughput. Crucially, they often neglect other metrics essential for real-life deployments, such as memory consumption and training cost. To overcome these limitations, we propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours. Inspired by the human natural language generation process, $PPD$ approximates outputs generated at future timesteps in parallel by using multiple prompt tokens. This approach partially recovers the missing conditional dependency information necessary for multi-token generation, resulting in up to a 28% higher acceptance rate for long-range predictions. Furthermore, we present a hardware-aware dynamic sparse tree technique that adaptively optimizes this decoding scheme to fully leverage the computational capacities on different GPUs. Through extensive experiments across LLMs ranging from MobileLlama to Vicuna-13B on a wide range of benchmarks, our approach demonstrates up to 2.49$\times$ speedup and maintains a minimal runtime memory overhead of just $0.0004$%. More importantly, our parallel prompt decoding can serve as an orthogonal optimization for synergistic integration with existing speculative decoding, showing up to $1.22\times$ further speed improvement. Our code is available at https://github.com/hmarkc/parallel-prompt-decoding.
Navigable Graphs for High-Dimensional Nearest Neighbor Search: Constructions and Limits
Venue: arXiv cs.DS 2025
Year: 2025

There has been significant recent interest in graph-based nearest neighbor search methods, many of which are centered on the construction of navigable graphs over high-dimensional point sets. A graph is navigable if we can successfully move from any starting node to any target node using a greedy routing strategy where we always move to the neighbor that is closest to the destination according to a given distance function. The complete graph is navigable for any point set, but the important question for applications is if sparser graphs can be constructed. While this question is fairly well understood in low-dimensions, we establish some of the first upper and lower bounds for high-dimensional point sets. First, we give a simple and efficient way to construct a navigable graph with average degree $O(\sqrt{n \log n })$ for any set of $n$ points, in any dimension, for any distance function. We compliment this result with a nearly matching lower bound: even under the Euclidean metric in $O(\log n)$ dimensions, a random point set has no navigable graph with average degree $O(n^α)$ for any $α< 1/2$. Our lower bound relies on sharp anti-concentration bounds for binomial random variables, which we use to show that the near-neighborhoods of a set of random points do not overlap significantly, forcing any navigable graph to have many edges.
To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability
Venue: arXiv cs.LG 2025
Year: 2025

The massive computational costs associated with large language model (LLM) pretraining have spurred great interest in reduced-precision floating-point representations to accelerate the process. As a result, the BrainFloat16 (BF16) precision has become the de facto standard for LLM training, with hardware support included in recent generations of accelerators. This trend has gone even further in the latest processors, where FP8 has recently been introduced. However, prior experience with FP16, which was found to be less stable than BF16, raises concerns as to whether FP8, with even fewer bits than FP16, can be a cost-effective option for LLM training. We argue that reduced-precision training schemes must have similar training stability and hyperparameter sensitivities to their higher-precision counterparts in order to be cost-effective. However, we find that currently available methods for FP8 training are not robust enough to allow their use as economical replacements. This prompts us to investigate the stability of reduced-precision LLM training in terms of robustness across random seeds, learning rates, and datasets. To this end, we propose new evaluation techniques and a new metric for quantifying loss landscape sharpness in autoregressive language models. By simulating incremental bit reductions in floating-point representations, we analyze the relationship between representational power and training stability with the intent of aiding future research into the field.
Face processing emerges from object-trained convolutional neural networks
Venue: arXiv cs.CV 2025
Year: 2025

Whether face processing depends on unique, domain-specific neurocognitive mechanisms or domain-general object recognition mechanisms has long been debated. Directly testing these competing hypotheses in humans has proven challenging due to extensive exposure to both faces and objects. Here, we systematically test these hypotheses by capitalizing on recent progress in convolutional neural networks (CNNs) that can be trained without face exposure (i.e., pre-trained weights). Domain-general mechanism accounts posit that face processing can emerge from a neural network without specialized pre-training on faces. Consequently, we trained CNNs solely on objects and tested their ability to recognize and represent faces as well as objects that look like faces (face pareidolia stimuli).... Due to the character limits, for more details see in attached pdf
PILA: Physics-Informed Low Rank Augmentation for Interpretable Earth Observation
Venue: arXiv cs.LG 2025
Year: 2025

Physically meaningful representations are essential for Earth Observation (EO), yet existing physical models are often simplified and incomplete. This leads to discrepancies between simulation and observations that hinder reliable forward model inversion. Common approaches to EO inversion either ignored this incompleteness or relied on case-specific preprocessing. More recent methods use physics-informed autoencoders but depend on auxiliary variables that are difficult to interpret and multiple regularizers that are difficult to balance. We propose Physics-Informed Low-Rank Augmentation (PILA), a framework that augments incomplete physical models using a learnable low-rank residual to improve flexibility, while remaining close to the governing physics. We evaluate PILA on two EO inverse problems involving diverse physical processes: forest radiative transfer inversion from optical remote sensing; and volcanic deformation inversion from Global Navigation Satellite Systems (GNSS) displacement data. Across different domains, PILA yields more accurate and interpretable physical variables. For forest spectral inversion, it improves the separation of tree species and, compared to ground measurements, reduces prediction errors by 40-71\% relative to the state-of-the-art. For volcanic deformation, PILA's recovery of variables captures a major inflation event at the Akutan volcano in 2008, and estimates source depth, volume change, and displacement patterns that are consistent with prior studies that however required substantial additional preprocessing. Finally, we analyse the effects of model rank, observability, and physical priors, and suggest that PILA may offer an effective general pathway for inverting incomplete physical models even beyond the domain of Earth Observation. The code is available at https://github.com/yihshe/PILA.git.
Optimization-based Proof of Useful Work: Framework, Modeling, and Security Analysis
Venue: arXiv cs.DC 2025
Year: 2025

Proof of Work (PoW) has extensively served as the foundation of blockchain's security, consistency, and tamper-resistance. However, long has it been criticized for its tremendous and inefficient utilization of computational power and energy. Proof of useful work (PoUW) can effectively address the blockchain's sustainability issue by redirecting the computing power towards useful tasks instead of meaningless hash puzzles. Optimization problems, whose solutions are often hard to find but easy to verify, present a viable class of useful work for PoUW. However, most existing studies rely on either specific problems or particular algorithms, and there lacks comprehensive security analysis for optimization-based PoUW. Therefore, in this work, we build a generic PoUW framework that solves useful optimization problems for blockchain maintenance. Through modeling and analysis, we identify the security conditions against both selfish and malicious miners. Based on these conditions, we establish a lower bound for the security overhead and uncover the trade-off between useful work efficiency and PoW safeguard. We further offer the reward function design guidelines to guarantee miners' integrity. We also show that the optimization-based PoUW is secure in the presence of malicious miners and derive a necessary condition against long-range attacks. Finally, simulation results are presented to validate our analytical results.
Differentially Private Clustered Federated Learning
Venue: arXiv cs.LG 2025
Year: 2025

Federated learning (FL), which is a decentralized machine learning (ML) approach, often incorporates differential privacy (DP) to provide rigorous data privacy guarantees. Previous works attempted to address high structured data heterogeneity in vanilla FL settings through clustering clients (a.k.a clustered FL), but these methods remain sensitive and prone to errors, further exacerbated by the DP noise. This vulnerability makes the previous methods inappropriate for differentially private FL (DPFL) settings with structured data heterogeneity. To address this gap, we propose an algorithm for differentially private clustered FL, which is robust to the DP noise in the system and identifies the underlying clients' clusters correctly. To this end, we propose to cluster clients based on both their model updates and training loss values. Furthermore, for clustering clients' model updates at the end of the first round, our proposed approach addresses the server's uncertainties by employing large batch sizes as well as Gaussian Mixture Models (GMM) to reduce the impact of DP and stochastic noise and avoid potential clustering errors. This idea is efficient especially in privacy-sensitive scenarios with more DP noise. We provide theoretical analysis to justify our approach and evaluate it across diverse data distributions and privacy budgets. Our experimental results show its effectiveness in addressing large structured data heterogeneity in DPFL.