GenAI4ELab/papercli-papers huggingface dataset:
Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression
Authors: Michael Crawshaw; Blake Woodworth; Mingrui Liu
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: We analyze two variants of Local Gradient Descent applied to distributed logistic regression with heterogeneous, separable data and show convergence at the rate $O(1/KR)$ for $K$ local steps and sufficiently large $R$ communication rounds. In contrast, all existing convergence guarantees for Local GD applied to any problem are at least $\Omega(1/R)$, meaning they fail to show the benefit of local updates. The key to our improved guarantee is showing progress on the logistic regression objective when using a large stepsize $\eta \gg 1/K$, whereas prior analysis depends on $\eta \leq 1/K$.
MamKO: Mamba-based Koopman operator for modeling and predictive control
Authors: ZHAOYANG LI; Minghao Han; Xunyuan Yin
Venue: ICLR 2025 (ICLR 2025 Spotlight)

Abstract: The Koopman theory, which enables the transformation of nonlinear systems into linear representations, is a powerful and efficient tool to model and control nonlinear systems. However, the ability of the Koopman operator to model complex systems, particularly time-varying systems, is limited by the fixed linear state-space representation. To address the limitation, the large language model, Mamba, is considered a promising strategy for enhancing modeling capabilities while preserving the linear state-space structure.
In this paper, we propose a new framework, the Mamba-based Koopman operator (MamKO), which provides enhanced model prediction capability and adaptability, as compared to Koopman models with constant Koopman operators. Inspired by the Mamba structure, MamKO generates Koopman operators from online data; this enables the model to effectively capture the dynamic behaviors of the nonlinear system over time. A model predictive control system is then developed based on the proposed MamKO model. The modeling and control performance of the proposed method is evaluated through experiments on benchmark time-invariant and time-varying systems. The experimental results demonstrate the superiority of the proposed approach. Additionally, we perform ablation experiments to test the effectiveness of individual components of MamKO. This approach unlocks new possibilities for integrating large language models with control frameworks, and it achieves a good balance between advanced modeling capabilities and real-time control implementation efficiency.
Robustness Auditing for Linear Regression: To Singularity and Beyond
Authors: Ittai Rubinstein; Samuel B. Hopkins
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: It has recently been discovered that the conclusions of many highly influential econometrics studies can be overturned by removing a very small fraction of their samples (often less than $0.5\%$). These conclusions are typically based on the results of one or more Ordinary Least Squares (OLS) regressions, raising the question: given a dataset, can we certify the robustness of an OLS fit on this dataset to the removal of a given number of samples?

Brute-force techniques quickly break down even on small datasets. Existing approaches which go beyond brute force either can only find candidate small subsets to remove (but cannot certify their non-existence) [BGM20, KZC21], are computationally intractable beyond low dimensional settings [MR22], or require very strong assumptions on the data distribution and too many samples to give reasonable bounds in practice [BP21, FH23]. 

We present an efficient algorithm for certifying the robustness of linear regressions to removals of samples. We implement our algorithm and run it on several landmark econometrics datasets with hundreds of dimensions and tens of thousands of samples, giving the first non-trivial certificates of robustness to sample removal for datasets of dimension $4$ or greater. We prove that under distributional assumptions on a dataset, the bounds produced by our algorithm are tight up to a $1 + o(1)$ multiplicative factor.
Can Transformers Do Enumerative Geometry?
Authors: Baran Hashemi; Roderic Guigo Corominas; Alessandro Giacchetto
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: We introduce a Transformer-based approach to computational enumerative geometry, specifically targeting the computation of $\psi$-class intersection numbers on the moduli space of curves. Traditional methods for calculating these numbers suffer from factorial computational complexity, making them impractical to use. By reformulating the problem as a continuous optimization task, we compute intersection numbers across a wide value range from $10^{-45}$ to $10^{45}$. To capture the recursive nature inherent in these intersection numbers, we propose the Dynamic Range Activator (DRA), a new activation function that enhances the Transformer's ability to model recursive patterns and handle severe heteroscedasticity. Given precision requirements for computing the intersections, we quantify the uncertainty of the predictions using Conformal Prediction with a dynamic sliding window adaptive to the partitions of equivalent number of marked points. To the best of our knowledge, there has been no prior work on modeling recursive functions with such a high-variance and factorial growth. Beyond simply computing intersection numbers, we explore the enumerative "world-model" of Transformers. Our interpretability analysis reveals that the network is implicitly modeling the Virasoro constraints in a purely data-driven manner. Moreover, through abductive hypothesis testing, probing, and causal inference, we uncover evidence of an emergent internal representation of the the large-genus asymptotic of $\psi$-class intersection numbers. These findings suggest that the network internalizes the parameters of the asymptotic closed-form and the polynomiality phenomenon of $\psi$-class intersection numbers in a non-linear manner.
Large Language Models Assume People are More Rational than We Really are
Authors: Ryan Liu; Jiayi Geng; Joshua Peterson; Ilia Sucholutsky; Thomas L. Griffiths
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: In order for AI systems to communicate effectively with people, they must understand how we make decisions. However, people's decisions are not always rational, so the implicit internal models of human decision-making in Large Language Models (LLMs) must account for this. Previous empirical evidence seems to suggest that these implicit models are accurate --- LLMs offer believable proxies of human behavior, acting how we expect humans would in everyday interactions. However, by comparing LLM behavior and predictions to a large dataset of human decisions, we find that this is actually not the case: when both simulating and predicting people's choices, a suite of cutting-edge LLMs (GPT-4o \& 4-Turbo, Llama-3-8B \& 70B, Claude 3 Opus) assume that people are more rational than we really are. Specifically, these models deviate from human behavior and align more closely with a classic model of rational choice --- expected value theory. Interestingly, people also tend to assume that other people are rational when interpreting their behavior. As a consequence, when we compare the inferences that LLMs and people draw from the decisions of others using another psychological dataset, we find that these inferences are highly correlated. Thus, the implicit decision-making models of LLMs appear to be aligned with the human expectation that other people will act rationally, rather than with how people actually act.
Small-to-Large Generalization: Training Data Influences Models Consistently Across Scale
Authors: Alaa Khaddaj; Logan Engstrom; Aleksander Madry
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Choice of training data distribution greatly influences model behavior. Yet, in
large-scale settings, precisely characterizing *how* changes in training
data affects predictions is often difficult due to model training costs. Current
practice is to instead extrapolate from scaled down, inexpensive-to-train proxy
models. However, changes in data do not influence smaller and larger models
identically. Therefore, understanding how choice of data affects large-scale
models raises the question: how does training data distribution influence model
behavior across compute scale? We find that small- and large-scale language
model predictions (generally) *do* highly correlate across choice of
training data. Equipped with these findings, we characterize how proxy scale
affects effectiveness in two downstream proxy model applications: data
attribution and dataset selection.
LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation models
Authors: Ziqi Lu; Heng Yang; Danfei Xu; Boyi Li; Boris Ivanovic; Marco Pavone; Yue Wang
Venue: ICLR 2025 (ICLR 2025 Spotlight)

Abstract: Emerging 3D geometric foundation models, such as DUSt3R, offer a promising approach for in-the-wild 3D vision tasks.
However, due to the high-dimensional nature of the problem space and scarcity of high-quality 3D data,
these pre-trained models still struggle to generalize to many challenging circumstances,
such as limited view overlap or low lighting.
To address this, we propose LoRA3D, an efficient self-calibration pipeline to *specialize* the pre-trained models to target scenes using their own multi-view predictions.
Taking sparse RGB images as input, we leverage robust optimization techniques to refine multi-view predictions and align them into a global coordinate frame.
In particular, we incorporate prediction confidence into the geometric optimization process, 
automatically re-weighting the confidence to better reflect point estimation accuracy. 
We use the calibrated confidence to generate high-quality pseudo labels for the calibrating views and fine-tune the models using low-rank adaptation (LoRA) on the pseudo-labeled data.
Our method does not require any external priors or manual labels. It completes the self-calibration process on a **single standard GPU within just 5 minutes**.
Each low-rank adapter requires only **18MB** of storage. 
We evaluated our method on **more than 160 scenes** from the Replica, TUM and Waymo Open datasets,
achieving up to **88\% performance improvement** on 3D reconstruction, multi-view pose estimation and novel-view rendering.
For more details, please visit our project page at https://520xyxyzq.github.io/lora3d/.
Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation
Authors: Linda He; Jue WANG; Maurice Weber; Shang Zhu; Ben Athiwaratkun; Ce Zhang
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Large Language Models (LLMs) struggle with long-context reasoning, not only due to the quadratic scaling of computational complexity with sequence length but also because of the scarcity and expense of annotating long-context data. There has been barely any open-source work that systematically ablates long-context data, nor is there any openly available instruction tuning dataset with contexts surpassing 100K tokens. To bridge this gap, we introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of LLMs while preserving their general task performance. Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of available real-world data, which effectively addresses the scarcity of raw long-context data. 
Through a step-by-step rotary position embedding (RoPE) scaling training strategy, we demonstrate that our model, with a context length of up to 1M tokens, performs well on the RULER benchmark and InfiniteBench and maintains robust performance on general language tasks.
Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric
Authors: Toshimitsu Uesaka; Taiji Suzuki; Yuhta Takida; Chieh-Hsin Lai; Naoki Murata; Yuki Mitsufuji
Venue: ICLR 2025 (ICLR 2025 Spotlight)

Abstract: In typical multimodal contrastive learning, such as CLIP, encoders produce one
point in the latent representation space for each input. However, one-point representation
has difficulty in capturing the relationship and the similarity structure of a
huge amount of instances in the real world. For richer classes of the similarity, we
propose the use of weighted point sets, namely, sets of pairs of weight and vector,
as representations of instances. In this work, we theoretically show the benefit
of our proposed method through a new understanding of the contrastive loss of
CLIP, which we call symmetric InfoNCE. We clarify that the optimal similarity
that minimizes symmetric InfoNCE is the pointwise mutual information, and show
an upper bound of excess risk on downstream classification tasks of representations
that achieve the optimal similarity. In addition, we show that our proposed
similarity based on weighted point sets consistently achieves the optimal similarity.
To verify the effectiveness of our proposed method, we demonstrate pretraining of
text-image representation models and classification tasks on common benchmarks.
EqNIO: Subequivariant Neural Inertial Odometry
Authors: Royina Karegoudra Jayanth; Yinshuang Xu; Ziyun Wang; Evangelos Chatzipantazis; Kostas Daniilidis; Daniel Gehrig
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Neural network-based odometry using accelerometer and gyroscope readings from a single IMU can achieve robust, and low-drift localization capabilities, through the use of _neural displacement priors (NDPs)_. These priors learn to produce denoised displacement measurements but need to ignore data variations due to specific IMU mount orientation and motion directions, hindering generalization.
This work introduces EqNIO, which addresses this challenge with _canonical displacement priors_, i.e., priors that are invariant to the orientation of the gravity-aligned frame in which the IMU data is expressed. We train such priors on IMU measurements, that are mapped into a learnable canonical frame, which is uniquely defined via three axes: the first is gravity, making the frame gravity aligned, while the second and third are predicted from IMU data.  The outputs (displacement and covariance) are mapped back to the original gravity-aligned frame. To maximize generalization, we find that these learnable frames must transform equivariantly with global gravity-preserving roto-reflections from the subgroup $O_g(3)\subset O(3)$, acting on the trajectory, rendering the NDP $O(3)$-_subequivariant_. We tailor specific linear, convolutional, and non-linear layers that commute with the actions of the group. Moreover, we introduce a bijective decomposition of angular rates into vectors that transform similarly to accelerations, allowing us to leverage both measurement types. Natively, angular rates would need to be inverted upon reflection, unlike acceleration, which hinders their joint processing. We highlight EqNIO's flexibility and generalization capabilities by applying it to both filter-based (TLIO), and end-to-end (RONIN) architectures, and outperforming existing methods that use _soft equivariance from auxiliary losses or data augmentation on various datasets. We believe this work paves the way for low-drift and generalizable neural inertial odometry on edge devices. The project details and code can be found at [https://github.com/RoyinaJayanth/EqNIO](https://github.com/RoyinaJayanth/EqNIO).
MANTRA: The Manifold Triangulations Assemblage
Authors: Rubén Ballester; Ernst Röell; Daniel Bin Schmid; Mathieu Alain; Sergio Escalera; Carles Casacuberta; Bastian Rieck
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: The rising interest in leveraging higher-order interactions present in complex systems has
led to a surge in more expressive models exploiting higher-order structures in the data,
especially in topological deep learning (TDL), which designs neural networks on higher-order domains such as simplicial complexes. However, progress in this field is hindered
by the scarcity of datasets for benchmarking these architectures. To address this gap, we
introduce MANTRA, the first large-scale, diverse, and intrinsically higher-order dataset for
benchmarking higher-order models, comprising over 43,000 and 250,000 triangulations
of surfaces and three-dimensional manifolds, respectively. With MANTRA, we assess
several graph- and simplicial complex-based models on three topological classification
tasks. We demonstrate that while simplicial complex-based neural networks generally
outperform their graph-based counterparts in capturing simple topological invariants, they
also struggle, suggesting a rethink of TDL. Thus, MANTRA serves as a benchmark for
assessing and advancing topological methods, paving the way towards more effective
higher-order models.
Aligning Visual Contrastive learning models via Preference Optimization
Authors: Amirabbas Afzali; Borna khodabandeh; Ali Rasekh; Mahyar JafariNodeh; Sepehr Kazemi Ranjbar; Simon Gottschalk
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Contrastive learning models have demonstrated impressive abilities to capture semantic similarities by aligning representations in the embedding space. However, their performance can be limited by the quality of the training data and its inherent biases. While Preference Optimization (PO) methods such as Reinforcement  Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have been applied to align generative models with human preferences, their use in contrastive learning has yet to be explored. This paper introduces a novel method for training contrastive learning models using different PO methods to break down complex concepts. Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task. In particular, we focus on enhancing model robustness against typographic attacks and inductive biases, commonly seen in contrastive vision-language models like CLIP. Our experiments demonstrate that models trained using PO outperform standard contrastive learning techniques while retaining their ability to handle adversarial challenges and maintain accuracy on other downstream tasks. This makes our method well-suited for tasks requiring fairness, robustness, and alignment with specific preferences. We evaluate our method for tackling typographic attacks on images and explore its ability to disentangle gender concepts and mitigate gender bias, showcasing the versatility of our approach.
Exponential Topology-enabled Scalable Communication in Multi-agent Reinforcement Learning
Authors: Xinran Li; Xiaolu Wang; Chenjia Bai; Jun Zhang
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: In cooperative multi-agent reinforcement learning (MARL), well-designed communication protocols can effectively facilitate consensus among agents, thereby enhancing task performance. Moreover, in large-scale multi-agent systems commonly found in real-world applications, effective communication plays an even more critical role due to the escalated challenge of partial observability compared to smaller-scale setups. In this work, we endeavor to develop a scalable communication protocol for MARL. Unlike previous methods that focus on selecting optimal pairwise communication links—a task that becomes increasingly complex as the number of agents grows—we adopt a global perspective on communication topology design. Specifically, we propose utilizing the exponential topology to enable rapid information dissemination among agents by leveraging its small-diameter and small-size properties. This approach leads to a scalable communication protocol, named ExpoComm. To fully unlock the potential of exponential graphs as communication topologies, we employ memory-based message processors and auxiliary tasks to ground messages, ensuring that they reflect global information and benefit decision-making. Extensive experiments on large-scale cooperative benchmarks, including MAgent and Infrastructure Management Planning, demonstrate the superior performance and robust zero-shot transferability of ExpoComm compared to existing communication strategies. The
code is publicly available at [https://github.com/LXXXXR/ExpoComm](https://github.com/LXXXXR/ExpoComm).
Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?
Authors: Seth Aycock; David Stap; Di Wu; Christof Monz; Khalil Sima'an
Venue: ICLR 2025 (ICLR 2025 Spotlight)

Abstract: Extremely low-resource (XLR) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books. Machine Translation from One Book (Tanzer et al., 2024) suggests that prompting long-context LLMs with one grammar book enables English–Kalamang translation, an XLR language unseen by LLMs—a noteworthy case of linguistics helping an NLP task. We investigate the source of this translation ability, finding almost all improvements stem from the book’s parallel examples rather than its grammatical explanations. We find similar results for Nepali and Guarani, seen low-resource languages, and we achieve performance comparable to an LLM with a grammar book by simply fine-tuning an encoder-decoder translation model. We then investigate where grammar books help by testing two linguistic tasks, grammaticality judgment and gloss prediction, and we explore what kind of grammatical knowledge helps by introducing a typological feature prompt that achieves leading results on these more relevant tasks. We thus emphasise the importance of task-appropriate data for XLR languages: parallel examples for translation, and grammatical data for linguistic tasks. As we find no evidence that long-context LLMs can make effective use of grammatical explanations for XLR translation, we conclude data collection for multilingual XLR tasks such as translation is best focused on parallel data over linguistic description.
Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
Authors: Xiang Yue; Yueqi Song; Akari Asai; Seungone Kim; Jean de Dieu Nyandwi; Simran Khanuja; Anjali Kantharuban; Lintang Sutawika; Sathyanarayanan Ramamoorthy; Graham Neubig
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Despite recent advances in multimodal large language models (MLLMs), their development has predominantly focused on English- and western-centric datasets and tasks, leaving most of the world's languages and diverse cultural contexts underrepresented.  
This paper introduces PANGEA, a multilingual multimodal LLM trained on PANGEAINS, a diverse 6M instruction dataset spanning 39 languages. PANGEAINS features: 1) high-quality English instructions, 2) carefully machine-translated instructions, and 3) culturally relevant multimodal tasks to ensure cross-cultural coverage. 
To rigorously assess models' capabilities, we introduce PANGEABENCH, a holistic evaluation suite encompassing 14 datasets covering 47 languages. 
Results show that PANGEA significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. Ablation studies further reveal the importance of English data proportions, language popularity, and the number of multimodal training samples on overall performance.  We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs, promoting equity and accessibility across a broader linguistic and cultural spectrum.