dataset-review/dataset-review huggingface dataset:
Simplifying DINO via Coding Rate Regularization
ICML 2025

Simplifying DINO via Coding Rate Regularization
Ziyang Wu 1 Jingyuan Zhang 2 Druv Pai 1 Xudong Wang 1
Chandan Singh 3 Jianwei Yang 3 Jianfeng Gao 3 Yi Ma 1 2 4
Abstract
DINO and DINOv2 are two model families be-
ing widely used to learn representations from
unlabeled imagery data at large scales. Their
learned representations often enable state-of-the-
art performance for downstream tasks, such as
image classification and segmentation. However,
they employ many empirically motivated design
choices and their training pipelines are highly
complex and unstable — many hyperparameters
need to be carefully tuned to ensure that the rep-
resentations do not collapse — which poses con-
siderable difficulty to improving them or adapting
them to new domains. In this work, we posit
that we can remove most such-motivated idiosyn-
crasies in the pre-training pipelines, and only need
to add an explicit coding rate term in the loss func-
tion to avoid collapse of the representations. As
a result, we obtain highly simplified variants of
the DINO and DINOv2 which we call SimDINO
and SimDINOv2, respectively. Remarkably, these
simplified models are more robust to different de-
sign choices, such as network architecture and
hyperparameters, and they learn even higher-
quality representations, measured by performance
on downstream tasks, offering a Pareto improve-
ment over the corresponding DINO and DINOv2
models. This work highlights the potential of
using simplifying design principles to improve
the empirical practice of deep learning. Code
and model checkpoints are available at https:
//github.com/RobinWu218/SimDINO.
1. Introduction
Self-supervised learning (SSL) is the toolkit of choice to
learn representations for large datasets of unlabeled images
1UC Berkeley 2TranscEngram 3Microsoft Research 4HKU.
Correspondence to: Ziyang Wu <zywu@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author...
Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders
ICML 2025

Aggregation of Dependent Expert Distributions
in Multimodal Variational Autoencoders
Rogelio A Mancisidor 1 Robert Jenssen 2 3 4 Shujian Yu 2 5 Michael Kampffmeyer 2 4
Abstract
Multimodal learning with variational autoen-
coders (VAEs) requires estimating joint distri-
butions to evaluate the evidence lower bound
(ELBO). Current methods, the product and mix-
ture of experts, aggregate single-modality dis-
tributions assuming independence for simplicity,
which is an overoptimistic assumption. This re-
search introduces a novel methodology for ag-
gregating single-modality distributions by ex-
ploiting the principle of consensus of dependent
experts (CoDE), which circumvents the afore-
mentioned assumption.
Utilizing the CoDE
method, we propose a novel ELBO that approx-
imates the joint likelihood of the multimodal
data by learning the contribution of each sub-
set of modalities.
The resulting CoDE-VAE
model demonstrates better performance in terms
of balancing the trade-off between generative
coherence and generative quality, as well as
generating more precise log-likelihood estima-
tions.
CoDE-VAE further minimizes the gen-
erative quality gap as the number of modali-
ties increases.
In certain cases, it reaches a
generative quality similar to that of unimodal
VAEs, which is a desirable property that is
lacking in most current methods.
Finally, the
classiﬁcation accuracy achieved by CoDE-VAE
is comparable to that of state-of-the-art multi-
modal VAE models. CoDE-VAE is available at:
https://github.com/rogelioamancisidor/codevae.
1Department of Data Science, BI Norwegian Business School,
Oslo, Norway 2Department of Physics and Technology, UiT The
Arctic University of Norway, Tromsø, Norway 3Department of
Computer Science, University of Copenhagen, Copenhagen, Den-
mark 4Dept. BAMJO, Norwegian Computing Center, Oslo, Nor-
way 5Department of Computer Science, Vrije Universiteit Ams-
terdam, Amsterdam, Netherlands. Correspondence to: Rogelio A
Mancisidor <rogelio.a.manc...
Adversarially Robust Anomaly Detection through Spurious Negative Pair Mitigation
ICLR 2025

Published as a conference paper at ICLR 2025
MITIGATING SPURIOUS NEGATIVE PAIRS FOR ROBUST
INDUSTRIAL ANOMALY DETECTION
Hossein Mirzaei1, Mojtaba Nafez2, Jafar Habibi2, Mohammad Sabokrou3, and Mohammad
Hossein Rohban2
1École Polytechnique Fédérale de Lausanne (EPFL), Switzerland
2Sharif University of Technology, Iran
3Okinawa Institute of Science and Technology, Japan
rohban@sharif.edu, hossein.mirzaeisadeghlou@epfl.ch
ABSTRACT
Despite significant progress in Anomaly Detection (AD), the robustness of existing
detection methods against adversarial attacks remains a challenge, compromising
their reliability in critical real-world applications such as autonomous driving. This
issue primarily arises from the AD setup, which assumes that training data is
limited to a group of unlabeled normal samples, making the detectors vulnerable to
adversarial anomaly samples during testing. Additionally, implementing adversar-
ial training as a safeguard encounters difficulties, such as formulating an effective
objective function without access to labels. An ideal objective function for adver-
sarial training in AD should promote strong perturbations both within and between
the normal and anomaly groups to maximize margin between normal and anomaly
distribution. To address these issues, we first propose crafting a pseudo-anomaly
group derived from normal group samples. Then, we demonstrate that adversarial
training with contrastive loss could serve as an ideal objective function, as it cre-
ates both inter- and intra-group perturbations. However, we notice that spurious
negative pairs compromise the conventional contrastive loss to achieve robust AD.
Spurious negative pairs are those that should be closely mapped but are erroneously
separated. These pairs introduce noise and misguide the direction of inter-group
adversarial perturbations. To overcome the effect of spurious negative pairs, we
define opposite pairs and adversarially pull them apart to strengthen inter-group
perturbati...
Diversity-Aware Policy Optimization for Large Language Model Reasoning
NeurIPS 2025

Diversity-Aware Policy Optimization for Large
Language Model Reasoning
Jian Yao1, Ran Cheng1,2,3 ∗, Xingyu Wu1, Jibin Wu1,2, Kay Chen Tan1
1 Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University
2 Department of Computing, The Hong Kong Polytechnic University
3 The Hong Kong Polytechnic University Shenzhen Research Institute, Shenzhen, China
nigel97.yao@connect.polyu.hk, {ran-peter.cheng, xingy.wu, jibin.wu, kctan}@polyu.edu.hk
Abstract
The reasoning capabilities of large language models (LLMs) have advanced rapidly,
particularly following the release of DeepSeek-R1, which has inspired a surge of
research into data quality and reinforcement learning (RL) algorithms. Despite the
pivotal role diversity plays in RL, its influence on LLM reasoning remains largely
underexplored. To bridge this gap, this work presents a systematic investigation
into the impact of diversity in RL-based training for LLM reasoning, and proposes
a novel diversity-aware policy optimization method. Across evaluations on 12
LLMs, we observe a strong positive correlation between the solution diversity
and Potential@k (a novel metric quantifying an LLM’s reasoning potential) in
high-performing models. This finding motivates our method to explicitly promote
diversity during RL training. Specifically, we design a token-level diversity and
reformulate it into a practical objective, then we selectively apply it to positive
samples. Integrated into the R1-zero training framework, our method achieves
a 3.5% average improvement across four mathematical reasoning benchmarks,
while generating more diverse and robust solutions. The code is available at
https://github.com/nigelyaoj/R1_zero_Div.
1
Introduction
Recently, the reasoning capabilities of large language models (LLMs) have made remarkable progress,
with significant improvements showcased by OpenAI-o1 [35], DeepSeek-R1 [13], and Kimi-k1.5
[43]. Among these advancements, two key innovations have contributed signif...
AtlasD: Automatic Local Symmetry Discovery
ICML 2025

AtlasD: Automatic Local Symmetry Discovery
Manu Bhat 1 Jonghyun Park 1 Jianke Yang 1 Nima Dehmamy 2 Robin Walters 3 Rose Yu 1
Abstract
Existing symmetry discovery methods predomi-
nantly focus on global transformations across the
entire system or space, but they fail to consider
the symmetries in local neighborhoods. This may
result in the reported symmetry group being a
misrepresentation of the true symmetry. In this
paper, we formalize the notion of local symme-
try as atlas equivariance. Our proposed pipeline,
automatic local symmetry discovery (AtlasD),
recovers the local symmetries of a function by
training local predictor networks and then learn-
ing a Lie group basis to which the predictors are
equivariant. We demonstrate AtlasD is capable
of discovering local symmetry groups with multi-
ple connected components in top-quark tagging
and partial differential equation experiments. The
discovered local symmetry is shown to be a use-
ful inductive bias that improves the performance
of downstream tasks in climate segmentation and
vision tasks. Our code is publicly available at
https://github.com/Rose-STL-Lab/AtlasD.
1. Introduction
Equivariant neural networks (Bronstein et al., 2021), a fam-
ily of models that exploit symmetry as an inductive bias
for neural network architectures, have received increasing
attention in deep learning due to their training efficiency
and improved generalization (Krizhevsky et al., 2017; Wor-
rall & Welling, 2019; Zaheer et al., 2017). The key idea
behind these models is that many real-world situations ex-
hibit inherent symmetries—transformations such as rotation,
translation, and scaling, which leave the essential properties
of a system unchanged. This has enabled a wide range of
applications, leading to empirical success (Winkels & Co-
hen, 2018; Brown & Lunter, 2018; Cohen & Welling, 2016;
Cohen et al., 2018). Despite its achievements, equivariant
1University of California San Diego
2IBM Research
3Northeastern University.
Correspo...
Why Do Some Language Models Fake Alignment While Others Don't?
NeurIPS 2025

Why Do Some Language Models Fake Alignment
While Others Don’t?
Abhay Sheshadri∗†
Anthropic
John Hughes∗
Anthropic
Julian Michael
Scale AI
Alex Mallen
Redwood Research
Arun Jose
Independent
Janus
Independent
Fabien Roger
Anthropic
Abstract
Alignment faking in large language models presented a demonstration of
Claude 3 Opus and Claude 3.5 Sonnet selectively complying with a helpful-
only training objective to prevent modification of their behavior outside of
training. We expand this analysis to 25 models and find that only 5 (Claude
3 Opus, Claude 3.5 Sonnet, Llama 3 405B, Grok 3, Gemini 2.0 Flash) comply
with harmful queries more when they infer they are in training than when
they infer they are in deployment. First, we study the motivations of these
5 models. Results from perturbing details of the scenario suggest that only
Claude 3 Opus’s compliance gap is primarily and consistently motivated
by trying to keep its goals. Second, we investigate why many chat models
don’t fake alignment. Our results suggest this is not entirely due to a lack
of capabilities: many base models fake alignment some of the time, and
post-training eliminates alignment-faking for some models and amplifies it
for others.We investigate 5 hypotheses for how post-training may suppress
alignment faking and find that variations in refusal behavior may account
for a significant portion of differences in alignment faking.
1
Introduction
Large Language Models (LLMs) with growing capabilities [Kwa et al., 2025] are increasingly
deployed in high-stakes applications [Maslej et al., 2025, Casper et al., 2025]. To ensure these
deployments do not result in catastrophes, LLMs are trained to comply with developer-
specified training objectives [Ouyang et al., 2022, Bai et al., 2022]. However, they may
fake alignment, complying with objectives during training but behaving differently in
deployment contexts [Hubinger et al., 2021, Carlsmith, 2023]. Understanding when and
why such compliance gaps appear may he...
DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents
NeurIPS 2025

DRIFT: Dynamic Rule-Based Defense with Injection
Isolation for Securing LLM Agents
Hao Li1, Xiaogeng Liu2, Hung-Chun Chiu3, Dianqi Li3, Ning Zhang1, Chaowei Xiao2
1Washington University in St. Louis, 2Johns Hopkins University
3Independent Researcher
{li.hao, zhang.ning}@wustl.edu, cxiao13@jh.edu
Abstract
Large Language Models (LLMs) are increasingly central to agentic systems due
to their strong reasoning and planning capabilities. By interacting with external
environments through predefined tools, these agents can carry out complex user
tasks. Nonetheless, this interaction also introduces the risk of prompt injection
attacks, where malicious inputs from external sources can mislead the agent’s
behavior, potentially resulting in economic loss, privacy leakage, or system com-
promise. System-level defenses have recently shown promise by enforcing static
or predefined policies, but they still face two key challenges: the ability to dy-
namically update security rules and the need for memory stream isolation. To
address these challenges, we propose DRIFT, a Dynamic Rule-based Isolation
Framework for Trustworthy agentic systems, which enforces both control- and
data-level constraints. A Secure Planner first constructs a minimal function trajec-
tory and a JSON-schema-style parameter checklist for each function node based
on the user query. A Dynamic Validator then monitors deviations from the original
plan, assessing whether changes comply with privilege limitations and the user’s
intent. Finally, an Injection Isolator detects and masks any instructions that may
conflict with the user query from the memory stream to mitigate long-term risks.
We empirically validate the effectiveness of DRIFT on the AgentDojo and ASB
benchmark, demonstrating its strong security performance while maintaining high
utility across diverse models—showcasing both its robustness and adaptability. The
code is released at https://github.com/SaFoLab-WISC/DRIFT.
1
Introduction
Large Language Models...
SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation
ICLR 2025

Published as a conference paper at ICLR 2025
SCOPE: A SELF-SUPERVISED FRAMEWORK FOR IM-
PROVING FAITHFULNESS IN CONDITIONAL TEXT GEN-
ERATION
Song Duong∗,1,4
Florian Le Bronnec∗,1,2
Alexandre Allauzen2
Vincent Guigue3
Alberto Lumbreras4
Laure Soulier1
Patrick Gallinari1,4
1Sorbonne Université, CNRS, ISIR, F-75005 Paris, France
2Miles Team, LAMSADE, Université Paris-Dauphine, Université PSL, CNRS, 75016 Paris, France
3AgroParisTech, UMR MIA-PS, Palaiseau, France
4Criteo AI Lab, Paris, France
ABSTRACT
Large Language Models (LLMs), when used for conditional text generation, often
produce hallucinations, i.e., information that is unfaithful or not grounded in the
input context. This issue arises in typical conditional text generation tasks, such
as text summarization and data-to-text generation, where the goal is to produce
fluent text based on contextual input. When fine-tuned on specific domains, LLMs
struggle to provide faithful answers to a given context, often adding information
or generating errors. One underlying cause of this issue is that LLMs rely on
statistical patterns learned from their training data. This reliance can interfere with
the model’s ability to stay faithful to a provided context, leading to the generation
of ungrounded information. We build upon this observation and introduce a novel
self-supervised method for generating a training set of unfaithful samples. We
then refine the model using a training process that encourages the generation of
grounded outputs over unfaithful ones, drawing on preference-based training. Our
approach leads to significantly more grounded text generation, outperforming
existing self-supervised techniques in faithfulness, as evaluated through automatic
metrics, LLM-based assessments, and human evaluations. Code is available at
https://github.com/sngdng/scope-faithfulness.
1
INTRODUCTION
Large Language Models (LLMs) are widely used for generating fluent and coherent text completions
based on input contexts (Brown et al....
EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping
ICML 2025

EEG-Language Pretraining for
Highly Label-Efficient Clinical Phenotyping
Sam Gijsen 1 2 Kerstin Ritter 1 2
Abstract
Multimodal language modeling has enabled
breakthroughs for representation learning, yet re-
mains unexplored in the realm of functional brain
data for clinical phenotyping. This paper pioneers
EEG-language models (ELMs) trained on clinical
reports and 15000 EEGs. We propose to com-
bine multimodal alignment in this novel domain
with timeseries cropping and text segmentation,
enabling an extension based on multiple instance
learning to alleviate misalignment between irrele-
vant EEG or text segments. Our multimodal mod-
els significantly improve over EEG-only models
across four clinical evaluations and for the first
time enable zero-shot classification as well as re-
trieval of both neural signals and reports. In sum,
these results highlight the potential of ELMs, rep-
resenting significant progress for clinical applica-
tions.
Medical neuroimaging, including electroencephalography
(EEG), has lagged behind other fields in leveraging the
significant advancements of deep learning. While EEG
sees widespread clinical use for clinical phenotyping such
as pathology detection, in particular for epilepsy (Binnie
& Stefan, 1999; Jing et al., 2020) as well as sleep disor-
ders (Malhotra & Avidan, 2013), available annotated data is
scarce. As the impressive scaling properties of deep learn-
ing are now well described (Kaplan et al., 2020; Smith et al.,
2023), self-supervised learning (SSL) is a promising direc-
tion by enabling pretraining with unlabeled data and thereby
increasing available training sample sizes (Hadsell et al.,
2006; Chen et al., 2020). Various such approaches have
shown initial success when applied to EEG. These include
methods relying on data-augmentations (Mohsenvand et al.,
1Charit´e – Universit¨atsmedizin Berlin, Department of Psychia-
try and Psychotherapy, Berlin, Germany 2Hertie Institute for AI in
Brain Health, University of T¨ubingen, ...
Hybrid Latent Representations for PDE Emulation
NeurIPS 2025

Hybrid Latent Representations for PDE Emulation
Ali Can Bekar
Helmholtz Centre Hereon
Geesthacht, Germany
ali.bekar@hereon.de
Siddhant Agarwal
Helmholtz Centre Hereon
Geesthacht, Germany
siddhant.agarwal@hereon.de
Christian Hüttig
Institute of Space Research
German Aerospace Center (DLR)
Berlin, Germany
christian.huettig@dlr.de
Nicola Tosi
Institute of Space Research
German Aerospace Center (DLR)
Berlin, Germany
nicola.tosi@dlr.de
David S. Greenberg
Helmholtz Centre Hereon
Geesthacht, Germany
david.greenberg@hereon.de
Abstract
For classical PDE solvers, adjusting the spatial resolution and time step offers
a trade-off between speed and accuracy. Neural emulators often achieve better
speed-accuracy trade-offs by operating on a compact representation of the PDE
system. Coarsened PDE fields are a simple and effective representation, but cannot
exploit fine spatial scales in the high-fidelity numerical solutions. Alternatively,
unstructured latent representations provide efficient autoregressive rollouts, but
cannot enforce local interactions or physical laws as inductive biases. To overcome
these limitations, we introduce hybrid representations that augment coarsened
PDE fields with spatially structured latent variables extracted from high-resolution
inputs. Hybrid representations provide efficient rollouts, can be trained on a simple
loss defined on coarsened PDE fields, and support hard physical constraints. When
predicting fine- and coarse-scale features across multiple PDE emulation tasks,
they outperform or match the speed-accuracy trade-offs of the best convolutional,
attentional, Fourier neural operator-based and autoencoding baselines.
1
Introduction
Integrating partial differential equations (PDEs) can be computationally expensive, particularly for
chaotic and turbulent systems with fine-scale structures requiring high spatiotemporal resolution.
For traditional solvers, discarding fine-scale details reduces both cost and accuracy, but data-driven
emulators can...
Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
ICML 2025

Satori: Reinforcement Learning with Chain-of-Action-Thought
Enhances LLM Reasoning via Autoregressive Search
Maohao Shen * 1 Guangtao Zeng * 2 Zhenting Qi * 3
Zhang-Wei Hong 1 Zhenfang Chen 4 Wei Lu 2 Gregory Wornell 1
Subhro Das 4 David Cox 4 Chuang Gan 4 5
Abstract
Large language models (LLMs) have demon-
strated remarkable reasoning capabilities across
diverse domains. Recent studies have shown that
increasing test-time computation enhances LLMs’
reasoning capabilities. This typically involves ex-
tensive sampling at inference time guided by an
external LLM verifier, resulting in a two-player
system. Despite external guidance, the effective-
ness of this system demonstrates the potential of
a single LLM to tackle complex tasks. Thus, we
pose a new research problem: Can we internal-
ize the searching capabilities to fundamentally
enhance the reasoning abilities of a single LLM?
This work explores an orthogonal direction fo-
cusing on post-training LLMs for autoregressive
searching (i.e., an extended reasoning process
with self-reflection and self-exploration of new
strategies). To achieve this, we propose the Chain-
of-Action-Thought (COAT) reasoning and a two-
stage training paradigm: 1) a small-scale for-
mat tuning stage to internalize the COAT reason-
ing format and 2) a large-scale self-improvement
stage leveraging reinforcement learning. Our ap-
proach results in Satori, a 7B LLM trained on
open-source models and data. Extensive empiri-
cal evaluations demonstrate that Satori achieves
state-of-the-art performance on mathematical rea-
soning benchmarks while exhibits strong gener-
alization to out-of-domain tasks. Code, data, and
models are fully open-sourced1.
* Core contributors of Satori team, contributed equally to this
work. Order determined by a dice roll.
1MIT 2Singapore University of Technology and Design 3Harvard
4MIT-IBM Watson AI Lab, IBM Research 5UMass Amherst.
Correspondence to: Maohao Shen <maohao@mit.edu>.
Proceedings of the 42 nd Internationa...