dataset-review/dataset-review huggingface dataset:
Compressing tree ensembles through Level-wise Optimization and Pruning
ICML 2025

Compressing Tree Ensembles through Level-wise Optimization and Pruning
Laurens Devos * 1 2 Timo Martens * 1 2 Deniz Can Oruc¸ 1 2 Wannes Meert 1 2 Hendrik Blockeel 1 2 Jesse Davis 1 2
Abstract
Tree ensembles (e.g., gradient boosting decision
trees) are often used in practice because they offer
excellent predictive performance while still being
easy and efficient to learn. In some contexts, it
is important to additionally optimize their size:
this is specifically the case when models need
to have verifiable properties (verification of fair-
ness, robustness, etc. is often exponential in the
ensemble’s size), or when models run on battery-
powered devices (smaller ensembles consume less
energy, increasing battery autonomy). For this
reason, compression of tree ensembles is worth
studying. This paper presents LOP, a method for
compressing a given tree ensemble by pruning or
entirely removing trees in it, while updating leaf
predictions in such a way that predictive accuracy
is mostly unaffected. Empirically, LOP achieves
compression factors that are often 10 to 100 times
better than that of competing methods.
1. Introduction
Ensembles of decision trees, a.k.a. decision forests, are
frequently used in practice because of their ease of training,
excellent overall performance, and high efficiency. Their
training requires little or no hyperparameter tuning, the
computational cost of both training and prediction is very
small, and for tabular data, they are usually among the best
performing models (Grinsztajn et al., 2022).
This paper is about compressing decision forests: reducing
their size as measured by, e.g., the total number of leaves
in the forest, while maintaining their accuracy. In the light
of the above successes, one might wonder whether this is
worthwhile: if forests are already so efficient, can we gain
*Equal contribution
1KU Leuven Department of Computer
Science, Leuven, Belgium 2Leuven.AI, KU Leuven Institute for
Artificial Intelligence, Leuven, Belgium.
Co...
Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
ICML 2025

Bongard in Wonderland:
Visual Puzzles that Still Make AI Go Mad?
Antonia W¨ust 1 Tim Woydt 1 Lukas Helff 1 2 Inga Ibs 3 4 Wolfgang Stammer 1 2 Devendra S. Dhami 5
Constantin A. Rothkopf 2 3 4 Kristian Kersting 1 2 4 6
Abstract
Recently, newly developed Vision-Language
Models (VLMs), such as OpenAI’s o1, have
emerged, seemingly demonstrating advanced rea-
soning capabilities across text and image modal-
ities.
However, the depth of these advances
in language-guided perception and abstract rea-
soning remains underexplored, and it is unclear
whether these models can truly live up to their
ambitious promises. To assess the progress and
identify shortcomings, we enter the wonderland
of Bongard problems, a set of classic visual rea-
soning puzzles that require human-like abilities of
pattern recognition and abstract reasoning. With
our extensive evaluation setup, we show that while
VLMs occasionally succeed in identifying dis-
criminative concepts and solving some of the
problems, they frequently falter. Surprisingly,
even elementary concepts that may seem trivial to
humans, such as simple spirals, pose significant
challenges. Moreover, when explicitly asked to
recognize ground truth concepts, they continue to
falter, suggesting not only a lack of understanding
of these elementary visual concepts but also an
inability to generalize to unseen concepts. We
compare the results of VLMs to human perfor-
mance and observe that a significant gap remains
between human visual reasoning capabilities and
machine cognition. 1
1AIML Lab, TU Darmstadt 2Hessian Center for AI (hessian.ai)
3Institute of Psychology, TU Darmstadt 4Centre for Cognitive
Science, TU Darmstadt 5Uncertainty in AI Group, TU Eindhoven
6German Center for AI (DFKI). Correspondence to: Antonia W¨ust
<antonia.wuest@cs.tu-darmstadt.de>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1Our
code
is
available
at
https://github.com/...
Diffusion Federated Dataset
NeurIPS 2025

Diffusion Federated Dataset
Seok-Ju Hahn1
Argonne National Laboratory
hahns@anl.gov
Junghye Lee2
Seoul National University
junghye@snu.ac.kr
Abstract
Diffusion models have demonstrated decent generation quality, yet their deploy-
ment in federated learning scenarios remains challenging. Due to data heterogeneity
and a large number of parameters, conventional parameter averaging schemes of-
ten fail to achieve stable collaborative training of diffusion models. We reframe
collaborative synthetic data generation as a cooperative sampling procedure from
a mixture of decentralized distributions, each encoded by a pre-trained local dif-
fusion model. This leverages the connection between diffusion and energy-based
models, which readily supports compositional generation thereof. Consequently,
we can directly obtain refined synthetic dataset, optionally with differential privacy
guarantee, even without exchanging diffusion model parameters. Our framework
reduces communication overhead while maintaining the generation quality, realized
through an unadjusted Langevin algorithm with a convergence guarantee.
1
Introduction
Federated learning (FL [1]) enables clients (i.e., data owners) to collaboratively train a statistical
model by exchanging locally updated parameters with a central server over iterative communication
rounds, thereby preserving data privacy. While this model-centric FL paradigm is well-established,
sharing public data can substantially enhance FL performance by mitigating statistical heterogeneity
arising from non-independent and identically distributed (non-IID) local data distributions [1–4]. For
instance, public or synthetic datasets can homogenize disparate local distributions and serve as direct
signals for server-side pretraining. This facilitates client-side transfer learning or data augmentation,
both improve the overall utility of FL.
While these data-centric FL scheme offers clear advantages over purely model-centric approaches,
there remain challen...
Decoupled Subgraph Federated Learning
ICLR 2025

Published as a conference paper at ICLR 2025
DECOUPLED SUBGRAPH FEDERATED LEARNING
Javad Aliakbari1
Johan ¨Ostman2
Alexandre Graell i Amat1
1Chalmers University of Technology
2AI Sweden
ABSTRACT
We address the challenge of federated learning on graph-structured data distributed
across multiple clients. Specifically, we focus on the prevalent scenario of in-
terconnected subgraphs, where interconnections between different clients play a
critical role. We present a novel framework for this scenario, named FEDSTRUCT,
that harnesses deep structural dependencies. To uphold privacy, unlike existing
methods, FEDSTRUCT eliminates the necessity of sharing or generating sensitive
node features or embeddings among clients. Instead, it leverages explicit global
graph structure information to capture inter-node dependencies. We validate the ef-
fectiveness of FEDSTRUCT through experimental results conducted on six datasets
for semi-supervised node classification, showcasing performance close to the cen-
tralized approach across various scenarios, including different data partitioning
methods, varying levels of label availability, and number of clients.
1
INTRODUCTION
Many real-world data are graph-structured, where nodes represent entities and edges capture their re-
lationships. For example, in anti-money laundering, nodes symbolize accounts and edges correspond
to confidential transactions.
Graph neural networks (GNNs) are specialized neural networks for graph-structured data, showing
success in fields such as drug discovery, social networks, or traffic flow modeling (Stokes et al.,
2020; Fan et al., 2019; Jiang and Luo, 2022). In many practical applications, graph data is inherently
distributed across multiple clients, i.e., the global graph encompasses multiple, non-overlapping
subgraphs. For example, in anti-money laundering, local graphs represent internal transactional
networks of financial institutions.
For this and many other applications, data sharing among clients is ...
Optimal Sensor Scheduling and Selection for Continuous-Discrete Kalman Filtering with Auxiliary Dynamics
ICML 2025

Optimal Sensor Scheduling and Selection for Continuous-Discrete Kalman
Filtering with Auxiliary Dynamics
Mohamad Al Ahdab 1 2 John Leth 1 Zheng-Hua Tan 1 2
Abstract
We study the Continuous-Discrete Kalman Filter
(CD-KF) for State-Space Models (SSMs) where
continuous-time dynamics are observed via multi-
ple sensors with discrete, irregularly timed mea-
surements. Our focus extends to scenarios in
which the measurement process is coupled with
the states of an auxiliary SSM. For instance,
higher measurement rates may increase energy
consumption or heat generation, while a sensor’s
accuracy can depend on its own spatial trajec-
tory or that of the measured target. Each sensor
thus carries distinct costs and constraints asso-
ciated with its measurement rate and additional
constraints and costs on the auxiliary state. We
model measurement occurrences as independent
Poisson processes with sensor-specific rates and
derive an upper bound on the mean posterior co-
variance matrix of the CD-KF along the mean
auxiliary state. The bound is continuously differ-
entiable with respect to the measurement rates,
which enables efficient gradient-based optimiza-
tion. Exploiting this bound, we propose a finite-
horizon optimal control framework to optimize
measurement rates and auxiliary-state dynam-
ics jointly. We further introduce a deterministic
method for scheduling measurement times from
the optimized rates. Empirical results in state-
space filtering and dynamic temporal Gaussian
process regression demonstrate that our approach
achieves improved trade-offs between resource
usage and estimation accuracy.
1Department of Electronic Systems, Aalborg University, Aal-
borg Øst 9220, Denmark. 2Pioneer Centre for Artificial Intelli-
gence, Copenhagen 1350, Denmark.. Correspondence to: Mo-
hamad Al Ahdab <maah@es.aau.dk>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1. Introduction
State-space...
The Logical Implication Steering Method for Conditional Interventions on Transformer Generation
ICML 2025

The Logical Implication Steering Method for Conditional Interventions on
Transformer Generation
Damjan Kalajdzievski 1
Abstract
The field of mechanistic interpretability in pre-
trained transformer models has demonstrated sub-
stantial evidence supporting the “linear represen-
tation hypothesis”, which is the idea that high
level concepts are encoded as vectors in the space
of activations of a model. Studies also show that
model generation behavior can be steered toward
a given concept by adding the concept’s vector
to the corresponding activations. We show how
to leverage these properties to build a form of
logical implication into models, enabling trans-
parent and interpretable adjustments that induce
a chosen generation behavior in response to the
presence of any given concept. Our method, Log-
ical Implication Model Steering (LIMS), unlocks
new hand-engineered reasoning capabilities by
integrating neuro-symbolic logic into pre-trained
transformer models.
1. Introduction
In this work, we are motivated by advances in mechanis-
tic interpretability which deepen our understanding of the
representations within pre-trained transformer models, as
well as by the goal of bridging neuro-symbolic and algo-
rithmic capabilities with these representations. Harnessing
the extensive knowledge and intuitive processing embedded
in these models in a high-level, interpretable, and algorith-
mic way, holds the potential to unlock significantly greater
reasoning capabilities and control.
Driven by these motivations, we introduce the Logical Impli-
cation Model Steering (LIMS) method, which extracts and
uses concept vectors to allow a user to “program” logical
implication into model generation behavior in a transparent
and interpretable manner. LIMS achieves this by implement-
ing the atomic building blocks for a general neuro-symbolic
1Salesforce. Correspondence to: Damjan Kalajdzievski <dam-
jank7354@gmail.com>.
Proceedings of the 42 nd International Conference on Machine
Learning...
A-Mem: Agentic Memory for LLM Agents
NeurIPS 2025

A-Mem: Agentic Memory for LLM Agents
Wujiang Xu1, Zujie Liang2, Kai Mei1, Hang Gao1, Juntao Tan1, Yongfeng Zhang1,3
1Rutgers University
2Independent Researcher
3AIOS Foundation
wujiang.xu@rutgers.edu
Abstract
While large language model (LLM) agents can effectively use external tools for
complex real-world tasks, they require memory systems to leverage historical
experiences. Current memory systems enable basic storage and retrieval but lack
sophisticated memory organization, despite recent attempts to incorporate graph
databases. Moreover, these systems’ fixed operations and structures limit their
adaptability across diverse tasks. To address this limitation, this paper proposes
a novel agentic memory system for LLM agents that can dynamically organize
memories in an agentic way. Following the basic principles of the Zettelkasten
method, we designed our memory system to create interconnected knowledge
networks through dynamic indexing and linking. When a new memory is added, we
generate a comprehensive note containing multiple structured attributes, including
contextual descriptions, keywords, and tags. The system then analyzes historical
memories to identify relevant connections, establishing links where meaningful
similarities exist. Additionally, this process enables memory evolution – as new
memories are integrated, they can trigger updates to the contextual representations
and attributes of existing historical memories, allowing the memory network to
continuously refine its understanding. Our approach combines the structured
organization principles of Zettelkasten with the flexibility of agent-driven decision
making, allowing for more adaptive and context-aware memory management.
Empirical experiments on six foundation models show superior improvement
against existing SOTA baselines.
Code for Benchmark Evaluation:
https://github.com/WujiangXu/AgenticMemory
Code for Production-ready Agentic Memory:
https://github.com/WujiangXu/A-mem-sys
1
Introduction
Large Lang...
Breaking Mental Set to Improve Reasoning through Diverse Multi-Agent Debate
ICLR 2025

Published as a conference paper at ICLR 2025
BREAKING MENTAL SET TO IMPROVE REASONING
THROUGH DIVERSE MULTI-AGENT DEBATE
Yexiang Liu1,2, Jie Cao1∗, Zekun Li3, Ran He1,2, Tieniu Tan1,2,4
1MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences
2School of Artificial Intelligence, University of Chinese Academy of Sciences
3University of California, Santa Barbara
4Nanjing University
liuyexiang2023@ia.ac.cn, jie.cao@cripac.ia.ac.cn
ABSTRACT
Large Language Models (LLMs) have seen significant progress but continue to
struggle with persistent reasoning mistakes. Previous methods of self-reflection
have been proven limited due to the models’ inherent fixed thinking patterns.
While Multi-Agent Debate (MAD) attempts to mitigate this by incorporating mul-
tiple agents, it often employs the same reasoning methods, even though assigning
different personas to models. This leads to a “fixed mental set”, where models rely
on homogeneous thought processes without exploring alternative perspectives. In
this paper, we introduce Diverse Multi-Agent Debate (DMAD), a method that en-
courages agents to think with distinct reasoning approaches. By leveraging diverse
problem-solving strategies, each agent can gain insights from different perspec-
tives, refining its responses through discussion and collectively arriving at the op-
timal solution. DMAD effectively breaks the limitations of fixed mental sets. We
evaluate DMAD against various prompting techniques, including self-reflection
and traditional MAD, across multiple benchmarks using both LLMs and Multi-
modal LLMs. Our experiments show that DMAD consistently outperforms other
methods, delivering better results than MAD in fewer rounds. Code is available at
https://github.com/MraDonkey/DMAD.
1
INTRODUCTION
In the past few years, Large Language Models (LLMs) have experienced unprecedented develop-
ment (Brown et al., 2020; Touvron et al., 2023a;b; Chowdhery et al., 2023; Anil et al., 2023; Achiam
et al., 2023; Open AI, 2024a),...
Riemann Tensor Neural Networks: Learning Conservative Systems with Physics-Constrained Networks
ICML 2025

Riemann Tensor Neural Networks: Learning Conservative Systems with
Physics-Constrained Networks
Anas Jnini 1 Lorenzo Breschi 1 Flavio Vella 1
Abstract
Divergence-free symmetric tensors (DFSTs) are
fundamental in continuum mechanics, encoding
conservation laws such as mass and momentum
conservation. We introduce Riemann Tensor Neu-
ral Networks (RTNNs), a novel neural architec-
ture that inherently satisfies the DFST condition
to machine precision, providing a strong induc-
tive bias for enforcing these conservation laws.
We prove that RTNNs can approximate any suffi-
ciently smooth DFST with arbitrary precision and
demonstrate their effectiveness as surrogates for
conservative PDEs, achieving improved accuracy
across benchmarks. This work is the first to use
DFSTs as an inductive bias in neural PDE surro-
gates and to explicitly enforce the conservation
of both mass and momentum within a physics-
constrained neural architecture.
1. Introduction
Partial Differential Equations (PDEs)
Partial Differen-
tial Equations (PDEs) are central to the mathematical model-
ing of complex physical systems, including fluid dynamics,
thermodynamics, and material sciences. Traditional nu-
merical methods, such as finite element and spectral meth-
ods, often require fine discretization of the physical domain
to achieve high accuracy. These approaches can become
computationally expensive, particularly in engineering ap-
plications where systems must be solved repeatedly under
varying parameters or initial conditions. Recent advances in
machine learning (ML) have shown promise in addressing
these challenges by leveraging neural networks (NNs) as po-
tential alternatives or enhancements to traditional numerical
solvers (Kovachki et al., 2021; Li et al., 2020).
1 Department of Information Engineering and Computer Sci-
ence, University of Trento, Trento, Italy. Correspondence to: Anas
Jnini <anas.jnini@unitn.it>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancou...
Mahalanobis++: Improving OOD Detection via Feature Normalization
ICML 2025

Mahalanobis++: Improving OOD Detection via Feature Normalization
Maximilian M¨uller 1 Matthias Hein 1
Abstract
Detecting out-of-distribution (OOD) examples
is an important task for deploying reliable ma-
chine learning models in safety-critial applica-
tions. While post-hoc methods based on the Ma-
halanobis distance applied to pre-logit features
are among the most effective for ImageNet-scale
OOD detection, their performance varies signif-
icantly across models. We connect this incon-
sistency to strong variations in feature norms,
indicating severe violations of the Gaussian as-
sumption underlying the Mahalanobis distance es-
timation. We show that simple ℓ2-normalization
of the features mitigates this problem effectively,
aligning better with the premise of normally dis-
tributed data with shared covariance matrix. Ex-
tensive experiments on 44 models across diverse
architectures and pretraining schemes show that
ℓ2-normalization improves the conventional Ma-
halanobis distance-based approaches significantly
and consistently, and outperforms other recently
proposed OOD detection methods. Code is avail-
able at github.com/mueller-mp/maha-norm.
1. Introduction
Deep neural networks have demonstrated remarkable perfor-
mance across a variety of real-world tasks. However, when
faced with inputs that fall outside their training distribu-
tion, they can behave unpredictably and even result in high-
confidence predictions (Hendrycks & Gimpel, 2017; Hein
et al., 2019). These so-called out-of-distribution (OOD) in-
puts are often misclassified with high confidence as belong-
ing to the in-distribution (ID) classes, creating significant
risks for real-world deployments. OOD detectors aim to
identify and reject such anomalous inputs — potentially
prompting human intervention, transitioning to a safe state,
or declining to provide a prediction — while still allowing
1University of T¨ubingen and T¨ubingen AI Center.
Corre-
spondence to: Maximilian M¨uller <maximilian.mueller@...
From Pixels to Perception: Interpretable Predictions via Instance-wise Grouped Feature Selection
ICML 2025

From Pixels to Perception:
Interpretable Predictions via Instance-wise Grouped Feature Selection
Moritz Vandenhirtz 1 Julia E. Vogt 1
Abstract
Understanding the decision-making process of
machine learning models provides valuable in-
sights into the task, the data, and the reasons be-
hind a model’s failures. In this work, we propose
a method that performs inherently interpretable
predictions through the instance-wise sparsifica-
tion of input images. To align the sparsification
with human perception, we learn the masking in
the space of semantically meaningful pixel re-
gions rather than on pixel-level. Additionally, we
introduce an explicit way to dynamically deter-
mine the required level of sparsity for each in-
stance. We show empirically on semi-synthetic
and natural image datasets that our inherently in-
terpretable classifier produces more meaningful,
human-understandable predictions than state-of-
the-art benchmarks.
1. Introduction
Knowledge is power (Bacon, 1597). Knowing how ma-
chine learning models make their predictions empowers
users to understand the underlying patterns in the data, iden-
tify potential biases, reason about their safety, and build
trust in the models’ decisions. In high-stakes domains,
where model decisions can have significant consequences,
the field of interpretability is crucial to provide this under-
standing (Doshi-Velez & Kim, 2017). In this work, we focus
on providing inherent interpretability – a subfield where the
explanations are necessarily faithful to what the model com-
putes (Rudin, 2019) – through the lens of sparsity in the
input features (Lipton, 2016; Marcinkeviˇcs & Vogt, 2023).
This constraint makes it easier to understand which specific
inputs contribute to an output (Miller, 1956), leading to
more human-understandable predictions.
1Department
of
Computer
Science,
ETH
Zurich,
Switzerland.
Correspondence
to:
Moritz
Vandenhirtz
<moritz.vandenhirtz@inf.ethz.ch>.
Proceedings of the 42 nd International Conference on ...