Behavior Learning (BL)
Authors: Zhenyao Ma
Yue Liang
Dongxu Li
Venue: ICLR 2026 Poster (2026)
Keywords: Inverse Optimization
Energy-Based Models (EBMs)
Intrinsic Interpretability
Identifiability
Behavioral Modeling
Primary area: interpretability and explainable AI
TL;DR: Inspired by behavioral science, we propose Behavior Learning (BL), a general-purpose machine learning framework that learns interpretable and identifiable (hierarchical) optimization structures from data.

Abstract: Inspired by behavioral science, we propose Behavior Learning (BL), a novel general-purpose machine learning framework that learns interpretable 
and identifiable optimization structures from data, ranging from single optimization problems to hierarchical compositions. It unifies predictive performance, intrinsic interpretability, and identifiability, with broad applicability to scientific domains involving optimization.  BL parameterizes a compositional utility function built from intrinsically interpretable modular blocks, which induces a data distribution for prediction and generation. Each block represents and can be written in symbolic form as a utility maximization problem (UMP), a foundational paradigm in behavioral science and a universal framework of optimization. BL supports architectures ranging from a single UMP to hierarchical compositions, the latter modeling hierarchical optimization structures that offer both expressiveness and structural transparency. Its smooth and monotone variant (IBL) guarantees identifiability under mild conditions. Theoretically, we establish the universal approximation property of both BL and IBL, and analyze the M-estimation properties of IBL. Empirically, BL demonstrates strong predictive performance, intrinsic interpretability and scalability to high-dimensional data. Code: https://github.com/MoonYLiang/Behavior-Learning; installable via pip install blnetwork.

Published as a conference paper at ICLR 2026
BEHAVIOR LEARNING (BL)
Zhenyao Ma
Yue Liang
Dongxu Li Xi' an Jiaotong University
Xiamen University
University of Tübingen
ABSTRACT
Inspired by behavioral science, we propose Behavior Learning (BL), novel general-purpose machine learning framework that learns interpretable and iden- tifiable optimization structures from data, ranging from single optimization prob- lems to hierarchical compositions. It unifies predictive performance, intrin- sic interpretability, and identifiability, with broad applicability to scientific do- mains involving optimization. BL parameterizes compositional utility func- tion built from intrinsically interpretable modular blocks, which induces data distribution for prediction and generation. Each block represents and can be written in symbolic form as a utility maximization problem (UMP), a founda- tional paradigm in behavioral science and universal framework of optimization. BL supports architectures ranging from a single UMP to hierarchical compo- sitions, the latter modeling hierarchical optimization structures that offer both expressiveness and structural transparency. Its smooth and monotone variant (IBL) guarantees identifiability under mild conditions. Theoretically, we estab- lish the universal approximation property of both BL and IBL, and analyze the M-estimation properties of IBL. Empirically, BL demonstrates strong predictive performance, intrinsic interpretability and scalability to nigh-dimensional data. Code: g github. com www./oom//ooccooeec.oooooooooooe MMonnIoooa ang/ //BB/aavvrrreeaaa Lninning installable via pip install blnetwork.
INTRODUCTION
Scientific research often grapples with phenomena that resist precise formalization (Anderson, 1972;
Mitchell, 2009), including human and social domains (Simon, 1955; Arthur, 2009). Such phenom- ena are difficult to predict and even harder to falsify through theory alone. Interpretable machine learning (Interpretable ML) (Molnar, 2020), with its powerful approximation capabilities and built- in transparency, offers a promising alternative for modeling such phenomena. Yet a long-standing tension remains unresolved: model predictive performance and intrinsic interpretability often trade off challenge commonly known as the performance-interreee 2020). High-performing models such as deep neural networks (LeCun et al., 2015) typically lack transparency, while intrinsically interpretable models struggle to capture complex nonlinear patterns.
trade-off (Arrieta et al.,
trade-off, and the main (i) Additive models (Caruana et al., 2015; Hastie, 2017; Nori et al., 2019; Agarwal et al., 2021; Chang et al., 202 1). (ii) Concept- based models (Alvarez Melis & Jaakkola, 2018; Kim et al., 2018; Koh et al., 2020). (ii) Rule- and score-based systems (Ustun & Rudin, 2016; Angelino et al., 2018) (iv) Shape-constrained neural networks (You et al., 2017) Recent additional interpretable modeling frameworks include Kraus et al. (2024); Liu et al. (2024b); Plonsky et al. (2025). These approaches demonstrate varied strengths.
Some efforts have been made to mitigate the performance-interprttaahi existing approaches can be broadly grouped into four categories.
However, two fundamental limitations remain, restricting their scientific applicability. (i) Insufficient alignment with scientific theories. Most approaches focus on extending existing machine learning methods to achieve interpretability, rather than developing a scientifically grounded framework (c.g., based on optimization problems or differential equations). This often hinders alignment with sei- entific theories and limits the ability to extract scientific knowledge from learned models (Roscher et al., 2020; Bereska Gavves, 2024; Longo et al., 2024). (ii) Non-uniqueness of interpretations.
"Correspondence zhenyaoma stu.xuuuum xww.edu.cc

---

Published as a conference paper at ICLR 2026
(4)
Human Behavior
(d)
Single UMP
Environment
Utility Maximization Problem (UMP)
Behavior yeR
x xER R
BL. (Single)
max 0 (x,,)
((x,y) 20 T(x,f) =0
Behavior Learning
UMP Composition
Leurning Scheme
BL (Shallow)
Compl BL(x,))
Each B Learnable
(x, y
(()
max
U(xy))
Learnable
[C(x,y)
Learnable
(xx,y)
Learnable
Interpretability & Inference
(Deep)
Intrinsic Interpretability
Symbolic Form of B
Visualization
g
U (xx,y)
2m, + Sz,
85
((xxy) = =, 50 [Tx,y)
8.6.
Prediction
Generation
(x.y)
argenne Bl.s.))
gu. xylxx) x (x)
Figure 1: Behavior Learning (BL). (a) Human behavior modeled as a UMP. (b) Learning scheme of BL. where CompU denotes the compositional utility function. (c) BL offers intrinsic interpretability (via symbolic form as optimization problem), identifiability (via unique parameterization) and inference capability. (d) Three architectural variants of BL, from single UMP to deep compositions.
Most models are non-identifablle-thii interpretations are not uniquely determined by observable predictions in mathematical sense (Ran & Hu, 2017; Méloux et al., 2025), As result, such models cannot support reliable estimation ground-truth parameters (Newey & McFadden, 1994; Van der Vaart, 2000), and may even lack Popperian falsifiability (Popper, 2005), ultimately limiting their scientific credibility. These limitations naturally raise a key question: can we design an interpretable MI. framework that mitigates the performane-interreaablllyy trade-off while being scientifically grounded and identifiable?
Inspired by behavioral science, we propose Behavior Learning (BL): general-purpose machine learning framework that learns interpretable and identifiable (hierarchical) optimization structures from data. It unifies high predictive performance, intrinsic interpretability, and identifiability, with broad applicability to scientific domains involving optimization. As illustrated in Figure 1, BL builds of the most fundamental paradigms in behavioral science-utihe maximization-whiie posits that human behavior arises from solving a willity maximization problem (UMP) (Samuelson, 1948; Debreu, 1959; Mas-Coleh al., 1995). Motivated by this paradigm, BL learns interpretable optimization structures from data. It models responses (y) as drawn from a probability distribution induced by UMP a composition of multiple interacting UMPs. This distribution parameterized by compositional utility function BL(x,y), constructed from intrinsically interpretable modular blocks (xx, y). Each block a learnable penalty- based formulation that represents an optimization problem (UMP), which can be written in symbolic form and offers transparency comparable to linear regression.
BL. admits hierarchical structure, mainly in three architectural variants: BL.(inglle, defined by single block; BL(Shallow), a moderately layered composition blocks; and (Deep),) a deep hier- archical composition of multiple blocks. The latter two model, and can be symbolically interpreted as, hierarchical optimization structures. All variants are trained end-to-end to induce a condi- tional Gibbs distribution for prediction and generation. By refining the penalty functions in each block into smooth and monotone forms, we develop Identifiable BL (IBL), the identifiable variant of BL. Under mild conditions, IBI. guarantees unique intrinsic interpretability. This property ensures
2

---

Published as a conference paper at ICLR 2026
the scientific credibility of its explanations and further supports recovery of the ground -ruth model under appropriate conditions.
While motivated by behavioral science, BL is not domain -ommain-ppecffic It applics broadly any scientific domain where observed outcomes arise as solutions to optimization problems- such as macrocco- nomics (Ramsey, 1928; Ljuugqvist & Sargent, 2018), statistical physics (Gibbs, 1902; Landau & Lifshitz, 2013), or evolutionary biology (Wright et al., 1932; Fisher, 1999). This generality is sup- ported by key theoretical insight (Theorem 2.2): any optimization problem can be equivalently written as a UMP. This makes BL a general-purpee purpose modeling framework for data-driven inverse optimization (Ahuja Orlin, 2001) across diverse scientific disciplines.
We study BL both theoretically and empirically. Theoretically, we show that both BL and IBL admit universal approximation under mild assumptions (Section 2.2). For IBL, we further establish its M- estimation properties (Section 2.3), including identifiability, consistency, and universal consistency. Empirically, we evaluate BL across three tasks. Standard prediction tasks (Section 3.1) demonstrate its strong predictive performance. A qualitative case study (Section 3.2) illustrates its intrinsic inter- pretability, Prediction on high-dimensional inputs (Section 3.3) further demonstrates its scalability to high-dimensional data. Due to limited space, further discussion and related work are deferred to Appendix B and Appendix C, respectively. We also provide guidance on how to scientifically explain (L(ee)p) and architectural details in Appendix A and Appendix D, respectively.
Overall, our key contributions are threefold. (i) We propose Behavior Learning (BL), a novel general-purpose machine learning framework inspired by behavioral science, which unifies high predictive performance, intrinsic interpretability, identifiability. and scalability. (ii) For scientific re- search, BL offers scientifically grounded, interpretable, and identifiable machine learning approach for modeling complex phenomena that defy precise formalization. BL applies broadly to scientific disciplines involving optimization. (iii) At the paradigm level, BL learns from data the optimization structure of either a single optimization problem or a hierarchical composition of problems through distributional modeling, contributing new methodology to data-driven inverse optimization.
BEHAVIOR LEARNING (BL)
2.1 UTILITY MAXIMIZATION PROBLEM (UMP)
The modeling of human behavior, particularly in behavioral science and decision theory, often be- gins with the assumption that observed outcomes arise from a latent optimization process. A canon- ical formulation of this idea is the Utility Maximization Problem (UMP) (Mas-Colell et al., 1995), in which an agent selects actions y y in response to contextual features X' by solving:
max y s.t. y(x,y) s 0, T(x,y) =0 yEy
Here, U-) denotes a subjective utility function encoding the agent's 's internal preferences or goals. The inequality constraint C(-) captures resource constraints, while the equality constraint T(-) en- codes either endogenous belief consistency or exogenous conservation laws.
The UMP can be recast as a bent-ft framework, where the agent trades off utility gains against constraint violations. Formally, under mild regularity conditions, it admits an unconstrained penalty reformulation at the level of local optimality (Han & Mangasarian, 1979), as formalized below.
Theorem 2.1 (Local Exact Penalty Reformulation for UMP). Let X C Rd and y C Rd, be nonempty compact sets, and let U X' y - IR, C: X x y - Rmm and : X x y + Rp be C Assume that for a given x E X, the Han-Mangasarian constraint qualification holds at any strict local maximizer y of the UMP. Then there exist No > 0, X₂ Rm and d2 E TRP, such that y* is a local maximizer of
max No AU(( - XT - -1 (x),)) yEy
(2)
Here Φ : R-+ R. is strictly increasing and C²,, p(z) == maxfz, 0}. and (z)) : |z/.
The proof is provided in Appendix E. This unconstrained reformulation offers greater tractability for both theoretical analysis and model training.
3

---

Published as a conference paper at ICLR 2026
While motivated by behavioral modeling, the UMP formulation is not domain It applies to any setting where observed outcomes are solutions to optimization problems. This because any optimization problem can be equivalently formulated a UMP. We state this in the following result, while the formal statement and proof are provided in Appendix E.1.
Theorem 2.2 (Universality of UMP). Any optimization problem of the form maxyey f(x, or minyey yx, subject to equality and inequality constraints, is equivalent to a UMP.
2.2 BL ARCHITECTURE
Figure l(b-d) illustrates the architecture of BL. We consider samples (x.y) ~ D, where x E R denotes contextual features and y is the response, represented as (yddise, y yoont) E Valse Rm cap- turing its hybrid structure. Responses are assumed to be stochastically generated by solving multiple interacting UMPs, each with a penalty- based formulation, which together compose compositional utility function BL(x, y). On this basis, we model the data using conditional Gibbs distribution (Gibbs, 1902) parameterized by (x,
(y x; exp (L (x,
(x(x;)) (xx(BLo(x,))/r)) dy'
(x +)
Here the temperature parameter т > 0 controls the randomness of the response. As T - 0, the distribution in equation converges Dirac measure supported on arg maxy BI, y), thereby recovering the deterministic best response obtained by solving the composed UMPs.
Model Structure of yL(x,y). To represent the composition of multiple UMPs, build BL(x, y) by composing fundamental modular blocks B(x,y). Each block provides a penalty-based formulation of single UMP, and together they yield the overall compositional utility function. Motivated by Theorem 2.1, we parameterize B(x,y) as
y; (Uog (x, X, p(Cn (x. - X/ v) (xo,(x,))
(4)
where 0 : (Ao, A1 λ2, By dc, 0T) denotes the complete set of learnable parameters. Following Theorem 2.1, & is an increasing function; P penalizes inequality violations; and a captures symmet- ric deviations. Each block can be written as a well-defined UMP.
We then compose BL(x, y) from multiple B-blocks through hierarchical composition to improve its representational power for optimization structures, yielding three main architectural variants, as illustrated in Figure (dd).
1. BL((ingle) applies single instance of B(x, y) defined in equation 4, without any additional layers. It can be viewed as learning a single UMP, and offers maximal interpretability.
2. BL(Shallow) uses B(x, y) as the fundamental modular block construct shallow net- work. It introduces one or two intermediate layers of computation. Each layer B. stacks multiple parallel Br,, blocks to produce vector in Ed, i.e., (x, 0z) == [xx, y; fe,))..... ..... Be,d, yx 0e,,),, The output of By is directly fed into the next B and only the final output is passed through learnable affine transformation.
BL(Deep) extends the BLhhalllow) architecture to more than two layers, enabling richer hicrar- chical compositions of UMPs while maintaining the same recursive structure. As before, only the final output is affine transformed.
The overall structure of BL((hallow) and BL(Deep) be expressed in unified form, where the shallow case corresponds to L < and the deep case to L > 2:
BL(x,y) = WL BL (2(
(5)
Learning Objective. The response y may contain both discrete and continuous components. For discrete responses, we directly apply cross-entropy (Kullback & Leibler, 1951) on ylise, For contin- uous responses, since the compositional utility function is analogous an energy function (LeCun et al., 2006), we employ denoising score matching (Vincent, 201 on cont, The final objective combines the two with nonnegative weights 7a. Te:
(()) a E [ llolog P- Py x)] 76 E. V goomt log Pogp-(S (y cont x) (f )| (6)
4

---

Published as a conference paper at ICLR 2026
Implementation Details. Here, describe the key implementation choices for the general form of BL, taken as defaults unless otherwise noted. Further details are provided in Appendix D.3.
Function Instantiation. Following equation 4, we instantiate the function x(x,y)
y) = Ng (xnh(p.(xx.))) ReL
yx(
where Pu: Pe: Ps are polynomial feature maps of bounded degree, providing interpretable rep- resentations of utility, inequality, and equality terms, respectively. The bounded tanh reflects the principle of diminishing marginal utility (Jevons, 2013). a commonly assumed principle in behavioral science, while ReLU and introduce soft penalties for constraint violations.
Polynomial Maps. In BL(Single), the structure of polynomial maps is optional. BL(Shallow) and BL(Deep), each B-block employs affine transformations as its polynomial maps, with higher-degree and interaction terms omitted by default for computational efficiency.
Skip Connections. For deep variants, skip connections can be optionally introduced to improve representational efficiency.
More detailed architectural descriptions for this section are provided in Appendix D.
Theoretical Guarantees. Under the given architecture, the BL framework has universal approxi- mation power: it can approximate any continuous conditional distribution arbitrarily well, provided that BL has sufficient capacity, as stated below. The proof is given in Appendix E.2.
Theorem 2.3 (Universal Approximation BL). Let C IRd and y C R be compact sets, and let p"(y x) be any continuous conditional density such that p" (y x) >> 0 for all (x,y) E X x y. Then for any t >> and E >> O, there exists a finite BL architecture (with depth and width depending on E) and parameter 0 such that the Gibbs distribution in equation 3 satisfies
sup KL(p"( (x1x) [| xx)))) <E. xEX
(8)
Interpretability. Alongside its expressive power, BL also exhibits strong intrinsic interpretability. (i) Each B-block can be expressed in symbolic form as an optimization problem (UMP): the tanh term defines the objective, the ReLU term corresponds to an inequality constraint, and the absolute- value term corresponds an equality constraint. Thus, BL(Single) can be directly expressed as a symbolic UMP, whereas deeper architectures can be interpreted as compositions of UMPs, with each block retaining interpretability. (ii) The polynomial basis ensures level of transparency comparable to linear regression, as both objectives and constraints can be represented as linear combinations of polynomial features. It can further be visualized as a computational graph (Fig- ure 6), in which each input's influence every B-block is traceable through compositional path- ways. (iii) (Deee))) composes B-blocks in a layered manner, forming a hierarchical optimization structure. Interpretation proceeds in bottom-up fashion, where the relation between any secutive layers can be viewed as aggregation or coarse coarse-grained grained observation. Overall, the interpretive pathway is: input features - micro-level optimization blocks - macro-level aggregation or coarse-gre saarse-grained behavioral constructs -> macro-level optimization systems. Appendix A provides a detailed description of this interpretation procedure. (iv) BL also offers multiple architectural de- grees of freedom that provide flexibility but simultancously affect the resulting interpretability. In deep variants, skip connections introduce cross-layer dependency structures that are modeled in sta- tistical physics (Yang & Schoenholz, 2017). Replacing polynomial maps with affine transformations preserves the underlying optimization semanties but reduces symbolic granularity, yielding a more qualitative rather than symbolic interpretation of each block. (v) BL can be interpreted as a single UMP when the final layer contains only one B-block, since all lower-layer structures aggregate into unified optimization problem. When the final layer contains multiple B-blocks, BL corresponds to a linear trade-off among multiple optimization problems.
2.3 IDENTIFIABLE BEHAVIOR LEARNING (IBL)
Beyond prediction and interpretability, the BL framework supports third fundamental goal: the identification of ground-truth parameters, which turn endows BL with the capacity for scientifi- cally credible modeling. We refer to this setting Identifiable Behavior Learning (IBL). In the
5

---

Published as a conference paper at ICLR 2026
Interpreting (S(ningle)
(c) Interpreting (L(Deep)
Buyen' Preferenios
Busgeet Conntmintt
Bayern Beliek
Ubildg in (1 ad.
Condrrint: C (x <o
Belef Equatin: 7 (k.)) m1l
Beherior
Luiity Landsoape
Curpeon
Heprescntativv Agent
T
Ceorse Grainieg
UMP of Duyers
tealy (1 (1 RM0 R.)
Buyers' Preferencaa
Localis-
Contter Baye
Tradooff Boaer
Trade-offs
D(P RM)
Badger Constraint
Course Grainins
+ (2224 CRUI BPP+
Buyers' Bellefs
Interpreting BL Laye
Ride- Seasitive Buses
Zaning Corine Biger
Scaative
Sewans Deper
shstera ikpe:
Graining
Princitive Preferences
Envirsnncot
Fiotturee
(d)
BDeee) Application: Hierarchical Optimization Systems in Science
Hierarchy of Needs
Hierarchical Social Organization
Renormalization in Physies
Microscopic
Coarse- Graining
Macroscopic
Langth
Figure 2: (a) Visualization and symbolic form of BL((ingle) trained the Boston Housing dataset, modeling the UMP (max U s.t. C < 0, T 0) of a representative buyer in Boston housing (details Section 3.2). Top: computational graphs of the polynomials inside the three penalty functions-tand (preference), ReLU (budget), and - (belicf). Each graph is respectively centered on tanh (U)), C. and T from left to right, with surrounding nodes representing input features. Directed edges (shown only if coefficient 2 0.3) indicate how each feature contributes to the corre- sponding term. Bottom: approximate symbolic formulation of the trained BL model as UMP. (b) The 1L[ architecture. Layer identifies two key micro-level preference types: the Economic- sensitive Buyer and the Location-sensitive Buyer. Layer 2 aggregates these two components into an effective representative buyer. (c) The BL(Dcep) [5,3,1] architecture. Layer 1 recovers five distinct micro-level housing preference types. Layer 2 identifies three macro-level trade-off types captur- ing different ways these primitive preferences interact. Layer 3 aggregates them into the overall representative buyer. Table 8 provides detailed descriptions of each type. BL(Deep) provides hi- erarchical explanation consistent with the coarse-graining principle (Kadanoff, 1966) in statistical physics, reconstructing the full micro-to-macro optimization hierarchy. In addition, the preference and trade-off patterns uncovered by BL(Deep) are well documented in the classical economics lit- erature (see Table 9). (d) BL can be applied broad class of hierarchical optimization structures in science, including hicrarchical need structures, hicrarchical social ooginlorgaaizatioonal structures, and renormalization style coarse- grained structures in physics.
IBL setting, we define the modular block as
(x. = -n AT (Uo, (x, - X (C (x.))) u (To (x,
Unlike BL, which uses general nonlinearities. the IBL architecture imposes stricter structural con- straints: dd and pd are strictly increasing, while Rd is symmetric and strictly increasing in In addition, all three functions are C'.. These properties ensure that each UMP block stays responsive

---

Published as a conference paper at ICLR 2026
and adjusts smoothly to objectives and constraints. In practice, we instantiate equation 9 as 1eth (x. y) = Agg (ann(pp(x,)) y)) - Ao (oftplus(p.(x.x)) ())
(10)
where (.)2 denotes elementwise square.
We design IBL in three architectural forms. Similar to BL, the IBL(Single) directly uses (x, y) as the compositional utility function. The [BI/Shallow) and (BI((Dep)) variants are defined recursively
yB) = wi B ..... BB (Bi (x,
121
(11)
where 18 stacks multiple parallel blocks B (x, y), and Wi is learnable affine transformation without bias. All other design choices follow the BL setting.
Theoretical Foundation. IBL admits favorable properties for ground truth identification. We be- gin by establishing identifiability, which is fundamental for statistical inference. We first state our key assumption (see Assumption E.1 for details).
Assumption 2.1. Let T denote the quotient space of atomic parameters. We assume that the map J + xx'' V + gt, is injective, and that any finite set of distinct atoms linearly indepen- dent. We further restrict attention to minimal representations with no duplicate atoms and fixed canonical ordering.
Theorem 2.4 (Identifiability of IBL), Under Assumption E.I, the architectures (BI(Single), (SL(Slallow) and IBL(Deep) are identifiable in the parameter quotient space O.
Theorem 2.5 (Loss Identifiability of IBL), The IBI. model is parameterized by 0 E O. Suppose θ is compact. Then under Assumption E.1, the population loss L defined in equation 6 satisfies:
Ne 0, it admits unique minimizer in the quotient space θ;
2/ 0, admits a unique minimizer in the scale-invariant quotient space +.
Theorems 2.4 and 2.5 together establish the identifiability IBL. Theorem 2.4 shows that if two IBL models of the same structure induce the same compositional utility, then their parameters coincide up to an equivalence class. Theorem 2.5 further extends this result to loss-based identifiability. These results jointly imply that IBL admits a unique parameter estimate up to an equivalence class, and thus yields intrinsic interpretability that is unique up to the same class.
Building on identifiability, Theorem 2.6 establishes statistical consistency of IBL: under com- pactness of the parameter space, the learned parameters converge in probability to minimizer of the population loss as the sample size n + oc. the model is correctly specified, the estimator further converges the ground-truth parameter. recovering the true underlying model, thereby endowing IBL with the potential to recover the ground-truth model.
Theorem 2.6 (Consistency of IBI.). Let E\t denote the relevant parameter quotient space: = le > and E θ % 0. Let 0n, E arg mingee (,) denote the empirical minimizer, and let 6* E arg minsee M(0) denote the population minimizer. Then under the conditions of Theorem E.5,
0, E, 0" in E,
M((.)) D, M(
Moreover, if the model is correctly specified (i.e., the data distribution is realized by some 6* E O), then 6* = 6r in E, and thus 0,, E, 0*
Correct specification is a strong and often unrealistic assumption. Fortunately, the IBL frame- wirk-like BL-also enjoys universal approximation guarantee (Theorem E.6). Building this result, we further establish the universal consistency of IBL: even under misspecification, IBL. capable of recovering the ground-truth model with sufficiently large sample sizes.
Theorem 2.7 (Universal Consistency of IBL). Under the conditions of Theorem E.7, for any admis- sible data-gent gentaaggnerttigg distribution p satisfying the regularity assumptions of Theorem E.6, the IBL posterior sequence {P satisfies
sup KI(p (x) Pl (. (x)) E, ZEN
Lee. the learned conditional distributions' (pg_) converge in KL. to pt uniformly over x.

---

Published as a conference paper at ICLR 2026
Specifically, this result implies that, even under model misspecification, the learned predictive dis- tribution P parameterized by the IBL model, converges uniformly in KL to the true conditional distribution pl, provided that the capacity of the IBL architecture grows with the sample size Formal statements and proofs of theorems in this part are deferred to Appendix E.3.
3
EXPERIMENTS
In this section, we conduct four groups of experiments to systematically evaluate the capabilities of BL. Due to space constraints, details are provided in Appendix F.
3.1 STANDARD PREDICTION TASKS
Performance Scores
Performance Scores duisets F1-Macra hocpla!
Performance Ranks
seeda AUC hosplat
10 dussets seeds
P1-Marm
DT
DT
DT
PolyLoginic
NAM
NAM
Logistic
RF
Polylaggiicc
Logistic
ElsttcNNt
Loginic
Polyl.gginic
ElasticNet
ElastiNNet
TakNa
SVGP
SVGP
SVGP
TabNet
MLP
MLP
MLP
TobNet
LightGBM
LightGBM
BL/Singte)
[Singgg)))
BLJSingte)
BLSShlllww
Liae
Reyonia Malled
BL (S)alllo))
BL(Shallow)
LightGDM
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.2
0.4
0.6
6
8
10
euppt
rank
Figure 3: Predictive performance of BL. and baselines. Left/Middle: relative AUC and F1-Macro gains over DT, sorted by mean (excluding BL). Right: mean F1-Macro ranks (1 better). BL achieves first-tier performance in both metries. Its variants rank second and third in mean F1-Macro rank, with (Shalloow) showing no statistically significant difference from state-of-the-art models.
BL accurate enough for standard prediction tasks? In this part, we evaluate the predictive performance of BL on 10 datasets (Table 2), covering diverse sample sizes, feature dimensions, and scientific domains. For fair comparison, we consider two BI. Prriants-Bl(Sinee BL((Sallow)-ang and compare them against 10 baseline models (Table 3) drawn from five method- ological families: neural networks, tree-based models, gradient boosting methods, Bayesian meth- ods, and linear regressors. All methods share unified preprocessing and tuning pipeline.
and
Predictive Performance. Figure 3 shows that BL attains first-tier predictive performance overall, achieving the best results among intrinsically interpretable models. Notably, BI.(Shallow) surpasses MLP, highlighting that BL delivers interpretability without sacrificing performance.
3.2 INTERPRETING BL: A CASE STUDY
How can BL be interpreted in practice? This part presents a case study using the Boston Housing dataset, where we train a supervised (L(Single) model with a degree-2 polynomial basis, BL[2, model (i.e.. a two-layer BL with two B-blocks in the first layer and one in the second layer), and a BLeDeep) model with a [5,3,1] architecture to predict median home values. We illustrate how the internal structure of BL can be interpreted as explicit optimization problems and their hier- archical versions, accompanied by complementary visualizations. Further details are provided in Appendix E3 and F.6.
8

---

Published as a conference paper at ICLR 2026
Symbolic Form of BL(Single) UMP. As shown in Figure 2, the trained BL((ingle) model can be interpreted as the UMP of a representative buyer in the Boston Housing market, comprising single objective, inequality, and equality term. Each term is represented by an estimated quadratic polynomial. For parsimony, we extract approximate symbolic expressions by retaining only the monomials with the largest (2-5) absolute coefficients, while collecting the remaining terms (in- cluding constants) into residual term R. For example, the utility term can be written as:
P.. 0.56 p2 0.6 RM + 0.57 RM- P R. (1 P) P- RR) + R.
We similarly simplify the budget and belief terms to recover an approximate UMP for the buyer. The full symbolic form is illustrated at the bottom of Figure
Interpreting BL(Single) via Model Visualization. Visualizations of each term's polynomial re- veal how features constitute the UMP. Three insights emerge from the visualizations in Figure 2. (i) Median housing price (MEDV) and average number of rooms (RM) are dominant across all terms-MMDD negatively affects utility in a near-quadratic form, while RM modulates its marginal effect. (ii) Proportion of lower-income residents (LSTAT) features prominently in the budget con- straint, reflecting implicit resource limitations.
(iii Crime rate (CRIM) appears only in the belief term, suggesting that buyers treat it as influencing others' behavior rather than their own preferences.
Interpreting BL(Deep). (1) Figure 2 (b) illustrates the optimization problems learned by the BI[2,1] model. Layer 1 identifies two micro-level preference types: an Economic-sensittve Buyer, whose utility and constraint terms load primarily on ZN (Large-lot residential share) and LSTAT (Proportion of lower-income residents); and a Location-sensitive Buyer, driven mainly by CHAS (Charles River indicator) and RAD (Highway accessibility). Layer 2 aggregates these basic pref- erences, yielding effective "representative buyer" that integrates the two preference types. (2) Figure 2 (c) presents the internal structure of the BL[5,3,1] model. In Layer 1, BL recovers five distinct micro-level preference types characterizing heterogeneous patterns in the housing market. Layer 2 identifies three macro-level representative agents, each capturing a different macro-level trade-off' among the basic preferences. Layer then aggregates these components into single high-level mechanism, yielding the overall representative buyer. Table 8 provides detailed descrip- tions of each type. (3) Beyond interpretability, we find that each preference pattern and trade-off recovered by BLeeeep) aligns with established findings in the economics literature (see Table 9). This indicates that BL successfully reconstructs underlying scientific knowledge.
3.3 PREDICTION ON HIGH-DIMENSIONAL INPUTS
Is BL scalable to high-dimenssonal inputs? We evaluate BL against the energy-based MLP (E- MLP) baseline across network depths d E {1, 3,3, with all models implemented without skip connections. Experiments are conducted on four datasets spanning both image and text domains, and are evaluated using six metrics: in-distribution accuracy, calibration metrics (ECE and NLL), and OOD robustness metrics (AUROC, AUPR, and FPR @95). For OOD evaluation, we adopt symmetric ID++OOD splits, using MNIST (LeCun et al., 2002) and Fashion-MNIST (Xiao et al., 2017) as one pair, and AG News and Yelp Polarity (Zhang et al., 2015) as another. E-MLP and BL are controlled to have comparable parameters.
Scalability on High- Diigh-Dimesiionll Inputs. Figure 4 and Table present results for BL and E-MLP across network depths. On image datasets, the two models exhibit comparable in-distribution ac- curacy, while BL generally achieves stronger out-of-distribution detection performance on Fashion- MNIST at similar accuracy levels. On text datasets, BL consistently improves ID accuracy over E-MLP across depths. However, OOD detection behavior varies by dataset: BL outperforms E- MLP on Yelp, whereas E-MLP shows better OOD discrimination on AG News. BL also achieves better calibration metrics (ECE and NLL; Table 13).
Downward Shift of the Pareto Frontier. Table II reports the parameter counts of BL and E-MLP across four tasks, and Tables 12 summarize their runtimes. The two models have highly comparable parameter sizes. Across datasets. BL exhibits slightly higher training time than E-MLP. Combining these results with their comparable predictive performance and the intrinsic interpretability of BL, in contrast with the black-box E-MLP, indicates that BL achieves a downward shift of the Pareto frontier.

---

Published as a conference paper at ICLR 2026
Bl E-MLP Image Daanda
Table I: ID accuracy and OOD AUROC (%) on image and text datasets. BL and E-MLP are evaluated at depths 1-3 with matched parameter counts, both without skip connec- tions. Top-two per column are blue and red.
Image Datasets
Model
MNIST
Fashion MNIST
Accuracy
OOD AUROC
Accuracy
OOD AUROC
OODAUROC
E: E.MLP (depth=1)
98.15 = 0:07
88.72 1.36
88.79 + 0.29
90.57 1
Bl. vu, E-MP on Text Duteetts
(depth=1)
97.97 # 0.18
91.17
2.88
89.26
+ 0.22
91.89
0.71
4.4
E: E-MLP (deptli=2)
98.11
90.32 +
88.88
0.36
84.61
BL (depth=2)
98.05 = 0.12
90.57
+: 2.49
88.96
+: 0.39
89.87
2.45
E-MLP (depth=3)
98.14 1 0.11
87.76
2.55
89.33
0.25
83.13
1.90
700 10
BL. (depth=3)
97.93 0.27
92.92
88.79
0.25
89.24
Text Datasets
Model
AG News
Yelp
Accuracy
OOD AUROC
Accuracy
OOD AUROC
E-MLP (depth=1)
86.05 0.15
87.95 + 2.2
89.86 :: 0.09
79.35
0.86
(depth=1)
0.12
83.15
91.53 + 0.02
84.31
E-MLP (depth=2)
85.11
0.29
84.92 5.40
89.59 + 0.22
77.55
2.16
Comparison of BL and E- MLP on image and text datasets; d de- notes model depth.
Figure 4:
BL (depth=2)
87.72 1 0.64
78.68
1: 5.09
90.77 1 0.19
81.80
1: 1.06
E-MPP (depth:3)
89.59
+ 0.53
78.55
BL (depth=3)
88.00 0.30
82.33
90.99
0.37
83,58
3.4 CONSTRAINT ENFORCEMENT TEST: HIGH- DIMENSIONAL ENERGY CONSERVATION
To evaluate whether the learnable penalty terms in BL enforce near-hard constraints under finite tem- perature, we isolate the penalty mechanism and test it on a high-dimensions energy-conservan constraint. The utility term is removed, allowing violations to be analyzed functions of tempera- ture 7 and penalty scale À (details in Appendix F.5).
Constraint enforcement. Figure 5 shows that BI. achieves near-hard constraint enforcement der finite temperature and penalty scaling. Violations decrease substantially as 7 decreases or increases. At around À 25 and v 0.01, the 64-dimensional energy-conservation constraint is enforced within 10-2 error. Curves remain mostly smooth and monotone in 64 dimensions, indicat- ing stable Langevin sampling and effective penalty enforcement.
Vialation r (2- 25.0.)
Violation vs. (rr-A(==00000
meenn IC percentlle ict
10
70
10
10
10
menn ICT
percentile
10
10
10
10"
10
10"
10
10'
Temporature
Peralit weight
feasible fraction vs. z (1=25.0)
feasible fraction v ( 0.050)
0.8
P(ltsyy) 50.1)
0.8
0.6
0.4
0.2
0.2
0.0
10
10
Temperaltur +
Penalty weight 2
Figure 5: Constraint enforcement test of the BL penalty block on energy congy-conserrvatio constraint.
The figure reports violation statistics [T y)1 when varying the temperature (left side of panel) and the penalty weight À (right side of panel).
10

---

Published as a conference paper at ICLR 2026
4 ACKNOWLEDGEMENTS
We would like to thank Prof. Dr. Philipp Hennig, Shu Liu, and Prof. Dr. Sen Geng for their helpful discussions and valuable suggestions. We are also grateful to participants of the Xi'an Jiaotong University seminar for their constructive feedback.
We acknowledge the public computational resources provided by the University of Tübingen.
Finally, we sincerely thank all anonymous reviewers for their insightful comments. In particular, we appreciate Reviewer sGAR for the highly constructive advice. The energy conservation constraint experiment was added following a suggestion from Reviewer sGAR.
any errors remain, they are solely our responsibility.
REFERENCES
Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. arXiv preprint arXiv:2004. 13912, 2020.
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34:4699 4711, 2021.
Ravindra K Ahuja and James Orlin. Inverse optimization. Operations research, 49(5):771-783, 2001.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631, 2019.
Genevera I Allen, Luqin Gan, and Lili Zheng. Interpretable machine learning for discovery: Statis- tical challenges and opportunities. Annual Review of Statistics and Its Application, 11, 2023.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018
Philip Anderson. More is different: broken symmetry and the nature the hierarchical structure of science. Science, 177(4047):3333390) 1972.
Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning certifiably optimal rule lists for categorical data. Journal of Machine Learning Reseurch, 18(234); 1-78, 2018.
Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6679-6687, 2021.
Alejandro Barredo Arricta, Natalia Díaz Dodz-Rodriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges to- ward responsible ai. Information fusion,
W Brian Arthur. Complexity and the economy. In Handbook of Research on Complexity. Edward Elgar Publishing. 2009.
Anil Aswani, Zuo-Jun Shen, and Auyon Siddiq. Inverse optimization with noisy data. Operations Research, 66(3):870-892, 2018.
Santiago R Balseiro, Omar Besbes, and Gabriel Y Weintraub. Dynamic mechanism design with
hudget-constrained buyers under limited commitment. Operations Research, 67(3):711-730, 2019.
Patrick Bayer, Fernando Ferreira, and Robert McMillan. A unified framework for measuring pref- erences for schools and neighborhoods, Journal of political economy, 115(())588 638, 2007.
11

---

Published as a conference paper at ICLR 2026
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety-a review. arXiv preprint Stti:22404.148822 2024.
Steven T Berry, James A Levinsohn, and Ariel Pakes. Automobile prices in market equilibrium:
Part i and ii, 1993.
Sandra E Black. Do better schools matter? parental valuation of clementary education. The quarterly journal of economics, 114(2)5777-59, 1999.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemic Elhadad. Intel- ligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1721-17300 2015.
Timothy CY Chan, Rafid Mahmood, and Ian Yihang Zhu. Inverse optimization: Theory and appli- cations. Operations Research, 73(2):10461100 1074, 2025.
Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. Node-gam: Neural generalized additive model for interpretable deep learning. arXiv preprint arXiv:2106.01613 2021.
Kenneth Chay and Michael Greenstone. Does air quality matter? evidence from the housing
market. Journal of political Economy, 113(2):376-424 2005.
Gerard Debreu. Theory of value: An axiomatic analysis of economic equilibrium, volume 17. Yale University Press, 1959.
Finale Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 1702.08608, 2017.
Jeffrey A Dubin and Daniel L McFadden. An econometric analysis residential electric appliance holdings and consumption. Econometrica: Journal of the Econometric Society, pp. 345-362, 1984.
Ronald Aylmer Fisher. The genetical theory of natural selection: complete variorum edition. Oxford University Press, 1999.
Randy A Freeman and Petar V Kokotovic. Inverse optimality in robust stabilization. SIAM journal on control and optimization, 34(4):1365-139) 1996.
Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
Stephen Gibbons and Stephen Machin. Valuing rail access using transport innovations. Journal of urban Economics, 57(1): 168 2005.
Josiah Willard Gibbs. Elementary principles in statistical mechanics: developed with especial ref- erence to the rational foundations of thermodynamics. C. Scribner's sons, 1902.
Edward L Glaeser and Joseph Gyourko. The impact of building restrictions housing affordability. Federal Reserve Bank of New York, Economic Policy Review, 2002:1-19, 2002.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings. 2010.
S-P Han and Olvi L Mangasarian. Exact penalty functions in nonlinear programming. Mathematical programming, 17(1):511269, 1979.
W Michael Hanemann. Discrete/continuous models of consumer demand. Econometrica: Journal of the Econometric Society, pp. 541-561, 1984.
Trevor Hastie. Generalized additive models. Statistical models in S, pp. 249-307, 2017.
12

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffry E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8)1171-11800, 2002.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Jaehwi Jang, Minjae Song, and Daehyung Park. Inverse constraint learning and generalization by transferable reward decomposition. IEEE Robotics and Automation Letters, 9(1):279-286, 2023.
William Jevons. The theory of political economy. Springer, 2013.
Leo P Kadanoff. Scaling laws for ising models near Physics Physique Fizika, 2(6):263, 1966.
Rudolf Emil Kalman. When is linear control system optimal? 1964.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422-440, 2021.
Amr Kayid, Nicholas Frosst, and Geoffrey E Hinton. Neural additive models library, 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Arezou Keshavarz, Yang Wang, and Stephen Boyd. Imputing convex objective function. In 2011 IEEE international symposium on intelligent control, pp. 613-619. IEEE, 2011.
Been Kim, Martin Wattenherg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp. 2668-2677. PMLR, 2018.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference machine learning, pp- 5338-5344 PMLR, 2020.
Mathias Kraus, Daniel Tschernutter, Sven Weinzierl, and Patrick Zschech. Interpretable generalized additive neural networks. European Journal of Operational Research, 317 2024.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79 86, 1951.
Lev Davidovich Landau and Evgenii Mikhailovich Lifshitz. Statistical Physics:: Volume 5, volume 5. Elsevier, 2013.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2224 2002.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy- based learning. Predicting structured data, 100), 2006.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553):436 444, 2015.
13

---

Published as a conference paper at ICLR 2026
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of inter- pretability is both important and slippery. Queue, 16(3):31-57,01 2018.
Guiliang Liu, Sheng Xu, Shicheng Liu, Ashish Gaurav, Sriram Ganapathi Subramanian, and Pascal Poupart. A comprehensive survey on inverse constrained reinforcement learning: Definitions, progress and challenges. arXiv preprint arXiv:2409.07566 2024a.
Shicheng Liu and Minghui Zhu. Meta inverse constrained reinforcement learning: Convergence guarantee and generalization analysis. International Conference on Learning Representations, 2024.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks. arXiv preprint vrXiv:2404.19756 2024b.
Lars Ljungqvist and Thomas J Sargent. Recursive macroeconomic theory. MIT press, 2018.
Luca Longo, Mario Breic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger, et al. Explainable artificial intelligence (xai) 2.0: A manifesto of open challenges and interdisciplinary research directions. Information Fusion, 106: 10:30033,, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement
learning. International conference on machine learning, pp. 7390-7399, PMLR, 2021.
Andreu Mas- MaleC,,ell, Michael Dennis Whinston, Jenry R Green, et al. Microeconomic theory, vol- ume 1. Oxford university press New York, 1995.
Daniel McPadden. Conditional logit analysis of qualitative choice behavior. 1972.
Daniel McFadden. Modelling the choice of residential location. 1977.
Maxime Méloux, Silviu Maniu, François Portet, and Maxime Peyrard. Everything, everywhere, all at once: Is mechanistic interpretability identifiable? arXiv preprint arXiv:2502.20914. 2025.
Melkamu Mersha, Khang Lam, Joseph Wood, Ali K Alshami, and Jugal Kalita. Explainable artificial intelligence: A survey needs, techniques, applications, and future direction. Neurocomputing, 599: 128111, 2024.
Melanie Mitchell. Complexity: A guided tour. Oxford university press, 2009.
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Hand- book of econometrics, 4:21 11-2245, 1994.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2, 2000.
Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv: 2019.
Daehyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. Conference robot learning, pp. 1005-1014, PMLR, 2020.
Ori Plonsky, Reut Apel, Eyal Ert, Moshe Tennenholtz, David Bourgin, Joshua C Peterson, Daniel Reichman, Thomas Griffiths, Stuart Russell, Even Carter, et al. Predicting human decisions with behavioural theories and machine learning. Nature Human Behaviour, pp. 1-14, 2025.
14

---

Published as a conference paper at ICLR 2026
Karl Popper. The logic of scientific discovery. Routledge, 2005.
Frank Plumpton Ramsey. A mathematical theory of saving. The economic journal, 38(152):543- 559, 1928.
Zhi- Yong Ran and Bao-Gang Hu. Parameter identifiability in statistical machine learning: a review. Neural Computation, 29(5):1111-1203 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Ribana Roscher, Bastian Bohn, Marco Duarte, and Jochen Garcke. Explainable machine learning for scientific insights and discoveries. leee Access, 8:42200-42211 2020.
Sherwin Rosen. Hedonic prices and implicit markets: product differentiation in pure competition. Journal of political economy, 82(1):34-55, 1974.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206-215, 2019.
Paul Anthony Samuelson. Foundations of economic analysis. Science and Society, 13(1), 1948.
Herbert A Simon. A behavioral model of rational choice. The quarterly journal of economics, pp- 99-118, 1955.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33: 12422-282448 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint wwXiv:2011.13455 2020.
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference Machine learning, pp. 1064- 2008.
Berk Ustun and Cynthia Rudin. Supersparse linear integer models for optimized medical scoring systems. Machine Learning, 102(3):349-3)) 2016.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674) 2011.
Sewall Wright et al. The roles of mutation, inbreeding, crosshreeding, and selection in evolution. 1932.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprint
2015.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprini arXiv: 170.:0777 2017.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30, 2017.
Scungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. Deep lattice networks and partial monotonic functions. Advances neural information processing systems, 30, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
15

---

Published as a conference paper at ICLR 2026
A SCIENTIFIC EXPLANATION OF BL(DE)P)
BL(Deep) provides a form of interpretability that is consistent with hierarchical optimization struc- tures. In BL, each layer performs coarse-gratee -aaaiiingn of the optimization structure implemented by the layer below. An intuitive analogy is a corporate organizational hierarchy: lower-layer managers solve their own local optimization problems, while higher-layer managers aggregate and coordi- nate the outcomes of many such lower-layer problems to achieve broader organizational objectives. BL(Deep) follows the same principle-highe layers summarize, reorganize, and coordinate the so- lutions formed at lower layers.
This perspective aligns with many scientific domains characterized by multi- complexity, in- cluding (i) the formation of representative behavioral agents in behavioral sciences, and (ii) renor- malization in statistical physics, where fine-scale interactions compressed into effective coarse- scale potentials.
We describe the explanation procedure below. archical optimization structure-thh may refer to a multi-layer organizational structure composed of individual agents, or multi-scale physical system composed of interacting particles.
To build intuition, let us first consider generic hier-
Step 1: Bottom-layer interpretation.
Each bottom-layer block is optimization problem that directly receives inputs from the environ- ment. These blocks correspond to micro-level behavioral mechanisms, such as the decision rules of individual agents performing environment-facing tasks in an organization, the motion laws governing a single particle in statistical physics. Examining these bottom-layer blocks reveals the fundamental optimization principles followed by all units that directly interact with the environment.
Step 2: Layer-wise coarse-graining and micro-to-macro macro aggregation.
Blocks the next layer aggregate the outputs of lower-layer optimization problems through new optimization step, producing a coarse- -rasind behavioral summary. Each higher-level block repre- sents the effective optimization system that emerges from the interactions among many lower-level units, thereby capturing macro-level regularities distilled from micro-level mechanisms.
This micro-to-men to-mmcro transition consistent with many well-established scientific principles, includ- ing:
(i) Aggregation and coordination: in hierarchical organizations, the outputs of lower-level agents are aggregated, reallocated, and coordinated by higher-level agents to achieve improved organizational objectives.
(ii) Coarse-grained observation: in hierarchical behavioral systems, individual agents are grouped into categories that share characteristic optimization patterns; in statistical physics, many particles collectively form systems whose coarse sanseegraiied behavior is governed by effective potentials induced by microscopic interactions.
Step 3: Bottom-up reconstruction.
A global explanation obtained by tracing the hierarchy upward, following the model's micro- to-macro abstraction path: raw input features -> micro-level optimization blocks + macro-level aggregation and coordination or coarse-grained behavioral constructs - maero-level optimization system.
At each layer, we inspect the characteristics of each block and its associated optimization objective, as well as how these optimization problems evolve across layers. This reveals how each higher layer aggregates, coordinates. or coarse- coarse-grains grains the outputs of the layer below. Together, these observations yield compact multi-scale interpretation in which BL is understood as hierarchical optimization structure.
B DISCUSSION
In what follows, we discuss the limitations and future directions of Behavior Learning from the perspectives of theoretical foundations, architecture, and applications.
16

---

Published as a conference paper at ICLR 2026
Scalability of theoretical assumptions. The seatifaabbiiit---lltee statistical theorems constitute the core theoretical pillars of IBL, ensuring uniqueness of the interpretability and supporting its scientific credibility. Although these results hold under mild conditions, their behavior in large-scale, highly over-paree wver-parameterized architectures remains less well understood. This highlights the need for systematic investigations into the robustness, potential failure modes, and empirical boundaries of these guarantees when applied to modern large-scale learning systems.
Choice of basis functions. Polynomial basis functions enhance expressivity while preserving symbolic interpretability in BL (Single). However, high-order polynomials may introduce optimiza- tion instability, exacerbate sensitivity to initialization and normalization, and complicate training dynamics, Future work may explore alternative basis families-such as trigonometric, spline-based, or neural basis functions-and develop conditioning or normalization strategies that improve nu- merical stability without sacrificing interpretability.
Interpretable generative modeling. BL integrates several training techniques from energy-based models while retaining intrinsic interpretability. enabling interpretable generative modeling for vi- sion (e.g., image or video generation) and language (e.g.. large language models). Extending BL explicitly generative architectures in which outputs correspond directly to lumannbbddrsttaddablle and scientifically meaningful blocks represents compelling direction. Such extensions could yield generative systems with greater transparency, controllability, and scientific credibility compared to traditional black-box models.
Hybrid architectures for partial interpretability. A promising direction for future work is to develop hybrid architectures that integrate BL with black-box models in a principled way to achieve partial interpretabiliiy. Three avenues are particularly worth exploring: (i) Feature-level integra- tion. Black-box neural networks can serve as high-capacity feature extractors, while BL operates on the resulting learned representations to impose structured, optimization-based semanties. (ii) Decision-critical integration. BL blocks may be inserted specifically at high-risk or decision-critical components of the model, substantially reducing the interpretability and reliability risks associated with purely black-box architectures. (iii) Mechanism-lewe
integration. Because BL provides an inductive bias aligned with many real-world mechanisms, selectively applying BL to the parts of the system where such inductive bias is essential may yield models that better cap- ture the underlying ground-truth processes while retaining the flexibility of deep networks, thereby improving generalization performance.
optimization-drive
BL for scientific and social-scientific modeling. BL represents data composition of opti- mization problems, closely resonating with modeling paradigms in the natural and social sciences. Its competitive performance, intrinsic interpretability, and statistical rigor position BL as promis- ing framework for scientific machine learning. Future research may apply BL to domains such as statistical physics, evolutionary biology, computational neuroscience, and climate dynamics, well as behavioral science, economics, sociology, and political science-pane particullay in settings involving complex, partially formalized, or cognitively meaningful structures.
C RELATED WORK
C.I INTERPRETABILITY
Interpretability has become increasingly vital in machine learning (Lipton, 2018; Molnar, 2020), especially for scientific domains (Doshi- Velez & Kim, 2017; Roscher et al., 2020). Ensuring inter- pretability fosters transparency and reproducibility, and may further provide insights into underly- ing scientific principles. The ideal form of interpretability is intrinsic interpretability. in which a model's structure or parameters are directly understandable to humans. However, intrinsic inter- pretability is challenging to achieve in some widely used high-capacity models such as deep neural networks (LeCun et al., 2015). This has motivated post-hoc interpretability methods (Ribeiro et al., 2016; Lundberg & Lee, 2017), which seek to explain pre- pre-trained black-box model. While more broadly applicable, such explanations are often considered less suitable for scientific research (Rudin, 2019), they may compromise stability and faithfulness the model's 's decision process.
17

---

Published as a conference paper at ICLR 2026
performance-Interpettaal Trade-off. The limited intrinsic interpretability observed in high- capacity models has long been recognized as a central challenge. This is commonly framed as the ceeermaannce-nttereeaaaaae sion between predictive performance and intrinsic interpretability. High-performing models such as deep neural networks often lack transparency, whereas intrinsically interpretable models strug- gle to capture complex nonlinear patterns, Several efforts have sought to mitigate the perfor- mance-interpreetbbiity trade-off, which can be broadly categorized into four groups. (i) Additive models. Classical GAMs (Hastie, 2017), modern GA2Ms/EBMs (Caruana et al., 2015; Nori et al,, 2019), and neural variants such NAM (Agarwal et al., 2021) and NODE- GAM (Chang et al., 2021) preserve interpretabilityy by decomposing predictions into main effects and low-order interac- tions. (ii) Concept-
trade-off (Rudin, 2019; Arrieta et al., 2020), which posits ten-
models. Concept Bottleneck Models (Koh et al., 2020), TCAV (Kim et al., 2018), and SENN (Alvarez Melis & Jaakkola, 2018) map inputs into human-interpretable latent concepts and use them as intermediate predictors. (iii) Rule- and score-based systems. SLIM (Us- tun & Rudin, 2016) and CORELS (Angelino et al., 2018) generate transparent scoring functions or rule lists with provable optimality guarantees. (iv) Shape-constraings networks. Deep Lattice Net- works (You et al., 2017) and related monotonic architectures impose monotonicity and calibration constraints to encode domain priors while retaining flexibility.
Limitations in Scientifically Credible Modeling. The above approaches demonstrate strengths, yet two fundamental limitations restrict their applicability in scientific research. First, most methods are tool-centric modifications of machine learning architectures rather than frameworks grounded in scientific theory (e.g., optimization, dynamical systems, conservation laws). As recent surveys emphasize (Roscher et al., 2020; Karniadakis al., 2021; Allen al., 2023; Bereska & Gavves, 2024: Longo et al., 2024; Mersha et al., 2024), genuine scientific insight requires models linked to mechanistic principles, yet many interpretability techniques remain detached from such principles. Second, these approaches are typically non-identifiable (Ran & Hu, 2017; Méloux et al., 2025), meaning that multiple distinct parameterizations can explain the same data. This lack of unique- ness undermines their reliability for recovering ground-truth mechanisms and, in statistical terms, complicates consistency guarantees. As a result, the trained model may fail to converge to the true data-gent data-generating process as sample size increases (Newey & McFadden, 1994; Van der Vaart, 2000).
Relation to BL. BL also mitigates the aerformance-interrrttaabe trade-off. Unlike prior meth- ods, it is principle-driven and scientifically grounded, learning interpretable latent optimization structures directly from data. The framework applies broadly to domains where outcomes arise as solutions to (explicit or latent) optimization problems. It is also identifiable: its smooth and mono- tone variant, Identifiable Behavior Learning (IBL), guarantees identifiability under mild conditions, ensuring the scientific credibility of its explanations and supporting recovery of the ground-truth model under appropriate conditions.
C.2 DATA-DRIVEN INVERSE OPTIMIZATION
Inverse optimization (IO) (Ahuja & Orlin, 2001; Chan et al., 2025) is a core paradigm for learning latent optimization problems from observed data. Traditional IO aims to construct objectives or constraints that exactly rationalize small set of deterministic decisions. In contrast, data-driven 10 (Keshavarz et al., 2011; Aswani et al., 2018) focuses on statistically recovering the underly- ing problem from large-scale, noisy observational data. Inverse optimal control (IOC) (Kalman, 1964; Freeman & Kokotovic, 1996) extends this paradigm to dynamic settings, seeking to infer sequential decision processes from expert trajectories. Within machine learning. inverse reinforce- ment learning (IRL) (Ng et al., 2000; Wulfmeier et al., 2015) and inverse constrained reinforcement learning (ICRL) (Malik et al., 2021; Liu et al., 2024a) are prominent instances of data-driven IOC: Typically, IRL assumes fixed constraints and learns a reward function, whereas ICRL reverses this role. Both require repeatedly solving for (near-) )optimal policies and matching with expert demon- strations-incuriig high computational cost. In the behavioral sciences, particularly economics, numerous studies can be viewed as instances of the data-driven 10 paradigm. Foundational work (McFadden, 1972; Dubin & McFadden, 1984; Hanemann, 1984; Berry al., 1993) and related stud- ies typically posits theoretically grounded, parametric utility maximization problems (UMPs) and estimates their structural parameters from observed behavior.
18

---

Published as a conference paper at ICLR 2026
Relation to BL. The BL framework also falls under the paradigm of data-driven inverse optimiza- tion but differs notably from prior related work in both machine learning and behavioral science. Compared with IRL and ICRL, BL does not rely matching expert-demonstratee policies with the aim of improving task-specific performance. Instead, it is proposed as a general-purpose, scien- tifically grounded, and intrinsically interpretable framework that operates via low-cost end-to-end training with a hybrid CE-DSM objective. It jointly learns utility functions and constraints- direction that has received little attention in IRL and ICRL (Park al., 2020; Jang al., 2023; Liu & Zhu, 2024). Meanwhile, in behavioral science, related work typically formulates distinct utility maximization models under varying assumptions for specific decision contexts, and estimate their parameters accordingly. However, to the best of our knowledge, structure-free framework for learning UMPs that generalizes across contexts. BL fills this gap with a structure-free, data-driven approach that does not rely on fixed UMP structures.
existing work proposes
C.3 ENERGY-BASED MODELS (EBMs)
Energy-based models (EBMs) (LeCun et al., 2006) are a prominent data-driven IO scheme, rooted in the principle of energy minimization from statistical physics. They learn an energy function (z, y) that parameterizes the compatibility between imputs and outputs, inducing Gibbs distri- bution po (y x) xx exp(- Eag (x., })} that favors outcomes corresponding to low-energy solutions. In practice, this energy function is almost always instantiated by high-capacity neural networks, en- dowing the learned landscape with strong expressive power but also a black-box nature. Training EBMs typically relies objectives that circumvent the intractable partition function, with classi- cal approaches including contrastive divergence (Hinton, 2002), persistent contrastive divergence (Tieleman, 2008), and contss-conttastive estimation (Gutmann & Hyvärinen, 2010). A particularly influential line of work is score matching (Hyvärinen & Dayan, 2005) and its denoising variant (DSM) (Vincent, 201 1), which have underpinned breakthroughs in score-based generative modeling (Song& & Ermon, 2019; 2020) and laid the foundation for modern diffusion methods (Song et al., 2020).
Relation to BI. BL and EBMs exhibit a principled correspondence: BL is grounded in behav- ioral science and rooted in utility maximization, while EBMs are grounded in statistical physics and based energy minimization. BL adopts several training techniques common to EBMs, such as Gibbs distribution modeling and denoising score matching (DSM). However, the two frameworks differ substantially in model structure. EBMs primarily focus generative quality and typically employ black-box neural networks to learn an opaque energy function with little regard for inter- pretability. In contrast, BL is built on the utility maximization problem (UMP) and its equivalence to penalty formulations, yielding a principled and scientifically grounded framework. Its architecture composed of intrinsically interpretable blocks, each of which can be explicitly expressed in sym- bolic form as a UMP- foundational paradigm in behavioral science and a universal optimization framework. These properties enable BL to jointly achieve high predictive performance, intrinsic interpretability, and identifiability. thereby supporting scientifically credible modeling that extends beyond mere generative capability.
D ARCHITECTURE DETAILS
D.1 LEARNING SCHEME DETAILS
Input and output of the BL function. We formulate BL as a direct mapping from input-output pairs to compositional utility representations:
BL X x
- Rout
(z. + BL(x,)) E Rdo.
where the output dimension dout is chosen according to the modeling choice. This formulation intentionally allows BL to return either a sealar vector for each (xr,)) y); the following cases most common:
Scalar per candidate (pointwise evaluation). Set dout 1. Here BL(x,y) R is a scalar compositional utility evaluated for the single candidate y. This view is natural for continuous (regression density estimation) when one prefers to evaluate candidates individually.
19

---

Published as a conference paper at ICLR 2026
Vectorized over finite candidate set. y {y1 ( .U....... Mm is finite, one choose dout and define the vector vector-valued output by stacking evaluations over the candidate set:
BL(x,
Ba(x)
E R
BL (z, ymm)
This vectorized form is convenient for classification: evaluates all class candidates at once and yields single compositional utility vector per
Flexibility and equivalence. The scalar and vector modes are compatible: the vectorized form is simply batch of pointwise evaluations. Conversely, scalar pointwise evaluator can be used to assemble vector by repeated calls over candidate set. 'hhe choice between pointwise (scalar) and vectorized outputs is therefore an engineering choice that trades off computational efficiency and convenience.
Given a dataset D = B.) training and inference may use either mode: vectorized compu- tation where feasible (e.g., small finite Y), or pointwise evaluation when y is large or continuous.
Conditional Gibbs model. Let (x, v) ~ 'D with x E Rd and = (yاiic yydlscycomm yoont) E Veisc x Rm (discrete, continuous, or hybrid). BL induces conditional Gibbs distribution with temperature
T T>0 0:
Py xx) (x, (x)
(x exp(BL(s, [
For discrete y = (1), 3m we choose the vector-outpur formulation, we define
BL(x)= == [BL(2, 91)..... BL(x, ym)]] E R
so that the conditional distribution reduces to a softmax over this compositional utility vector:
P- - x) softmaxx ( BL(x)).
Behaviorally, 7 encodes noisy rationality; as r + 0, (- x) concentrates on arg max, BL v), corresponding to the deterministic optimal choice implied by the learned model.
Supervised, unsupervised, and generative uses. BL accommodates multiple regimes. (i) Su- pervised: take æ as input and y as label. For discrete y. one may either (a) adopt the vector- output formulation, where BL(x)= E I yields compositional utility vector over all classes and the likelihood is given by a softmax, or (b) adopt the scalar-outpl formulation, where BL(x, y) is evaluated separately for each candidate and then normalized across classes. For continuous y, BL naturally operates in the scalar-output mode, treating BL(x,, y) E IR as a compositional utility field. (ii) Unsupervised / generative: model marginal p(y) (xp(BI(y)/7) (empty x:) or a joint p(z, y) x (BL(BL(z, w)/T}; sampling the Gibbs distribution yields generator.
Learning objective. Since the response y may contain both discrete and continuous components, we estimate 0 by minimizing type-specific risk:
(()) [ log (1 disc |x)] + 7e IE V geont log py (1) (cont [x) + 22
(ydinc
y cont
where the first term is cross-entropy on the discrete component and the second is denoising score matching (DSM) the continuous component with grout yoout + 8, [N00 o²1). Set (Td. %e) = (1, 0) for purely discrete outputs, (0, 1) for purely continuous outputs, and (> 0, 0) for hybrids.
D.2 MODEL STRUCTURE DETAILS
In the main text adopted compact notation for BL: here present equivalent, more explicit matrix/vector formulation that makes dimensions, linear maps, and the per-head parameterizations explicit, which is useful for formal proofs and for implementation details.
20

---

Published as a conference paper at ICLR 2026
Fixed bases and head pre-activations. For a block imput (specified below), let
my (z) E R
m²(2) RR, m=(=) R
denote fixed basis (e.g., monomial) vectors. Learnable linear maps produce head pre-activations: u(z)= == Ma (z))) + bu E R" d() == m²)= b e R', t(=) := Mr me(z)) I2",
with M, E I. Brr.dd, Me E RRred M1 E Riixd and optional biases b..
Single BL block.
single modular block
B(2) = No (u(()) Lo dc(z)) - Xo v(t(2)),
(12)
where Ao E RR", A1 E R"c X2 E R are learnable weights, and Φ, va act coordinatewise with the roles specified in Theorem 2.1 (increasing for utility, penalty p for inequality violations, symmetric zb for equalities). Identifying
Uso (z, v) u(= = (z, (2, = dz = v)), Toz (z, v) = 6(z = (x, v)),
substituting into equation 12 recovers the main-text parameterization in equation 4.
Layer of parallel blocks. A layer By stacks de parallel copies of equation 12 with (possibly) distinct parameters Oz,z:
Be (zt)
Be((e)
E ERd
BBB. (ze)
We adopt the standard layered (feedforward) form:
21 = (x, y), so that each layer's input is simply the previous layer's output. This is the canonical feedforward architecture.
ze+1 Be(ee) (e=1....,, --1,
Optionally, one may allow each layer explicitly access the original inputs:
21
(x. v),
2e+1 TBr ((2, v), 22).
To improve trainability one may also use residual connections:
== 25 + Be(z))
Shallow/Deep composition and final affine readout. For depth L 2 1, the BL compositional utility is produced by final learnalee affine transformation of the top layer:
BL(x, v) W,. Br, (ZI.) + br.
(13)
with W,. Rixdz for scalar output or W, E Rmxds for vector output, and bias by. of matching dimension. The cases L (with d, 1), L < 2, and L > 2 correspond to BL(Single), BL(Shallow), and BL(Deep). respectively, exactly as described the main text.
D.3 IMPLEMENTATION DETAILS
D.3.1 FUNCTION INSTANTIATION
Default instantiation. (tanh, R l):
In practice, we instantiate equation 4 with the specific choice (0, p. v)
Bix, 0) A tani( ()) XX, (eL 2T Ter (z,,))]
(14)
Here λo, Å1 A2 learnable nonnegative weights. The bounded tanh captures saturation effects and diminishing returns in the utility head (Jevons, 2013), while ReLU and | impose asymmetric (one-sided) and symmetric (two-sided) penalties for inequality and equality violations.
21

---

Published as a conference paper at ICLR 2026
Variants and simplifications. Several variants of equation 14 often useful:
Identity utility head. Set Φ = id so the utility head uses raw polynomials:
B Uou ReLU(Cac) - Tezl-
Smooth penalty alternatives. Replace ReLU with softplus yield smooth inequality penalties, replace with Huber or squared penalties to modulate sensitivity near zero for equality terms.
Dropping heads. The framework is modular, may omit heads depending the task:
No T head: ignores symmetric deviations, yielding constrained maximization with only inequality penalties.
No C' head: if the T head is retained, the model reduces a maximization problem with only equality constraints; if T is also removed, it becomes a fully unconstrained maximization.
No U head: produces a pure (soft-)constraint model focusing on feasibility.
Strikingly, removing both U and 7 leaves only piecewise-linear ReLU penalties; when fol- lowed by final affine readout, the resulting architecture becomes highly similar to standard MLP-ste suggessgg that MLPs may be viewed as a closely related special instance within the broader BL framework.
D.3.2 POLYNOMIAL FEATURE MAPS AND LINEAR REDUCTIONS
We adopt pragmatic default: low-degree polynomial maps for single- blogk- models to maxi- mize interpretability, and use affine (degree- 1) maps inside blocks for shallow/deep stacks to control parameter growth and compute. Below state the instantiations and give the final block formulas used experiments.
BL(Single) polynomial instantiation. Let mD y) denote a fixed basis of monomials up to total degree D (e.g. D < 2):
mn(r, v) = [ur, vec(xx T) vec(ryT) T) vec(yy) ))....
Parameterize each map as a linear map on this basis:
Uno y). My (p +bu. Coo (z, y) Mc mp +bo,
Tor (2. Mr (2, 8) bbr,
with learnable matrices M. and biases b. The block becomes
B(x, 0) Ag 6(Mump bu) > p(McmD +bc) Ay (MMmD + br).
BL (Shallow/Deep) - linear-by-layer instantiation. For stacked architectures (Shallow/Deep) we use affine maps inside each block to keep per-layer complexity low:
Uey (a. y) Au [z; vl + bc.
(x. Ac [x; vl + bo. (x, Ar [z; vl br.
with learnable A. and b.. The corresponding block is
B(x, y: 0) 2n (sAu [x; + bu) >1 p(Ac(x bc) - No (Ar (2; y] bb).
On-demand higher-order terms. If diagnostics domain knowledge indicate underfitting, optionally augment the affine maps with selected higher-order terms or interactions. Concretely, this is done by appending small set of monomials (e.g. T,Jli, x². v the input vector [x:; y) and re-estimating the same affine maps A. This targeted augmentation preserves the base aftine parameterization, increases expressivity only where required, and keeps both computational and statistical costs modest while retaining interpretability.
D.3.3 SKIP CONNECTIONS
Skip connections are optional in our implementation. When beneficial, we often consider two pat- terns tailored to BL: DenseNet-style (concatenative) variant and ResNet-style (additive) variant.
22

---

Published as a conference paper at ICLR 2026
F
+
F az to
F= +ay d
Figure 6: Visualization of polynomial feature maps computation graphs, where nodes represent variables or outputs and edges represent their effects. The left panel illustrates the linear form == in which the single edge x + F directly encodes the marginal effect of on J. The middle panel shows the quadratic form F = ax.² + t where æ not only has direct edge x - but also acts on its own edge ("x + F"), thereby modifying the strength of its self-effect through higher-order contribution. The right panel depicts the interaction form F = az + by + cry + d, where y has an edge y/ - and, in addition, æ acts on this edge ("y -")",, thereby modulating the strength of y's contribution to FF. Symmetrically, y may act on the edge ("x FF), that each variable can reshape the other's 's effect through the interaction term.
Dense skip connections (DenseNet-style, concatenation). concatenation all preceding representations, mirroring DenseNet (Huang et al., 2017). Let
This variant feeds each layer with the
21 = ax; yl:
81 = (z1) E Rd,
For f 2 2,
:= x; y: s1:... :8-
8t :===Bl Be(z)) E Rd
The final compositional utility is read out
(I, = WL 8L + bL-
Pros. By exposing all earlier block outputs explicitly as inputs to later blocks, dense skips preserve a transparent feature trail: one can trace which intermediate B-block outputs enter downstream computations and the final affine readout. This often improves feature reuse and yields favorable interpretability at the block level.
Residual skip connections (ResNet-style, addition). This variant adds an identity (or projected) shortcut to each layer, as in ResNet (He et al., 2016). Define
21 = [x; y]:
81 = -B((2 E Rd
and for f 2 2,
:= 15 + 1I, 81-1, II, E deexde-1
where HTg is the identity d; de or a bias-free learnable projection otherwise. The readout again
BL(x,y) = WL 8L + +b
Skip Connections and Interpretability. Skip connections introduce explicit cross-layer depen- dency structures, form widely studied statistical physics and other scientific domains. Such structures enhance scientific interpretability by making long-range rang-range influences transparent. In behav- ioral and organizational sciences, they capture situations which lower-level agents directly affect higher-level decision makers without routing through intermediate layers. In physics, microscopic parameters can exert direct effects on macroscopic behaviors across multiple scales. Architecturally, ResNet-style skip connections model linear cross-layer dependencies, whereas DenseNet-style con- nections realize concatenative finfornation-epplicatnng dependencies. These mechanisms provide flexible yet interpretable pathways for representing hierarchical interactions.
E
PROOFS OF THEOREMS
E. UTILITY MAXIMIZATION PROBLEM (UMP)
Theorem 2.1 (Local Exact Penalty Reformulation for UMP)
23

---

Published as a conference paper at ICLR 2026
Let X C Rd and y C By be nonempty compact sets, and let U x x - R. : y - R
and T X x + Rp be C². Consider the Utility Maximization Problem (UMP)
max yxx s.t. C(x,y) S O, T(x.))
(15)
yEy
Assume there exisly feasible point y* E inn())) which is strict local maximizer of equation 15 and the Man-aaagaaa constraint qualification (2.1) holds at y (in the notation of Han & Mangasarian (1979)). Let ó R + IR be strictly increasing and C², and define p(z) = max 0} and (2)) = |z| (componentwise on R and ).). Then there exist Xa > x1 E R7+ and 22 E R2+++ such that y* is local maximizer of
max do (U((xy)) (Td((x.)) N3 (T((T(x,))) yEy
(16)
Proof. Fix x E and abbreviate
g(y) == C(x,y) R", My) = T(x,)) E R
Feasibility of y" means g(y") < 0 componentwise and h(y") 0.
Step 1: Convert to a constrained local minimization problem in the ambient space. Pick any Ao > 0 and define
f(y) = = =UX((())) yU(
(17)
Since strictly increasing, for any y1, Y2 we have U(x. > U(x. y2) if and only if f(y1) < f(y2). Hence y" is strict local maximizer of equation 15 if and only if is a striet local minimizer of
min f(y) s.t. g(y) S h(y) = 0.
(18)
yey
Now use the interior-point assumption y* E int( there exists > 0 such that B:, (y*) C V. Therefore, the notion of strict local minimizer over y at y* coincides with the ambient-space notion: for any function F, there exists E E (0, Eol such that
F(y") < Fly) Vy V (y(nBB))))")) \ (yy)
and only if
F(y*) Fly) E \ (y*}.
Hence y" is a strict local minimizer of the constrained problem equation 18 in the ambient-space sense.
Moreover, by assumption, the triple (f.g. h) is continuously diffcrentiable on neighborhood y".
Step 2: Embed vector weights into norm and build Han-Mangasarian penalty. Define the positive part 9+ (y) E RT componentwise by (9+ (y)); == max(g: (y)), 0}. Let A E E" and Ag E R4 be arbitrary for the moment, and define a norm on R by the weighted fn-orrm
[1(u, e)l AT |u| + (ol, (u, R. R.,
(19)
where | is componentwise absolute value. Since λ1 d2 have strictly positive entries, || llx is indeed norm.
Choose the scalar penalty function Q [0, + [0, oo) as Q(t) t. Then Q satisfies the penalty regularity condition (1.3) in Han Mangasarian (1979), in particular Q'(0+) > 0. Define for 2 0 the penalty function
a(y,a) a= f(y) +
(20)
Expanding equation 20 using equation 19 and Q(t) = yields
Ply, a) = - (U + \ (x)+ Xx [Th(y))]
(21)
Since p(g(y)) = (y) and x((())))) = [ componentisse, equation 21 can be rewritten as a)y,a) = - (U - ce (xTTTTTTM)))
+ + +
(22)
24

---

Published as a conference paper at ICLR 2026
Step 3: Apply Han Han-Mangasarian Theorem 4.4. By Steps 1-2, the functions f.g. h are C' neighborhood of y" is strict local minimizer (ambient-spacee sense) of the constrained problem equation 18, and the Han Han-Mangasaring constraint qualification (2.1) holds at y*. Therefore, by (Han & Mangasarian, 1979, Thm. 4.4), there exists ā 2 0 such that for every a 2 a. y is local minimizer of ax)
Step 4: Return to local maximization and ensure strictly positive vector weights. Choose any
max 0},
(23)
so in particular a > 0. Define the penalized maximization objective
Fly) == a(), = No (U((x a (oly)) a N (h(y))
Since y* is a local minimizer a(), it is a local maximizer of F. Finally set 1= a R++
X a e ET
Then Fly) equals
Aa (U((x)))) - (X1) (C(x,)))
- (x))) -X6) (Txx,)))
which is precisely the objective in equation 16. Hence y* is local maximizer of equation 16 over V. Since y" E i))), this is equivalent to local maximality in the ambient space sense. This completes the proof.
Theorem 2.2 (Universality of UMP). Let X and y be arbitrary nonempty sets. Let f X x x be an objective and let
for fgiliesg: {ōk fuktk
{hi}seJ
be (possibly empty, countable, or uncountable) families of real-valued constraint functions on X x
y. For each fixed X, consider the optimization problem
sup v(x.y) yEy
s.t. yx S 0 E Iz), yx 2 0 E2),, x,(x,))=00f) = (24)
Define (with the convention sup Ø == -oo and maxima taken in the extended reals)
y) = f(x,y)
C(x,y) == max{o 0, sup g)(x,y), sup
(-āx(x, y))
velg
kelz
T(x,y) == maxf0, 0, sup [x,(x,
jEJ
Then for every x E X, problem equation 24 is equivalent to the utility problem sup U yx yey
s.t. C(x,y) S0,
T(x,y) O,
(25)
in the sense that the feasible sets of equation 24 and equation 25 coincide; hence the optimal values coincide, and whenever maximizers exist, the argmax sets coincide. For minimization problems, replace U by -f.
Proof.
Fix X E X. Let
F(x) = fy= gx(xy)) SoVi Is, (x 2 4A4 E Iz, (xx,))=ovj-
denote the feasible set of equation 24, and let
f(x) = {y Ey: C(x,y) S T(x,)) = =)
denote the feasible set of equation 25. We prove that F(x) f(x).
(i) F(x) S F(x). Let y F(x). Then gi (x, y) S 0 for all i Ig, hence
sup g, y) < 0. sele
25

---

Published as a conference paper at ICLR 2026
Similarly, (x. y) 2 0 for all &t E Iz
implies -9k (xx - 0 for all hence
(-)x(x, y)) 0. kETz
sup
Moreover, yy( = 0 for all j E J implies (x) 0 for all E J, hence
sup
jeJ | (x, y) S 0
(with the convention sup Ø -00).
By definition,
C(x,y) - -maxy) 0, sup gi(x, y); sup
(x, (x) O, kelz
iels
and
T(x,y) ( 0, sup
[x(( O.
jEJ
Thus y F(x).
(ii) (x) C F(x). Lety F(x). Set
A A sup gi (x,
B: sup ketz
(x,
S = (x. y)l. jeJ
ielz
Then
C(x,y) max(0, A, B} 0.
Since 0 max(0, (,, A, B} always holds, we have max {0, A, B} 0, and in particular A S 0 and
B S Using the basic property the supremum, for every E Lg we have
gi y) $ sup gx A S 0,
vels
and for every k E 1> we have
(x, I sup
y)) = =
kElz
ie., ÿk (X. 2 O. Next, T(x, y) 0 means
= T(x, y) = (0, S},
hence S < Since 1h,(x, y) for every j E J and [h, (x, < 0, it follows that yx) 0 for all E J,
equivalently (x. = 0 for all j E J. Therefore y E F(x).
Combining (i) and (ii) yields F(x) fxx). Since U(x,y) = f(x,y) (and for minimization problems one may equivalently optimize -),, the two problems optimize the same objective over the same feasible set. Consequently, their optimal values coincide, and whenever maximizers exist, their argmax sets coincide.
E.2 BL ARCHITECTURE
Theorem 2.3 (Universal Approximation of BL). Let X C Idd and y C Rd be compact sets, and let py x) be any continuous conditional density such that p" x) > 0 for all (x,y) E X x V. Then for any T >> 0 and = > O, there exists a finite BL architecture (with some depth and width depending on E) and parameter 6* such that the Gibbs distribution
eBp 8))T
Pr xxx0)) = Jy exp BL (x,Y")/T)dy
(26)
satisfies
sup KL x (- xxx0)) < <E. xex
(27)
26

---

Published as a conference paper at ICLR 2026
Proof. Step 0 (bounded log-density). Define fxx,y) logp" and strictly positive on the compact set X' x J/ it attains positive minimum and finite maximum.
(y x). Since p* is continuous
Hence C(x x )) and is bounded.
Step 1 (the BL block contains a one-hidden-layer tanh network). Recall the clementary block
(xx, y; 0) X (pa - ReLLU v)) X (x)
(28)
Set A1 0 and A2 0. Choose (x, y) to be affine in [x:y], i.e. px((x,) W[x:)] +
TR
for some A E N. Then
( y; 0) dy tanb [x:y|++]]
(29)
which is a standard one-hidden-layee tanh network on the compact domain X x V.
If Ao I is unconstrained, equation 29 the classical universal approximation class. If instead one imposes Ao 2 0 componentwise, the same expressivity is retained because tanh is odd: for any scalar R, write Since -h is affine whenever h is affine, negative coefficients can be realized by duplicating hidden units and keeping the corresponding output weights nonnegative. Thus, up to a constant-factor increase in width, the block class contains signed linear combinations of tanh units.
a+
with a O, and note tanh(h) a* tanh(h) + a tanh(-h).
Step (uniform approximation of the target energy). By the universal approximation theorem for Lingle-hidden-lae networks with nonpolynomial activation (e.g., tanh), for any ö > 0 there exist a width and parameters such that
sup (x,y)xX'x3'
B(x,y:0) 0) -f((x < 8.
(30)
Define y(x,y) = B( and 17 5/T. Then equation 30 is equivalent
sup (xx))xx'x))
g(x,y) y(x <n.
(31)
Step 3 (uniform KL control). For each x E X' define
g(y x) = exp(g(x,)))
(32)
Sy exp(s(xx)))) dy'
The normalizer in equation 32 is finite because is continuous and y is compact. Let (x) = fy exp(9(x, y')) dy'. Since p-( (- x) is density, without loss of generality we may normalize the energy so that Sy (xxx' dy = From equation 31, for all (x, y).
(xx,y)
ex(xy) S e
Integrating over y E y yields
e - < (x(x)= S Se"
hence
|log Zx(x) Sn, Vx X.
(33)
Moreover,
log y" (y x)
(x.y)
= log eg(x.y) /2,xx))/2((()) (x) (f(x, - y)) + log Zy (x).
q(y x)
Taking expectation under p²( (- x) and using equation 31 and equation 33 gives
(T. x) g(( x)) = (xx)(1x)[((x Y) g(x, Y)] + log (x)
- 5 カ 2n. E
(34)
Step 4 (choose 5 and embed into BL). Choose 5 == ET c4/4, so that 1j - E/ Then equation 34 implies
KL xEX
x) q( xx) 2y = 8/2< <.
sup
Finally, the density q(- x) equals the Gibbs distribution equation 26 with energy BLg- (x,y)
((x, y; 8) (a finite BL architecture containing a single block), and temperature т. This proves the [
claim.
27

---

Published as a conference paper at ICLR 2026
E.3 IDENTIFIABLE BEHAVIOR LEARNING (IBL)
E.3.1 SETUP AND ASSUMPTION
Input-output space and data. Let C Rd and y C RR be compact sets. Assume the data distribution Px.y is supported X' y, and that there exists point 20 (zo, yo] in the interior of its support; that is, some open neighborhood of 2g has positive Px,y-measure All expectations taken with respect Px,y unless otherwise specified.
Parameter space and polynomial feature maps. The parameter space factorizes as
θ == Or x Oc x Or x W.
For Oy E ou, Oc E cc. and Or E Or we define polynomial feature maps
Pu X R*. Pe x R Ps P x R4
each of fixed degree and injective in their coefficients (i.e., distinct coefficients yield distinct func- tions). For single block, By, fc, Oy correspond to the parameters of the U, C, and T terms together with their respective external multipliers (e.g., penalty weights A). For deep network composed of multiple blocks,
(6y, fc. 0r) denotes the collection of all block-level parameters across the hierarchy, where Ou aggregates the parameters of all U-terms, Oc those of all C-terms, and Or those of all T T'termms (each including their associated multipliers).
The output component W. corresponds to the affine transformation in the final layer: W. = Rdd for single-output prediction, and W. = Rd xm for m-way classification, where d' is the output dimension induced by the preceding network, whether shallow or deep.
Identifiable base block. Let Au RR",, d1 Rd, and X2 Rd denote nonnegative weight vectors, treated as learnable parameters. We instantiate the identifiable modular block
gid (x, = Ao (an)(,(<x,)) sooppplluspp.((. v)) - X1 (px(,,
(35)
where (.)02 denotes elementwise squaring. By construction, the tanh and softplus heads are strictly monotone their arguments, while the quadratic head even.
We assume that each polynomial feature map (* u) contains no nonzero monomial independent of y: that is, no feature is pure function of æ or a constant. This ensures that 3dd(r, (x. y) is noncon- stant in unless all weights vanish.
We implement IBL. in three architectural forms, each producing compositional utility function over (z.y).
Architectures.
IBL(Single): A single block is used as the compositional utility,
IBL(x, y) = (x.
IBL(Shallow): Shallow IBL uses one or two stacked layers of parallel blocks. For instance, first layer
(x v) = [ w),,)...... Bid (x, 3)] E Rd4
feeds into bias-free affine map
IBL (z. y) W Bjd (z.
v),
where Wi e mmxd for classification and Wi R 1xd1 for scalar output.
IBee()ep):: Deep IBL extends the construction to depth L > 2, recursively defined as id (L v)) w))...)
IBL(x,)) wi. BBd (...
where each Bid stacks parallel blocks Bid (z. y). and W a bias-free affine transformation. The cases L = 1 and recover the Single and Shallow architectures, respectively.
28

---

Published as a conference paper at ICLR 2026
Induced conditional model.
Let IBL( y) denote the compositional utility function produced by the chosen architecture (Single, Shallow, or Deep). It induces the conditional Gibbs distribution
(Discrete
(m]) ply æ) = softmax,, (x, y)}.
(36)
(I /)/r}
(Continuous y)
ply x) = Jy exp 0)/T) dÿ
T >> 0 fixed.
(37)
Here т is a fixed temperature parameter. Thus, IBL predicts by defining a compositional utility landscape whose Gibbs distribution governs y given
Quotient parameter space.
Definition E.1 (Symmetry Quotient Space). Define the equivalence relation on θ as the smallest relation satisfying
0, 0%
(z, vs 020 112 (x,
ys (x,
020
for all i and (x,
The corresponding quotient space is
= 0/~ ~~
Explanation. The T --mompnentt is designed encode equality constraints, which are symbolically equations. Flipping the overall sign of such a constraint leaves the equation unchanged, so different parameterizations that differ only by sign should be regarded as equivalent.
Definition F.2 "Scale-Invarian Quotient Space). Define the equivalence relation ss on A by
0z 0'
3e such that s(x, ys 0) cs(x, y; 0')
The scale-invariant quotient space is then given by
θ 0==0/2
Explanation. classification, predictions depend only on relative compositional utility differences between candidate labels. From technical perspective, quotienting out global shifts uniform scalings is necessary: without this identification, the cross-entropy loss admits redundant parame- terizations that differ only by such transformations. At the same time, this quotient is natural and harmless, since it does not eliminate informative ratios between classes but merely discards absolute levels or scales that play no role in the softmax decision rule.
Loss Functions. We adopt a hybrid loss to simultancously accommodate discrete and continuous outputs. Speeifically, cross-ent roos-entroppy (CE) applied to discrete targets, while denoising score match- ing (DSM) is applied to continuous targets. Let Ye, Yd 2 0 with + > 0. The population risk, defined on the quotient parameter space, is given by
M(0) = >a El- po X)] + Te E[SDSSs X)), θ (0),
(38)
where π denotes the canonical projection from the original parameter space onto its quotient.
For continuous outputs Y E y C RRd., DSM implemented by perturbing the target with additive Gaussian noise y = Y + s, e ~ N(0, o2I,, and penalizing the squared discrepancy between the model score and the corresponding denoising score:
Vy X) 2 - -m1l x
V
SDSM((: X)
Er
(39)
20
In classification-only settings we set 7/ 0 (pure CE), while in regression-only settings we set Ta 0 (pure DSM).
For a single observation = (X, Y), we define the per-sample loss as
{(0; Z) na log PY X)] + %o SDSM (0; X).
(40)
The empirical criterion then takes the standard M estimmatiio form
(n) = n
Σ 1=1
{(0 Z.),
Z, = (X,, Y.).
(41)
29

---

Published as a conference paper at ICLR 2026
Key Assumptions.
Assumption E.1 (Global Atomic Independence and Injectivity). Let Ap be the atomic parameter quotient.
1. Injectivity the quotient. The map v - XR & 9 is injective.
2. Linear Independence. Atomic linear independence. Any finite collection of pairwise distinct atoms {9g, Jimi with V. E is linearly independent in xxx
3. Minimality. In all model instances we only consider minimal representations: no duplicate atoms and its corresponding linear coefficient the mixture is nonzero.
4. Canonical ordering. For each model instance, a fixed canonical ordering is imposed on the atom list.
Explanation. Assumption E, treats each identifiable block Bdd as an atomic building unit and im- poses four structural requirements representations built from these atoms. Together, these four conditions define a non-ambiguous, non-redundant, and canonical algebra of atoms: after quotient- ing by the natural symmetries, every model constructed from B-blocks admits a unique minimal representation (up to the prescribed equivalences). This structural regularity is the foundation on which identifiability statements are built: it guarantees that observing the model output (or the ob- jective it optimizes) allows one, in principle, to recover the underlying atomic components and their coefficients in the appropriate quotient sense.
Practical remark. In practice, these conditions can be encouraged or approximately enforced in two complementary ways. First, the design of atomic classes (choice of polynomial bases, interaction terms, and activation heads) can be chosen that injectivity and linear independence are more plausible by construction. Second, model selection and post-processing (e.g., pruning atoms with near-zero coefficients, enforcing deterministic tie-breaking rule for ordering) can be applied after training to realize minimality and canonical ordering. These practical measures make the theoretical assumptions operationally meaningful in empirical applications.
E.3.2 PROOF OF THEOREMS
Lemma E.1 (Identifiability of Linear Combinations). Let Z be set. For each 1,..., m, let dy be a parameter space and define atomic functions
9 == f(;:)));
+ = (j.bg)) E EV,
D3 is the disjoint union. Let be the quotient atomic parameter space, and denote its elements by v T.
where =- Unm
Define the quotient parameter space of the model as
12 E = ((R ((R (0)) 1)
((a1, (1)......... (ams Fm)).
j=1
The associated linear combination model is
S =
By virtue of Assumption E. the model is identifiable in the quotient parameter E: if = S on Z, then e
Proof. Suppose Sg Sg. on Z, i.e.
Σ aj ((3,))
Σ d (6
30

---

Published as a conference paper at ICLR 2026
Let U be the set of distinct atoms in the quotient v that appear either side, and for each E 44 let
B(6)
Σ aj Σ dy
j:
5':
be the net coefficient gú. Then
Σ NEW
B(b) gs 0.
By the linear independence condition (Assumption E. of pairwise distinct atoms in V we must have ((vb)= for all v U.
Furthermore, by the Minimality requirement (Assumption E. E.):3), each & appears exactly once on each side and with nonzero coefficient. Thus the two sides contain the exact same list of coef- ficient-atom pairs {(a,, d5)]]= and since a canonical ordering is imposed (Assumption E:1: follows that
=
Theorem E.1 (Identifiabilityy of IBL((Snglee). The IBL(Single) architecture uses the atom set (aanh(pu,), softplus(pes), (pr,))= i ..... dui ... doi 1,...
Under Assumption E.I, the model identifiable in the quotient space θ: if Blg = Big on X x y, then 0' in O
Proof. Write
m ByJ = Σ aj f(;;(()))))))) j=1
m m: du + do + de.
where each f(:: Φ5) is one of the atoms tanh(Pu,s). a)) softplus(po.). or (pr,)) and aj is the corre- sponding entry in (Au, Å1, A2), with a fixed ordering over all indices.
If BBj = B on X' y, then Lemma E.1 and Assumption E.1 imply that all atoms and coefficients must agree in the quotient atomic space V. Since the ordering fixed, this implies 0 I in O. L
Theorem E.2 (Identifiability of (Sh(Slallow)), The IBL(Shallow) architecture uses the alom set
Bid (x. }==1"
where each Bid X x y + IR is a single-block IBL module parametrized by 01 E O₁. The full parameter is denoted
= ((..., 01.d,). Wi) E :0:=(())) x mxxd,
Under Assumption 1. the mapping 0 + IBLshattow, is identifiable in the quotient space θ: if Shatton (x, 0) = IBI (x. y; 0') X x y,
then
0 0' in O
Proof.
Write the k-th output component linear combination of atoms:
di
(x, v)
Σ w
(1)
Bid (I.
lc
m.
where w/th denotes the (k, (tt entry of W
Suppose parameter tuples (Wi, (01, (01)))=)) and (Wi', (0'', (0,)))1 yield identical vector scores (k)
X' y. Then for each k have s(k) eg X y.
31

---

Published as a conference paper at ICLR 2026
Fix any k. Under Assumption E.I, Lemma E.1 ensures that the coefficient-aton pairs (K) {(u)) B0,, )== are uniquely determined (up to equivalence in the quotient +). In particular, for each j = 1,..., .....,d d1, we must have
(k) " 3
(K)
B
B
m, it follows that Wi Wi' and 01, f1 in the quotient Because this holds for all h 1, ,,,,,,,, parameter space for all
Thus 0' in 6, establishing full identifiability under fixed ordering.
Theorem E.3 (Identifiability of IBL(Deep)). Fix integers > and widths .i...,,d-1., The [BL(Deep) architecture uses the final-layer atom set
{ Bid or. (z, v) 1/th CR
where each B Rdd-- - R is scalar-valued block applied to the oulput of layer L-1. Only the first-layer blocks (E = 1) IBL(Single) modules in Theorem E.I. For architectures with skip connections, the final-layer atoms can be extended to include features (e.g.. from d2
earlier layers), which are treated as elements of BBd (x.y) 8==1
The full parameter
L
W E II (01)* 0 = ((0t))est
L.de
de x mxd2
E=1
Under Assumption F. the mapping 0 - IBI ys 0) identifiable in the quotient space +
Proof. Under the given architecture, the IBL(Deep) model ultimately takes the form
dv. (x, = Σ w
Bid
(x, v),
== 1,...,, m,
j=1
where each Big, is a scalar-valued function applied to the output of preceding layers. By treating the set [Bid #L. (I, 4) as the atom set, we reduce the model to an IBL(Shallow) form:
s(x, v) W our By v)-
Under Assumption E.1. Theorem E.2 applies, implying that the full parameter 0 ((Deghes, Wau) is identifiable the quotient space θ
Theorem 2.4 (Identifiability of IBL). Under Assumption E. the architectures IBL(Single), IBL (Shallow), and (Beep) are all identifiable the quotient space +
Proof. Immediate from Theorems E.1, E.2, and E.3.
Theorem 2.5 (Loss Identifiability of IBL). Let IBL y) denote an IBL model, and consider the conditional Gibbs distribution
exp(IBLo(a) y)) Po x)) Jy exp((BLe (z, y²)) dy"
Define the population risk on the symmetry quotient O in equation 38. Assume that the parameter space θ is compact. Then, under Assumption E.1, the following holds:
(i) Te the risk functional M admits unique minimizer in O. Moreover,
M( = =M(22)
0 02-
32

---

Published as a conference paper at ICLR 2026
(ii) Ye = 0. the risk functional M admits unique minimizer in the scale-invariant quotient θ
Moreover.
M(01) = M( = 0 =2.
Proof. Under Assumption E.I, the IBL, architecture identifiable modulo the symmetry group de- fined by θ as established in Theorem 2.4. Let 8" E arg mingge MM(0) and set --] == Pa (. r). Since is compact and the loss M contimuous, global minimizer exists. show that unique in the stated quotient.
Case To > 0. At any minimizer we have both po(: x)) = =p- x)) and Vy logpol | x)) v (- x)) a.c. Since
Vy log (3 æ) = V V,IBLo y) - v, y log (ι V yI,LLo (),
(the partition function (x)x)) is y-independent), score equality yields V, (TBI.p IBI.go (us a.c. IBL contains no g-independent terms. Therefore,
(x, y) IBLg (xz v)
a.c.
By Theorem 2.4 (identifiability in θ), the minimizer is unique in θ; in particular, M(Ø) M(22)
0 02.
Case To Here, M reduces to the cross-entropy risk, which minimized if and only Po(- I) p²(( z) almost everywhere. The cross-entropy loss depends on IBL y) only through its relative values across y. and is invariant under additive shifts and positive rescalings of the compositional utility, Hence, the loss depends only on the equivalence class E G}. As
result,
M( M((2)
0 0g
i.e. the minimizer is unique in θ.
Hence, the minimizer is unique in the stated quotient space. This completes the proof.
Theorem E.4 (Uniform M-estimation consistency (Newey McFadden, 1994, Theorem 2.1)). Let (A, d) be a compact metric space, and let Ln A + R be a sequence of random objective functions, with population objective L A + R such that:
Laa) is uniquely minimized at a² A;
is compact;
3. L(a) is continuous;
4. (a) 2 L(a) uniformly in a E A.
Then any sequence ân argminaee (a) satisfies &n 24 a*.
Theorem E.5 (Consistency IBI.) Let M be the population risk defined in equation 38, and let Mn denote its empirical analogue. Suppose:
{(X,, 1.) are i...d. samples;
is compact;
3. 0 + M(8) is continuous, and the loss class admits an integrable envelope such that
Mm. - -M()) E, 0:
sup
Let E denote the relevant quotient space ( if 0. if 0), and let 0 arg mineee My( and E arg mingee M()). Then
0, P 0 E
M().) E. M(8").
If the model is correctly specified (the data law is realized by some 0* E θ), then 8 = 0* in E, so 0. 2, 0*.
33

---

Published as a conference paper at ICLR 2026
Proof. Let E denote the relevant quotient space: E θ if %c > and E π 6 be the canonical quotient map. Since is compact and π is continuous and onto, E is compact. By assumption, M and M, are invariant under the corresponding symmetry, hence they factor through x:
= = = % = 0. Let
==))==mM)
Mn(()=== M.) (any 0 E x (S))-
These well-defined and continuous E\r because M is continuous on O. Moreover,
sup MM)(5) ((())) sup lee
MM, M(O) & 0,
so uniform convergence in probability holds on E.
By Loss Identifiability of IBI. (Theorem 2.5), M has a unique minimizer 8* =. Let &n arg millgg (5) (equivalently, choose 0n arg mingee (0) and set En x(,))) Then the conditions of Theorem E,4 hold on the compact metric space (E, d), whence
En 2 g" Since M is continuous on E and M(En) = M((.). M(E*) = _M(8") for any representative 0 π (S*), we also obtain
M( = M((n) ES M(gg) M(8").
If the model is correctly specified (there exists 0 θ inducing the data law). the striet propricty of the CE/DSM terms implies that the unique minimizer in the quotient is the class of 8*; hence 6, converges in probability to 0* in the corresponding quotient space.
Theorem E.6 (Universal Approximation of IBL), Let X C RRd and y C T be compact sets, and let p" (y) x)) be any continuous conditional density such that p" x)) 0 for all (æ, v) X x V. Then for any т > O and >> O, there exists finite IBL architecture (with some depth and width depending on E) and parameter 0* such that the Gibbs distribution
exp exp(IBLo (z, (2T
Pr (y 2z 0²) = Jy exp [BL (x, (z.1)//T)d dy
(42)
satisfies
sup KT. (.1 p- x; 0*) <
(43)
xEX
Proof. The argument follows the same construction in the proof of Theorem 2.3, with only nota- tional modifications due to the IBL parameterization. For brevity, the details are omitted.
Lemma E.2 (Sieve Approximation Lemma), Lei θ + [0, oc) be a complexity measure on the parameter space, and let (Cn)n21 be nondecreasing sequence with cn t sc. Define the sieve
e, = {0 θ C(0) S ca).
and for fixed data- generating distribution pt, set
8n(p')
inf sup KLp'' (- x)) po(. x)). o aEx
Then the following equivalent:
Sieve universal approximation: For every e > 0 there exists a constant C2 < oo such that
inf sup KT(p'( (x) po 12)) < e. d: 0:C(0)SC, aE
2. Vanishing approximation enoo: (xt)) 0 as oc.
Moreover, if each θ, is compact and + sup, KL(p" [[po) is continuous on θ,, then the infimum in 5,, (pt) is attained for every n.
34

---

Published as a conference paper at ICLR 2026
Proof. (1) (2). Fix 0 and let Ce(p))
) be as in (i). Since Cn 1 oc, there exists N such that
Cn 2 Ce(p)) for all n 2 N. Hence ,, 2 {0 C(0) I Ce(p") for all n 2 N, and therefore
8x(p²) inf sup (p)))) 6E
sup KL 0:c(0)CC.(p) x
inf
E,
for all n 2 N. Since (6,) nonincreasing in n (because O f), it follows that 8n (p(p)) & O.
(2) (1). Fix e O. By (ii) choose N such that 8.v (p²) Set Ce(pT) ex- Then sup (12o)))lpo) & inf sup KL (1po) = 8x(p)) < e,
inf (.C())
8e8x
which is (i).
The attainment statement follows immediately from compactness of On and continuity of sup, KL (p) on O...
Theorem E.7 (Universal Consistency of IBL), Consider parameter space O for class of IBL. models, and let C θ + [0, oo) be lower semi-continsous complexity measure (e.g. network depth, width, or parameter norm). Let (Cn)n21 be nondecreasing sequence with on t oo, and
define the sieve
O == 1 S c.
Assume:
1. The map 0 sup, (T( is continuous each compact On.
2. The sequence of empirical minimizers {0,} is relatively compact in Un On. as ensured by the uniform L.N together with compuctness and continuity.
Then for any admissible data-generating distribution p satisfying the regularity assumptions of Theorem E.6, the IBL posterior sequence (Po,) satisfies
KL(p' (-1 11 (- (-1x)) 2 0, #EN
sup
i.e. {Pa,) converges to p uniformly in x (in KL).
Proof.
Fix an admissible data law pl (satisfying the regularity of Theorem E.6). For E U, O define
F(0) sup KL(p'( po x)), 8, inf F(0).
=Ex
nEE,
Then Theorem E.6 and Lemma E.2 together imply that 5, 4 By assumption 1, F is continuous on each compact θ,,
Let 0, E mingee, (0) be any sequence of ERM solutions. We show F()..) 2, 0.
Step I (subsequence reduction and precompactness). Take an arbitrary subsequence (0n, )x- By assumption there exists a further subsequence, still denoted (8.,) k. and a (possibly k-dependent) index set N& < n& with a parameter limit 0xc @N (for some finite N) such that in.. + doc in probability. Passing to a further subsequence if needed, we may assume N₂ N.
Step 2 (risk domination against θ -approxximants) For each k pick 0g E ON with F(Ox) < n + 1/k (attainment follows from compactness and continuity of F on ①x). By the ERM property and uniform LLN on ON.
M((n.)) < .M(O0) + (1)
(k+oo)
Assume (w.l.o.g.) the CE component is present with a positive weight, so that the population risk decomposes as
M(0) const 7a Ex [KT(p (.1 X) 1l X))] 7a (D)
35

---

Published as a conference paper at ICLR 2026
with "a 0 (the Dnlly case is handled analogously by replacing KL with Fisher divergence).
Using Ex [KL & F(-). we obtain
lim sup Ex KL(p' (. (x) Po., (-1x)) < lim sup F(tx) & 8N-
k-+00
Hence, along the subsequence,
Ex KUp (-1x) (. (-1x))] 2 0.
Step 3 (identification of the subsequential limit). By continuity of the model map 0 + pa(- xx) (from Theorem E.6 regularity) and bounded convergence,
Ex [ (- x) Po (-])) 0.
Thus f(z) == KL(pt( 2) Po. (- x)) equals 0 for Px-a.e. æ. Since f continuous on compact X (by the same regularity) and Px has full support (admissible law), we conclude f(x) = 0 on X, i...
F(0x) sup f(x) = 0. xEX
Step 4 (conclude F(O..)) 0 in probubility, hence F(Oa) in probability). By assumption continuity F on ON and Onz + 3o in probability, we have F(ô..) 2g F(f=2) = O. Since the original subsequence arbitrary and every subsequence admits further subsequence with F(Omg) Sg the full sequence satisfies F(O,)
Therefore,
sup KI(p' (. (=1x)] [)) Pa. (- (=)=)) 2 0,
ie. Pa., (-1x) + p'( (x) uniformly in in KL.
F EXPERIMENTAL DETAILS
F.1 HARDWARE
Most experiments are conducted on a single NVIDIA L40S GPU. A small number of runs are performed laptop equipped with NVIDIA GeForce RTX 2050 GPU and Intel Core i7- 12700H CPU.
F.2 STANDARD PREDICTION TASKS
Datasets. In the Standard Prediction Task, we use 10 OpenML datasets across diverse application domains. Details are given in Table 2.
Baseline Models. For comparison, we include the following baselines: MLP. Neural Additive Model (NAM) (Agarwal et al., 2020; Kayid et al., 2020), ElastioNet, Random Forest, Stochastic Variational Gaussian Process (SVGP) (Gardner al., 2018), Logistic Regression, Decision Tree, TabNet (Arik & Pfister, 2021), Polynomial Logistic Regression, and LightGBM (Ke et al., 2017).
Data preprocessing. For all ten datasets, we apply consistent preprocessing strategy. Ordinal categorical variables are mapped to integer levels to preserve their inherent order. Nominal cat- egorical variables without natural ordering are transformed using one-hot encoding- Continuous variables standardized to zero mean and unit variance. Each dataset is randomly partitioned into train/vliiationntess splits with a 7:1:2 ratio.
Hyperparameter Tuning Protocol. We perform hyperparameter optimization for most models using the TPE sampler from the Optuna package (Akiba et al., 2019). with 50 trials per dataset. For each model and dataset, the tuned configuration is evaluated under random seeds.
36

---

Published as a conference paper at ICLR 2026
Table 2: Standard OpenML datasets used in our task. #Features denotes the number of input vari- ables (excluding the target and ID).
Name
Size
#Features
Task type
Field
German Credit
1,000
20
Binary cls.
Finance
Adult Income
48,842
14
Binary cls.
Economics
COMPAS (two-years)
5,278
13
Binary cls.
Law & Society
Bank Marketing
45,211
16
Binary cls.
Marketing
Planning Relax
182
12
Binary cls.
Psychology
EEG Eye State
14,980
14
Binary cls.
Neuroscience
MAGIC Gamma Telescope
19.020
10
Binary cls.
Physies
Electricity
45,312
8
Binary cls.
Electrical Engineering
Wine Quality (Red)
1,599
11
Multiclass
Chemistry
Steel Plates Faults
1,941
27
Multiclass
Industrial Engineering
Table Overview of baseline models in the standard prediction task
Methodological Family
Model Name
Standard MLP
Neural networks
Neural Additive Model (NAM)
TabNet
ElasticNet
Linear regressors
Logistic Regression
Polynomial Logistic Regression
Random Forest
Tree-based models
Decision Tree
Gradient boosting methods
LightGBM
Bayesian methods
Stochastic Variational Gaussian Process (SVGP)
37

---

Published as a conference paper at ICLR 2026
BL Model Hyperparameter Space. For BL(Single) and BL(Shallow), optimize cross-entropy loss for classification. Both Adam (Kingma, 2014) and AdamW (Loshchilov Hutter, 2017) op- timizers considered, and the better-perte setter-performing variant is reported for each dataset. No data augmentation is applied. Batch sizes are chosen in a dataset-specific manner.
BL(Single): A unified setting is reported across all experiments:
degreey [2]
degreec = [2, 2, 2].
degreer [2, 2].
params 0.01,
= 0.01, a) = 0.01, O = 0.01.
Here, degreey, degreec. and degree, denote the polynomial degrees the blocks that param- eterize (ar, y). C(r, y). and (sr, y) respectively. Lists indicate both the number of blocks and each block's degree: degrecy = [2] means a single quadratic block for U, degree = [2. 2. means three quadratic constraint blocks, and degree, = [2. 2] means two quadratic belief blocks, params initializes coefficients of all polynomial blocks, while Ag DAa. and a22 ini- tialize the UMP weights (Ao, λI, A2). The search grid is reported in Table 4.
BL(Shallow): We use global gradient clipping of 1.0 and an early stopping patience of 20 epochs without validation improvement. Shallow architectures with depth L I are considered. The search grid is reported in Table 5.
Baseline Model Hyperparameter Spaces. For baseline models, we also consider both Adam and AdamW for the neural network-based variants, and report results with the better-performing optimizer on each dataset. Batch sizes are tuned separately for each dataset. The detailed hyperpa- rameter search spaces are summarized in Table 6.
Table 4: Hyperparameter tuning space for BL(Single)
Model
Parameter
Search space
learning_rate
{le-3, 3, le-1] (64, 128, 256, 512) [1.0, 2.0, 5.03
BL(Single)
batch.size max.grad.norm
Table 5: Hyperparameter tuning space for BL(Shallow)
Model
Parameter
Search space
learning.rate
LogUniform 5e- 5e-3) {64, 128, 256, 512} Uniformint (1 3}
batch.size
n.layers n.first.layer
BL(Shallow)
{24, 30, 36, 403
n middle layer
[8, 6.
n last layer
{2, 4, 6)
we weight_decay
LogUniform 1e-1)
Table 6: The hyperparameter tuning space for baseline models used the standard prediction tasks
Model
Parameter
Search space
learning rate
LogUniform Ic Ic-5, 5, le-1} {32, 64, 128, 256}
batch.size
MLP
n_layers
UniformInt (2, 4)
hidden_size
Uniformint 32, 256)
weight.decay
LogUniform 1e-6, le-2]
learning.rate
LogUniform (eeremmmme--- (le-3, le-1) {128, 256, 512, 1024)
NAM
batch_size
patience
Uniformint( 30}
38

---

Published as a conference paper at ICLR 2026
Model
Parameter
Search space
alpha
LogUniform ( 1e-4, le+2]
11.ratic
Uniform(O.0. 1.0]
max marr
Uniformint 100, 2000}
tol
LogUniform 6, le 1e-2)
FlasticNet (SGD)
fit_intercept
(true, false]
learning.rate
{optimal, constant, invscaling, adaptive) LogUniform le-4, le-1}
eta0
validatioon.factionn
Uniform (0.05, 0.303
n.iter.no.change
UniformInt 20}
degree
{2. {2z, ε1. "elasticnet") LogUniform 1e+2]
penalty
C
PolyLogistic
11 ratic
(0.1 ("liblinear", "Ibfgs". "newton-cg", "saga") UniformInt(500 2000)
solver
max_iter
tol
LogUniform 5, 1c-33
C
LogUniform le- 3, 1e+2]
11.ratic
Uniform(0.0. 1.0}
Logistic (ElasticNet)
max-iter
Uniformint 100, 2000}
tol fit.intercept
LogUniform ( (1e-6, 6. le le-2) {true, false}
solver
("liblinear", "Ibfgs", "sag")
C max-iter
LogUniform/ 1c+2]
Uniformint 100, 2000}
LogisticRegression
tol
le LogUniform {True, False) Uniform 10.0)
le 2]
fit.intercept
intercept.scaling
learning.rate
LogUniform le 3e-20
batch.size
{128. 256, 512, 1024] {64, 128) Uniformint 16, 64)
TabNet
n_d=n_a
n_steps
Uniformint 3, 7}
ganma
Uniform 1.2, 1.7]
lambda.sparse
LogUniform le le- 6, le-33
criterion
"nnrropy, "logloss"" UniformInt 20}
max-depth
min.samples.split
UniformInt (2, 20)
min.samples.leaf min frattito leaf max features
Uniformint 1, 10)
DecisionTree
fraction
Uniform(0.0. 0.5}
("sqrt", "log2")
max_leaf_nedes
Uniformint 10, 1000)
min.impurity.decresee ccp.alpha
Uniform (0.0, 0.1
Uniform (0.0, 0.1]
kernel
{rbf, matern, rational -qaadrraiicuaccc LogUniform 10.0)
lengthscale
rq.alpha
Log 10.1, 5.0}
GP (SVGP)
num.indueing
Uniformint 100, 500)
learning rate
LogUniform 21
training iters
Uniformint 200)
n estimators max nexth
Uniformint 100, 500}
UniformInt 30)
RandomForest
max_features
("sqrt", "log2")
min.samples.leaf
Uniformint 1, 10)
39

---

Published as a conference paper at ICLR 2026
Model
Parameter
Search space
min samples split
Uniformint (2, 20)
F.3 INTERPRETING BL: A CASE STUDY
F.3.1 INTERPRETING BI((DEP)): HIGH-LAVEL, OVERVIEW
Deeper variants of BL are constructed by stacking multiple BL(Single) modules into hierarchical layers, followed by final affine transformation. This forms system of interacting UMPs (each of which can be viewed as an agent), where each internal block B represents a single interpretable UMP. As shown in Figure 7, first-layer modules correspond to individual UMPs, while the second- layer module performs optimal coordination by aggregating or allocating their outputs. This layered structure offers compositional interpretation of deeper BL models systems of interacting, inter- pretable UMPs.
y
max E R"
C(x,y) 0
s.t.
T(x,y) 0
Figure 7: Interpreting deeper BL architectures as hierarchical systems of interacting agents. Each block represents an interpretable agent solving its own UMP, while a layer corresponds to a set of heterogeneous agents operating in parallel. The next layer then aggregates and reallocates the neg- ative energies from the previous layer, thereby performing higher-level coordination across agents. This layered organization provides natural compositional interpretation of deep BL: bottom-layer modules encode local objectives, while upper layers synthesize these into collective outcomes. Anal- ogous structures arise in biological and social systems- for example, in ant colonies, individual ants (first-layer agents) follow simple local rules, yet their collective behavior is coordinated through higher-level interactions (second-layer aggregation), yielding globally efficient resource allocation and task division.
F.3.2 CASE STUDY: ADDITIONAL DETAILS
40

---

Published as a conference paper at ICLR 2026
Table 7: Boston Housing dataset variables and descriptions.
Variable
Description
CRIM
Per-capita crime rate by town
ZN
Proportion of residential land zoned for lots 25,000 sq.ft.
INDUS
Proportion non-retail business acres per town
CHAS
Charles River dummy variable (=1 if tract bounds river)
NOX
Nitric oxide concentration (parts per 10 million) Average number of rooms per dwelling
RM
AGE
Proportion of owner-occupied units built prior 1940
DIS
Weighted distances five Boston employment centers Index of accessibility to radial highways
RAD
TAX
Full-value property-tax rate per $10,000 Pupil teacher ratio by town 1000( B2 - 0.63)² where B, the proportion of Black residents by town
PTRATIO
LSTAT
Percentage of lower-status population
MEDV
Median value of owner-occupied homes in $1000s
Table 8: Semantic roles of blocks in the deep BL architecture.
Layer
Block
Representative preference
Location-Sensinee Buyer
Values river access, transport accessibility, and neighbor- hood amenities.
Risk-Sensitive Buyer
Averse to local disamenities such as pollution and envi- ronmental risk.
Cconomic-Sensitiee Buyer
Sensitive school quality and neighborhood socio- economic composition.
Layer
Zoning-Contrnn Buyer
Responds to zoning and land-use patterns that shape local housing supply. Strongly prefers high prices.
Affordability Prretbbi Buyer
affordable housing and dislikes
Integrated Economic Buyer
Jointly evaluates location and socio-economic attributes in an integrated way.
Layer
Budget-Conflict Buyer
Exhibits strong preferences for desirable locations but faces binding budget constraints.
Balanced Tradc-off Buyer
Jointly considers multiple housing attributes in balanced manner.
Layer
Representative Composite Buyer
Aggregates all lower-level preference components into representative houschold.
Table 9: Each block in the deep BL architecture is aligned with classic preference mechanism
documented in the economics literature.
Layer Block
Representative reference
Layer 1: Senastiveee Buyer
Gibbons Machin (2005)
Layer 1: Risk-Sensitive Buyer
Chay & Greenstone (2005)
Layer 1: Sccnomic-Sensitivve Buyer
Black (1999)
Layer 1: Zoning Conttrat Buyer
Glaeser & Gyourko (2002)
Layer 1: Affordability Preeeernn Buyer
McFadden (1977)
Layer 2: Integrated Location Economic Buyer
Bayer al. (2007)
Layer 2: Budget-Conflict Buyer
Balseiro al. (2019)
Layer 2: Balanced Trade-off Buyer
Rosen (1974)
41

---

Published as a conference paper at ICLR 2026
F.4 PREDICTION ON HIGH- HIGH-DIMENSIONAL INPUTS
Datasets Description and Preprocessing. For image datasets, we use the official train/test splits of MNIST and Fashion-MNIST: Inputs are converted to single-channel images scaled to [0, 1] and standardized with dataset-specific statistics. No resizing or data augmentation is applied. Training uses shuffled mini hini-batchess of size 64. For text datasets, we apply the following procedures:
1 Data sources and official splits. We use the official training and test splits for AG News and Yelp Review Polarity without any custom re-partitioning. Both datasets are class-balanced across labels, and we do not perform any resampling.
2 Dataset sizes. AG News: 120,000 training 7,600 test samples with four balanced classes. Yelp Review Polarity: 560,000 training 38,000 test samples with two balanced classes.
3 Label mapping. AG News: labels 1-4 are mapped to 0-3. Yelp Review Polarity: labels 1-2 mapped to 0-1.
Text preprocessing and feature representation. All texts are lowercased and tokenized the word level. The vocabulary is built with unigrams and bigrams, discarding words that appear fewer than two times in the training corpus. The vocabulary size is capped (AG News: 200,000; Yelp: 100,000). We compute TF-IDF weights on the training split and apply the learned weights to the test split. Dimensionality is reduced to 128 latent components using truncated singular value decomposition (SVD). Features are standardized to zero mean and unit variance and finally l-- normalized. We fix the random seed for reproducibility and reuse the learned preprocessing components across runs.
Additional OOD Detection Results. addition accuracy and AUROC, we also report AUPR and FPR 95 for both image and text datasets; the results are shown in Table 10. On image datasets, BL (depth=1) achieves the best overall balance: ranks first on Fashion-MNIS" AUPR and second on Fashion-MNIST FPR 95. On MNIST, it is second in AUPR but underperforms in FPR @95 compared with E-MLP (depth=2). These results suggest that BL yields separable score distributions, particularly Fashion-MNIST although its 95% FPR threshold admits more OOD samples than E- MLP at the same recall. On text datasets, OOD detection performance is dataset- -eeendee--deeddett E-MLP performs better AG News, whereas BI. achieves stronger OOD performance on Yelp.
Table 10: OOD AUPR and FPR@95 (%) on image and text datasets. BL and E-MLP are evaluated depths 1-3 with matched parameter counts, both without skip connections. Top-two per column are blue and red.
MNIST
Fashion-MNIST
Model
AUPR
FPR @95
AUPR
FPR @95
E-MLP (depth=1)
89.37 + 1.52
35.57 5.87
91.35 1.25
28.24
4.37
BL (depth=1)
91.57 2.39
47.81 = 11.29
91.79 0.90
38.86
2:57
E-MLP (depth=2)
1
28.89
2.85
86.19 2.27
47.72
4.79
BL (depth=2)
91.20 +: 1.22
52.71 # 18.66
89.30
2.47
42.65
9.53
E-MLP (depth=3)
90.04
31.92
5.76
84.30
54.49
2.74
BL. (depth=3)
92.36 2.03
32.32 + 5.76
88.41 + 4.04
41.19 + 13.36
AG News
Yelp
Model
AUPR
FPR @95
AUPR
FPR 95
E-MLP (depth=1)
44.52 + 15.10
33.82 4.89
3.31
54.21
2.11
BL (depth=1)
18.68 1: 16.48
42.03
1: 5.99
12.70 1: 2.29
40.95
1.56
E-MLP (depth=2)
31.48 1: 23,94
40.20
9.82
1.47 2.11
57.80
4.88
BL (depth=2)
10.76 + 15.94
53.71
+ 9.68
6,73 +
46.54
1.86
B-MLP (depth=3)
51.24 +
32.96
3:23
3.14
55.24
5.33
BL (depth=3)
16.99 + 17.12
45.24
6.11
10.96 1.10
42.27 1.94
42

---

Published as a conference paper at ICLR 2026
Number of Parameters. To ensure fair comparison between E-MLP and BL, match the number trainable parameters as closely as possible for models with the same depth (see Table 11).
Running Time. To evaluate computational cost, we compare the training time BL and Energy- based MLP across image and text datasets (Table 12). Under comparable parameter budgets, BL generally requires slightly higher training time than E-MLP across datasets. In particular, BL is moderately slower image datasets and AG News, while exhibiting comparable running time Yelp.
Calibration We report ECE and NLL metrics to assess calibration quality, and the results are presented in Table 13. On image datasets, BL provides substantially better calibration, with BL models occupying the top two positions in each column. On text datasets, calibration performance broadly comparable, with BL showing slightly lower NLL on Yelp. Overall, these results indicate that BL delivers strong predictive performance together with reliable probability estimates.
Table 11: Number of trainable parameters for E-MLP and BL models across high-dimension datasets.
Dataset
Model
Parameters
E-MLP (depth=1)
203,530
BL (depth=1)
208,384
E-MLP (depth=2)
235,146
MNIST & FashionMNIST
BL. (depth=2)
219,264
E-MLP (depth=3)
238,314
BL (depth=3)
221,684
E-MLP (depth=1)
136,196
BL (depth=1)
149,720
E-MLP (depth=2)
386,284
AGNews
BL (depth=2)
397,568
E-MLP (depth=3)
230,788
BL. (depth=3)
224,128
E-MLP (depth=1)
134,146
BL. (depth=1)
148,960
E-MLP (depth=2)
385,770
Yelp
BL (depth=2)
397,312
E-MLP (depth=3)
230,530
BL (depth=3)
224,000
Table 12: Training time (seconds) of BI. vs. E-MLP on high-dimensional datasets (mean + std).
Model
MNIST
FashionMNIIT
AG News
Yelp
E-MLP (depth=1)
100.59
73.57
14.69 + 0.40
179.37 0.73
BL (depth=1)
110.63 3.34
96.52 + 290
17.20 + 0.06
181.07 + 1.80
E-MLP (depth=2)
102.64 0.26
78.25 0.28
15.76 0.06
179.22
0.66
BL (depth=2)
122.85 3.95
114.43 1 3.72
21.78 1 0.08
180.38
1.44
E-MLP (depth=3)
104.52 0.30
85.57 1.19
16.95 1 0.85
178.99
1.42
BL (depth=3)
140.17 4.42
130.03 + 4.96
26.29 + 0.24
180.36 091
F.5 CONSTRAINT ENFORCEMENT TEST: ACHIEVING DIGE-DDMENSONAL ENERGY CONSERVATION
To evaluate whether the learnable penalty terms in BL are capable of enforcing near-hard constraints under finite temperature, we isolate the penalty mechanism and test it on a high-dimensional energy- conservation constraint. This diagnostic experiment removes the utility term and focuses solely
43

---

Published as a conference paper at ICLR 2026
Table 13: ECE and NLL on image and text datasets. BL and E-MLP are evaluated at depths 1-3 with matched parameter counts. Top-two per column are blue and red.
MNIST
Fashion-MNIST
Model
ECE
NLL
ECE
NLL
E-MLP (depth=1)
0.02 0.00
0.20 0.02
0.08 0.00
0.74 + 0.01
BL (depth=1)
0.02 + 0.00
0.26 = 0.01
0.05 = 0.00
0.36 + 0.01
E-MLP (depth=2)
0.02 0.00
0.23 0.02
0.09 0.00
0.89 0.03
BL (depth=2)
0.02 1: 0.00
0.16 + 0.01
0.07 1 0.00
0.44 1 0.01
E-MLP (depth=3)
0.02 0.00
0.16 0.02
0.09 0.00
0.85 + 0.04
BL (depth=3)
0.02 + 0.00
0.13 + 0.02
0.07 = 0.00
0.49 + 0.02
AG News
Yelp
Model
ECE
NLL
ECE
NLL
E-MLP (depth=1)
0.02 0.00
0.40 0.01
0.01 0.00
0.24 +
BL (depth=1)
0.02 +: 0.00
0.31 4: 0.01
0.00 +: 0.00
0.20 4: 0.00
E-MLP (depth=2)
0.02 0.00
0.42 0.01
0.00 0.00
0.25 1:
BL (depth=2)
0.06 + 0.01
0.43 = 0.03
0.02 = 0.00
0.23 + 0.01
E-MLP (depth=3)
0.01 0.00
0.41 0.02
0.00 = 0.00
0.25 + 0.01
BL (depth=3)
0.05 0.01
0.39 0.02
0.02 + 0.00
0.22 +: 0.00
the penalty term, providing a characterization of how the penalty term controls constraint violations as function of temperature and penalty scale X.
Experiment setup. We sample E R64 i.i.d. from a standard Gaussian x ~ NO, Toa) and define a pure penalty compositional utility
T(x, v)
Ilvll ||0|
BL( v) -))
which plays the role of energy-conservation residual and its quadratic penalty.
We target the Gibbs distribution
ply x) x exp(BL(z,))/7
using overdamped Langevin dynamics with step size 17 10-4
yk++ = yk + 1) BL yu /// + v 21F Ek.
EA (0 T64).
512 parallel chains, each for 1500 Langevin steps (500 burn-in). We sweep over temperatures < {2.0, 1.0, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01, 0.005) at fixed penalty À 25, and over penalty weights E 1,, 10, 30, 100, 200, 500) at fixed temperature
For each pair 7A,
0.05.
For each configuration we record the residual magnitude [T' y) from the final state of every chain. We then report three summary statisties: (i) the mean violation E[] y)[]. (ii) the 95th percentile of [Tx, y)1, and (iii) the empirical probability of near-feasible samples. We declare a sample to satisfy the constraint approximately if
[T(,, v)1 S Evol with Ewl 10-1,
and estimate P(IT(2, v) Evo) across chains. This tolerance scale chosen be small relative to the typical unconstrained residuals, so that the near-feasibl regime corresponds to a practically constraint.
tight
Constraint enforcement. Figure 8 shows that BL achieves near-hard constraint enforcement un- der finite temperature and penalty scaling. Violations decrease substantially as 7 decreases or increases. At around À= 25 and - 0.01, the 64-dimensional energy conservvtion constraint is enforced within 10 error. Curves remain mostly smooth and monotone in 64 dimensions, indicat- ing stable Langevin sampling and effective penalty enforcement.
44

---

Published as a conference paper at ICLR 2026
Violation vs. (A=25.0)
Violation (==0.050)
mean IC)
10
95th percentile
10
mxx
17G.y)
10
10"
10
IC)
95th percentile (CI
10
10
10"
10
10
10
10
10
Temperature
Penalty weight A
Near-feasible fraction vs. r (2:05)))
Near-feasible fraction vs. r= 0.050)
$0.
0.8
0.6
0.6
0.2
0.0
10
10
10
0.0
10
10
-
10
10
10'
10"
Temperature r
Penalty weight
Figure 8: Constraint enforcement test of the BL penalty block on energy-conservation constraint. The figure reports violation statistics [2' y)1 when varying the temperature (left side of panel) and the penalty weight (right side of panel).
F.6 ESTIMATION RESULTS OF BL. ON THE BOSTON HOUSING DATASET
G USE OF LARGE LANGUAGE MODELS (LLMs)
We used large language models (LLMs) solely for minor grammar correction and polishing of awk- ward sentences. They were not used in any other part of the research process.
45

---

Published as a conference paper at ICLR 2026
Table 14: Estimated UMP block parameters learned by the BL model (layer = [2, IJ) on the Boston Housing dataset. For each block, denotes the Utility component, C the Inequality-Come Constrain component, and T the Equality- Monaiit-Conntrrintt component.
Block 11
Block 12
Variable
Un
Cn
Ti
U12
C12
Ti2
1.003
0.997
0.999
0.997
1.003
1.000
per capita crime (CRIM)
0.21
0.14
0.03
0.12
0.09
0.25
residential land proportion (ZN)
0.23
-0.04
-0.27
0.25
0.09
non-retail business acreage (INDUS)
-0.06
0.21
0.25
0.16
0.22
0.27
Charles River dummy (CHAS)
0.25
0.04
-0,24
-0.12
-0.20
-0.23
nitric oxide concentration (NOX)
-0.06
-0.13
0.21
0.16
0.02
-0.28
average rooms per dwelling (RM)
0.06
0.07
0.05
0.05
-0.19
-022
proportion of older units (AGE)
-0.13
-0.12
-0.09
0.14
-0.18
distance to employment centres (DIS)
0.16
-0.03
0.17
-0.17
-0.09
0.11
radial highway accessibility (RAD)
0.24
-0.11
0.04
-0.28
0.09
0.10
property tax rate (TAX)
-0.20
0.18
0.22
-0.11
-0.06
0.23
low-income population (LSTAT)
0.05
-0.09
0.23
-0.19
median home value (MEDV)
0.21
-0.08
0.07
0.08
-0.17
0.15
Constant term (C)
0.03
-0.17
-0.07
0.11
-0.16
-0.12
Block 21
Variable
U21
C21
T21
1.000
1.003
0.999
Black output (b1.1)
0.428
-0.551
0.147
Block output (b1,2)
-0.168
-0.356
-0.178
Constant term (C)
0.406
0,219
0.421
46

---

Published as a conference paper at ICLR 2026
Table 15: Estimated UMP parameters for the Layer blocks of the BL model (layer [5, 3, 1J) trained on the Boston Housing dataset. Here, U denotes the Utility component, C' the Inequality- Constraint component, and 7 the Equality-Cone Constraint component.
Variable
Un
U12
U13
U14
U15
1.000
0.998
1.003
1.002
1,000
per capita crime rate (CRIM)
0.12
0.17
-0.09
0.06
residential land proportion (ZN)
0.23
0.25
-0.07
-0,22
-0.16
non-retail business acreage (INDUS)
-0.06
0.16
0.16
0.23
-0.14
Charles River dummy (CHAS)
0.25
-0.12
-0,22
-0.05
-0.01
nitric oxide concentration (NOX)
-0.06
0.16
-0.14
0.24
0.16
average rooms per dwelling (RM)
0.05
0.05
0.08
0.09
-0.07
proportion of older units (AGE)
-0.13
0.14
0.06
-0.23
-0.15
distance to employment centres (DIS)
0.16
-0.17
-0.07
0.19
-0.10
radial highway accessibility (RAD)
0.24
-0.27
0.17
-0.08
-0.20
property tax rate (TAX)
-0.20
-0.11
0.19
-0.11
0.10
low-income population (LSTAT)
0.05
0.23
-0.15
-0.28
-0.26
median home value (MEDV)
0.21
0.08
0.25
0.08
0.06
Constant term (C)
0.03
0.12
-0.09
-0.06
0.15
Cn
C12
C13
C14
C1s
0.999
1.001
1.000
0.997
1.002
per capita crime rate (CRIM)
0.13
0.09
-0.10
0.11
0.05
residential land proportion (ZN)
-0.04
-0.01
-0.27
-0.23
-0.10
non-retail business acreage (INDUS)
0.21
0.22
-0.16
0.20
0.15
Charles River dummy (CHAS)
0.04
-0.19
0.07
-0.20
0.15
nitric oxide concentration (NOX)
-0.13
0.02
-0.04
-0.05
0.11
average rooms per dwelling (RM)
0.07
-0.19
-0.20
0.06
-0.05
proportion of older units (AGE)
-0.13
0.08
0.01
0.14
-0.07
distance employment centres (DIS)
-0.03
-0.09
-0.19
0.22
0.03
radial highway accessibility (RAD)
0.08
-0.23
0.25
-0.05
property tax rate (TAX)
0.18
-0.06
-0.15
-0,22
-0.08
low- low-income population (LSTAT)
-0.13
-0.17
-0.18
-0.12
0.24
median home value (MEDV)
-0.08
-0.17
0.28
-0.03
-0.03
Constant term (C)
-0.17
-0.16
0.05
-0.21
-0.06
Th
Ti2
Tis
Ti4
Tis
0.999
1.002
0.999
1.004
1.001
capita crime rate (CRIM)
0.03
0.25
0.08
0.25
0.00
residential land proportion (ZN)
-0.27
0.10
-0.26
-0.20
-0.02
non-retail husiness acreage (INDUS)
0.25
0.26
-0.18
0.15
0.07
Charles River dummy (CHAS)
-0.23
-0.23
-0.09
0.10
0.08
nitric oxide concentration (NOX)
0.21
-0.28
0.04
0.09
-0.25
average rooms per dwelling (RM)
0.05
-0.21
-0.24
-0.15
-0.10
proportion of older units (AGE)
-0.09
-0.19
-0.12
0.26
0.24
distance to employment centres (DIS)
0.12
-0.17
0.10
radial highway accessibility (RAD)
0.04
0.10
0.00
0.04
-0.01
property tax rate (TAX)
0.22
0.23
-0.10
-0.24
-0.17
low-income population (LSTAT)
-0.09
-0.19
-0.19
-0.04
-0.25
median home value (MEDV)
0.08
0.15
-0.19
-0.13
-0.09
Constant term (C)
-0.11
-0.16
0.10
47

---

Published as a conference paper at ICLR 2026
Table 16: Layer 2 and Layer 3 UMP parameters (U, C, 2) for Blocks in the BL model (layer = [5, IJ).
Block 21
Block 22
Block 23
Variable
U21
C21
T21
U22
C22
T22
U23
C23
T23
À
1.000
1.000
1.000
0.999
1.003
1.002
1.001
1.002
0.999
Block 11 output (bi,1)
0.28
0.06
-0.20
-0.31
0.24
0.18
-0.29
-0.08
0.22
Block 12 output (b,.2)
0.21
-0.11
-0.09
-0.44
0.12
-0.22
0.15
-0,22
0.20
Block 13 output (b1.))
-0.40
0.18
-0.44
-0,36
-0.01
-0.09
-0.13
-0.14
0.32
Block 14 output (b,.4)
-0.27
-0.17
0.30
0.33
-0.34
-0.26
0.28
-0.42
-0.34
Block 15 output (b1,s)
-0.07
-0.29
0.34
0.22
-0.38
-0.08
-0.14
0.25
0.32
Constant term (C)
0.43
0.33
0.16
0.38
-0.42
-0.32
-0.33
-0.31
-0.21
Variable
U31
Ca1
T51
1.002
0.998
1.000
Block 21 output (b,.1)
0.21
-0.13
0.36
Block 22 output (b2.2)
0.54
-0.48
0.43
Block 23 output (b2,3)
-0.08
0.28
0.55
Constant term (C)
-0.01
-0.58
-0.14
48


Reviews:
[{"review_id": "pNBMKwFemW", "reviewer": "Reviewer_jMN4", "summary": "The paper proposes to learn data distribution P(x | y) by using a specific form of parametrized probability families, motivated from the Lagrangian reformulation of constraint programs. The paper moves on to develop relevant theories for this parametrized probability class, including approximation results, uniqueness of optima of the loss constructed from this class, and certain estimation results including consistency.", "questions": "1. In Eq 4, what are the exact forms of phi, rho, and psi? Are the the same as those in Theorem 2.1? Are they hyper-parameters to be chosen? If so how to choose them? \n2. In section 3.2, do the authors consider endogeneity in the dataset, or do they assume exogeneity? If latter, then the problem is just simple regression problem.\n3. In all the numerical comparison, are the number of parameters similar? Using similarly complex models ensures a fair comparison. \n4. In the numerical part, does BL recover the relevant utility and constraint functions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is very well written, and compares with many benchmarks. In appendix there are abundant introduction of the dataset used and the competing methods.  \n2. The proposed parametrization has utility maximization interpretation. Learning the model is uncovering agents' constraint, utility function at the same time.", "weaknesses": "1. The statistical results part is standard. A more interesting theory would be to show, the parametrization is effecient if the data distribution is indeed generated from some utility maximization process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfcyWgTNFP", "reviewer": "Reviewer_sGAR", "summary": "This paper proposes _Behavior Learning (BL)_, a specialization of Energy-Based Models which provides a natural decomposition of an EBM into constrained utility optimisation problems (\"blocks\") each of which can be interpreted as a utility term minus  loss terms which penalise solutions that do not satisfy constraints.\nSpecifically, it parameterises a joint distribution $p_{\\tau}(x,y)$  energy $BL_\\theta(x,y)$, where the energy is built by composing “utility–minus–constraint-penalty” blocks.\nEach block encodes a *utility* term $U$ and *constraint* terms $C$ (inequality) and $T$ (equality), passed through fixed activations and linearly combined into a single scalar compositional utility. \nTraining uses cross-entropy for discrete $y$ and denoising score matching (DSM) for continuous $y$, thereby avoiding explicit partition-function estimation.\nInference for fixed $x^0$ uses the Gibbs form $p_{\\tau}(y\\mid x^0) \\propto \\exp(BL_\\theta(x^0,y)/\\tau)$.\nA sub-type (IBL) restricts the activations to smooth families and gains identifiability of parameters to shape-constrained (monotone/smooth) forms to obtain guarantees of identifiability of parameters.\nAs such BL is perhaps best understood as a collection of  sub-families of EBMs which guaranteeing utility/constraint semantics.\nExperiments (classification/regression/causal tasks; some higher-dimensional inputs) aim to show competitive accuracy with stronger interpretability and sometimes improved OOD behaviour.", "questions": "### How does Deep BL work in practice?\n\nI'm trying to map my understanding of the composed DeepBL to standard deep nets, and have gotten stuck on the paper. Can the authors clarify for me what happens in various cases? \nI have questions both about how the deep network works, and what it means.\n\nIn the default case (around eq (5)) each block outputs a single scalar, so at layer $\\ell$ we have a state vector of size $d_{\\ell}$.\nThat is, later layers operate on a low-dimensional vector of utilities, not on rich features.\nIs that correct? This is a serious disanalogy with blocks in the first layer which actually observe the input features and labels.\n\nWe learn that introducing $(x,y)$ as inputs (\"skip-connections\") to later blocks restores capacity but it looks like weakens the “everything is a UMP” interpretation, since later blocks are no longer functions purely of upstream utilities.\n\nAnyway, my question is: have I got that right? I got lost in the details of the Deep BL section and would appreciate some clarifications.\n\nAlso, why are Deep and Shallow BL treated differently? They are both the same but $>2$ right? I'm not sure I understand the choice to spend previous column inches on introducing more subfamilies than absolutely necessary in this crammed paper.\n\n### Interpretability-performance tradeoffs\n\nI don't think the authors claim to have eliminated the performance-interpretability tradeoff but I think the paper makes a claim to have implicitly shifted the pareto frontier of this tradeoff downwards.\nHave they quantified this?\n\nNot that I am proposing new experiments, but I think the paper needs to be clearer about the tradeoffs it is making, and to what extent the interpretability gains come at a cost in accuracy or efficiency.\n\nThe BL formalism seems to include a lot of design choices that affect this tradeoff (Skip connections, affine versus polynomial bases etc), and I think the paper needs to be clearer about what happens at scale to the interpretability guarantees.\n\nFor example to isolate the value of the BL parameterization, the paper could compare against capacity-matched EBM/Deep declarative network/ symbolic regression/Variational bayes baselines, and see how complex the hypothesis space can grow while still retaining \"interpretability\" in some sense, for each of these.\nThis is probably out of scope for the current paper, but what is the closest approximation to this that could be done in the current body of work?\n\n### Constraints\n\n**Constraint satisfaction under noise.**  \n*Q:* How large must penalty weights or how small must $\\tau$ be to achieve near-hard constraint satisfaction in practice?  \n*Request:* Provide plots of feasibility violation vs $\\tau$/penalty scale and discuss numerical stability.\n\nIt would be super nice if you for example imposed a physics residual constraint (e.g. conservation of mass/energy) and showed how well it is satisfied in practice as a function of $\\tau$ or penalty scale in  a high dimensional problem.\n\n### CausalBL\n\nI have many questions about the CausalBL variant; is there any chance you can squeeze an actual description of it in to the body text?\nIf not, should it even be in the paper? \nThese results seem powerful, and potentially very significant, but also barely explained. Maybe there is a whole other paper there?\n\n### Calibration\n\nSince the model is probabilistic, how well-calibrated are the predictive distributions?  \nAny chance we could get proper scoring metrics (e.g. ECE/NLL/Brier)  vs EBM/MLP baselines?\n\n### Hybrid approaches for partial explainability\n\nIt seems that this model gives us some desirable explainability qualities under simple settings (essentially the IBL setting: polynomial bases, if the network is deep then block communicate only by utility signals etc) but in larger practical problems the authors relax these simplifications (using affine bases over large input spaces, skip connections and so on). This suggests that maybe \"partial explainability\" is potentially within reach, selecting a subset of model space in which IBL blocks are used, but then allowing \"non-explainable/identifiable\" blocks to participate in inference, for example if we do not need explainability over all model interactions, but desire it for a small subset. Potentially we could absorb arbitrary black box interactions that way. Have the authors considered this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "### Originality\n\nThe architectural idea is, it seems, novel, even though it is closely connected to other attempts at the same goal. Reparameterizing an energy function as a composition of relaxed constrained utility maximisation programs is neat. Universal approximation theorems this reassure us that a BL program the capacity to well-approximate both any conditional density and rather general problem constraints, albeit relaxed via penalties. We can view this as guaranteeing interpretable behavioral model for random distributions.\n\n**Identifiable variant (IBL)**: In the IBL sub-family, not only does this interpretation exist, but we can read it off the learned parameters under reasonable assumptions. \n\nThis synthesis of constrained optimisation theory seems simple in hindsight but I cannot find it in the literature. As such: cool insight.\n\n### Quality\n\nThe model has a strong  Mechanistic narrative: “utility minus penalties” + Gibbs distribution = stochastic bounded-rational behavior. The existence of such a disentangling is satisfying, and they show that in some cases it can even be recovered.\nThe Gibbs form makes the model a bona fide conditional density; the $\\tau\\to 0$ limit recovers deterministic best response.\n\nWe have other networks that naturally encompass constrained solutions; for example Gould's Deep Declarative networks: compared with inverse RL/bi-level pipelines that differentiate through argmax/KKT by the implicit function theorem, BL’s training objective stays first-order and stable, using standard autodiff on $BL(x,y)$.\nThe theory for IBL** seems promising:  identifiability and consistency and asymptotic efficiency are unusual strengths for high-capacity models.\n\n### Clarity\n\nPresentation is generally clear (although see below for some notes regarding the Deep BL and CausalBL variants). \nExamples solid. Prose style is clear.\n\n### Significance\n\nDefinitely a first step in an interesting direction.\nThe notion here of “interpretable EBMs” is appealing, and the framework seems flexible enough to accommodate a variety of domain theories via the choice of block heads and compositions.\nCausal extensions and the promise of interpretability are high value in high-risk or sensitive or legally constrained domains, or where the natural way to incorporate prior knowledge is via constraints.", "weaknesses": "There is a lot crammed into this paper (identifiability results, convergence results, universal approximation bounds...) and the authors have made some judgment calls about what to include in the body of the paper, which makes it difficult to evaluate the paper in terms of the body text claims.\nIdeally it would have been two papers; there is enough material here for two.\n\nIMO the places where this is toughest are \n\n1. Causal BL: Impressive, but it is entirely relegated to the appendix, as well as the considerable machinery that it requires. I understand that this result is super cool, but we can only evaluate it base on the main text. If this is to remain in the paper something else should be cut to make room for a fuller treatment. \n2. Deep BL: This feels to me like the least intuitive part of the paper, and the treatment in the main text is light. I have many questions about the mechanics of this setting (*see below) \n\n**Interpretability trade-offs**\n\nThe model runs straight at the performance-interpretability tradeoff challenge, but there are some open questions about how well it navigates this tradeoff in practice.\n\n- If I understand correctly composition of BL block results in different behaviour in later layers than in earlier ones; it seems there are bottlenecks in the layers and that practical high capacity networks throw out many of the interpretability guarantees; we probably want to know more about this tradeoff surface.\n\nAFAICT, cost of using monomial bases for $U,C,T$ can explode combinatorially with input dimension and degree. The paper partly sidesteps this (affine heads in deep/high-dim settings), but then the identifiability and symbolic clarity are less compelling. The trade-off deserves a tighter discussion and ablation.\n\nAt finite temperature, Gibbs does not enforce constraints exactly; samples can violate $C\\le 0, T=0$. This weakens the normative “optimizer” story unless $\\tau$ is very small or penalties are very large; then training/sampling may suffer from stiffness. We probably need to know more about this,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PiwbtOgGNY", "reviewer": "Reviewer_3vXd", "summary": "The paper introduces Behavior Learning (BL), a machine learning framework for prediction (primary focus) and generation, designed to be intrinsically interpretable. BL models are built from modular computation blocks, each mathematically equivalent to a utility maximization problem (UMP). This structure enables direct interpretability: the model’s output can be understood as the result of maximizing a utility function subject to constraints.\n\nThe paper establishes theoretical contributions for BL. First, it defines a subclass called Identifiable Behavior Learning (IBL), where penalty functions are smooth and monotone. It shows that the model parameters in this class are identifiable up to an equivalence class, ensuring a unique and consistent interpretation of the learned behavior. Second, both BL and IBL possess universal approximation capabilities, demonstrating that interpretability does not come at the cost of predictive performance. For IBL, the authors additionally prove desirable statistical properties, including consistency, universal consistency, asymptotic normality, and asymptotic efficiency.\nFinally, the paper empirically validates BL across a range of prediction tasks, including those with high-dimensional inputs, and shows that BL can accurately estimate causal effects.", "questions": "**Q1**. In the shallow and deep BL architectures, multiple UMP blocks (Eq. (4)) are computed in parallel and then combined through an affine transformation. After this aggregation step, is the entire composed model still guaranteed to be equivalent to a single utility maximization problem (UMP)? If this is not the case, in what sense can we still interpret each individual block as performing a UMP, especially when the affine weights may change the sign or scale of the penalty/utility terms?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**S1**. The framework is novel in that it leverages utility maximization problems (UMPs) to construct models with intrinsic interpretability, allowing outputs to be directly understood as utility-maximizing decisions under constraints.\n\n**S2**. The theoretical analysis is thorough and solid, including universal approximation guarantees, identifiability results for IBL, and other properties of IBL. \n\n**S3** The experiments are comprehensive, evaluated on diverse datasets with strong baselines, and demonstrate that BL performs well not only on prediction tasks but also on causal effect estimation.", "weaknesses": "**W1**. Although BL is intrinsically interpretable at the block level, the utility and constraint functions are fully learned during training, making it difficult to connect the resulting functions to real-world meaning. For instance, in the interpretability case study (Fig. 2), the learned utility $\\tanh((1-P)(1+P-RM) + R_u)$ lacks a clear semantic interpretation. As BL becomes deeper, it introduces more learned symbolic functions that may reduce the practical interpretability of the model.\n\n**W2**. The paper lacks sufficient case studies that demonstrate interpretability in practice. Despite interpretability being a key claimed advantage of BL, the experiments focus mainly on prediction accuracy, with only limited analysis of interpretability (e.g., Section 3.3). The case study for deeper BL architectures (Fig. 10) remains high-level and does not include a concrete real-world interpretability example.\n\n**W3**. The causal inference section (CausalBL) is confusing in definition and terminology. Section 3.2 and Appendix E describe the setup, method, and experimental results, but the treatment of potential outcomes and propensity scores is inconsistent. Lines 1860–1861 suggest the goal is to estimate $p(t \\mid x), y_0, y_1$, implying an ATE formulation. However, lines 1944–1955 describe a setup where y becomes input to the propensity score and the target becomes ITE, not ATE. The paper should use causal inference terminology more carefully to avoid confusion, even if ATE estimation can be derived from ITE in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttAzRCDt89", "reviewer": "Reviewer_za71", "summary": "This paper tackles the central challenge of building interpretable machine learning models without sacrificing accuracy. It proposes Behavior Learning, which models predictions as solutions to stacked utility maximization problems, yielding latent structures that are interpretable by design. The authors also introduce Identifiable Behavior Learning, an architecture with identifiability guarantees for recovering ground truth parameters under certain assumptions. Across predictive and causal inference benchmarks, the method delivers performance on par with strong baselines while providing intrinsic interpretability and demonstrating scalability to high-dimensional data.", "questions": "1. For BL(Deep), what is the procedure to generate scientific explanation from the network of composed blocks?\n2. Could BL be easily extending to consider non polynomial feature maps?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of providing more interpretable ML systems without sacrificing performance is a well motivated problem. The proposed algorithm is also quite interesting, relating the predictive methodology of their approach to solving a system of behavioral science motivated MDPs.\n2. The theoretical results are well presented and all proofs look sound.\n3. BL outperforms competitive methods across multiple prediction and causal inference problems, while still providing interpretability. The Boston Housing case study provides an interesting and necessary demonstration for how to extract the human interpretable structure from the model.", "weaknesses": "1. It seems like the work needs some moderate restructuring. The IBL identifiability result hinges on Assumption D.1, but it only appears in the appendix. Related work coverage in the main text is also sparse. The paper ends abruptly without a final discussion of their contributions.\n2. While the Boston Housing case nicely illustrates BL(Single) and individual blocks, there’s no intuitive instruction for how to interpet layers of blocks as in BL(Deep). It is unclear how if the model becomes seeming more difficult as more layers, how this approach would still claim to be providing intrinsic interpretability.\n3. The implementation uses bounded-degree polynomials, however many real systems have non-polynomial structure and complex interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)