FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Models
Authors: Md Kowsher
Nusrat Jahan Prottasha
Shiyun Xu
Shetu Mohanto
Niloofar Yousefi
Ozlem Garibay
Chen Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Bidirectional Language Models
Information Bottleneck
Mutual Information
FlowNIB
Layer-wise Analysis
Context Understanding
Natural Language Understanding
Primary area: foundation or frontier models, including LLMs
TL;DR: Bidirectional LMs retain more mutual information per layer than unidirectional ones, and our FlowNIB method measures this to explain—and predict—their superior downstream performance.

Abstract: Bidirectional language models (LMs) consistently show stronger context understanding than unidirectional models, yet the theoretical reason remains unclear. We present a simple information bottleneck (IB) perspective: bidirectional representations preserve more mutual information (MI) about both the input and the target, yielding richer features for downstream tasks. We adopt a layer–wise view and hypothesize that, at comparable capacity, bidirectional layers retain more useful signal than unidirectional ones. To test this claim empirically, we present Flow Neural Information Bottleneck (FlowNIB), a lightweight, post-hoc framework capable of estimating comparable mutual information values for individual layers in LMs, quantifying how much mutual information each layer carries about the input and target. FlowNIB takes three inputs—(i) the original LM’s inputs/dataset, (ii) ground–truth labels, and (iii) layer activations—simultaneously estimates the mutual information for both the input–layer and layer–label pairs. Empirically, bidirectional LM layers exhibit higher mutual information than similar—and even larger—unidirectional LMs. As a result, bidirectional LMs outperform unidirectional LMs across extensive experiments on NLU benchmarks (e.g., GLUE), commonsense reasoning, and regression tasks, demonstrating superior context understanding.

Published as a conference paper at ICLR 2026
FLOWNIB: AN INFORMATION BOTTLENECK ANALYSIS OF BIDIRECTIONAL VS. UNIDIRECTIONAL LANGUAGE MODELS
Md Kowsher'; Nusrat Jahan Prottasha"; Shiyun Xu², Shetu Mohanto Niloofar Yousefi', Ozlem Garibay', Chen Chen' University of Central Florida UUniversity of Pennsylvania DDelinate Inc.
ABSTRACT
Bidirectional language models (LMs) consistently show stronger context under- standing than unidirectional models, yet the theoretical reason remains unclear. We present a simple information bottleneck (IB) perspective: bidirectional representa- tions preserve more mutual information (MI) about both the input and the target, yielding richer features for downstream tasks. We adopt a layer- wise view and hypothesize that, comparable capacity, bidirectional layers retain more useful signal than unidirectional ones. To test this claim empirically, we present Flow Neural Information Bottleneck (FlowNIB), lightweight, post-hoc framework capable of estimating comparable mutual information values for individual layers in LMs, quantifying how much mutual information each layer carries about the input and target. FlowNIB takes three inputs-(i) (i) the original LM's inputs/dataset, (ii) ground truth labels, and (iii) layer activations simultaneously estimates the mutual information for both the input- bidirectional LM layers exhibit higher mutual information than similar-ant even unniireoaiiiit LMs. As a result, bidirectional LMs outperform unidi- rectional LMs across extensive experiments on NLU benchmarks (c.g., GLUE), commonsense reasoning, and regression tasks, demonstrating superior context understanding. Code: github. com/ wttp://:eeeeiiiiiie./e/
and layer-label pairs, Empirically,
INTRODUCTION
Large language models have brought significant advancements in natural language understanding (NLU) tasks. Among them, bidirectional models such as BERT have demonstrated superior perfor- mance in natural language understanding, while unidirectional models like GPT dominate generation tasks, As shown in Table Devlin et al.] (2019), the BERT- base model outperforms GPT (Radford et al, 2018) across all GL.U benchmarks (Wang et al. 2018) despite having comparable model size- for example, achieving 66.4% accuracy on the RTE task versus GPT's 56.0%. Moreover, the empirical evidence (Li al. 2022 Liu al. 2019 Raffel al. 2020 Clark et al. 2020) consistently demonstrates that bidirectional LMs outperform unidirectional
LMs on a wide range of NLU tasks.
While the empirical advantage of bidirectional models is well documented, a clear theoretical account remains limited. We adopt an information -theoretic view based on the Information Bottleneck (IB) principle (Tishby et al., 2000). Let Z be layer representation and write [X; Z) for the mutual information between the input X and Z, and T(Z;Y) for the mutual information between 7 and the label Y. In IB, desirable representations compress the input (small T(X; Z)) while preserving task-relevant content (large 1 YY;
Our claim that, at comparable capacity, bidirectional layer retains more information about the input and transmits more information relevant to predicting the target than unidirectional layer; formally, for corresponding layers E: I(X; 2i+) 2 I(X; Z,*) [(;;;; Z I(Z,";Y0 Y') with strict inequalities under mild conditions (c.g., when future context reduces input uncertainty or contributes predictive signal). Intuitively, the bidirectional representation Z conditions on both past
"Equal contribution

---

Published as a conference paper at ICLR 2026
2000
2000
0.08
1750
1750
0.07
1500
1500
0.06
1250
1250
((21)
0.00
0.2
0.04
750
0.03
500
500
250
250
0.8.
0.2
0.4
0.6
0.8
1,0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
(0G2)
1002)
Figure I: information-plamt trajectories under FlowNIB training for (left) DeBERTaVas DO-ERTaV3-Base (bidi- rectional) and (right) MobileLLM-350M (unidirectional) MRPC. Each curve shows mutual information I(Ze; Y) versus [X; Ze) over training epochs, colored by epoch index. A small cumu- lative horizontal offset is added to I(X; Ze) for successive layers (+0.11 per layer the left, +0.05 on the right) to visually separate layer-wise trajectories; this shift for visualization only and does not affect the underlying MI values. The green line connects the Optimal Information Coordinate (OIC) for each layer, from lower to upper layers.
and future tokens, whereas the unidirectional representation Z conditions only on the past. Since conditioning reduces entropy (Madiman & Tetali 2010 H(X) 2,²)) S H(X Zz²), and therefore I(;; ) 2 [(x; "."). To make the IB analysis applicable to LMs, we formalize the following:
Definition 1.1 (A valid information plane (post hoc)). Let language model (LM) have hidden layers with layer-f output Ze for I,..., L. input X, and target Y under data distribution p(z, y). Let {1() J:20 denote a mutual information estimator family (e.g., MINE, InfoNCE) obtained by training the estimator for t internal steps on (X,Ze) and (Ze, Y) while the LM is frozen. Define the epoch-t information plane as Z = { (X; Ze), (Ze Y)) {= I.....,,, L} R², We say Iid is well-defined if, for all (i) Finite-valuedness: (X; and (Z; are finite. (ii) Layerwise indexability: Each point is associated with its layer index (ties in coordinates allowed). (iii) Temporal consistency: Across i, the same estimator (iv) Differentiability: The and the same p(z, y) are used, so )12 is a well-defined sequence.
maps driving ((0) are a.e. differentiable in their inputs so that gradients exist when backpropagating through Zr-
Remark 1.2 (Dynamics). Empirical "fitting" (both r(X; Zz) and 7 7(2;; rise) and "compression" [(X; Ze) decreases while T(Z6; Y) continues rise) patterns are diagnostic and not required for well-definedness.
Recent work has used the IB to improve training (Alemi al. 2016 Nguyen & Choi 2017 Achille & Soatto 2018) and to visualize training dynamics (Shwartz-Ziv & Tishby 2017 et al. (019). Applying IB to language models remains challenging: layer representations
high-dimensional
MI estimation is expensive. Very recent work applies IB to LMs but is largely descriptive such as explaining the model behavior (Wang al.) 2025 Wu al. 2025), iitribution-focusel studies (Jiang et al. 2020). in-context learning Yang et al.] 2025). and pruning oriented work (Fan et al.) 2021) which is limited to estimating empirical MI of a layer between input-layer and layer-output pairs. However, to test our claim empirically, we require a joint empirical assessment that captures layer's information-carrying capacity-hoo much information preserves from the input and how much it conveys the target at time which helps to show bidirectional layers exhibit higher joint information capacity than unidirectional layers.
We estimate mutual information using MINE (Beighazi et al. 2018). which optimizes lower-bound objective on the true MI For a layer Ze, MINE can independently estimate either [Xx; Ee) or Y) However, goal is understand how much information a representation Z, carries about both the input and the target simultaneously. Independent MINE critics yield values that are
'The Donsker-Vann Varadhan objective underlying MINE is lower bound in theory; however, with finite data, finite critic capacity, and imperfect optimization, the resulting estimates not calibrated and depend the critic's expressiveness. Thus MINE should be interpreted as providing relative, rather than absolute, MI values.
2

---

Published as a conference paper at ICLR 2026
incomparable across layers due to different optimization dynamics and critic capacities, making joint interpretation difficult.
To address this, introduce FlowNIB, simple extension of MINE that jointly approximates [[;; Ze) and [Ze;; ) within a unified optimization process and produce relative information. FlowNIBB trains two critics using curriculum parameter (t) that initially emphasizes T(X; Z and gradually shifts toward T(Z;; Y) over T iterations. This produces a continuous information-forn trajectory: (x; Zz), (Z;)))
t= 1,... T C RR2 which places both MI quantities on the same geometric path and makes them directly comparable across layers. From this trajectory, we select the point where the representation jointly maximizes information about X and Y; we refer to this coordinate as the Optimal Information Coordinate (OIC).
Definition 1.3 (Optimal Information Coordinate (OIC). Let each epoch E {o..... T yield Jt [(0 (X; Zz) and yr = 7 (Z;; Y). For trade-off weight y E [0, 1]. we define OIC for layer E L. where is set of hidden layers.
(1) E arg max y + (1 11)-() Vt.
OIC, : (1)) Me)()))
A scale-balanced choice is y² Ho where Rg = max; 2t - ming 2e and Ry = mmax Ve min; Vt-
Ry
We then compare OICs after fine-tuning on the same dataset between bidirectional and unidirectional LMs to see which carries more information for both input and output. In Figure [ we see the bidirectional LM has a higher OIC than the unidirectional LM. Beyond the theoretical explanation, we empirically compare OICs using FlowNIB across diverse datasets and show clear benefits for downstream tasks. In particular, on standard benchmarks such as GLUE, commonsense reasoning, and regression tasks, small bidirectional model outperforms a larger unidirectional model.
Contributions. (i) We provide a theoretical explanation for why bidirectional language models achieve better context understanding, showing that they can carry higher mutual information than unidirectional models. (ii) To estimate relative mutual information in high-dimensional LM represen- tations, we propose FlowNIB, simple and testable framework that jointly estimates I [X; Ze) and T(ZzY), quantifying the information capacity of (ii) Empirically, we show that downstream task performance is strongly correlated with mutual information: models (and layers) with higher mutual information about both the input X and the target Y consistently achieve higher accuracy.
2 METHODOLOGGY
Unidirectional language models, such as GPT, construct each hidden representation using only left-to-right context (Radford et al. 2018). In contrast, bidirectional models like BERT encode each token using both past and future context (Devlin et al. 2019 He et al.] 2021 Liu et al. 2019). This architectural asymmetry raises a natural question: can bidirectional representations carry more information?
Let X (51 (#i,....,2: .n) denote the input sequence. For layer f, let 7,- ( be the forward (causal) representations, where 2," depends only on I<t. Let 2/- = (zI..... be the backward (anti-causal) representations, where 2 depends only on T2t. A unidirectional model uses 2 whereas bidirectional model augments this with Z,- and forms the full bidirectional representation Z = (Z,", 2 (e.g.. by concatenation or another fusion). We measure repre- sentational quality via mutual information: T(X: Z) = H(X) - H(X 2), where H(X Z) is the conditional entropy of the input given Z. Because Z includes strictly more context than Z it can, in principle, reduce uncertainty about X more effectively. This follows from the monotonicity of conditional entropy: conditioning on more information reduces entropy (Theorem A.2 In this sense, Z defines an information-tie theoretic upper bound on how much information any representation obtained by deterministically merging the forward and backward directions can retain about the input, and this upper bound is at least as large as that of the purely forward representation Z," Therefore, bidirectional models can, in principle, produce latent representations that retain at least as much (often strictly more) information about the input sequence as purely unidirectional models.
Theorem 2.1 (Full version in Appendix [A.3). Bidirectional representations preserve more mutual in- formation about the input and the output: T(X; 2T7) T(X; Z,²) ) and T T () 2 1(Z7;Y).
3

---

Published as a conference paper at ICLR 2026
While mutual information quantifies how much information representation Ze preserves about the input or the target, it does not describe the internal structure or complexity of that representation. To complement MI, we analyze the spectral properties of Ze via effective dimensionality, which captures how many orthogonal directions in representation space carry significant variance. This helps characterize how richly each layer encodes information.
Definition 2.2 (Generalized Effective Dimensionalityy. Let Σx, Cov(Z,) and let AI...., da be its nonzero eigenvalues, where n rank(E (Zzz,) ). Define the normalized spectrum pt == X1/ 2j=1* 2j. The generalized effective dimensionality of Ze under spectral functional M is
daa M) == exp (M(p)),
where M(p) is real-valued function of the spectrum that satisfies: (i) nonnegativity: M(p) 2 0; (ii) maximality: M(p) & logn, n, with equality iff p; = 1/n for all i; (iii Schur-concavity: if pf > then M(p) S M(p).
Examples. (1) Shannon entropy: M(p) - Ps Ps yields den((z)) = ex)(H(p))
(Roy & Vetterli 2007 (2) f participation ratio: M(p) = log(1/ Ei- pē) gives dett (Zr)
EE No) ا X Intuitively, the &g participation ratio measures how many eigen-directions are effectively active: if the spectrum is spread out over many eigenvalues, dett is large, whereas if most variance concentrates on a few eigenvalues, deft becomes small. Unless otherwise stated, we adopt the f2 version the default. The effect of alternative measures is explored in Appendix C..]
Lemma 2.3 (Bidirectional Representations Exhibit Higher Spectral Complexity). Let E R de- note the unidirectional representation and Z == (Z 22;) E RD2DD the concatenated bidirectional representation of an input X. If Cov ( Zz is nonsingular, then deat (Z, :M) 2 dea (Zz"; :M), is conditionally redundant given Zz", i.e., Cov(Zi Zr²) =
with equality iff z
See Appendix
A.5 for the proof and Appendix C.2 for ablation.
Key Finding
Bidirectional representations retain at least as much (and typically strictly more) mutual infor- mation about the input than unidirectional representations. They also exhibit higher effective dimensionality throughout depth, reflecting richer and more expressive latent spaces.
NIB. For empirical validation of Theorem 2.1 we need quantitative way to measure mutual information and check whether bidirectional model yields higher MI than unidirectional model at each layer. We therefore use FlowNIB, which is based on MINE. Because we work with finite data and neural network critic, the resulting MI values are only approximate and should be viewed as relative scores, not exact or perfectly calibrated quantities. We use these scores to compare different layers and model types (e.g., bidirectional vs. unidirectional) and to study how they correlate with downstream accuracy, rather than to claim new exact information-thhoretti bounds. After fine-tuning the LM dataset, FlowNIB approximates the mutual information of each layer, quantifying how much information a layer carries about both the input and the target. FlowNIB is simple: it trains two independent MINE critics by minimizing a single objective with a time- vime-ngrying weight:
(() -(a(l) Zx)
(1))()) ((z)))).
Here (tt) : {O..... T) + [0. 1] is a discrete, monotonically non-increasing schedule. A key motivation for the schedule is that training two separate MINE critics- one for I(X; Ze) and one for I(Zr; Y)-produces MI values that are not comparable: neural MI estimators depend strongly on critic capacity, learning rate, and training duration, so two independently trained critics may operate at different scales.
FlowNIB resolves this by using critic network that estimates both quantities along one continuous training path. The schedule a(t) controls which MI the critic focuses on at each stage. Early in training (a 1), the objective emphasises [(X; Ze), so the critic learns how much information the representation Zg retains about the input X. As (xt)) decreases, the emphasis gradually shifts toward [[Z; Y), and the critic instead learns how much of that information is predictive of the target Y.
Because both estimates come from the same network under shared optimization samiiionn--mmeee capacity, same learning rate, same training history-they more comparable than estimates from independently
4

---

Published as a conference paper at ICLR 2026
trained critics. This does not eliminate all estimation bias, but it ensures that differences between [[;; Z and I(2;; Y) reflect genuine properties of the representation Ze rather than artifacts of separate optimization runs.
At each iteration the critic produces pair (7²) (x; Zz), (Zg; Y)). We define the OIC as the point t* along this trajectory where (tt)) has transitioned enough that neither estimate dominates the optimization, so the critic has had sufficient exposure to both objectives. At this point, the pair (x( Ze), T(*) (Z;; Y)) serves as a relative measure of how much information Ze retains about the input and how much converts into target-relevant structure. Because the same critic produces both values under shared training dynamics, these OIC coordinates can be meaningfully compared across layers and across models.
Formally, att) "......, T - [0, 1] is discrete, monotonically non-increasing schedule. We use (0) = 1 and a(t+1) = max 0, (t) - 5), where 8 > 0 is a small step (e.g.. = 0.001); if T is small, larger& 5 ensures the schedule covers [1. 0J within T steps (see Appendix C.I for an ablation on 8). At each step we record the information-plant coordinate (X() Zt), (z:
During training, we optionally normalize [[x; Ze) by the per-layer effective dimension der' (Zt) and T( Y) by deff (Y) to reduce scale effects. This normalization is used only for optimization and does not affect the MI values we report. Figure a shows that the effective dimension depends strongly on the size of the output space. When the label space of Y is small, dett (Ze) starts at a moderate value and typically drops as we go deeper, because the task does not require the network to maintain a large amount of information. When Y is high-dimensional, d de (Zz) instead increases with depth as the network needs richer representations solve the task.
A similar pattern appears in Figure & for mutual information, consistent with our Key Finding For oow-dimensional Y. I(X; Z) usually decreases with depth, while I(Z; Y)) increases only slightly For high-dimensional Y. T(2: Y) rises much more sharply and tends to saturate later. These trends explain the apparent scale imbalance in Figure [] on GLUE (where labels take only 2-3 values),
[(X; Z) often appears much larger than [(; Y) simply because the label space is small. Since effective dimension correlates with the amount of mutual information a layer can realistically encode, normalizing by (-)(-) provides simple, task-aware scaling that keeps the FlowNIB objective in equation balanced (see Proposition B.3 and Ablations C.2
and C.4 for details).
Over all epochs t = O...., T, we then select the OIC for each layer, which summarizes the layer's capacity to jointly capture information about the input and the target, as detailed in Figure
In Practice. (i) Fine-tune the LM on a dataset with inputs X and targets Y. (ii) Run the model once to cache (X,Y. Y. Zz) for all layers& f. (iii) For each f, fit two crities on this fixed cache-one for T(X; Z and one for T(Z;; Y the same neural MI setup (iv) Train the crities by minimizing equation with the schedule (t). (v) Compute the OICs, We report these as relative measurements (e.g.. for OIC selection) rather than absolute MI values Full details are in Appendix B
3 EXPERIMENTS
This section presents empirical evidence for our theoretical findings. We conduct two complementary evaluations. First, after fine-tuning each model on a dataset, we apply sequence-level FlowNIB to every layer f, obtaining per-epoch coordinates ( (X; Zt), (Ze; Y)). In practice, we take X to be the embedded input sequence (token embeddings plus positional embeddings from the model's embedding layer), and Zz(x) to be a sequence-level representation at layer f. For each layer we then select the OIC to summarize its joint ability to retain input information and align with the target; comparing OICs across layers, we want to show that bidirectional LMS consistently achieve higher information than unidirectional LMs. Second, because large bidirectional LMs are limited, we perform downstream fine-tuning under a matched parameter budget (5600M parameters) on both classification and regression benchmarks, and compare task performance test whether the information advantage translates into end-task gains. To ensure fair comparison, all models use identical data splits, training budgets, and a common PEFT recipe, RoCoFT (Kowsher al. 2025a), which updates a small subset of existing weight rows without introducing new adapter parameters
All MI numbers neural hound estimates with fixed hyperparameters across layers and models; no additional noise or quantization is added.
5

---

Published as a conference paper at ICLR 2026
(we update three rows per linear layer). This setup is closer to full fine-tuning in parameterization while preserving pretrained information and keeping the fine-tuning footprint comparable across architectures. In contrast, adapter-biss based PEFT methods add new parameters that confound comparisons. Additional results with LoRA appear in Appendix T For FIBwNIB, we report relative MI quantities (for OIC selection and comparison) using the same estimator architecture, batch size, negative sampling scheme, optimizer, and training steps across layers and models; absolute MI numbers are not the focus.
Clasaifier
Clossifier
CLB
X
X
X
X
Figure 2: Illustration of representation extraction methods: (a) prediction from CLS-token (bidirec- tional), (b) prediction from pooled embedding (unidirectional), (c) prediction from masked token (bidirectional), and (d) prediction from next-token generation (unidirectional),
Model framework. While standard approaches apply a pooling operation by averaging over final hidden states followed by classifier, we adopt an alternative strategy inspired by the PredGen framework (Kowsher et al., 2025b). Instead of averaging. PredGen follows the native behavior of LMs-e.g.. masked prediction or next-token generation- for prediction tasks. PredGen demonstrates that leveraging the model's ' generative or masking capability, rather than relying solely on pooled representations, retains higher mutual information with the input and improves prediction quality. However, a key limitation of PredGen is the increased computational cost of multi-token generation, especially for regression-type tasks.
To address this, we modify this framework to use a single-token generation or masked prediction setup for both the downstream task and mutual information estimation, as illustrated in Figure 2 (right). Specifically, the model predicts single masked token at a designated position, from which we extract the corresponding final hidden state. This representation is then passed through lightweight MLP classifier. In Table 37 we compare single-token prediction with PredGen across diverse datasets; see Appendix L for details.
In short, we focus on answering the following three research questions: (i) Do bidirectional models preserve more useful information than unidirectional models? (ii) Does higher mutual information lead to better context modeling? (iii) Does predicting single token (e.g., masked token or next token) lead to better performance than traditional methods?
Key Finding
We illustrate a simplified variant of the PredGen framework that replaces multi-token genera- tion with single-token generation masked prediction. This approach achieves comparable performance to PredGen while substantially reducing inference cost and training complexity. See Appendix Table 37 for the comparison between single coken-based prediction and PredGen.
We evaluate our models across 15 diverse NLP datasets spanning classification and regression tasks to ensure a comprehensive analysis of representational learning under the information bottleneck framework. For classification, we include SST-2, MRPC, QNLI. RTE, MNLI, and CoLA from the GLUE benchmarks (Wang et al., 2018), as well as BoolQ CClak et al. 2019). HellaSwag (Zellers et al. 2019), and SIQA (Sap et al. 2019), covering range of linguistic challenges such as sentiment analysis, natural language inference, grammatical acceptability, question answering, and commonsense reasoning. The regression tasks comprise STS-B (Cer al. 2017).
Datasets:
SICK (Marelli et al. 2014a), WASSA (Mohammad & Bravo- Travo-Marquer 2017). LCP (Shardlow et al. 2020), CWT (He et al. 2021) and Humicroedit (Hossain et al. 2019). addressing semantic textual

---

Published as a conference paper at ICLR 2026
6.0
Direction
GP1-2 medium
deberta
Unidirectional Bidlrectional
GPT-2 large
deberta-v V-laage
SmolLMZ-133M
roberta -base
SmolLM2-360M
large
MM-25eLLX-125M
moden bert basc
MobileLLM
10::21
MobileLLM BOOM
3.0
0.0
SST-2
MAPC
RTE
HellaSwag
SICK
COLA
STS-B
Avg.
GPT-2_medium
deberta 433
1.2
GPT-2_larg
deberta v3 large
(0-0LM22-13580
SmoILM2-36BD
roberttallrige
bodden-bert-bate
MobileLLM-35000
modem bert large
0.8
MobiieLLM-6GO
0.4
0.0
HellaSwag
Model Avg.
Figure 3: Average OIC T(X; Z) (top) and I(Z;Y) (bottom) across all layers for unidirectional and bidirectional LMs over multiple datasets. Bars show dataset-wise and average values, comparing information flow differences between architectures.
0.6
2.5
0.5
2.0
1.50
0.3
1.5
1.25
0.2
1.0
1.00
0.3
0.0
0.0
10
0.25
100/2]
0.40
1.6
0.35
2.4
0.25
0.30
1.2
0.20
0.25
1.0
0.20
0.8
0.6
0.10
0.4
0.05
0.00
0.00
0.0
(06.2
10K,20
Figure Mutual information flow comparison between bidirectional (top) and unidirectional (bottom) models across three datasets. The first column shows results on the SICK dataset using DeBERTa- base and MobileLLM-330M. The second column shows SST-2 results using RoBERTa-borr and MobileLLM-330M. The third column presents results the CoLA dataset using DeBERTa-v3-1 and MobileLLM-600M.
7

---

Published as a conference paper at ICLR 2026
5ST-2
MRPC
QNLI-1
RTE
COLA
QNLI-2
(a) RoBERTa-Base
(B) RoBERTa-Large
Smo1LM2-135M
(d) SmoILM2-360M
100
90
Accuracy
80
Layers 10 12 10 15
20
100
20
30
10
20
30
Layers
Layers
Layers
Figure 5: Layer-wise linear prohe accuracy. For each layer E, a logistic regression classifier is trained on frozen representations Zz(x) to predict dataset labels. Accuracy increases with depth across all architectures and tasks, indicating that deeper layers encode stronger task-aligned information.
similarity, lexical complexity prediction, and humor detection. Dataset sizes range from approximately 2,500 to 400,000 examples, with either binary multi-class classification labels, or continuous- valued targets for regression. We exclude generation-bass tasks because bidirectional language models are not designed for auto rettereerees generation; instead, focus tasks requiring strong contextual representations to assess representational sufficiency under the information bottleneck. Additional dataset statistics are provided in Table T0 in the Appendix. In addition, the details of used models architecture, hyperparameters, evaluation metrics, and environment setup are provided in Appendix Appendix Appendix
H and Appendix G respectively.
MI results. To measure layerwise information, we first fine-tune each model on the target dataset, then run single pass cache triplets (X, Y, Ze) for every layer & L, where Ze denotes the layer's activations on X. Given this fixed cache, we instantiate two identical two- fully connected layer (nn.Linear() in PyTorch) estimator networks (same widths, nonlinearity, and initialization): one estimates (X; Ze) and the other estimates (Ze; Y). Both estimators are trained jointly under the common FIowNIB objective in equation| [ with a discrete schedule (t)) that linearly decays from 1 to 0: (0) 1, (l++))
max(0, (t)) - -], ō 0.001. Unless noted otherwise, we batch size of 128, T = 2000 training epochs, run each experiment with 10 random seeds, and the same optimizer and negative-sampling scheme across all layers and models. At each step record the information- plane coordinate (I(+) (X: Zt), [(t) Y)). After training, for each layer l we select its OIC from these coordinates; the OIC summarizes the layer's 's capacity to jointly capture input and target information. We apply the same estimator architecture, schedule, and hyperparameters to all bidirectional and unidirectional models, enabling a like-for-like comparison. The full procedure is given in Algorithm
Figure 3 compares the average OIC across all layers between bidirectional and unidirectional LMs. We observe that bidirectional models consistently retain higher mutual information for both ((;; Z) and I(Z Notably, even smaller bidirectional models (c.g., RoBERTa-base, 125M) surpass larger unidirectional models (e.g.. MobileLLM-600MM Smo1LM2-360M) in OIC on many datasets. To further elucidate this behavior, Figure 4 visualizes the information-pian trajectories layer by layer over the estimator training horizon T, contrasting bidirectional and unidirectional models on multiple datasets. Across layers and epochs, bidirectional models trace trajectories with systematically higher [(;; 2) and I(Z; Y). aligning with their larger OICs. addition, Figure 9 shows a token-leve MI analysis from the final layer (after fine-tuning SST-2), which further highlights the representational advantage of bidirectional models.
To quantify how much task-relevant information each layer encodes, we conduct a standard layer-wise linear probing analysis. Given a fine-tuned model with layers {( 1.... L}, let Zz(x:) dd denote the hidden representation at layer e for input example For Zx(xx))
Layer- wise Linear Probing.
each dataset D = { y:) } we extract frozen representations Z,
N.....N
For every layer f, train logistic regression on the fixed representations (x(x: while keeping all
model parameters frozen. The procedure consists of three steps: (i) extract (x.) for each training
example, (ii) train a classifier on the pairs (Ze(x.). y1), and (iii) evaluate the trained probe on the held-out split to obtain layer-specffic classification accuracy. This metric quantifies how linearly decodable the labels are from each layer. a simple linear classifier achieves high accuracy at depth then Ze contains strong task-relevant structure.
8

---

Published as a conference paper at ICLR 2026
Figure shows the resulting waser-wise probe accuracies. Across all architectures (RoBERTa: CaBBRRaa-Base RoBERTa-Large SmoILM2-133MM SmolLM2-36600 360M) and across all datasets (SST-2, MRPC, QNLI- RTE, CoLA, QNLI-2), we observe consistent trend: bidirectional models exhibit higher probe accuracy than unidirectional models at nearly every depth, including the earliest layers. This indicates that bidirectional representations retain higher OIC, leading to richer and more linearly decodable features throughout the network. The linear probing results therefore provide independent evidence supporting our main claim that bidirectional architectures produce more informative representations than unidirectional ones.
task-relevant mutual information and hence
Key Finding
OIC strongly correlated with model performance: representations with higher OIC values- high mutual information with both the input and the output consutsenttssitttyccccc yield better downstream task accuracy.
Evidence for the MI-Performance Link. clear pattern emerges: bidirectional mod- els (top block of Table 1 consistently achieve higher average accuracy and lower regression loss than unidirectional mod- els (bottom block), even when the latter have comparable or larger parameter counts. In particular, smaller bidirectional models such DeBERTa-v3- -a--RTa,----B,,,. RoBERTa-Base, and ModernBERT-BB outperform larger unidirectional models (e.g.. GPT-2 Large, MobileLLM--600M1 on both classification and regression metries. This indicates that bidirectional architectures provide more effec- tive context understanding under the same or lower compute budget. These findings align closely with our mutual-information analy- sis in Section [ FlowNIB shows that lay- ers bidirectional models carry systemati- cally higher MI with both the input X and the target than their unidirectional counter- central claim:
Model DeBERTa-v3-
Method
Acc.
MAE/MSE
Pooling
Masking
81.52
0.197/0.298
DeBERIB.V3-Lagg
Pooting
80.96
0.187 0.225
Masking
84.73
0,184 / 0.282
RoBERTs-Rase
Pooling
76.53
0.218 0.311/ 0.314
Masking
79.95
0.206 0
RoBERTa- Laree
Pooling
80. 14
0.197 / 0.298
Masking
0.195 0.297
ModemBEKT-Ber
Pooling
76.74
0.229 0.324
Masking
79.73
0.220 0..20
ModemBERT
Pooling
80.35
0,200 / 0.305
Masking
83.84
0.197 0.300
GPT-2 Medium
Pooling
0.313
Generation
72.04
0.300 / 0.375
GPT-2 Large
Pooling
71.26
0.288 0.366
Generation
72.07
0.279 / 0.354
SmolLLM2 135M
Pooling
71.37
0.218 0.322
Generation
72.82
0.210/0.311
SmolLM2-330M
Pooling
72.95
0.213/0.314
Generation
0.207 0.310
MobileLLM-225M
Pooling
70.48
0,211 / 0.320
Generation
71.92
0.205 0.314
MobileLLM330M
Pooling
71.89
0,200 0.300
Generation
73.73
0.198 0.334
MobieeLLM-00M
Pooling
74.50
0,193 003//0.300 0.302
Generation
76.55
0,193 0.393/0.302
parts. Therefore it supports models with higher MI about the imput and target yield better downstream task perfor- mance, and bidirectional models benefit from this advantage more strongly than unidirec- tional models. The full results are provided in Appendix
Table 1: Accuracy (%). Acc. denotes the average
accuracy over all classification tasks (detailed results in Table {) and the average MAE/MSE over all re- gression tasks (detailed results in Table
Model
Layer
Heads
Embd. Dim
Max Length
Vocab Size
Params
FLOPS
MAC'
Time
ModerrBERT--uss
30368
28.258
ModermBERT-large
28
16
1024
8192
50368
395M
87.883
43.923
2.53
RoBeRTa-base
768
512
125M
21.760
RoBERTh-large
24
16
1024
512
50265
355M
77.344
38. 656
6.06
DeBERTa-y3-basc
768
512
184M
39.275
DeBERTa-v3-barg.
24
16
1024
512
128100
435M
136.943
68.451
6.48
GPI2-medium
24
16
1024
1024
50257
345M
77.342
38.655
6.04
GPT2-large
36
20
1280
1024
50257
762M
181.254
90.597
12.46
SmolLLM2-135M
30
9
576
2048
49152
135M
27.185
13.590
2.52
Smoll M2-360M
32
15
960
2048
49152
360M
80.541
40.265
7.04
MobileLLM-1255
30
9
S76
2048
32000
125M
31 900
15. 9,5950
3.83
MobleLLMM-60M1
154,408
Table 2: Overview of bidirectional (top) and unidirectional (bottom) model architectures evaluated in experiments, including FLOPs, MACs and Training time (s/step).

---

Published as a conference paper at ICLR 2026
Bidirectional vs. Unidirectional Model Efficiency. Although bidirectional Transformers theo- retically more expensive- due to full-sequence self-attention at every layer-the the empirical results in Table reveal different practical trend. When controlling for the same training conditions (learning rate 2 x 10-5, batch size 64, and two epochs on SST-2 using a single H100 GPU), several smaller bidirectional models not only train faster but also achieve higher accuracy than larger unidirectional models.
For example, RoBERTa-base (125M, bidirectional) requires only 21.8 GFLOPs and trains in 2.11 per step--ubstantialll faster than MobileLLM-I255M (3.83 s) and even the larger GPT2-medium (6.04 s), despite both being unidirectional models. Similarly, ModernBERRT-bas. (149M) achieves competitive compute cost (28.3 GFLOPs, 1.15 s) compared to unidirectional models of similar or larger size, such as SmolI (2.52 s) or GPT2-large (12.46 s). This pattern continues at larger scales: Ta-laree (355M, 6.06 s) trains faster and with fewer FLOPPs than SmolL.M-330M (7.04 s) and significantly outperforms GPT2-large (762M, 12.46 s). Even the largest bidirectional model, DeBERTa- -a-agee,, remains competitive with the unidirectional MobileLLM-600M despite having more expressive capacity.
Combined with our mutual information analysis and linear probing results, these findings demon- strate that bidirectional architectures encode richer, more task-relevant information at each layer. Consequently, smaller bidirectional model can match exceed the performance of much larger unidirectional model-while requiring less compute and achieving faster training time. Thus, in practical settings, bidirectional models offer more efficient and powerful trade-off between compu- lational cost and representational quality.
4 RELATED WORK
Information bottleneck in deep learning The IB principle has been studied from both practical and theoretical perspectives in deep learning. On the practical side, (Alemi al.) 2016 Higgins ct al. 2017 Achille & Soatto 2018) formulated the IB problem as deep learning objective and
introduced variational approximations enable optimization via gradient descent. On the theoretical side, (Tishby& & Zaslavsky 2015; Shwartz-Viv & Tishby 2017) provided an information- -thhorrtccc
framework for understanding deep learning, establishing the IB as a foundational tool for analyzing representation learning and generalization in deep learning. These fundamental ideas have inspired wide range of follow-up works (Goldfeld & Polyanskiy] 22000 Saxe et al., 2019) Shwartz-Ziv 2022)
that further investigate deep learning dynamics through the lens of information theory.
Mutual information estimation Mutual information quantifies the statistical dependence between two random variables and plays an important role in the IB principle. However, the mutual information is notoriously difficult to estimate between continuous high- nigh-dimensional random variables. Traditional nonparametric approaches (Fraser & Swinney [986;; Moon et al. 1995 Darbellay & Vajda 1999] Suzuki et al.) 2008 Kwak & Choi [002 Kraskov et al.) 2004) typically are not scalable with dimension and sample size. To achieve an effcient estimator, recent work (Nguyen et al.) 2010) Nowozin et al. 2016) characterized the mutual information of two random variables with the Kullback-
Leibler (KL-) divergence (Kullback] 1997) between their joint distribution and the product of the marginals and used dual representations to cast the KI divergence. The Mutual Information Neural Estimator (MINE) (Belghazi et al. 2018)) utilized the dual representation of the KL divergence and estimated mutual information via gradient descent over neural networks and thus scaled well.
CONCLUSION
This work investigates why bidirectional models outperform unidirectional ones in natural language understanding and context modeling, combining theory with empirical evidence. We introduce FlowNIB, dynamic, IB-based framework that tracks layer-wise mutual information over training. Our results show that bidirectional models retain more input information and more predictive information, yielding stronger representations and better downstream performance. FlowNIB offers principled explanation for this advantage and suggests new directions for analyzing and improving deep language models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 2018.
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00 0040000 2016.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martin Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlívcek, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big -eg-dtt--centric training of small language model. arXiv preprint arXiv:2502.00273, 2025.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference machine learning, pp. 531-540. PMLR, 2018.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpln and Lucia Specia. Semeval-2017 task
I: Semantic textual similarit--uuttiingaal and cross-lingual focused evaluation. arXiv preprint 2017.
Hao Cheng. Dongze Lian, Shenghua Gao, and Yanlin Geng. Utilizing information bottleneck to evaluate the capability of deep neural networks for image classification. Entropy, 21(5):456, 2019.
Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 19%1v49905.10044 2019.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint 0rXiv:2003.10555 2020.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, 2006.
Georges A Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45(4):1315-11 1321, 1999.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Chun Fan, Jiwei Li, Xiang Ao, Fei Wu, Yuxian Meng, and Xiaofei Sun. Layer-wise model pruning based on mutual information. arXiv preprint arXiv:2108. /2594, 2021.
Andrew M Fraser and Harry L Swinney. Independent coordinates for strange attractors from mutual information. Physical review A, 33(2):1134 1986.
Ziv Goldfeld and Yury Polyanskiy. The information bottleneck problem and its applications in machine learning. IEEE Journal on Selected Areas Information Theory, 1(1):19-38, 2020.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-dieentangled embedding sharing. arXiv preprint arXiv:2111.09543 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference learning representations, 2017.
Nabil Hossain, John Krumm, and Michael Gamon. "president vows to cutj taxes; hair": Dataset and analysis of creative text editing for humorous headlines. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume (long and short papers), pp. 133-142, 2019.
11

---

Published as a conference paper at ICLR 2026
Zhiying Jiang, Raphael Tang, Ji Xin, and Jimmy Lin. Inserting information bottlenecks for attribution in transformers. arXiv preprint arXiv:2012.13838, 2020.
Md Kowsher, Tara Esmaeilbeig, Chun-Nam Yu, Chen Chen, Mojtaba Soltanalian, and Niloofar Yousefi. Rocoft: Efficient finetuning of large language models with row-column updates. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 26659-26611 2025a
Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat, Chun-Nam Yu, Mojtaba Soltanalian, Ivan Garibay, Ozlem Garibay, Chen Chen, and Niloofar Yousefi. Predicting through generation: Why generation is better for prediction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 26845-26871, 2025b.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E Statistical, Nonlinear, and Soft Matter Physics, 69(6):066138, 2004.
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Nojun Kwak and Chong- Goo Choi. Input feature selection by mutual information based on parzen window. IEEE transactions on paltern analysis and machine intelligence, 24(12):1667-1671, 2002.
Qintong Li, Zhiyong Wu, Lingpeng Kong, and Wei Bi. Explanation regeneration via information bottleneck. arXiv preprint arXiv:22/2.09666 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai. Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference Machine Learning, 2024.
Mokshay Madiman and Prasad Tetali. Information inequalities for joint distributions, with interpreta- tions and applications. IEEE Transactions on Information Theory, 56(6):2699-2713, 2010.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Lunguage Resources and Evaluation (LREC' pp. 216-223, Reykjavik, Iceland, May 201 4a. European Language Resources Associ- ation (ELRA). URL attp:/ /wwww rec- conf http://pooeeee..o////rr//////// 363 Paper.por
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. The sick (sentences involving compositional knowledge) dataset for relatedness and entailment. URL: https://doi. org/10.5281/zenode 2787612, 2014b.
Saif Mohammad and Felipe Bravo-Marquez, Wassa-2017 shared task on emotion intensity. In Proceedings of the 8th workshop on computational approaches to subjectivity, sentiment and social media analysis, pp. 34-49, 2017.
Young-ll Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52(3):2318, 1995.
Thanh T Nguyen and Jaesik Choi. Liyer-wise learning of stochastic neural networks with information bottleneck. arXiv preprint arXiv: 1rXiv:1712.012222 2017.
XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11) (6(11)558475861, 2010.
12

---

Published as a conference paper at ICLR 2026
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization, Advances in neural information processing systems, 29, 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 2020.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610.10 IEEE, 2007.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.097728 2019.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Matthew Shardlow, Michael Cooper, and Marcos Zampieri. Complex: A new corpus for lexical complexity prediction from likert scale data. arXiv preprint arXiv:2003.07008 2020.
Ravid Shwartz-Ziv. Information flow in deep neural networks. arXiv preprint arXiv:2202.06749, 2022.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint 17X3 703.00810, 2017.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual informa- tion by maximum likelihood density ratio estimation. In New challenges for feature selection in data mining and knowledge discovery, pp. 5-20. PMLR, 2008.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. Ieee, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicx/0004057, 2000.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rXiv:1804.07461, 2018
Shaojie Wang, Sirui Ding, and Na Zou. Rethinking the understanding ability across 1lms through mutual information. arXiv preprint arXiv:2505.23790, 2025.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, bet- ter, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXi::2412.1366. 2024.
Xuansheng Wu, Jiayi Yuan, Wenlin Yao, Xiaoming Zhai, and Ninghao Liu. Interpreting and steering lims with mutual information-based explanations on sparse autoencoders. arXiv preprint arXiv.2502.15576, 2025.
Zhou Yang. Zhengyu Qi. Zhaochun Ren, Zhikai Jia, Haizhou Sun, Xiaofci Zhu, and Xiangwen Liao. Exploring information processing in large language models: Insights from information bottleneck theory. arXiv preprint arXiv:2501.0099 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905.07800 2019.
13

---

Published as a conference paper at ICLR 2026
Haoyi Zhou, Shanghang Zhang. Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115,20 2021.
14

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Methodology
3 Experiments
Related Work
10
Conclusion
10
Bidirectional Unidirectional Representation
16
FlowNIB: Flow Neural Information Bottleneck
20
Ablation Study
24
C.I
Effect of step size à FlowNIB dynamics
24
Effective Dimensionality Across Models
24
Effective Dimensionality vs. Output Complexity;
24
Mutual Information Dynamics Across Output Dimensions and Layers:
25
C.5
Validating Generalized Effective Dimensionality
25
Stability Across Random Seeds
26
Bidirectional vs. Unidirectional Attention in Time-Series Forecasting
26
LoRA Based Performance Comparison
27
Details Results
27
Dataset
29
Environment Setup
29
H Evaluation Metries
30
Model Description
30
J Hyppepaarameeers
30
Model Profile Information
31
PredGen One- Token Generation:
One-Token
34
15

---

Published as a conference paper at ICLR 2026
A BIDIRECTIONAL VS UNIDIRECTIONAL REPRESENTATION
Theorem A.1 (Conditioning Reduces Entropy), Let and be continuous random variables with joint density (x, y), marginal densities fx (x), fy and conditional density fxly The differential entropy satisfies:
H(X) 2 H(XIY),
where (X) and HY) denote the marginal and conditional differential entropy, respectively. Cover & Thomas 2006
Proof. For continuous random variables, differential entropy is defined as:
H(X) -- fx(x) (x()), H( = - -I fx.r y) log fx1x (rly)dady.
Substituting (xlr( (x,y(a,)) into H(X(Y), we derive: (r(
H(XI)) // (x.r fx v) drdy
fr(
Expanding the logarithm:
-Iʃ fxry(1, v) log fx. v) drdy + /ʃ fx. log fo (v( drdy.
H(XIY)
B(X.)
The second term simplifies using the marginal fx,Y(E,)))) fr)
If fx, y) log fy dw)d
J (rrW)logfrr))d)
-H(
Thus,
H(XIY)= = - H(Y).
To show H(X) 2 H(XIV), we invoke the non-negativity of the Kullback-Leibler (KI.) divergence:
fx, v) log fx.((2.)) dxdy
(fx =
0.
fx(x)fr(y)
Expanding the integrand:
DRL. fx,y log (x, wywddydddddyxxxxx log fx (x))dady-Sxx -x, (2, v) log fr (y)drdy.
Recognizing the entropy terms:
DRL. = -H(X,Y) + H(X) + H(Y) > o
H(X) + H(Y) 2 H(X,Y).
Substituting H(X,Y) (XXI)) + HYY) into the inequality:
16

---

Published as a conference paper at ICLR 2026
H(X) 2 H(X))).
Theorem A.2 (Monotonicity of Conditional Entropy). Let X, Y, Z be continuous random variables. Then the differential entropy satisfies:
H(X V) 2 Y,2),
with equality if and only if X 1 Z ZI. More generally, for any sequence YI...., ......,,..
H(X Yi) H(X Y2)
H(X , Yn).
Proof. We begin with the definition of conditional differential entropy:
H(X Y) --=-] // fx,( log fxir (xr v) dx dy, H(X 22
y, 2) dx dy dz.
Recall that:
fxlr (xr v) - - / fx|x22(0 v, 22=) v) dz.
Now apply Jensen's 's inequality using the convexity of - log(-):
- llg (J fxivzaaa v. (2 1 dz) S s-f [2x(( s- y) log fx v, 2) dz.
Multiplying both sides by fx1Y(< v) and integrating over ær y. we obtain:
H(X r = - // fxx((z, () fx y (æ y) dx dy
2 fx,Y, 2) log (x v,2) dr dy dz
H(X YY,))
Equality holds iff Jensen's inequality becomes equality, which occurs and only if fx y, = f (x v) a.e. 2,
ic., X 1 Z Y.
For the generalization, apply this result inductively:
H(X Y)) 2 H(X Y2,Ys 2 2.. H(X Yy,...,y,,)
Theorem A.3 (Bidirectional Representations Preserve More Mutual Information). Let X denote .....,,,,
sequence imput 3τ, ......, Fr. of layer & from the forward context:
Let Z denote the unidirectional hidden representation constructed
Z (2T 22
x with ,. ((11...,
and Z the backward reprexentation:
z =2 x, with 2 = g(z
Let the bidirectional representation be:
Zitt (Z Zz
Then the mutual information between X and the bidirectional representation satisfies:
I(X; 2 (7";
with equality if and only if 7/ X 2
17

---

Published as a conference paper at ICLR 2026
Proof. We begin with the identity:
[ = H(X 2).
Apply this to both representations:
( H(X z7), - H ze' 7.)
Since - contains strictly more information than Zz we can invoke the monotonicity of conditional entropy A.2
H(X 2 Zz, zz)
with equality iff X 1. z Zt"
Subtracting both sides from (X) gives:
= H(X 22,47 H(X)
H(X (7)
Ze)
Thus:
2 2 I(x:
Equality holds iff:
H(X
which by the equality condition of monotonicity of conditional entropy holds iff:
X 1 z Zi.
Similarly with respect to output we can show:
2 Y ((Z
This completes the proof.
Theorem A.4 (General Bound on Representation Difference). Let Zz 25 RR denote the bidirectional and unidirectional representations of the same input token at a given layer, and define:
Az = Z Ze¹
Then the expected squared difference satisfies:
E|Aa|| tr Cov((,²)) + ir Cov (Z;z) 2 Cov 7,) + ||[
In particular, we have the following bound:
Ir Cov 27²t + tr Cov Cov(2 2| lr Cov ;i+)
<
< Cov(Zi") + Cov (Z() 2/ Cov (Ze Zi')l.
Proof. By the covariance identity, we have:
Cov(Az) = Cov + Cov(Z,*) - Cov(Z,t, - -Cov(Zi (ttt Taking the trace and noting that tr (AT) tr(A), we obtain: tr Cov(Az) = tr Cov( ( tr Cov(Zet) tr Cov( ;Z7)
The expected squared norm decomposes as:
E A ir Cov + |B|A2]12
18

---

Published as a conference paper at ICLR 2026
Substituting the expression for Cov(Az) yields the stated identity.
Finally, since for any real scalar a, we have -lal I a S |a], it follows:
-|tr Cov (Z z7) S tr Cov zr Σ tr Cov
which implies:
ir Cov(Az) E (rr Cov(Zi") + tr - - -2 Cov(Zi" Zi²)];
Ir Cov (Z + ir Cov(22 + 2/tr Cov( Z[
Substitute into the expectation equation to complete the proof.
Lemma A.5 (Effective Dimensionality Bidirectional Representations), Let Zi E RR denote the unidirectional representation and 7 == (7 7 TR2D the concatenated bidirectional representation of input X. Define h-norm-basedd effective dimension
(Z)(2) = (1)
Σ,
where X are eigenvalues of the covariance matrix of Ze. If Cov( -,T) is non-singular, then: des (Zt)) 2 dett (Zt)(ZT"))
with equality iff " conditionally redundant given Z (i.e., Cov(Z, (17)) = 0).
Proof. Let ; Cov RDD and D = Cov(Zi") E 2D2x2D denote the covariance
matrices of unidirectional and bidirectional representations, respectively.
By block structure:
y+
CT
Σ
where C' = C
Let {A,1)21 be eigenvalues E and {25²) 12 13=1 eigenvalues of Σ
Since ~ augments " with additional variables Z and cross-covariance C, by eigenvalue interlacing theorem (Cauchy's interlacing), we have:
2D Σ
j=l
im1
and
2D
>
Σ (2,))
1=1
1=1
with strict inequality if C or E is nonzero.
Applying definition:
EE Σ,(5リ)
des (2+²
Since numerator and denominator both increase under positive-definil augmentation, and quadratic- over-linear ratio increases under positive additive terms (Jensen's inequality), we conclude:
doft (2tt) 2 detf (Z,").
Equality holds iff - = and C' = 0, implying Z carries no additional variance or covariance beyond 7,2
19

---

Published as a conference paper at ICLR 2026
B FLOWNIB: FLOW NEURAL INFORMATION BOTTLENECK
We consider, for each layer f, the Markov chain
X
Ze
Y,
where X denotes the input, Ze the layer-l representation (induced by encoder pe (z(e) x)), and Y the target variable.
Our goal is to learn a representation Ze that:
compresses the input information by minimizing [(X; Zz).
preserves predictive information by maximizing 1 Y)
The classical Information Bottleneck (IB) principle (Tishby et al. 2000) Tishby & Zaslavsky 2015 formalizes this trade-off as
min [(x; Zx) 3 (1 Y), p(ae(a)
where B > 0 controls the balance between compression and prediction.
MI requires high- dimensional density ratios over p(æ, ze) vs. p(z)p(ze) and p(2,, y) vs. p(ze)p(v). which are intractable to compute exactly when X, Ze are high-dimensional. The KL divergence
DRI(P(æ, ze) p(z)p(ze))
is especially problematic because neither joint marginals are known in practice and must be estimated (Belghazi et al. 2018). In deep networks, deterministic real-valued layers can also lead to unbounded [[x; ; Ze) in the continuous setting; in practice, careful estimator training. These issues make vanilla IB difficult to apply directly to large models.
variational lower bounds and
FlowNIB approach. To address these challenges, we introduce FlOwNIB, which gradually shifts emphasis from input preservation to target prediction during training or post-hoc estimation. We use dime-dependent trade-off : [0, 1] that monotonically decays from 1 to 0 as the estimator training step t increases (the model can be frozen). The FlowNIB loss at step t for layer & is
t) = -aa(t) ((::) + (1- (l)a(t) I(Z
so early steps (a 1) emphasize r(X; Zz), while later steps (z) emphasize I(Z;; Y). Each mutual information term is
((x; Zz) = Dri(p(z, =e)
(Z = DwI(p((Ee v) ((()))(()))
with DKL the Bellback-Leibler divergence. Since exact KLs are infeasible in high dimensions, use variational lower bounds (MINE-style) (Belghazi et al. 2018):
2();2t) 2 Ep( Tax, z0)] - log Ep(x)pte Tea (mnar))
I(Z;; Dep(s..niTTTmm e v)] log Epts))p(ill (
where Taa, and Tay. are learned scalar-valued critics (small neural networks) trained on joint pairs and product-of-margine minibatches; use the same critic architecture, batch size, negative sampling, optimizer, and steps across layers and models for comparability.
pairs (implemented by shuffling). Expectations are estimated with
Because X, Ze, Y can have different scales and dimensions, normalize MI estimates using the effective dimension (participation-rat ratio effective rank) (Roy
(EA,)² Σ, A
dea (Ze)
where {1i} are the eigenvalues of Cov Zτ) (estimated via PCA). The normalized MI estimates are log Ep(enize) (TT...(0.44))
i(x;Zt) = Ep(r.))) Tes. ees.e(a 24)] dort 2
20

---

Published as a conference paper at ICLR 2026
Ep(aes)) T T2y,e(2e, y) - log (p(e))(()) (e"eneelxe))))
i(Z
detr (Y))
Remark. The delf (-)2 factor is practical normalization for scale- matching across layers/models; it does not change the fact that the estimates are variational lower bounds.
Thus, the final loss optimized during FlowNIB training is
(e -(a(t) (():))()) (1-a I(Ze: ((ze))) x)).
which, expanded, becomes
Ce(0,t) = - -(a(t) Bp(a,se) e( zr)] - log Bp(e)p(ss) Epteeptssseeeessseeee (ess dott (2))²
E Epla (Ty w)1 - log Ep (p()) e (e + (1 - (1-a(())
(Y)
Here, denotes the parameters the encoder po(zz x) (if trained end-to-end) and of the critics Tz Tzy &. In post-hoc setting, the encoder is frozen and I refers the critic parameters, (+t) is the estimator step index. All MI values are neural lower bounds and are used for relative comparisons across layers (e.g., for OIC selection), not absolute MI.
Theorem B.1 (Consistency under optimal critics (per layer)). Fix a layer and let (X, Ze) ~ p(x, ze) and (Ze, Y) p(zz, y) with the Markov chain X + Ze - Y. Assume p(z, 24) & ((2)p(Z)) and p(zz,,) V) & pl p(y), and that the relevant expectations are finite. Suppose the Donsker- Varadhan optima (unique up to additive constant) attained:
(2 24) log pex, + Czz,ks
T Teye(=e, v) = log p(ze, v)
p(x)p(Z()
p(z))p())
Czy.e-
Let the dimension-noormalzzer estimators be
E (Tx, z1)1 - log (p(())((e)) Feo
i(X;Z))
dott (22)2
i(Z;:Y) Ep(z en v)1 - log (p(z))p(y) (lyy)e
det (Y)²
where (.)(.) E (0, oo) are fixed scale factors (e.g., participation-ratie effective ranks). Then
((X;2) Tex.c-Ti... 1(X; Zz)
i(Zr;Y) T 1(2,; Y)
dea )?
dea (Y)2
Proof. We show the claim for (X, Ze); the (Ze, Y) case is identical. By the DV representation, I(X; 2x) = sup (p(r,.>) [Tx ]+]] log LE pe ((sso)))
Under the stated assumptions the supremum is achieved at T Tt,,e(t 21) = log p(z,ze) constant c, and the objective is invariant to C:
p(x)p(=)) + c for any
E T + cl
log T+ = E[T] log Ble"
Substituting T gives
Ep(e,se) log p(a,2e)
log (p())p(e))
p(z,ze) p(z)p(zz))
= I(X; 2,) - log 1(X;2))
p(=)p(=2)
By definition, the normalized estimator satisfies
Ep(r. [Tazel - log E (e
i(x; Zt)
dust dee Z2
Hence, as Trz, + Tis.e in function space, the numerator converges to I(X; 2z), so i(X; T(X; dn)
+
1
21

---

Published as a conference paper at ICLR 2026
Remark. If Y is discrete (e.g., class labels), may set (Y) or compute it from fixed embedding of Y; the theorem holds for any finite, positive normalizer.
Lemma B.2 (Non-Monotonic Dependence of Mutual Information on Output Dimension). Let X E &Rdx, Z E Rdd, and Y E Ro denote input, latent, and oulput variables, respectively, with dx, dz fixed and dy variable.
Then under FlowNIB optimization, the mutual information [(X; Z) and I(Z;Y) are non-monotonic functions of dy, satisfying:
(I(X; Z)
OT(X;
>0 0 for dy k.
<0 for dy > k Ody
and similarly for ), some critical threshold k = dx.
Proof Sketch. FlowNIB optimizes tradeoff between I(X; 2) and I(ZYY,, constrained by the model's representational capacity dz and data complexity.
When dy is small (dy < dxx). the predictive target contains limited information; thus T(Z;Y) is small and the latent representation does not need high complexity.
As dy increases toward dx, the predictive task demands richer information; both T(X; Z) and [(Z; Y) increase to capture relevant features.
However, once dy > dx, the output space exceeds the input manifold's capacity; the latent represen- tation Zg cannot fully carry the increased predictive information due fixed dz, leading to saturation and eventual decline in both [XX; and [22; Y) redundant noisy output components exceed representational limits.
This yields non-monotonic dependency of mutual information dy, peaking around dy dx. then declining as dy further increases.
Proposition B.3 (Effective Dimensionality Adaptation under FlowNIB). Let X E Rdx and Y Rdy be imput and output random variables with dimensions x dy. Let Ze denote the latent representation layer & produced by a model trained under FlowNIB.
Then, under optimal critic approximation and continuous optimization, the effective dimension dest (Ze) exhibits the following dependence on dy (with dx fixed):
< 0 if < dx
Odat (Ze)
0 if dy dx
Ody
0 if dy > dx
i.e., the effective dimension dea (Ze) decreases with dy when dy is small, plateaus when dy as dx. and increases when dy exceeds dx-
Proof Sketch. Under FlowNIB, the latent representation Ze is optimized to balance information preservation I(X; Zt) and predictive sufficiency I(Ze; Y), modulated dynamically by att).
When dy & dx, the predictive information I(Z;; Y) is small; the model prioritizes compressing irrelevant input variance, resulting in reduced des (Zz).
When dy z dx, the predictive complexity of Y matches the input complexity; the model maintains dea ZZ) to balance preserving input and predictive information.
When dy >> dx, the model must expand Z; to capture sufficient predictive capacity, increasing dett (Ze) to span higher-dimensional output manifold.
Empirical observations support this trend, where deu (7e) traces a non-monotonic dependency on dy, reflecting an intrinsic adaptation of latent geometry output complexity.
22

---

Published as a conference paper at ICLR 2026
Algorithm 1 FlowNIB: Flow Neural Information Bottleneck
Require: Dataset D {( y.)) pretrained model fe. MI critics T2s and Tzy, scheduler att).
number of training steps 7
Initialize FlowNIB parameters and critics
2: for 1 to I do
3: Sample mini-batch {(z, y)} from D
4: Compute hidden representation Z fo(x)
5: Estimate [X;; 2) using MINE:
I(X; + Ep(a,=) Tes (z, z)] - log E (p(x)p(=) (e
6: Estimate [(Z; Y) using MINE:
I(Z;Y) t- pp(z,)) (2, y)] - log Ep()) (eT
7:
Normalize MI by effective dimensions:
I,(X: Z) dar(2)² (z; 1(2
I(X:2)
d
8:
Compute dynamic loss:
CFlONNIB f- (a(t) (x; (1- - aa))-aa(t)iizz:i))))) (z:1))
9: Update schedule: att + 1) t max (t) - 6)
Backpropagate and update 0, Tzz. Tzg
11:
end for
0.74
Medels
MIRDC 951-2
0.5-
PROEBILEA JORM
0.4-
STaEEIN S60M
0.2
(IX:
Layer
Figure 6: (Leff) plane trajectories under varying step sizes 5 for a(t) in FlowNIB. Each curve shows the progression of mutual information T(X; Z) and (Z: Y) across 2000 training epochs. (Right) Effective dimensionality doff (Z) across layers for different models MRPC and SST-2, Bidirectional models show higher dett (Z) than unidirectional models at every layer.
512
Bulorre sus Shernce wntray H131 Hoddalii 13MM Dureer obt
3.0
10
Figure 7: Effective dimension and Shannon entropy network layers. Left: Effective dimension dett 2) across layers for different output dimensions Ydim- Right: Shannon entropy MM() across layers for RoBERTa-Base and MobileLLM- 125M. Both plots use bold markers and shadows to emphasize trends in representation capacity and information compression.
23

---

Published as a conference paper at ICLR 2026
34
52
6.8
4.4
Oetpett Divendos
5.8
Tow 203
Layer Index
Layer Indes
Figure 8: Visualization mutual information across layers for different output dimensions. The left plot shows T(X; Z) and the right plot shows I(Z; Y) for various output dimensions Yaim- Each curve represents a specific output dimension, with bold markers and shadows to highlight the trends. This analysis provides insights into the evolution of representation capacity and target alignment across network layers as the output dimension increases.
C ABLATION STUDY
C.I EFFECT OF STEP SIZE ON FLOWNIB DYNAMICS
We conducted an ablation study on the MRPC dataset to analyze the influence of the step size controlling the decay of (tt) in FlowNIB Specifically, we varied 8 logarithmically from 10 to 10-11 and measured the evolution of mutual information (X;; Z) and I (Z; throughout training. Figureßletti shows the corresponding trajectories in the Information Plane. We observe that large step sizes (e.g., = 10- ') induce rapid compression, sharply reducing [(X; Z) early in training but failing to preserve sufficient predictive information I(Z; Y). likely due to premature information loss. Conversely, very small step sizes (e.g.. 6 = 10-6 cause negligible decay of a<(), leading to nearly static representations that retain high T(X; Z) but fail increase T(Z; Y). Intermediate step sizes (e.g.. 8 10 10 3 to 8 10=4) achieve the most desirable balance, gradually reducing [(X; Z) while increasing I (2; Y). effectively steering the model toward the information bottleneck frontier. These findings empirically validate our theoretical insight that 8 serves critical control knob governing the speed and quality of information compression in FlowNIB.
C.2 EFFECTIVE DIMENSIONALITY ACROSS MODELS
We measure effective dimensionality dutt (Z) across layers for DeBERTaV3 (base, large). Modern- BERT (base, large), MobileLLMM (350M, 600M), and SmolLM2 (360M) on MRPC and SST-2. To ensure fair comparison across models with different depths, we normalize layer indices to a common scale of to 12. Figure 6 right) shows that der (2) decreases monotonically with depth for all models, reflecting progressive compression (reasons of decreasing in Ablation Study C.4).
Importantly, bidirectional models consistently exhibit higher derr (Z) than unidirectional models at every layer. For example, on MRPC, DeBERTaVV-Large starts at 8.73 and compresses to 1.98, while MobileLLM-600) starts at 5.38 and compresses to 1.44. Similar trends appear on SST-2. These findings empirically support Lemma 2.3 confirming that bidirectional representations retain richer and more expressive features throughout depth.
C.3 EFFECTIVE DIMENSIONALITY vs. OUTPUT COMPLEXITY:
We study how the effective dimensionality dar (Z) of the latent representations changes with different output dimensions using the lime-series forecasting dataset ETTh1 (Zhou et al. (02I) by following Proposition B.3 We use fixed 6-layer network with each layer having 128 units and keep the input dimension fixed at dx = 380. We vary the output dimension dy from very small (dy = 2) to much larger than the input (dy 2500). As shown in Figure when the output dimension is much smaller than the input (dy & dx)., the effective dimension desr (Z) decreases across layers, showing that the representation becomes more compressed. As dy grows closer to larger than dx.
24

---

Published as a conference paper at ICLR 2026
Model
2
MRPC
RTB
HlllaaSogg
CoLA
SICK
STS B
LCP
8.134 # 0.046
2.513 = 0.084
2.843
+ 0.084
2.634 + 0.042
4.120 = 0.043
2.754
# 0.047
2.645
= 0.096
3.102
= 0.046
GPT-22Jange
3.322 0.0.33
2:523 = 0.063
3:123
2: 0.035
2.701 2 0 045
4:955
0.061
4.101
1 0.045
2.532
1 0:007
3.112 : 0.040
SmelLM2 135M
2.908
0.038
2.532
0.041
2.732
0.048
2.711
0.082
4.592
0.053
2.847
0.082
2.743
: 0.002
3.104
2 0.041
SmotL LM2 360M
3.183
0.038
2.583
0.035
3.212
0.046
2.742
030
4.943
0.054
4.242
0.045
0.007
3.105
1 0.034
MonaLLLM-1250
8.882
2.578
0.048
0.087
0.043
0.088
2.654
8.308
0.041
Mowiee.MM-350M
3.773
2.612
0.048
2.732
0.058
0.055
MosieLLM 600M
3.937
0.055
2,623 0.001
3.323
0.045
5,748
0.049
4.353
0.058
2.533
0.040
3.112 0.044
DeBERTa-v3 base
4.212 0.040
2,622
0.047
3.522
0.055
2: 783
833
5.194
0.051
4.372
0 067
3.143
00B
3.103
0.052
DuBERTa-x3-lage
4.872 # 0.048
2.6664
0.045
3.824
0.044
2.824
0.037
6.358
0.069
5.382
+ 0.061
2.563
+ 000
3.315
+ 0.050
robentz-hase
3.062 2: 0-036
2.522
0.037
3.212
2 0.045
2. 763
0.044
5.263
0.057
4.163
4 0 0.062
893
: 041
3.104 - 0.035
robetta large
4.012 1: 0.042
2.638
0.034
3.924
0.054
2 532
0 0.0
6.330
0.060
5.123
0 0.051
2. 893
0 0.034
3.112 1: 0.04T
modke--aart-bose
3.372
0.044
2.574
0.050
3.423
0.047
2. 792
0.039
6.123
0.057
4.772
0.000
583
0.031
3.107 + 0.047
ModemBBETT-lage
0.054
0.036
0.064
0.053
Table 3: Estimated (X; Ze) (mean + std over 10 seeds).
Model
MRPC
HellaSong
ColAA
SICK
STS-B
1.CP
GPT mediom
0.3144 :: 0.022
0.513 = 0.029
0.543
#: 0.085
0.534
# 0.032
0.754 = 0.030)
0.172 # 0.019
0:243 # U 0.028
0.1012 #: 0.025
GPT 2.aage
0.322 1: 0.026
0.523
0.029
1.123
2: 0.032
0.701
0. 0.030
0.901
0..03
0.235 1: 0.028
0.320
:: 0.021
0.112
2: 0..02
0.380 0.022
0.532
0.025
0.732
+ 0.085
0.711
0.026
0.747
0.002
0.152 + 0.023
0.243 + 0.023
0.103 + 0.01T
SmolLM2-3360M
0,483 1: 0.027
0.583
0.031
1,212
1- 0.045
0.742
0.003
0.902
0.051
0.243
0.025
0:431 1: 0 0.025
0.105 1: 0.027
Mob8eLLM-1220
0.812
0.024
0.573
0.030
0.643
0.032
0.025
0.025
0.308
MoDdeLLLM 350M
0.373
0.030
0.612
0.002
1.224
0.038
0.732
0.028
0.882
0.035
0.237
0.027
0.392
0.110
0.737
0.035
0.623
0.030
1.323
0.036
0.821
0.030
0.953
0.032
0.273
0.023
0:433
0.312 0.026
DeBERTa-ch-bost
1:212:1: 0.636
0.622
0.024
0.582
0.037
0.763
037
1,072
0.040
0.434
0.022
0:513
024
0.103 1: 0.025
D:BERTa-33-arg
1.472 4: 0.042
0.654
0.028
1.154
0.041
0.824
0.034
1.482
0.046
0.553
+ 0.028
0.463
= 0 026
0.115 : 0.018
reberttshaee
1.062 0.033
0.522
0.023
0.912 #: 0.030
0.753
0. 0.031
0.983
0.030
0.343
+ 0.024
0.205
+ 0 022
0.104 = 0.021
sobests-laage
1.412
0.044
0.608
0.034
1.124
0.087
0.832
0.033
1.208
0.035
0.439
0.026
383
+ 0 020
0.112 + 0.019
modem Sert- hose
1.172
0:041
0.574
0.032
0.942
0.085
0. 702
028
0.982
0.036
0.342
0. 203
0 020
0. : 0.021
ModemBEXT large
1,308 0.038
0.680 0.034
0.033
0,864
0,030
0.034
0.026
1: 0.017
Table 4: Estimated (Z, Y) (mean + std over 10 seeds).
we observe non-monotonic trend: the dimension first compresses, then expands. When dy >> dx. the effective dimension increases across layers, suggesting that the model adjusts the complexity of its representations to match the complexity of the prediction task. This behavior occurs even without directly optimizing for it in FlowNIB, showing that the shape of the output affects how the model organizes its internal representations.
C.4 MUTUAL. INFORMATION DYNAMICS ACROSS OUTPUT DIMENSIONS AND LAYERS:
We explore how changing the output dimension Yaim affects mutual information and model per- formance by following Lemma B.2 We trained the same model with different output sizes: Yaim E {2. 24, 96, 128, 380, 512, 1038, 1450, 2500), and measured the mutual information between inputs and hidden layers / [(X; Z), and between hidden layers and outputs (2; Y), after training. As shown in Figure 8 [X: Z) generally decreases across layers, especially for larger Yaim- meaning more information is lost the network gets deeper. At the same time, I(; increases with depth, but for large Yaim. it saturates early- it's harder for the model to align with very high-dimensional outputs. Interestingly, models with intermediate output dimensions (like Ydim 96 or 128) show a better balance: they retain useful input information and achieve strong alignment with the output. This balance leads to better performance. Overall, we find that output dimensionality plays a key role in controlling how well the model balances input compression and predictive accuracy, making it an important hyperparameter to tune.
C.5 VALIDATING GENERALIZED EFFECTIVE DIMENSIONALITY
To validate our definition of generalized effective dimensionality, we compare the layerwise trends of (2) (based on the & 2n-mrrm participation ratio) and the Shannon entropy M(p) across two models: RoBERTa--BE Base and Mobilel_LMM- 125M. As shown in Figure 7 (Right), both metrics follow similar trends across layers confirming that higher entropy leads to higher effective dimension, consistent with our definition and effective dimension than MobileLLM-125M at every layer, reflecting its richer representational capacity. The first few layers show a sharp drop in entropy, followed by a stable regime, aligning with the known compression phase in transformer representations. This empirical behavior confirms that both the entropy and deff satisfy the expected monotonicity and boundedness properties outlined in Definition 2.2 including non-negativity and the Schur-concavity property.
:= (xp)) Notably, RoBERTa-Bas maintains higher entropy
25

---

Published as a conference paper at ICLR 2026
ETTh1
ETTh2
Horizon
Uni
Bi
Uni
Bi
24
0.65
0.60
1.40
1.32
96
0.80
0.75
1.74
1.65
128
0.95
0.90
2.01
380
1.18
1.10
2.51
2.40
512
1.38
1.30
2.95
2.80
1038
1.72
1.64
3.30
3.18
Table Forecasting MSE on ETTh1 and ETTh2 for different prediction horizons. bidirectional attention; Uni = unidirectional (causal) attention.
ETThl
ETTh2
Horizon
(Ze; Jui
1(Ze; Y )) 1(Ze; Vul
I(Ze; Y D)a
24
2.07
2.24
1.86
2.03
96
1.99
2.18
1.79
1.97
128
1.94
2.11
1.73
1.89
380
1.87
2.02
1.64
1.80
512
1.76
1.89
1.54
1.70
1038
1.61
1.73
1.42
1.59
Table Mutual information I(Z;; Y) on ETTh1 and ETTh2 for different prediction horizons. Bi bidirectional attention; Uni unidirectional (causal) attention.
C.6 STABILITY ACROSS RANDOM SEEDS
In this section, we examine the stability of FlowNIB with respect to random initialization and minibatch sampling. For each model and dataset, we run FlowNIB with 10 different random seeds and report the mean and standard deviation of the estimated mutual information. The results for T(X; Ze) and T(Z;; Y) are summarized in Tables and respectively.
Overall, the variance across seeds modest. For both I(X; Zz) and I(Z;; Y), the standard deviations small compared to the differences between models and to the gap between unidirectional and bidirectional architectures. particular, bidirectional models (e.g., DeBERTa-v3, RoBERTa, Mod- emBERT) consistently exhibit higher mean mutual information than unidirectional models across all datasets, and this ordering stable under different seeds. We do not observe cases where a model with lower mean MI surpasses higher-MI model once the standard deviations taken into account.
The same pattern holds within each model family and across datasets: models and layers that are identified more informative by FlowNIB retain that ranking when averaged over 10 runs, and the error bars do not change the qualitative conclusions. This supports our use of FlowNIB as a relative diagnostic tool: while do claim recover the exact true mutual information, the estimates stable enough across random seeds to reliably compare layers and architectures and to link higher MI (especially at the OIC) with improved downstream performance.
C.7 BIDIRECTIONAL VS. UNIDIRECTIONAL ATTENTION IN TIME-SERIES FORECASTING
To check whether our mutual-information findings also hold beyond NLU,, we conduct small case study on multivariate time-series forecasting using the ETThi and ETTh2 benchmarks. In all experiments,
the same Transformer architecture for both settings: 2-layer Transformer with hidden dimension 512 and input sequence length 256. The only difference is the attention pattern: Uni uses standard causal (unidirectional) attention, while Bi uses bidirectional attention the input window.
Table reppors the forecasting MSE for different prediction horizons ETThl and ETTh2. Across all horizons and on both datasets, the bidirectional model achieves consistently lower MSE than the unidirectional model. The gap is small for short horizons (e.g., 24, 96), and becomes more
26

---

Published as a conference paper at ICLR 2026
pronounced as the prediction horizon increases, showing that bidirectional attention provides more robust long-range forecasting.
6 the corresponding mutual information I YY) measured with FlowNIB for the same models and horizons. For both ETThl and ETTh2, the bidirectional model has higher 1 [Z Y) than the unidirectional model at every horizon, indicating that its representations Ze carry more target-relevant information. In addition, T(Z;Y) gradually decreases as the prediction horizon grows, matching the increase in MSE and reflecting the increased difficulty of the task.
Together, these results provide a simple but concrete example in a non-NLU setting where (i) bidirectional attention improves performance under matched architecture, and (ii) higher mutual information between Ze and the target Y aligns with better forecasting accuracy. This supports our main claim that bidirectional representations tend to encode richer task-relevant information, and that FlowNIB's s MI estimates track meaningful performance differences even outside standard language understanding benchmarks.
D LORA BASED PERFORMANCE COMPARISON
Table LoRA.
shows the performance comparison between bidirectional and unidirectional models using
Model
Method
SST-2
MRPC
QNLI
RTE
CoLA
MNLI
BoolQ
HellaSwag
SIQA
Avg.
DeBERIaex3-Buee
Pooling
95.12
88.75
91.75
82.35
85.43
85.96
63.55
55.22
46.74
77.15
Masking
96.22
90.03
93.10
85.92
88.55
88 10
65.05
68.33
61.92
81.81
DeBERIaav3-Large
Pooling
96.25
92.88
58.15
$2.04
81.82
Masking
96.94
94.95
95.35
90.85
93.05
91.96
65.12
74.10
66.41
85.30
RoBERTa-Base
Pooling
83.40
82.20
51.78
44.63
Masking
94.80
86.10
93.42
86.02
88.25
87.20
63.80
65.33
61.12
80.45
RoBERTa-Large
Pooling
95.12
56.23
79.66
Masking
96.67
91.98
95.10
88.45
95.33
90.92
64.25
70.35
62.45
83.83
ModemBEKTT-BB
Pooling
93.70
82.40
90.25
81.52
84.22
86.02
62.00
54.18
45.70
75.78
Masking
94.92
84,05
92.88
85.00
85.80
88.55
61.35
62,00
60.00
78.95
ModemBERRT
Poaling
95.00
88.55
93.50
87.32
90.25
92.80
63.50
59. 00)
48.50
79.82
Masking
96.32
91.10
95.12
88.50
91.02
92.10
72.42
64.33
83.42
GPT-2 Medium
Pooling
92.70
84.32
90.42
68.50
79.15
78
62.33
36. 80
37.42
69.96
Generation
85.72
36.55
GPT-2 Large
Pooling
93.75
85.50
83.35
65.90
82.85
79. 79.55
63.50
39. 20
40.50
70.68
Generation
94.05
87.05
12
81.72
64.05
99.70
45.00
71.98
Smo1LM2-360M
Pooling 93.80
84,20
90.92
69.90
81.22
84,10
62.75
41.20
41.55
72.18
Generation
94.52
85.85
91.93
70.50
83.80
85.10
62.60
42.40
49.45
73.68
Smoll SmolL.MM2-
Pooling 91.90
83.05
89.43
67.55
80.16
81.51
61.35
37.00
40.25
70.13
Generation
92.80
83.85
90.05
68. 12
81.82
82.78
61.70
40.00
46.20
71.59
MobileLLM-125M
Pooling
92.25
81.42
68.42
32.30
40.40
Generation
92.98
82.35
90.22
68.92
80.42
82.20
60.25
36.12
47.33
70.53
MobileLL.MMMO
Pooling
82.65
69.55
Generation
94.10
82,98
90.85
70.25
82.62
83
62.85
39.20
50.05
72.15
MobileI.MMM 600M
Pooling
71.32
44,20
Generation
94.95
87.55
91.50
72.02
85.92
84,30
63.75
47. 80
57.32
75.68
Table Accuracy results across nine NLP classification tasks comparing bidirectional and unidirec- tional models under pooling, masking, and generation inference strategies using LoRA fine-tuning
E DETAILS RESULTS
Our results show that bidirectional models consistently outperform unidirectional models across both classification and regression tasks (Table Table For example, in classification, DeBERTa-vv3- Large achieves the highest average accuracy 84. 84.73% using masked token prediction, improving by +3.77% over its pooling-basee based variant. Furthermore, observe that even RoBERTa-buug outperforms MobileLLM-6001 in several tasks, highlighting a consistent trend with mutual information (MI): better MI is correlated with improved context modeling and task performance.
Overall, these findings highlight that masking inference yields stronger gains in bidirectional models, while generation provides modest improvements for unidirectional models but fails to close the accuracy and error gap, reinforcing the advantage of bidirectional context and masking for both classification and regression.
27

---

Published as a conference paper at ICLR 2026
1:33
8:18
3.21
0.13
6:12
5:17
8:13
3.31
1,00
1:18
9.23
6:43
6:18
0.38
1:22
3.59
0.95
6.19
6:18
9:1
0.15
611
121
6.35
9:38
6:10
1:04
9:39
0.22
8:28
9.28
0:52
9:36
0:30
6:37
L.00
0.77
6:34
2:0
0.38
633
9.6
F:2
6:18
1.23
R.24
622
2228
0.57
0.05
6:33
9:27
1,00
6:29
0.43
6.3
3:55
0:55
0.21
1.00
2.2
6.2
8.24
0.95
6:18
0.14
0.37
1.06
112
6
612
0.35
0.:0
6:12
0.00
0.11
6.12
1-2
0
221
0L21
0.00
0.23
0:28
2.24
1.00
0:30
60.00
Figure 9: Token-level mutual information matrix on the SST-2 dataset for sentiment classification, computed from the final hidden layer representations. (Left) RoBERTa-base; (Right) SmoILM2- 360M.
Model
Method
SST-2
MRPC
HellaSwag
SIQA
Avg.
Pooling
95.52
89.21
92.43
83.48
86.23
86.43
64,23
56.00
47.54
77.90
Masking
95.75
91.17
92.48
84.98
87.44
87.22
64.23
69.49
60.90
81.52
LeBaRTa-v3-Large
Pooling
95.67
93.45
93.58
88.38
93.34
90.76
64.73
57.34
51.43
80,96
Masking
96.11
94.04
94.14
89.93
92.95
91.43
64.98
73.43
65.53
84.73
RoBERTi-Base
Pooling
94.24
84,53
91.96
83.45
86.34
86.34
63.82
52.43
76.53
Masking
95.14
83.13
92.27
84.58
87.44
86.38
63.96
64.53
60.16
79.95
RoBERTa-Large
Pooling
89.54
86.32
48.69
Masking
96.21
91.25
94.38
87.84
95.83
91.13
63.82
71.43
63.67
83,95
ModemBERTTBBam
Pooling
94.35
83.33
46.81
Masking
95.38
85,43
92.43
12
84.43
88.21
62.17
63.54
61. 86
79.73
ModemBEEKT-Lare
Pooling
95.37
89.43
94..22
86.74
89.95
93.23
64.22
60.32
49.67
80.35
Masking
95.89
89.93
94..57
87.78
90.79
92.98
64.72
73.18
64.68
83.84
GPT-2 Medium
Pooling
93,80
85.78
91.17
69.67
80.24
78.81
63.43
37.83
38.45
71.02
Generation
94.14
85,93
91.93
69.83
81.43
80.18
63.54
37.93
43.45
72.04
GPT-2 Large
Pooling
93.97
86.27
84.01
66.
83.89
80.06
63.13
40.32
41.91
71.26
Generation
94.24
87.23
84.56
67.34
83.87
82.34
64.16
39.53
45.34
72.07
Smo1LM2-133MM
Pooling
92.58
84.59
90.56
68. 12
81.48
82.83
62.43
38.34
41.41
71.37
Generation
Smoll S M2.360M
Pooling
94.26
84,80
91.61
70.70
82.07
85.12
63.13
42.45
42.43
72.95
Generation
94.65
85,32
92.32
71.11
84.53
84.89
62.92
43.69
50.20
74.40
MobileLLM-125M
Pooling 93.05
82.43
90.58
69.32
80.29
82.98
60.73
33.45
41.45
70.48
Generation
93.15
83.35
90.54
69.53
80.53
83.24
61.26
37.42
48.23
71.92
Mobilel.I.MM33
Pooling 93.85
83.68
90.85
70.33
82.38
83.45
63.42
36.28
42.74
71.89
Generation
94.68
83.57
91.09
71.43
82.87
84.58
63.71
40.13
51.50 11.54
73.73
Mo6GIeLLM-600M
Pooling
94.86
87.34
91.34
72.45
64.18
45.32
74.50
Generation
95.14
87.87
91.37
72.29
86.30
84.79
64.12
48.53
58.54
76.55
Table 8: Accuracy(%) results across nine NLP classification tasks comparing bidirectional and unidirectional models under pooling, masking, and generation inference strategies.
28

---

Published as a conference paper at ICLR 2026
Model
Method
WASSA
SICK
STSB
LCP
CWI
Humicroedit
Avg.
DeBtRlaax3
Pooling
0.017A0.
0.16.00.97
4.5
0.42900.18
0,27880.412
0.20910.114
Masking
0.01389.911
0.135/0.277
0.575/0.46
0.006/0.060
0.385K3.478
0,274/0-423
0.197/0.298
DcBERTa-V3Laare
Pooling
0.016/0.18 102
0.140/0.22
0.353/0
0.007/0.073
0.345/9.457
0.26340.419
0.187/0.295
Masking
0.012/0.075
0. 2.133/0.274
0.348/0.44 414
0.005/0.051
0. 0,340/0.459
0. 0.268/0.42
0,184/0.222
RoBERTa-BBse
Proling
0.01610.097
0.168/0 300
1.364/044 452
0.00770.066
0.465/0.535
0.293/04118
0.218/0.314
Masking
0,015/0.094
11145/0.244
0.353/0 448
0.007/0.066 065
0,431/0.5177
0,289/0.4 431
0.206/0.300 308
RoBERTa-I
Proling
0.015/0.097
0.153/0 291
0.35 1/5110445 439
0.0060 060
0. 0.376/0.469
0.28330
0.197/0.299 298
Masking
0.01649.099
0.152/0.291
0.35000.45
0.006/0.059
0.166/0.475
0,281/0.431
0.195/0.297
ModemBERT-BBa
Pooling
0.01640.092
0.207/0.350
0.469/0.517
0.00%/0
0,376/0.468
0.302X0.448
0,229/0.524
Masking
0.173/0.33 0,01510.093
0.482/0.55. 536
0.006/ 067
0.36-40,471
0,281/0.430
0.220/0.320 320
ModemBRRRT-.am
Pooling
0.01690.093
0. 1616600.30 307
0 468
0.006/0 060
0. 00.41
0. 1.302/0-000 449
O. 0,200003000 305
Masking
0.015X0.094
0.150/0 292
0.57 1/0 462
0.006/0.053
0.344K).457
0.293/0.441
0.197/0.300
GPT-2 Medium
Pooling
0.019M0.11 112
0.662/0.619
0.427/0.49 499
0.908/0.088 0844
0,369K0.476
0.3948U.335
0.313/03387
Generation
0.018/0.111 111
0.673/0.633
0.412/0 490
0.008/0.083
0. 0.345/0.457
0.34470.449
0. 0,300N0.375
GPT-2 Large
Pooling
0.018/0. 105
0.623/0
0.442/0.522
0.007 080
0.32440.444 443
0.3 18/0 463
0,288/0
Generation
0.017/0. 107
0.58330 523
0.423/0.49 .99
0.000X0.078
1.326/0.446
0.323/0.473
12279/0.54
SmolLM2-133M
Pooling
0.017/0. 105
0.192/0.35
0,424/0.483
0.007A0.00 076
0. 0,36910,476
0,304/0.550
0.218/0.322 322
Generation
0.017/0. 106
0. 1..////.39
0.403/0.48 484
0.007X0.076
O. 0.366/0.475
O, 0.295/0.444 442
0,210/0.317
SmolL.M2-3600
Pooling
0,01780. 104
0.173/0.310
0.407/00.488 488
0.005/0.061
0.340N1.459
0.338/1.463
0.213/0.314
Generation
0.017A0.
0.170/0.299 298
0.394/0.48
0.006/0.066
0,332/0.454
0.323X0.468
0,207/0.310
MobIlLLLM-125M
Pooling
0.020/0. 111
0.197/0.355 354
0.419/0.499
0.006/0.070
0.323/0.446
0. 0.302/0.451
0.211/9.320
Generation
0.019/0. 113
0.192/0.30
0.410/0
0.006/0 068
0.312/0.4488
0.293/0. 442
0,205/0.314
Mobilel.LM.BB
Pooling
0.018/0. 104
0.191/0.336
0.394/0.488
0.006/0.063
0.310K0.4366
0,282/0.477
0,200N0.308
Generation
1015
0.187/0.320
0.391/0.477
0.006/0.063
0.209M0.437
0,278/0.411
0.198/0.304
MobileLLM-60MM
Pooling
0,017/0. 105
0.181/0.33 320
0.384/0.445 474
0.006/0 066
0.301/0.433
0,274/0.421
0.193/0.302
Generation
0.017/0. 105
0.
0.38 1/0.422
0.006/0.003
0,308/0.419
0,278/0.338
0.193/0 302
Table 9: Regression results (MAE/MSE) across six NLP regression tasks comparing bidirectional and unidirectional models under pooling, masking, and generation inference strategies.
F DATASET
The details of datasets are described in Table 10
Dataoot SST-2
Task Type
Domalla
Deserpiion
Wang al. 2018
Classiniation
Sentiment Analyss
The Stanford Sentimen Ticebank, bitary sestiment classifica. tion disaset laheling sentences positive negative.
MRPC
Clasttication
Paraphrase Detection
The Microsoft Research Paraparase
detecting whether
A quesion semanically squivaleut.
QNLI
Wing
Clasificution
Question Answering NNI
languge
dutaset built SQuAD,
determining consest sentence contaits answer.
RTE
Wing
Classification
Natural Language Inference
The Recapeszing Textul Fatailment dutrect determnning if byputhesis eatailed by a premise.
MNLT
Wang orA 2015)
Classincation
Nataral Lasguage Interence
MultiGeea Gense Natural Language Inference ditasct covering estail-
ment neutral, and contradiction relations acmoss muluppe genres.
CoLAA
Wing af oF
2018
Classification
Grammetical Accoptability
Corpis of Lingaistic Acceptability. evalusting whether seniencss conform to English grammatical rules.
Boo0
Classincation
Reoding Comprehensian
Boolean Questions
yestno questions based
Wikipodia paseages requiring reading comprehension.
HellsS ogg 22021
2019
Clasification
Commonsense Reasoning
Tests commonuense reasoning by selecting the most plausible contimation of a given scenario.
SIQA
Sap
2019
Classiicattion
Social Intelligence
Social IQu dittee evalusting models' undersanding of social sauations, emotions. and insenions.
WASS4 Mohninnt & Brave-Marque 2017
Regression
Faration Intensity
WASSA-201 dutset for prodicting emotion intensity scores for tweets across multiple emocions.
SICK
Mirenr ar 201RR)
Regnossion
Semontic Similarity
Sentences Involving Compositinal Knewlolge ditaset for mea- similarity and entailment.
STSB-
Regressian
Sammatic Similarity
Texnual Similariy Benchmack scored contimoos S,
scale
Shundlow et 20207
Regression
Lexical Complexity
Lexical Complexity Prediction dataset predicting the com- plesity of wnick within context.
CWI
116 c. JL. 2017
Regression
Comples Word ldentification
Complex Word Ideniiiiction dataset from SemEval. labeling woods as simple or complex in conext.
Homkrorant Missum etal. 2019
Regnession
Humor Perception
SemEvval humor datseei evalusting the impact of small test edits. (micro edits) un humor penception.
Table 10: Overview of the 15 benchmark datasets used in our experiments across classification and regression tasks.
G ENVIRONMENT SETUP
All experiments are conducted using PyTorch 2.0 and Hugging Face Transformers version 4.50. Training and evaluation are performed on single NVIDIA A100 GPU with 80GB of memory. We use Python 3.10 within an Anaconda virtual environment configured with CUDA 12.1. Key dependencies include NumPy, SciPy, scikit-leam, and tqdm for data processing and evaluation. Random seeds are fixed across all runs to ensure reproducibilityy
29

---

Published as a conference paper at ICLR 2026
H EVALUATION METRICS
We evaluate our models using task-specific metrics selected for their interpretability. relevance, and comparability to prior work. For classification tasks, adopt accuracy the primary metric, defined as the ratio of correct predictions to the total number of predictions:
Accuracy Number of correct predictions
Total number of predictions
Accuracy provides straightforward measure of model correctness and aligns with standard practices in classification benchmarks (Wang et al.) 2018).
For regression tasks, we report both mean squared error (MSE) and mean absolute error (MAE) to capture complementary aspects of prediction error. MSE emphasizes larger errors due to the squared term, while MAE reflects the average magnitude of errors:
N 1/
1
1 N
MSE
Σ U ;;);; N
MAE
Uel.
where N is the number of samples, y/1 is the ground-truth label, and ii the predicted value. These metrics ensure robust evaluation of both typical and extreme prediction errors (Cer et al. 2017 Marelli et al. 2014b).
In addition to task performance metrics, we measure the mutual information between the input X and the learned representation Ze. denoted [(;; Z). Mutual information quantifies how much information about the input is preserved in Ze. providing insight into the information bottleneck trade-off TTihhyy& & Zaslavsky 2015). We estimate / [X: Z) using a variational lower bound based on Mutual Information Neural Estimation (Belghazi al. 2018), following prior work in information- theoretic analyses of neural networks.
All metrics are computed using scikit-learn and official benchmark evaluation scripts. Model selection is performed based on validation set performance, with final metrics reported on the held-out test sets.
I MODEL. DESCRIPTION
We compare our method with a range pretrained language models covering both bidirectional and unidirectional architectures. The bidirectional baselines include DeBERTa V3-Base (He et al. 2021
V- (He et al. (. RoBERTa-Baae Base (Liu al. 2019)), RoBERTa-Large (Liu et al.
2019), ModernBERRI-Base (Warner et al.) 2024), and GonderBERRTS
(Warner al. 2024).
The unidirectional baselines include GPT-2 Medium (Radfford et al. 2019). GPT-2 Large (Rad
ford et al., 2019), MobileLLM--1255
Liu al. 2024), MobILeLLM-350M (Liu et al. 2024). MobileLLM-600M (Liu et al. 2024). MM-1LM2-133M (Allal et al.) 2025). and SmoILM2-
360M (Allal et al. 2025). These models are selected to cover range of sizes and architectures. enabling a fair and broad evaluation of representational learning. We focus on smaller model sizes to allow fair comparisons since large bidirectional models are not readily available. All baseline models are fine-tuned using RoCoFT adapters with an adapter rank of 3, enabling efficient fine-tuning without modifying the main model parameters. We use a cosine learning rate schedule for training.
J HYPERPARAMETERS
We select hyperparameters systematically ensure consistent and balanced evaluation across all tasks and models. For classification tasks, we set the learning rate to 1 x 10 with batch sizes between 8 and 16. For regression tasks, we increase the learning rate to x 10-3 3 with batch sizes ranging from 8 to 32. All models are fine-tuned using the Adam optimizer with cosine learning rate schedule, weight decay values in the range of 0. to 0.2, and a warmup ratio of O. Gradient accumulation steps are varied between I and 8 depending on GPU memory capacity. To improve training stability, gradients are clipped at a maximum norm of 1.0, and label smoothing with factor of 0.1 is applied where applicable. Each model is trained for 2 to 30 epochs, with warmup steps selected between 100 and 500. These hyperparameter settings are held consistent across experimental runs to ensure fair comparisons and reproducibility. This finding aligns with earlier work showing
30

---

Published as a conference paper at ICLR 2026
the benefits of bidirectional models for non-autoregrrssiiv NLP tasks. A detailed breakdown of the hyperparameters used for each dataset and model is provided in Appendix. including Table [1
(Humicroedit), Table 12 (WASSA), Table L3 (SICK), Table (STS-B), Table 13 (LCP), Table I6 (SST-2), Table 17 (MRPC), Table T8 (QNLD, Table T9 (RTE), Table 20 (CoLA), Table 21 (MNI.),
Table 22 (BoolQ) Q) Table 23 (HellaSwag), and Table 24 (SIQA).
Model
Learning Rate
Batch Size
Grad Accum
Weight Decay
LR Scheduler
Rank
Max Length
Epochs/ / Warmup Steps
MobieLLMM-SSOM.
bc-1
0.2
Cosine
512
107100 100
Smoil M2-
100
Sawill,
6o-4
8:2
Cosine
100
ModemBERT buse
6c-4
0.2
Cosine
512
10/ 100
med2-mm
6e-4
16
0.2
Cosine
512
[0/ 100
GPT2-lage
6o-4
16
0.2
Cosine
512
10/100
DeBERIa 1a-l--33-bua 3-base
0.2
Cosine
10/
roberia-longe
60-4
0.2
Cosine
100
DeBERIa v3-large
6c-4
0.2
Cosine
512
10/ 100
Mo 125M
6e-4
16
0.2
Cosine
512
107 100
Mobilel. LM 600
6--4
0.2
Cosine
512
10/ 100
ModemBEKT large
0.2
Cosine
10/ 100
Table 11: Hyperparameter settings for the Humicroedit dataset for each evaluated model.
Learning
Ratch
Accum
Weight Decay
Length
Kgochs/ /Warmpp Steps
Smhol ME--TMEE-135M
52-4
D.2
Coainz
312
1071100
MobileL.LM.35 3MMM0M
5e-4
14
0.2
Cosine
512
10/100
Smoil M2-360M
50-4
14
0.2
Cosine
512
10/100
GPT2-medium
Cosine
10/100
12-lare
MudenBERT-Ause
8.2
Cosine
100
60-4
0.2
Cosine
312
10/ 100
roberta-hise hase
62-4
16
0.2
Cosine
512
10/100
roberis-lung lurge
6o-4
16
0.2
Cosine
512
10/100
SOBERTo-33-lrgg
Cosine
MoM15eLLM-125M
MobileL.LM--e
6o-4
8.2
Cosine
512
100
ModenRERT-lagg
6o-4
16
0.2
Cosine
512
100
Table 12: Hyperparameter settings for the WASSA dataset for each evaluated model.
Model
Learning Kale
Batch
Grad
Weight Decay
Schedulerr
Kank
Max Length
Epochs/
Warmuup Sleps
SmxilL-M2-135M
8.2
Cosine
100
ModenBERT buse
16:3
2
0.2
Cosine
512
100
DcBERTa->3 M----a------s- buse
10:3
2
0.2
Cosine
512
20/ 100
mPP2-mm
10:3
14
8.2
Cosine
512
20/ 100
GP
0.2
Cosine
20/ 100
roberia- lange
1e:3
0.2
Cosine
100
DeBERIa-v3-lmm
16.3
0.2
Cosine
512
100
Mo 125M
1c:3
0.2
Cosine
512
20/ 100
MobileLLMM M- 600
16:3
0.2
Cosine
512
20/ 100
MedemBEEK large
0.2
Cosine
20/ 100
Table 13: Hyperparameter settings for the SICK dataset for each evaluated model.
K
MODEL PROFILE INFORMATION
We conduct comprehensive CPU profiling analysis of twelve transformer models to understand the computational bottlenecks and runtime behavior that influence performance. The models we
evaluate include DeBERTa-v3-Ball Table 25
DeBERTaa-v-LLaree Table 26 RoBERTa-Base Ta-
ble 27
RoBERTa-Large Table 28
ModemnBERT-BE
Table 29
Modern
Table 30
GPT-2 Medium Table 31 GPT-2 Large Table 32 SmolL 135M Table 33 SmolLM2- M20MMM Ta
Table
ble 34,
MobileLLMMI25M Table 35 and MobileLLM-600M Table 36 Our CPU profiling shows
that bidirectional models are often comparable to unidirectional models. For example, DeBERTa-v3-
Base and ModernBERT-BBan Table 29 complete inference in 502ms and 347ms, respectively,
while GPT-2 Medium Table 31 takes 1126ms more than double the time. Larger bidirectional Table and RoBERTa-Large Table have runtimes comparable to GPT-2 Large Table 32 in total execution time and compute distribution. Bidirectional models spread CPU usage evenly across attention, normalization, and embedding layers, whereas unidirec- tional models spend over 85% their time on addmm, suggesting less efficient resource utilization. Additionally, compact bidirectional models like M2-135M Table and
models like DeBERTa-v3-La.
31

---

Published as a conference paper at ICLR 2026
Midld
Weight
Schelller
Mas
Mebiiel MM-NIM
26-4
01
Courne
10/ 100
Small M121 NSM
7o-4
01
Conin
513
10/ 100
DeBERD 13 base
6c
16
0.2
Cosise
512
20/ 100
robenta base
6o
16
1.2
Cosine
512
20/ (00)
tobottbalage
Bood
16
0.2
Cosine
512
20/ 100
6c-4
16
0.2
Cusise
512
20/ 100
MobrieL IM-125M
bend
112
Cosine
512
20/ (00)
M--03
Bend
Ooeino
MedemBEXT
20/
medium
Costne
Meden
Table 14: Hyperparameter settings for the STSB dataset for each evaluated model.
Model
Learning Rale
Batch Size
Grad Accum
Weight Decay
LR Schedlller
Rank
Max Length
Epochs/ / /Warmup Steps
Smool M203
0.2
Conine
100
Mubilel MM-355M
Cosine
10/100
Savill
ModemBEKT-bars
5o-4
0.2
Cosine
512
10/ 100
GPP2-medium
So-4
8.2
Cosine
312
107 100
GPT2-lagge
5o-4
0:2
Cosine
512
10/100
roberta-hise
0.2
Cosine
10/100
10/100
0.2
Cosine
100
MobleLLM-I12MM
16-3
0.2
Cosine
512
10/ 100
M 600
1o-3
0.2
Cosine
512
107 100
ModenRERT-lang
16-3
0.2
Cosine
512
100
DeBERTo-v3-hsse
Cosine
10/100
Table 15: Hyperparameter settings for the LCP dataset for each evaluated model.
Model
Learninn Rate
Ratch Size
Grad Accum
Weight Deeay
IRR Schediller
Rank
Max Length
Epochs / /Warmpp Steps
D.1
37 7,500
3/
Sawill
8:1
Cosine
3/ 50000
ModenBERT--use
124
0.1
Cosine
512
37.500
16-4
16
4
0.00
Cosine
512
3/ 100
robertt--hie hise
16-4
16
4
0:00
Cosine
512
3/ 100
4
Cosine
3/ 100
DeBERTa-d3-lagg
3/
Mobilel.MM LM-
8.00
Cosine
3/ 100
Mobilel M-660
10.4
0:00
Cosine
512
3/ 100
ModenBERT-larg
10-4
16
0:00
Cosine
512
3/ 100
GPT2 mPT2-medium
10-4
0.1
Cosine
512
3/ 500
GPT2-large
Cosine
2/ 100
Table 16: Hyperparameter settings for the SST-2 dataset for each evaluated model.
Model
Learning Rale
Batch Slo
Grad Axcum
Weight Decay
LR Scherdiler
Rank
Max Length
Eguchs//
Warmiu Steps
360M
0.1
SmolLMM- 135M
5c-4
0.1
Cosine
512
10/ 100
ModenBERT base
Sc-4
0.1
Cosine
512
10/ 100
DeHEETnn-Y
16:3
64
0:00
Cosine
512
100
64
0.00
Cosine
10/100
large
DeBERTo-<3-lags
Cosine
GP GPT2-medium
5c-4
0.1
Cosine
512
10/ 100
GPT22-azze
1c-4
16
0.00
Cosine
512
10/ 100
Mobilel. L'M-125M
36-3
16
0:00
Cosine
512
57 100
MobileLLM-66
Cosine
57 100
large
Table 17: Hyperparameter settings for the MRPC dataset for each evaluated model.
Model
Learning Rate
Batch
Grad Accum
Weight Decay
Scheduler
Kank
Max
Epochs
WWrmmp Steps
20-4
MobileL
2--1
0.1
Cosine
2/ 1,000
SmolLLM2. 135M
22-4
0.1
Cosine
512
2/ 2/500
ModemBERT buse
2--
0.1
Cosine
512
2/ 2/500
GPT2 medium
2c-1
0.1
Cosine
512
2/500
GPT2-large
0:00
Cosine
2/ 100
DeBERTa.v3-baz
2/
roberia base
1e-4
0.00
Cosine
512
2/ 100
roberta-Inige
1244
0.00
Cosinz
312
2/ 100
SeBlERla-33-lagg
1044
12
0.00
Cosine
512
2/ 100
Mo 125M
0.00
Cosine
2/ 100
MobileLLMM-60
Cosine
2/ 100
ModemBEKTTHae
2/
Table 18: Hyperparameter settings for the QNLI dataset for each evaluated model.
32

---

Published as a conference paper at ICLR 2026
Model
Learning Kate
Batch Size
Grad Accam
Weight Decny
LR Scheduler
Rank
Max Length
Rpochs/ / Warmmp Steps
SmoiLM2-360M
0.00
Cosine
30/ 100
Mo
SmolLM2-155 135M
1e-4
0.00
Cosine
512
100
ModemBEKT base
12-4
0.00
Cosine
512
30/ 100
medium
1e-4
0.00
Cosine
512
30/ 100
GPT2-lage
16
0.00
Cosine
512
30 100
DeBEXIn-v3-hal
Cosine
30/
roberia large
1c-4
0.00
Cosine
512
100
DeSEXInox3-laage
12-4
0.00
Cosine
512
30/ 100
MobiteL 125M
1o-4
16
0.00
Cosine
512
30/ 100
MobileLLLM 600
0.00
Cosine
20/
ModemREKT-laer
Cosine
30/
Table 19: Hyperparameter settings for the RTE dataset for each evaluated model.
Model
Learning
Weight Decay
Eguchs/
Warmupp Steps
S50olLM2-300M
20.5
0.1
Cosine
512
107300
MMb
22-5
0.1
Cosine
312
10/:500
SmoiLLM2-11 M3555LM--13MM
20-5
0.1
Cosine
512
10/500
ModemBERR base
20.5
0.1
Cosine
10/500
GPT2-medium
0.1
Cosine
10/500
GPT2-large
DeBERRTa v3-bue
26-5
0.00
Cosine
512
10/ 100
roberta-hise
26-5
0:00
Cosine
312
107 100
roberta-large
26-5
0:00
Cosine
512
10/ 100
DeBEKTa large
0.00
Cosine
10/100
Mobilal M.-illl-MM-12500
0.1
Cosine
107
0:1
ModomBEKT-lagg large
5c-4
0.1
Cosine
512
10/100
Table 20: Hyperparameter settings for the COLA dataset for each evaluated model.
Model
Learning Rate
Batch Size
Grad Aecum
Weight Deeay
LR Scheduler
Rank
Max Lzngth
Epochs/
/Warmmp Steps
M2231M
25-4
0:00
Cosinz
512
27500
MobileLLM-
0:00
Cosine
2/500
SmolLM2
2/500
ModenBERT-bus
Cosine
2/
GPT2-medium
2--4
0.00
Cosine
512
2/500
GPT2 2-large
1c:3
32
1
0.00
Cosine
512
2/ 100
DeHEXTn-v3-hase
16:3
14
0:00
Cosine
512
2/ 100
Cosine
2/ 100
large
2/
Cosine
2/
MobilLLMM-12MM
12-3
0.00
Cosine
512
2/ 100
MobileLLM 600
10:3
0.00
Cosine
512
2/ 100
ModenRERT-larg
26-4
0:00
Cosine
512
2/ 500
Table 21: Hyperparameter settings for the MNLI dataset for each evaluated model.
Model
Learning Rale
Batch Size
Grad Accum
Weight Decay
LR Scheduler
Rank
Max Length
Eguchs/ / Warmuup Sleps
ModemBEKI
1007
Mobilel
Cosine
1007 100
Smvill M2-360M
32-4
128
0.00
Cosine
512
100/ 100
SmolLM2. 135M
3o-4
128
0.00
Cosine
512
100/ 100
GPT2-medium
3644
128
0:00
Cosine
512
1007 100
GPT2-large
128
Cosine
100 100
DeBERInav3-hsse
roberts-hise
128
8:00
Cosine
1007 100
roberta-larg
30.4
128
0.00
Cosine
512
100/ 100
DeBEEIIa-x3-ann
36-4
128
0.00
Cosine
512
1007 100
Mobilel.IM- MM-22M
30-4
128
0.00
Cosine
512
1007 100
Mobilel.IM-660
Cosine
100 100
ModemBERTT-aas
Table 22: Hyperparameter settings for the BoolQ dataset for each evaluated model.
Model
Learning Rate
Ratch
Grad Accum
Weight Decay
Schediller
Max Length
Egochi/
/Wrrmpp Steps
DXBEE I-sse-
12.4
16
D.00
Coaine
312
12.7 100
MobiteL.MM
LMMM30MM
1o-4
16
0.00
Cosine
512
12/ 100
SmolL.M2
1o-4
16
0.00
Cosine
512
12. 100
Smoll. M2-135M
Cosine
100
ModemBERTTbase
Cosine
GPT2-medium
16-4
0.00
Cosine
512
12/ 100
C1PT2-larg
16-4
16
0.000
Cosine
512
12/ 100
roberta-hise
10-4
16
0.00
Cosine
512
12/ 100
roberia-lanee lang
16
0.00
Cosine
512
12 100
DeBERTa-
Cosine
100
MM-125
Cosine
MubileL 600
16-4
0.00
Cosine
512
12/ 100
ModenBERT-lags Jarge
16-4
16
0.00
Cosine
512
12/ 100
Table 23: Hyperparameter settings for the HellaSwag dataset for each evaluated model.
33

---

Published as a conference paper at ICLR 2026
Model
Learning Rale
Batch Nior
Grad Accum
Weight Decay
LR Scheduler
Rank
Max Length
Epochs
/Warmup Steps
TXRFE In-se-hase
1544
16
0:00
Conine
312
400
MobileLLMMe LMbbleeLLMM3NIMMM
Cosine
4/ 100
Smoll
100
Smvil,
Cosine
4/
ModenRERT--use
30-4
0.00
Cosine
512
4/ 100
merd-mmeium
3c-4
16
0.00
Cosine
512
4/ 100
GPT2-lage
3o-4
16
0.00
Cosine
512
4/ 100
roberta-hise
Cosine
4/
lange
DeBanTa-<-lange
Cosine
4/ 100
M--12MM
R6-4
0.00
Cosine
512
4/ 100
Mo 600
3e-4
16
0.00
Cosine
512
4/ 100
MoskeBERTTlary
3--4
0.00
Cosine
512
4/ 100
Table 24: Hyperparameter settings for the SIQA dataset for each evaluated model.
ble 35 show runtimes similar GPT-2 Medium, indicating that this efficiency advantage holds at smaller scales.
Name
Self CPU %
CPU
CPU total %
CPU total
CPU time avg
of Calls
aten::lincar
0.51%
2.580ms
77.29%
388.420ms
4.046ms
96
aten:addmm
74.66%
375.212ms
76.25%
383. 177ms
3.99 1m9
96
aton::atmml
0.27%
1.333ms
8.83%
44.372ms
924,422uus
48
aten::bmm
8.25%
41.477ms
8.26%
41.502ms
864.622us
48
aten::copy-
4.84%
24.308ms
4.84%
24.308ms
79.180pu
307
aten:gather
2.73%
13.696ms
2.73%
13.696ms
570.650us
24
aten:cloone
0.12%
618. 618.044us
2.26%
11.360ms
135. 255.242us
84
aten::contiguus
0.04%
207.146us
2.08%
10.476ms
145.499ps
aten:repeat
0.12%
586.012ps
1.62%
8.156ms
339.848us
24
aten:add
1.17%
5.887ms
1.22%
6.136ms
84.054ps
73
Self CPU time total: 502.528ms
Table 25: CPU profiling results for DeBERTa-v3-Banl showing operation-wise breakdown of compu- tation time.
Name
Self CPU %
Self CPU
CPU total %
CPU total
CPU time avg
of Calls
aten::lincar
0.30%
4.865ms
82. 26.66%
1.329s
6.921ms
192
aten:addmm
80.79%
1.299s
82.08%
1.319s
6.872ms
aten::aamml
0.15%
2.466ms
7.37%
118.530ms
1.235ms
96
aten::bmm
7.03%
113.072ms
7.04%
113.1 11ms
1.178ms
96
aten:ccopy.
3.91%
62,848ma
3.91%
62.848ms
103.539ps
607
aten:gather
2.17%
34,856ms
2.17%
34.856ms
726. 126.164us
48
aten:cclone
0.07%
1.160ms
1.78%
28.664ms
170.619ps
168
aten::contiguous
0.03%
443.678ps
1.63%
26.265ms
182.397us
144
aten::repeat
0.08%
1.258ms
1.23%
19.738ms
411.214us
48
aten:add
0.88%
14.152ms
0.91%
14.626ms
100.871us
Sell CPU time total:
1608ms
Table 26: CPU profiling results for DeBERTa-v3-Large showing operation breakdown of computation time.
L PREDGEN VS. ONE-TOKER GENERATION:
The original PredGen framework (Kowsher et al.) 2025b) showed that generating multiple output tokens retains higher mutual information with the input, leading to better performance on regression and classification tasks compared to pooling-basse computational cost due to sequence-level decoding. To improve efficiency, propose simplified variant that performs single-token generation or masked prediction, predicting one specific token (e.g., via masked prompt-inserted position). We extract its hidden state and pass it through lightweight MLP for final prediction. This method achieves competitive results across six regression benchmarks (Table
methods. However, this approach incurs high
34

---

Published as a conference paper at ICLR 2026
Name
Self CPU
Self CPU
CPU total
CPU total
CPU time avg
of Calls
aten:linear
0.22%
2.379hms
92.35%
1.07%%
14.774ms
73
sten:addmm
91.46%
1.068s
91.93%
1,074s
14.706ms
73
aten:ocaled dol productt attention
0.02%
187 3p
5.13%
59. 890ms
1m991ms
12
wttp:::seaed.do.prooooooo flash attentionnn .op
5.04%
58, 58.850ms
5.11%
59, 39.703ms
4.975mm
12
aton:gellu
1.15%
13.426ms
1.15%
13.426ms
19ms
12
aten:layer_nom
0.03%
356 267
0.74%
8.673ms
346. 9366s
25
aten:native Jareeeeleeeerrmmeeeeeeeee
7.832ms
8.317ms
aten:copy.
0.42%
4 4.888ms
0.42%
4 4.888ms
61887uu
79
sten:add
0.25%
2,868ms
0.25%
2.878ms
106. 106.586ps
27
sten:ne
0.14
1.675ms
0.14%
1.675ms
1.675ms
CPU
Table 27: CPU profiling results for RoBERTa- Base showing operation breakdown of computa- tion time.
CPU avg
alen::lincar
0.39%
4,022ms
94.22%
962 099ms
6.773ms
145
aten:addmm
92.45%
963.708ms
93.46%
974.219ms
6.719ms
145
aten:scalei__do__p cat.olddo_ppddocttttteeoe antentit
0.03%
304,568ps
3.29%
34.249ms
1.427ms
24
scaled dol, prouucct flush attention for epu
3.13%
32.634ms
3.26%
33, 945mm
1.414ms
24
sten:ggelu
1.00%
10.469ms
1.00%
10.469ms
436, 198us
24
ston:copy.
0.93%
9.662ms
0.93%
9. 9.662ms
63 987ps
151
aten:llaee norm
434.620pus
7.775ms
18.6670ux
alen:native.. layer.mommm
0.63%
6.605mm
0.70%
7.340ms
149,800pps
49
atencadd
0.45%
4.657ms
0.45%
4.670ms
91.559iu
51
alen:vieww
0.22%
2.325ms
0.22%
2.325ms
4.754ps
489
Self CPU time total. 1042ms
Table 28: CPU profiling results for RoBERTa-Large showing operation- breakdown of computa- tion time.
Self CPU %
Self CPU
CPU
CPU
CPU time
Calls
aten:linear
0.13%
532.099ps
81.11%
282.066 ms
3.208ms
88
slen:maatmul
0.62%
2.164ms
81.03%
281.778ms
2.562ms
110
slen::mmn
79,88%5
277.768ms
79.89'5
277.5 4m
3.157ms
88
sten.sscaled der der.deooooeee..... product attention
0.07%
230. 230.328pu
6.25%
21.748ms
988. 988.5655u
22
sten:_scaled_do dor.pleedd atstttttiooooooooooooooo atttenttoonoor.oom
20.35
6.1%
21.518ms
978.096us
22
aten:layer norm
0.1%%
462, 996sss
2.60%
9.037ms
200.831ju
45
2.28%
8.574ms
190.542uu
sten.mmul
2. 2.17%
7.550ms
2.35%
8.189ms
53.177pa
154
stencadd
1.82%
6.327ms
1.82%
6.327ms
71.901js
88
slen:gellu
1.40%
4,852ms
1.40%
4.852ms
220.545us
22
Sell CPU time 347, m44s
Table 29: CPU profiling results for ModemnBERT-Bar showing operation wise breakdown of computation time.
Name
Self CPU %
CPU
CPU total
CPU total
CPU time
of Calls
alln::lineur
0.0.%%
818.323ps
81.17%
2.223%
19.850ms
I12
sten:matmul
0.14%
3,970ms
81.15%
2.223%
15.876ms
140
aten.:nm
80,90%
2.216s
80.90%
2,2168
19.785ms
sten:embeedding
0.00%
61 44664
12.23%
335.032ms
335. 335.032ms
aten:index, select
334.935ms
sten::layer norm
0.02%
470.737ps
2.22%
60.931ms
1.069ms
57
Mknn:eattvv_layer.norr
59.590ms
60.460ms
1.06lms
57
aten:scalled dor product attention
0.02%
564, 664.994ps
1.45%
39.851 ms
1.423ms
28
scaled dot product. flush attention for epu
1.38%
37.714ms
1.43%
39. 29,286ms
1.403ms
28
aten:gelu
24.332ms
24.352ms
068.9866us
28
Seif CPU time cotal: 2739ms
Table 30: CPU profiling results for ModemBERRT-arge showing operation-wist breakdown of computation time.
35

---

Published as a conference paper at ICLR 2026
Name
Self CPU %
Self CPU
CPU total %
CPU total
CPU time
# of Calls
alen:andimm
88.05%
96
sten::mul
3.18%
85.80m
3.35%
37.679ms
392 392,489pu
96
stcn:scalle. product atteution
396, 346.746us
2.76%
31,048ms
1.294ms
24
afen:_senled_dox wtt.duottoooootttooooomom www./aaaaaaaaaaaaaaaaaae itt.attttiioooor..ppoo
3.6%
25.255ms
2.72%
30.652ms
1.277ms
24
atenccopy.
2.07%
23.295ms
2.07%
23.295ms
80.886jus
288
alen:add
1.95%
21 21.947mms
1.99%
22.375mm
230,671pu
97
slen:contiguuus
0.03%
298, 298.059ps
1.01%
11.422ms
118, 118,9834s
96
sten:celone
482.4822s
11.124ms
115,879jus
96
slen::pow
0.87%
9,819ms
0.88%
9.867ms
411.125ps
24
aten.ttanh
0.79%
8.921ms
0.79%
8.921ms
371.720ps
24
Sell CPU time total: 1126ms
Table 31: CPU profiling results for GPT-2 Medium showing operation wise breakdown of computa- tion time.
Self
CPU
CPU avg
Calls
aten:addmm
87.92%
2.160s
89.08%
2.188s
15,196ms
144
aten:mmul
69.73
508.058px
aten:scaled wtte::celeeddo..predddoo htpp:loooooo.ccttttteeeetee Lttienton
0.02%
560.556pp
2.74%
67.311ms
1.870ms
36
scaled dot product ilash atention for opu
2.63%
64.497ms
2.72%
66.750ms
1.854ms
36
aten:copy.
1.82%
44. 44.776ms
1.82%
44.776ms
103 647us
432
atencadd
1.77%
43.543ms
1.80%
44. 44.286mm
305. 4222u
145
alen:comiiguus
0.02%
$48.391Iu
0.87%
21.351ms
148. 148.269ps
144
stenzclone
0.06%
1,422ms
0.85%
20.802ms
144.461lp
144
ston::ow
0.81%
19.877ms
0.81%
19 19.970ms
554.71440
36
aten:tanh
0,70%
17. 17.260ms
0.70%
17.2600m
479.437pu
36
CPU time total: 2456ms
Table 32: CPU profiling results for GPT-2 Large showing operation-wise breakdown of computation time.
Name
Seil' CPU %
Self CPU
CPU total
CPU Lotal
CPU time
Calls
alen::linar
0.35%
1.89mms
80.94%
441.637m 7m
1.00mm
210
sten:matmal
1.44%
7.8633s
79. 79.89%
435 925ms
2.066ms
211
aten:rrom
425.052ms
77.93%
alen:scaled doi, cool.ppduucttooo attention
0.07%
360.301ps
6.26%
14,133mms
1.138ms
30
scaled product flash attention epu
31.891ms
33.775ms
1.126ms
30
aten.mul
2.73%
14.91 Ims
2.74%
14,958ms
54,590ps
274
stencelone
0.18%
963.449|us
1.87%
198ms
84.981us
120
alen:coopy
1.54%
8.398ms
1.54%
8.398ms
34,277pus
245
sten:silu
1.51%
8.256ms
1.51%
8.256ms
275.200pu
30
atencadd
1.29%
7.025ms
1.48%
8.054ms
44.496u
181
Self CPU time roral. 545 639ms
Table 33: CPU profiling results for Mmo1LM2-113M showing operation-wise breakdown of computa- tion time.
Name
Self CPU %
Self CPU
CPU total
CPU total
CPU time avg
Calls
aten:lincar
0. 14%
1.401 ms
87.03%
895. 172ms
3.996ms
224
aten:matmmul
alen::mm
85.92%
883.710ms
85.93%
883. 526ms
3.946ms
224
aten:sccledd product
1.87
3.82%
39.269ms
1.227ms
afen: dowd...ceeed.o..roodee....c www.hhaaatttaaaaaaaaaa attontionffor.pppp
3.49%
33.847ms
3.64%
37.398ms
1.16ms
32
2.46%
65.292ms
2.46%
25.3199m
86.708ps
292
alen:ssilu
1.36%
13.992ms
1.36%
13.992ms
437.2600us
32
sten:add
1.07%
11.014ms
1.14%
11.728ms
60.769pu
193
aten.cllone
0.07%
706.6 63063
1.00%
10.261
80.1664s
128
ston:copy
0.87%
8. 8.908ms
0.87%
8.908ms
34.131ju
261
Table 34: CPU profiling results for SmolL.M--360M showing operation- wise breakdown of computa- tion time.
Name
CPl
CPU
CPU total %
CPU total
CPU time oo Calls
aten:lliee
0.13%
1.007ms
140ms
2.844ms
aton:.matmal
0.52%
3.615ms
86.62%
596. 756ms
2.815ms
212
85.81%
59 1.6ms
85,83%
591,306ms
2.802ms
211
scan:d dot prdductatttcttion
0.06%
386. 293
4.25%
29.303ms
976.77lus
30
aten:: scaled. dot product flash, attention for
4,04%
27 8323m
4.20%
28. 28.917ms
963 894ys
30
aten::mul
2.28%
15.710ms
15.770ms
$7.554us
aten:silu
1.45%
9 993ms
1.45%
9,993ms
333. 105ps
30
aten:add
0.98%
6.723ms
1.06%
7.271ms
40,174us
181
ston:cloucc
0.09%
62
0.91%
6,256ms
52.131us
120
aten:coppy
0.76%
5.25 Ims
0.76%
5.215ms
21.432us
245
CPU
Table 35: CPU profiling results for MobileLLM-125M showing operation-wing wise breakdown of computation time.
36

---

Published as a conference paper at ICLR 2026
8:45
0:30
1.45
1,00
8:38
0:00
8:43
3:40
0:09
2:43
0.10
1.43
6.21
0:31
0:14
0:23
0.14
5:7
6.18
533
6:00
9:22
8.09
0.14
8:15
0,00
6.45
5
0.20
cC
5
3:
0.1G
3.3L
1,00
0
5
0.27
0.37
2.
0.12
2.25
0.14
2.2
6:00
8338
8:34
0.15
9:37
100
6.11
2:33
C.O
2.11
0:08
1.04
0:35
6.44
6:35
6.45
0.64
1.45
0.35
6.45
0.55
2.43
1.30
5:37
0.29
6:11
0.13
6:13
018
X33
6.33
6:17
1/54
6:14
1.00
C12
17
6:10
3
1.0
0.40
20
6:15
2.
0:28
L/
02
3
0222
0.18
6:32
0.27
0.32
0.00
6:22
1.30
0.23
6.09
8.21
8:37
A15
6:14
6:19
0.22
3.60
0.35
3.23
E:44
0.22
6.2
0.11
8.09
0.33
1.43
1.30
Figure 10: Token-level mutual information the SST-2 dataset, computed using representations from layers 1, 8, 16, and 30 of MobileLLM. The figure highlights how information evolves across layers during fine-tuning.
2.2
3.2
3.0
2.0
2.8
2.6
0
2.4
1.6
1062)
2.6
2.4
1.8
1.7
2.4
2.2
3.6
3.5
1.4
3.8
3.3
1.8
1.4
7.0
7.5
(0K:2)
10C22
106:2)
Figure 11: Mutual information on the ETTh1 dataset for different prediction horizons: 24, 96, 128, 380, 512, and 1038. The figure illustrates how information flow varies as the prediction target becomes more distant.
37

---

Published as a conference paper at ICLR 2026
Name
Seif CPU %
CPU
CPU total %
CPU total
CPU time avg
# of Calls
aten:
0.10%
1.933mm
90.92%
18808
6.433ms
28T
mien:matmull
0.30%
6.000ms
90.62%
1,802s
6.389mms
282
atcn::mm
90.18%
1.793s
90.18%
1.793s
6.381ms
281
aten:scaled wwtp_/ooocucttttteeeeeooo anttentin
0.02%
431. 17pps
2.74%
54.424ms
1.36
40
2.62%
52.116ms
2.72%
33.992ms
1.350ms
40
aten:mmul
1.65%
32, 88.05mm
1.65%
32.838ms
90.2 4yys
364
aten:silu
1.46%
28.972ms
1.46%
28. 28.972ms
p24.307us
40
alen:add
0.77%
15,238ms
0.81%
16.094mms
66.778ps
241
sten:clone
0.05%
1,018ms
0.65%
13.012ms
81.325ps
160
stop:
0.55%
10.926ms
0.55%
10. 10.926ms
33.61 7us
325
Self CPU time total: 1988ms
Table 36: CPU profiling results for MobileLLM-6600 showing operation wise breakdown of computation time.
Model
PEPT LoRA
Method
WASSA
SICK
STSB
LCP
CWI
Humicroedit
Avg.
Predictor
0.45201151
0.8600 280
0.965R1950
0,336M..10 IGS
L.0400775
1.348/1.04
0,02289.553
Generator
0.090/0.023
0.340/. 195
0.610/0.630
0.900/0. JOS
0.46510 349
0.6500.505
0.509/0.301
PredGen
0.068/0.022
0.320/0.190
0.576/0.669
0.062/0.008
0,420/0.280
0.550/0.455
0.338/0.257
Genemtion
0.089/0023
0.315 192
0.58200 574
0.065/0:009
0.430/0 290
0.548/0.557
0.335/0.250
AdaLaRA
Predictor
0.424/0.44
0.845/0.270
0.950/0.935
0.918/0 100
1.0200 790
1.36011000
0.920/0.549
Llama2-78
Generalor
0.087/0.022
0.325/0.18
0.600/0.620
0.896/0.097
0.45510.353
0.6300.150
0.198/0.291
PredGen
0.060000000
0.305/.
0.575/0.570
0.0580.000
0,405/0
0.535/0.400
0.32690.244
Generation
0.079/0.020
0.308/0.188
0.578/0
0.057/0.006
0.410/0 274
0.5320.442
0.325V0.247
RoCoFT
Prodictor
0.42001148
0.55600274
0.958/0.912
0.924/0 102
0,990/0.770
1.340/1.000
0.
Generalor
0.065/0.02)
0,332/0 191
0.60500.233
0.895/0
0. 0.160/0 357
0.611/0.197
0.503/0.245
0.06:00.02
187
0.58200.580 380
0,060/0
0,40510
0.543/0.448
0.332/0.258
Generation
0.083/0.020
0,308/0.186
0.57800.375
0.061/0.00 008
0,110/0 278
0.54800.50
0.332/0.253
DoRA
Prodistor
0.511/0.150
0.850 275
0. 6.960009915
0.92240 104
0,98000.788 789
1.355/1.04
0.930M0.50
Generator
0.086600022
0,3300 190
0 62607A1625
0. 0.8850 100
0.462X0 338
016550,500
0.503/1.296 296
PredGen
0.065/0.021
0,301/0
0.5800.578
0,061/0.007
0.115/0
0,5100..44
0.333/0.22
Generation
0.08-0000
0,303/0 185
0.5840 580
0,062/0 008
0,41800 278
0.5380044
0.334M0 253
LoRA
Prodictor
0.37001130
0.80002200
0.92000910
0.88C
0.95000720
1 12801.000
0.86780317
Generator
0,075/0.188
0.310.0.17
0.580/0.590 550
0.850/0 090
0.130/0
0.60100160
0.17400.77
PrefGien
0,07/00188
0,287/0
0.5500
0,3800
0.500004400
0.308/0.231
Generation
0.073/0.018
0.289/0.170 17
0.553/0 542
2.051/0.000
0.38510.25
0,49500.402
0.309V0.232
AdaLoRA
Predicter
0,360/0.125
0.810/0225
0.93000.900
0. 0.896N0.095
0,960/0.733 750
1 1.300/1.010
A187540.522
Lloma2-138
PredCien
Generalor
0078M019
0,07600099
0.315/0.15 178
0,300/0
0.58500600
0,86000 095
0,05400000
0,3900
0.440/0 320
0.610/04470
0.5100,410
D. D.48 18) 280
D.31 3.315902366
Generation
0.077/0.019
0.302/0 766
0.52800.53 329
0,055/0.00 007
0,395/ 258
0.506/0.111
0.316/0.237
RoCoFT
A
PredCien
Generation
Generator
Prodicter
Predicter
0,071/0.018
0.7720017
0.3HDAD
0,07000017
0.365/0122
0.288/ 109
0.3050 172
0.80500 252
0,28600 170
0.790/0.244
0.348X
6,91160000 000
0.575X
015
(90
085400 0005400000 OCT
03450 088
0.87000088
0522400 102
0.0530000
0.37800
0,375M
0.4250 305
0,94000 710
1195507 7234
23150-240 248
177902224 250
0 1000110 000
O 29111-45
1 127000990
1 1I027
12050000 401
BX37700 077005777777
U. 0,30000 22
D. 86000.511
D. 0.86000511
0.307A0 237
PredGen
Generator
0.070/0.016
0.076/b.088
0.312/0 176
0.295/0.19
0,03300 0.5
025
0335500099
0,3850 252
1415K0319 111
0 405550440 40S
0200111445 465
D. 0.43900.200 279
0.31 1000001 130.233
Generttion
01669006
0,297/0 173
0.5580 550
0,0540 OCT
0WW00 254
0. 0.20200-00 400
2.91 220-244
LoRA
Prodicter
0.38D01100
0.20002200
0.9 019100922
0.91050.005
U9V0N0
131011200
03S5/0.331
Generator
0,881000199
0.320/ 180
0. 6159500610
0,87600 095
0:4000 525
062000 480
D. 0.4880.28 285
PredCien
047770019
0.298/ 173
0.565AX 555
0.0550
0.5950 260
SS00 420
0.318/0.290
Generntion
0.07//0019
0.300/.
0.562/1553
0.05410.00
0,5980 263
0518814419
0.3200.2400
A
Predicter
0.375Ab
0,8300 265
9,945/11930
0.910/0.00%
0,98000 750
1
1320/1 030
D 88330.535
LJama3-8B
PredGien
Generator
0.080000000
0.078/00099
0.30340.11 177
0.325/0 183
0.570A1 560
0.600
0.05240
1.7750.007
0.4000 265
0.4000 330
06300 485
0.50900.10
D. 0.493302288
0.323/0.243
Generation
0.077/0.019
0.305/0.178
0.573/0
0.058/0.007
0,403/0
0.505/0.412
0.322/0.242
RoCoFF
Predicter
0.3900.145
0.810A 285
0.935/1920
0.910/00098
0.9600 730
1 1.30111 D115
0.88400527
Generator
0.082/0.000
0.315/0.177
0.585/0.605
0.865/0.092
0.435/0 320
0.610/0.475
0.482X1.222
PredGen 0.079/0.020
0,288/0 169
0.565/0.558
0.058/0.00 007
0.385A0 255
0 0,530/0.225
0.317/0.238
Generation
0,290/0.170
0.567/0.559
0.059/0.008
0
0.528/0.426
0.31S/0.239
DoRA
Predictoe
0.385/0.38
0.82580226 261
0.950N0.935
0.90510.096
0.975X07745 745
1.315/L.02 025
0.893/0.533
Generator
0.078/b.019
0.322/0 179
0.592/0.608
0.880/0.096
0.445/0 328
0.625/0.462
0.490V0.285
PredGen
0.073/0.018
0.300/0.175
0.562/0.555
0.066/0.007
0.390/0 262
0. 0.525/0.425
0.319/0.241
Generation
0,072/0.18
0.30200176
0.56600.60
0.06510.007
0,393/0.265
0.523/0.426
0.320X0.42
Table 37: Regression performance of different PEFT methods across benchmarks, reported as MAE/MSE. Generation* denotes single-token generation.
USE OF LARGE LANGUAGE MODELS
We used a large language model (GPT) solely for minor writing assistance, such as grammar checking, language polishing, and improving readability, No content generation, ideation, experimental design, data analysis, or result interpretation was performed by the LLM. All research contributions, technical content, and results in this paper are entirely the work of the authors.
38


Reviews:
[{"review_id": "POAm2pv5t9", "reviewer": "Reviewer_9XA4", "summary": "The authors introduce a post-hoc framework that estimates how much information each model layer carries about the input and the label by training two mutual information critics. It summarizes each layer’s \"information plane\" trajectory with the \"Optimal Information Coordinate\" and uses this to compare models.\n\nAcross NLU and regression tasks, the paper reports that bidirectional encoders show higher OIC and typically higher accuracy than similarly sized unidirectional decoders.", "questions": "As mentioned in the weaknesses, I would suggest moving full tables with results to appendix, and keeping in the main part of the paper aggregated/selected results -- in a form of smaller, more readable table, or a figure. Table 1 with information about models is not _that_ relevant and could be completely moved to the appendix.\nThe correlation between OIC and performance could be shown explicitly to support the claim of higher OIC --> better performance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper adds a formal lens to a known empirical observation that bidirectional attention yields stronger representations than causal attention.\nThe work includes broad comparisons across many datasets and model families.", "weaknesses": "The authors claim that higher OIC is correlated with better accuracy of the model, but no quantitiave correlations or graphical comparisons are provided.\n\nThere are small issues with the presentation of the paper. Namely, the tables are hardly readable -- a lot of content with tiny font size. I would suggest moving these full tables to appendix, and leaving only the most important aggregated/selected values in the main text, or represented in a graphical form as a plot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dUTLrhqTIW", "reviewer": "Reviewer_8JaC", "summary": "This paper makes two contributions: (1) it introduces FlowNIB, a method for estimating mutual information in LM layers, (2) it examines the differences between unidirectional and bidirectional models, finding the latter enable higher MI, and linking this to improved downstream performance.", "questions": "- Definition 2.2: What does \"measure\" mean here? It's clearly not a measure in the sense of \"measure theory\". Some examples are provided, suggesting that \"measure\" here just means a mapping from p to real numbers?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I found the paper clearly written and overall easy to follow (with a few but critical exceptions mentioned below).\n- Given the recent dominance of unidirectional language models, it can be interesting to revisit bidirectional language models.", "weaknesses": "- I didn't understand a key point: line 099: \"in finding both information *simultaneously*\". What does this mean? Why not just use MINE to estimate I(X; Z_l) and I(Z_l; Y)? Why use a schedule emphasizing first one of these and then the other? I see that this yields a nice 2D plane, but what is the theoretical interpretation? This seems also key to understanding why the authors need to introduce FlowNIB, which otherwise seems unclear.\n- Figure 1: which ones are the lower vs upper layers? It seems that curves to the right indicate higher MI with both X and Y than curves on the left (depending on \\alpha(t)), which seems confusing, given that by the data processing inequality lower layers should have more infromation with X and upper layers should have more information with Y?\n- How nontrivial is the finding that higher MI is associated with better downstream performance? If a model has higher MI with Y, isn't this naturally almost the same as saying that the model performs better on predicting Y? I see that the authors use not MI with Y but the summary statistic OIC, but how much of a difference does this make?\n\nIf the authors can convincingly address these, I'd be happy to reconsider my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GLVj0eWOvR", "reviewer": "Reviewer_3k2z", "summary": "The authors analyze an important question, why bidirectional models (encoder-based transformers) performs better in natural language understanding tasks than uni-directional models (decoder-based transformers)? from an information theory perspective. They measure the mutual information between internal representations and inputs, I(X, Z), and between representations and output label, I(Z, Y). They claim that bidirectional LM layers exhibit higher mutual information than unidirectional ones, which could be the reason why they perform better on these tasks. However, I have severe concerns about some important aspects of the paper.", "questions": "Questions and suggestions:\n\n- When presenting equation 1 as the training objective, explicit say it is to be minimized, though it is later mentioned, but it might still cause confusion for readers. \n\n- After definition 2.2 when the L2 participation ratio is mentioned, it’s better to briefly explain the intuition behind it.\n\n- Regarding the scale of I(Z, Y) depends on the label space, maybe try normalizing it with  min(H(Y), H(Z)), which is usually H(Y), since the I(Z, Y) is upperbounded by that (the entropy can be reduced by observing the other variable), so the normalized version can be retreated as the proportion of entropy being reduced.\n\n- If X the embedding of a token, is it one-hot embedding or a learned token embedding? \n\n- In Figure 3, it shows average value across layers, can you also show it for the maximum value as well? which I feel would make more sense.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The paper uses and emphasizes the importance of mutual information, which is a metric well-supported by theory and could be very useful for analyzing neural models, but is somewhat overlooked by the research community.", "weaknesses": "- There’s an important question that is not clear to me, is the input variable X a sequence of tokens or just one token? From the architecture of MINE critics (2 layer MLP) mentioned in the paper, it seems it is similar to the original use case of MINE, where inputs are just pairs of vectors. But it means X is a single token, and Z is representation correspond to this token. Then this I(X, Z) is measuring how much information about the current token is encoded by its corresponding representation, which, to be honest, does not make sense to me. Because the representations in transformers are contextualized, they are meant to encode information about the whole input sequence. Measuring MI between the representation and a single token does not sound meaningful. I think it would make a lot more sense if X is the whole sequence, I(X, Z) measures information about the real input. Also, in the paper I(Z, Y) seems to be the average value across sequence length. In other words, in a certain layer of a transformer, there is a sequence of representation, and each representation is used to estimate MI and is averaged. But I think taking the maximum make more sense, because in decoder models many representations cannot even see the large part of the input.\n\n- In theorem 2.1 and lemma 2.3, authors assume concatenation of representations of forward and backward direction instead of addition of them, which is a big difference. As addition causes loss of information, many arguments might not follow. In the case of addition,  obtaining the information about future context is at the cost of information of previous context, I(X; Z→) is not necessarily lower than I(X; Z↔). In the case of concatenation, as the dimension gets larger, it is not a fair comparison. Importantly, this dicussion is only meaningful if X is a sequence.\n\n- In the paper, I(X,Z) and I(Z, Y) are measured by two separate critics (separate neural networks no weights sharing), changing a(t) that combines them to emphasize one over the other does not make a lot of sense to me. Because there is no trade-off for the networks, each network is always optimize only one thing (either I(X,Z) or I(Z, Y)) without considering the other. The only effect is that the loss is scaled differently throughout the training. I don’t see any benefits or important differences between training together and training separately. It seems to me it’s just funamentally equivalent in this paper. Not sure why give it a new name.\n\n- Writing and presentation are not good. Some important information is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ISCCYq3nfu", "reviewer": "Reviewer_vzyM", "summary": "This paper explores the information plane of bi- versus uni-directional LLLMs. The authors present FlowNIB and compare bi- with uni-directional LLLMs across classification and regression tasks. The authors' main claims are that bidirectional models show better MI values and thus better downstream performance.", "questions": "* How do you ensure that the observed MI differences are not simply a by-product of pretraining objectives or tokenizer differences rather than directionality per se?\n\n* How robust are your FlowNIB estimates across random seeds, estimator architectures, and datasets?\n\n* Have you validated FlowNIB on synthetic data or toy problems where ground-truth MI is known, to verify that it produces accurate or at least monotonic estimates?\n\n* Can you provide evidence that increasing OIC causally improves task accuracy — for instance, by pruning or freezing layers with low OIC and observing the resulting performance drop? Alternatively, can you show that models fine-tuned to explicitly maximize OIC achieve better generalization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The connection between conditioning reduces entropy and higher I(X;Z) in bidirectional models is rigorous and intuitive.\n\n* Consistent results across 16 datasets, covering both classification (GLUE, commonsense reasoning) and regression (STS-B, LCP, etc.). Layer-wise and token-level MI analyses support the theoretical claim.\n\n* The paper is well structured and easy to follow.", "weaknesses": "* Contribution (iii) is redundant. We all know the three talking point rule, but that's not a reason to try to make two contributions into three. \n\n* Theoretical novelty: The central theoretical claim (conditioning reduces entropy --> bidirectional > unidirectional) is mathematically sound but conceptually straightforward. It might be seen as an extension of standard information inequalities rather than a novel theoretical insight.\n\n* MINE-based estimators are known to be unstable and sensitive to hyperparameters.  While FlowNIB normalizes MI and uses consistent setups, no quantitative uncertainty analysis (e.g., variance across runs) is provided.  Moreover, the framework measures relative MI values but does not validate that estimated magnitudes correlate with true MI.\n\n* While the paper shows a correlation between MI and performance, it does not establish causation (e.g., whether higher MI directly improves task accuracy). Possible confounds such as architectural biases or tokenization differences are not fully controlled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rjbjw2dBZV", "reviewer": "Reviewer_ZBsL", "summary": "This paper introduces Flow Neural Information Bottleneck (FlowNIB) to analyze the information-theoretic properties of language models, by comparing bidirectional (e.g., BERT-like) and unidirectional (e.g., GPT-like) architectures. Grounded in the IB principle, the authors argue that bidirectional models retain more mutual information, leading to richer contextual understanding. FlowNIB estimates layer-wise MI between the input, hidden representations, and task labels, with the Optimal Information Coordinate to summarize each layer’s information-carrying capacity.", "questions": "1.\tHow do models with higher mutual information perform on non-NLU (e.g., generative or multimodal) tasks?\n\n2.\tCould you provide a quantitative comparison of computational costs between unidirectional and bidirectional models in the context of FlowNIB analysis?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper provides a clear and formal argument, grounded in the IB framework to explain why bidirectional models preserve more mutual information than unidirectional ones.\n\n2.\tThe evaluation is extensive, spanning 16 NLP datasets, multiple model families, and both classification and regression tasks.\n\n3.\tThe paper is generally clearly-written and easy to follow.", "weaknesses": "1.\tThe key insight of the paper that bidirectional layers retain more mutual information is largely intuitive and expected, given that unidirectional layers only attend to previous tokens, while bidirectional ones attend to both past and future tokens.\n\n2.\tThe comparison between unidirectional and bidirectional architectures is not entirely novel. Several prior works have examined similar distinctions, including:\n\n- [1] On the Role of Bidirectionality in Language Model Pre-Training. EMNLP Findings 2022\n\n- [2] Transforming decoder-only models into encoder-only models with improved understanding capabilities. Knowledge-Based Systems 2025\n\n- [3] The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training. ICML 2025\n\n- [4] What Limits Bidirectional Model's Generative Capabilities? A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning. ICML 2025\n\n3.\tA major practical limitation of bidirectional models is their substantially higher computational cost, which remains unaddressed. The paper would benefit from an analysis or discussion of this aspect.\n\n4.\tAlthough FlowNIB is described as “lightweight,” MI estimation remains computationally intensive, especially for large models and many layers. The paper does not fully quantify the runtime or scalability trade-offs.\n\n5.\tWhile FlowNIB identifies correlations between OIC and performance, it does not establish causality. High MI could stem from other architectural factors (e.g., attention span or parameter distribution), which the analysis does not disentangle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)