Learning to Abstract with Nonparametric Variational Information Bottleneck
Authors: Melika Behjati
Fabio James Fehr
James Henderson
Venue: EMNLP 2023 Findings (2023)
Keywords: Representation Learning
Analysis of Neural Networks
Nonparametric Variational Information Bottleneck
Deep Learning
TL;DR: We introduce a novel language representation model which is able to compress to different levels of abstraction within each layer of self-attention using Nonparametric Variational Information Bottleneck.

Abstract: Learned representations at the level of characters, sub-words, words, and sentences, have each contributed to advances in understanding different NLP tasks and linguistic phenomena. However, learning textual embeddings is costly as they are tokenization specific and require different models to be trained for each level of abstraction. We introduce a novel language representation model which can learn to compress to different levels of abstraction at different layers of the same model. We apply Nonparametric Variational Information Bottleneck (NVIB) to stacked Transformer self-attention layers in the encoder, which encourages an information-theoretic compression of the representations through the model. 
We find that the layers within the model correspond to increasing levels of abstraction
and that their representations are more linguistically informed. Finally, we show that NVIB compression results in a model which is more robust to adversarial perturbations.

Learning to Abstract with Nonparametric Variational Information Bottleneck
Melika Behjati 1.2
Fabio Fehri 1,2
James Henderson
Idiap Research Institute, Switzerland École Polytechnique Fédérale de Lausanne, Switzerland firstname last Iastname@iciap.cd
Abstract
Learned representations at the level of char- acters, sub-words, words and sentences, have each contributed to advances in understanding different NLP tasks and linguistic phenomena. However, learning textual embeddings costly as they are tokenization specific and require different models to trained each level of abstraction. We introduce novel language rep- resentation model which can learn to compress different levels of abstraction different lay- ers of the same model. We apply Nonparamet- ric Variational Information Bottleneck (NVIB) stacked Transformer self-attention layers in the encoder, which encourages an information- theoretic compression of the representations through the model. We find that the layers within the model correspond to increasing lev- els abstraction and that their representations more linguistically informed. Finally, show that NVIB compression results in model which more robust to adversarial perturba- tions.
Add Norm
Feed
Forward
Add Norm
Denoising Attention
K V
NVIB
Embedded
a (t-1)
Inputs
Figure I: Transformer encoder layer (I) including the NVIB layer and Denoising self-attention module.
stride pattern is fixed and thus can't be considered as learning to abstract. Behjati and Henderson (2023) recently introduced the task of learning higher level of abstraction in set-of-vector space by proposing Dynamic Capacity Slot Attention. In this work, we propose a novel character-lovel model of representation learning which learns dif- ferent levels of abstraction in different layers of the same model.
1 Introduction
Learning representations of language using self- supervision has become cornestone of NLP (Pen- nington et al., 2014; Peters et al., 2018; Devlin et al., 2019, inter alia). However, these represen- tations are specific to their tokenisation (e.g. Byte- Pair (Sennrich al., 2016), WordPiece (Schus- ter and Nakajima, 2012), SentencePiece (Kudo and Richardson, 2018), characters (Al-Rfou al., 2019), and even bytes (Xue et al., 2022)), which restricts the level of abstraction from the input text which their representations are able to con- vey. Work like CANINE (Clark et al., 2022) and Charformer (Tay et al., 2022) avoid problems with tokenisation by modeling individual characters or bytes, and thereafter pling reduce the representation length. The
We adapt the Nonparametric Vari- ational Information Bottleneck regulariser (NVIB) (Henderson and Fehr, 2023) for application self- attention in the stacked layers of Transformer en- coder.' The resulting model has greater abstraction than standard Transformer due to selectively drop- ping some vectors in higher attention layers. Inter- estingly, we observe that the learned abstract units
Contributions
btside downsam-
'The code is publically available at: https://at //gi com/ idiap/nvbb https: http://gitttubccccc httpshtww.c../ii////ee/aaaaaaaae fattention
EEqual contribution.

---

are intuitive, often corresponding to words. By employing different analysis methods, we demon- strate that our model is better at encoding seman- tically and linguistically meaningful information than a standard Transformer baseline. Moreover, it exhibits an enhanced level of robustness, further consolidating its ad advantage.
(μ Ruxp, o R of the base distribution. The model can drop entire vectors by setting their pseudo-counts to zero, thereby making the repre- sentation sparse. In addition, there is an 1+1 component of the base distribution for the prior, with parameters 1 ===0 and o²=1. The in- dividual pseudo-counts are both summed to get the DP's total pseudo-count and normalised to weight the components of the DP's base distribution. The NVIB layer then uses denoising attention access either a set of weighted vectors sampled from the DP (at training time), or the base distribution of the DP (at testing time).
2
The Model
Our model consists of standard Transformer encoder-decoder layers (Vaswani et al., 2017), where the encoder block has been augmented with an NVIB regulariser on the self-attention layers, as seen in Figure 1.
Henderson and Fehr (2023) use ReLU, linear and exponential activation functions to compute o, 1 and a, respectively. To adapt NVIB for stacked layers of self-attention tivation for the pseudo-count parameters with an exponential activation, and includes a multiplica- tive skip connection from the previous layer 1-1, as shown in Figure! 1:
2.1 NVIB for Self-Attention
model replaces the
Nonparametric Variational Information Bottleneck is an information-theorrtic regulariser for attention- based latent representations (Henderson and Fehr, 2023). It has been shown to induce smooth and sparse latent representations in the ross-attention layer of a Transformer encoder-decoder, where Henderson and Fehr (2023) used it to define a Vari- ational Auto-Encoder (VAE) (Kingma and Welling, 2014). It generalises attention over set of vectors denoising attention over a mixture of impulse distributions, and uses Bayesian nonparametrics to handle the fact that the number of vectors grows with the length of the text. NVIB uses Dirichlet Processes (DPs) to define distributions over these mixture distributions, and controls the information in the latent representation by sampling mixture distribution from the attention layer's DP, thereby adding noise which removes information.
w' +b+ log(a""))), (1)
where E BIxp and b E IK form the linear projec- tion. The exponential activation allows the model to be more stable in training. The skip connec- tion in between layers (-1 and helps coordinate the importance of vectors across layers. Keeping the pscudo-count parameters in log-space prevents overflow and improves precision when the parame- ters get larger. This results in multiplicative skip connection which emphasizes the communication between layers.
To compute self-attention, the DP parameters projected from all the individual vectors together define a single DP, and we take a single sample from this DP which all the individual vectors ac- cess via denoising attention. The queries for this de- noising self- olf-aatention are computed from the orig- inal n vectors Z E R=xp, before the NVIB layer. We also introduce the use of implicit reparameter- isation gradients (Figurnov al., 2018) for error backpropagation through the sampling step. See Appendix for the exact attention functions.
We extend the previous work by using implicit reparameterisation gradients (Figurnov et al., 2018) to improve learning, and by adapting NVIB for use in the stacked self oeff-attention layers of a Trans- former encoder. By extending NVTB's s information- theoretic regularisation to the series of latent repre- sentations inside the Transformer encoder, see increasingly abstract interpretable representations in the higher layers.
NVIB layer As with standard attention layer, an NVIB layer maps set of n vectors to an attention function. It first maps the n vectors Z Raxp the parameters of DP, which are a total pseudo-count for its Dirichlet distribution and mixture of Gaussians for its base distribution. Each of the n vectors is individually projected to a pseudo-count a E IR" and a Gaussian component
Training objective The NVIB loss regularises the attention-based representations so that the size of the representation at each layer is appropriate for the complexity of the representation being encoded at that layer. It has three terms, reconstruction
"Since the exponential function is never exactly zero, we threshold small values to introduce sparsity. See Appendix A.

---

loss Lic and two KL divergence terms: Lp for the pscudo-counts of the Dirichlet distributions, and LG for the parameters of the Gaussian com- ponents. The LR term is the supervised learning objective, which tries to make the latent representa- tion informative enough to predict the original text. The LG term tries to make the individual Gaussian components less informative, as in vector-space VAEs (Kingma and Welling, 2014). The LD term tries to push down the total pseudo-count, which pushes some of the individual pseudo-counts to zero, thereby effectively dropping their vectors and reducing the number of vectors in the latent repre- sentation. See Appendix C for loss equations. To apply NVIB to stacked self-attention layers, we want to allow the lower layers to compute with more vectors while encouraging the upper layers to compress fewer vectors, thereby encouraging ab- straction the higher layers. We therefore weight the loss terms differently at each layer:
is at the cost of longer sequence lengths and the need for greater model depth to reach the under- standing level of subword-based models. While CANINE (Clark et al., 2022) and Charformer (Tay et al., 2022) are some attempts to bypass these shortcomings, they do so by fixed architectural de- sign choices. Our work differs in that it allows the model to learn how to abstract and compress the input without hard-coded abstraction structure. Our inspiration comes from Behjati and Hender- son (2023) who introduced the task of learning a higher level of abstraction and proposed a method based on Slot Attention (Locatello et al., 2020) for this purpose. Our work is also related to HM- RNNs (Chung et al., 2017) as it tends to learn a hierarchy of units within its layers, though it does not make discrete decisions on unit boundaries. Our approach to learning meaningful disentangled abstractions by encouraging the models to learn compressed representations through a bottleneck is shared with VAEs (Kingma and Welling, 2014) and other work in that line (Alemi et al., 2017; Higgins et al., 2017).
L = LR + (ADLD + AGLG) 800)
(2)
for E (I,...,N}
E
4 Experiments
where 3(0 controls the degree of NVIB regularisa- tion for layer L, linearly increasing it for higher lay- ers. If a vector is dropped in the last self- attentonnnio layer (i.e. zero pseudo-count), then we also drop that vector in the pross-attention layer to the de- coder, but otherwise there is no NVIB regularisa- tion of the cross-attention.
Our proposed model's abstractness is analyzed qualitatively through attention visualisations (Sec- tion 4.2) and quantitatively through a challenging sub-topic classification task (Section 4.3.1). I). Each layer is probed to analyse the linguistic informa- tion captured (Section 4.3) and finally we examine the models' robustness to adversarial, synthetic noise (Section 4.4). We provide additional details of these experiments in the Appendices
During preliminary experiments, instead of the above formula for g(() we considered a uniform weight, as well as a doubling weight, per layer. These regularisation weights were either too weak or too strong, respectively. The values we con- sidered for the hyperparameter ÅD are given in Appendix B. When we increase this regularisation, the characters are grouped into fewer and fewer vectors until all characters are compressed into a single vector, much like a sentence embedding. we over-regularise, the representations collapse to the uninformative prior representation.
4.1 Experimental Setup
Data We train all models on the Wikitext-2 (Mer- ity et al., 2017) encyclopedia dataset at the char- acter level, with a noisy character deletion recon- struction objective (Lewis et al., 2020).
We compare the self-attention represen- tations from a standard Transformer encoder layer and our Transformer encoder layer with NVIB reg- ularisation. We consider models consisting of six stacked Transformer encoder layers to be in line with the base model from Vaswani et al. (2017). For the Transformer decoder we use only 2 layers so that the decoder not able to compensate for poor embeddings from the encoder. For simplic- ity of implementation and interpretation, we use only a single attention head. For the NVIB models,
Models
3 Related Work
Modeling language at the level of characters has the advantage of providing an end-to-end frame- work for the models to operate, without the need for tokenization as a preprocessing step (Xue et al., 2022; Ataman et al., 2020; Choe et al., 2019; Al- Rfou et al., 2019; Kawakami et al., 2017). This

---

F1
Transformer
95.51
56.51
64.52
NVIB
85.23
79.02
78.86
Table Word segmentation performance [%]]
level retains only average of ~ 35% of vectors.
This is because the stronger NVIB regularisation higher layers encourages the grouping of correlated characters, to reduce redundant information, and the strongest correlations are within words. We provide further examples in Appendix E.
We quantify the resemblance of the final-layer self-attention maps to words by extracting contigu- ous segments from the maps and computing the Fl measure between our segments and the words the sequence. In particular, we find the best align- ment between words and segments and compute the number of characters in the longest common substring between word and its corresponding discovered segment." Table 1 compares the per- formance of our model to the Transformer base- line. This impressive unsupervised performance (Fl of 78.86%) concurs with the attention visual- isations and quantitatively verifies that our model has learned to abstract to the level of words.
Figure 2: Self-attention patterns the layers of 6-layer Transformer encoders from bottom to top. Left: Standard self-attention. Right: With NVIB regularisa- tion. Sentence: "Whatever you are, be good one." Dark purple is 0 and light yellow is I for attention.
we only apply NVIB to the final three layers. To ensure comparability between our model and the baseline, we train the baseline to have the same denoising capability and thus the same validation cross-entropy when evaluated noised examples. For further details see Appendices and
4.3 Probing Analysis
This section uses different probing tasks to quan- titatively evaluate the abstraction capabilities of our model and analyse the linguistic information captured by the layers.
4.2 Attention Map Visualisations and Analysis
4.3.1 ArXiv Topic Classification
To qualitatively evaluate the model's ability to learn interpretable abstractions, we visualise the self-attention maps. Figure 2 compares the self- attention patterns of the the last 3 layers of: Transformer with 6 layers of standard attention (left); and Transformer with 3 layers of standard attention followed by layer of denoising attention with NVIB (right).
The ArXiv topic classification task (Hofmann al., 2022) is challenging task consisting of short in- put sentences with long technical words. For each subject, the classifier should classify the topic into 20 possible sub-areas. Following Behjati and Hen- derson (2023), we train an attention-based probe on the final layer of the models and report the F1 measure for performance on the ArXiv-I. dataset. Without finetuning the models, this classification task serves as probing high-level abstract linguistic properties (Hewitt al., 2021). As shown in Table 2, the NVIB layer results in the model learning more information about the meaning and semantics in the abstract representations than characters and therefore provides better units for performing the task.
Despite being trained solely on noisy reconstruc- tion at the character level, the NVIB layers com- press the self-attention representations through the layers into distinct groups. At lower levels, the model uses nearly all vectors (i.c. 99%) and di-
learns position-local information, shown agonal pattern. At higher levels the model drops some vectors (the blank columns) and groups char- acters (the vertical bars) in ways which strongly resemble subword units or even words. The last
"Sec Appendix I for further details and exact formulas.

---

Task
Transformer
NVIB
Mathematics
Computer science
44.02
42.33
47.13
44.47
9.4
-10
125
Physics
18.83
52.32
Average
45.06
47.97
NVIB Transfomer
0.5
5.0
0.4 0.4
0.5
088
1.0
5.9
0.2
24 Probobiity
0.6
0.3
1.2
Probability
Table 2: Fl score [%]] on Arxiv-L classification task.
Figure 4: Robustness plots showing relative perfor- mance change over increasing input perturbations.
BShit
Ratalive teak accurscy (N)
Coordhiv Tense
TepDonst
noise than a standard Transformer, with increased advantage as the level of noise increases.
10-
5 Conclusions
We propose a novel method for inducing abstract representations of text. We adapt the Nonparamet- ric Variational Information Bottleneck (Henderson and Fehr, 2023) regulariser for application to self- attention in the stacked layers of Transformer encoder. Our model learns how many vectors are needed each layer, thereby inducing different levels of abstraction in different layers of the same model. We find that these abstract units are intu- itive, more robust, and better at encoding semanti- cally and linguistically meaningful information.
Figure Relative performance of NVIB over Trans-
former for a subset of SentEval tasks.
4.3.2 Linguistic Probing
The SentEval task set is specifically designed to examine the linguistic information available a sentence representation at different levels, ranging from surface-level to semantic-level tasks (Con- neau et al.. 2018; Conneau and Kiela, 2018). We probe for linguistic information of our model and the baseline Transformer, across all layers. In gen- eral, the performance improves in deeper layers and increases further with the inclusion of NVIB in the layers. We highlight the results of four tasks in Figure 3, which to perform well in these tasks the representations must capture latent syntactic struc- tures (BShift), cluster them by constituent types (TopConst), or have an understanding of semantics (Tense) or broad discourse and pragmatic factors (CoordInv) (Conneau et al., 2018). The inclusion of our NVIB layers increases the relative perfor- mance over the Transformer baseline, showing be more linguistically informed. The complete set of results is in Appendix Table 4.
Limitations
While the models and training data are reasonable in size, the experiments do not include the very large scale training often found in work on repre- sentation learning in text. We anticipate that the advantages of NVIB on self-attention layers will only increase as the models and data are scaled up. since this should allow even more abstract represen- tations to be Icarned. In addition, the experiments are only done on English, but we would expect more improvements with more morphologically rich languages. In future work we plan to explore fine-tuning NVIB for sparsity and downstream per- formance, and consider different tokenizations be- yond characters only.
4.4 Robustness Analysis
Ethics Statement
We analyze the robustness of our models to syn- thetic noise injected into the input sequences (Belinkov and Bisk, 2017: Durrani et al., 2019). Namely, we evaluate the reconstruction quality when the inputs are perturbed by swapping, delet- ing, inserting, and substituting characters (Morris et al., 2020). We expect our model to be more ro- bust due to its compressed representations. Figure shows that our model is more robust to adversarial
We foresee no ethical concerns with our work.
Acknowledgements
Both Melika Behjati and Fabio Fehr were sup- ported by the Swiss National Centre of Competence in Research (NCCR) under the project Evolving Language, grant number "5INF40 1808888.

---

References
Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy
Guo, and Llion Jones. 2019. Character-level lan- guage modeling with deeper self-attention. In Pro- ceedings of the AAAI conference on artificial intelli- gence, volume 33, pages 3159-3166.
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. 2019. One size does not fit all: Comparing nmt representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1504-1516.
Alexander A. Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy. 2017. Deep variational information bottlencck. In International Conference on Learning Representations.
Duygu Ataman, Wilker Aziz, and Alexandra Birch. 2020. A latent morphology model for open- vocabulary neural machine translation. In Inter- national Conference Learning Representations (ICLR).
Michael Figumnv, Shakir Mohamed, and Andriy Mnih. 2018. Implicit reparameterivation gradients. Neu- ral Information Processing Systems.
James Henderson and Fabio James Fehr. 2023. A VAE for transformers with nonparametric variational in- formation bottleneck. In The Eleventh International Conference on Learning Representations.
Melika Behjati and James Henderson. 2023. Induc- ing meaningful units from character sequences with dynamic capacity slot attention. Transactions on Machine Learning Research.
John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher Manning. 2021. Conditional probing: measuring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine transla- tion. arXiv preprint arXiv: 1.1.02.
Dokook Choe, Rami AI-Rfou, Mandy Guo, Heeyoung Lee, and Noah Constant. 2019. Bridging the gap for tokenizer-free free language models. arXiv preprint arXiv:1908. 10322.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural net- works. In International Conference on Learning Rep- resentations.
Jonathan H. Clark, Dan Garrette, Iulia Ture, and John Wieting. 2022. Canine: Pre-training efficient tokenization-fre encoder for language representa- tion. Transactions of the Association for Computa- rional Linguisties, 10:73-91.
Valentin Hofmann. Hinrich Schuetze, and Janet Pierre- humbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained lan- guage model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers). pages 385-393, Dublin, Ireland. Association for Computational Lin- guistics.
Alexis Conneau and Douwe Kiela. 2018 SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LRFC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning create and reuse words in open- vocabulary neural language modeling. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1492-1502, Vancouver, Canada. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single S&!#" vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 2126-21366 Melhourne, Aus- tralia. Association for Computational Linguistics.
Diederik P. Kingma and Max Welling. 2014. Auto- encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Conference Track Proceedings, Banff, AB, Canada.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System

---

Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- lerly, 2(1-2)88-977
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal process- ing (ICASSP), pages 5149-5152 IEEE
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1725, Berlin, Germany, Association for Computational Lin- guistics.
BART: Denoising eequence-to-seqeene for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguistics.
Yi Tay, Vinh Q. Tran. Sebastian Ruder, Jai Gupta. Hyung Won Chung. Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. Charformer: Fast character transformers via gradient- based subword tokenization. In International Con- ference on Learning Representations.
Liyua Liu, Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In International Conference on Learn- ing Representations.
Francesco Locatello, Dirk Weissenbom, Thomas Un- terthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33: 11538
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards token-free future with pre- pre-treined byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics, 10:291-306.
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Pro- ceedings, Toulon, France. OpenReview.net.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A frame- work for adversarial attacks, data augmentation, and adversarial training in nip. Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126.
A Training Details
General Training All models are trained, with- out pretraining, using the same encoder and de- coder configuration for comparability. Our encoder size is defined by the base Transformer ( (Vaswani et al., 2017) such that we have a six layer Trans- former encoder. However, we use a two layer de- coder to ensure the task is not learned in the de- coder alone. We use a single attention head. The size for the word embedding vectors and model projections are 512 and feed forward dimensions 512, which leads models of approximately 12-17 million trainable parameters. An English charac- ter level tokeniser is used for tokenisation with a vocabulary of approximately 100 characters. Dur- ing training we use: learning rate of le with a cosine cool over all steps, RAdam optimiser (Liu et al., 2020) with mixed precision (FP16), batch size of 512, gradient norm clipping 0.1 and trained for 55 epochs (N2 8K steps). The number of steps were selected considering model conver- gence and minimising computation time. We use a dropout rate of 0.1. The input is noised at each
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Tiago Pimentel, Josef Valvoda, Niklas Stoehr, and Ryan Cotterell. 2022. Attentional probe: Estimating a module's functional potential. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 1145--14477 1472, Abu

---

batch with probability of character deletion of 0.1. Each model takes approximately 2.5hrs on single NVIDIA GeForce RTX 3090.
hyperparameters are reported in Table where the validation cross-entropy (CE) is matched for NVIB and baseline Transformers.
NVIB Training Training the models with the NVIB layers requires regularising the representa- tions. The introduction of the exponential activa- tion function (as opposed to ReLU) for the psuedo- count parameter requires threshold test time be exactly zero. We use a threshold for this at 0.1. During training and testing we enforce bottleneck between the encoder and decoder by masking the nal encoder representations by the aforementioned threshold.
Transformer
NVIB
NVIB layers
3
A
le
AD
1
a Training Steps
0.25
2.5K
8K
Val. CE
0.19
0.19
Table 3: Hyperparameters for final models evaluated.
The NVIB hyperparameters λc ÅD and A are selected through hyperparameter tuning. However, during training we only sample once from each component thus the approximation parameter is set to ق = 1. We use Kullback-Leibler annealing divergence strategy where the introduction of the KL divergence loss linearly introduced between 30% 60% of the training steps. This allows the model to learn initial representations, slowly intro- duce the regularisation and finally learn through the compressed latent representation.
The encoders 6 layers are inspired by the base model of Vaswani et al. (2017). For the Trans- former decoder use only layers such that the decoder is not able overpower the embeddings from the encoder it sees through cross attention.
NVIB Configuration For the NVIB layers dur- ing experimentation we considered: All layer in- cluding NVIB; the last layers including NVIB; and only the final layer including NVIB. When all layers were included it was challenging to get both compression and performance as the regularisation too strong. Only regularising the last layer managed reduce the number of vectors but often converged single sentence vector with lower, non-comparable validation cross-cntropy. Finally, we settled on only regularising the last layers as gave the model enough flexibility in the lower layers and progressive compression in the higher layers.
B Hyperparameter Tuning
The models are trained on the Wikitext-2 training dataset using the loss from Equation 2. They are tuned on the validation dataset with the aim to be able to reconstruct the character sequence from noisy input. Following from (Henderson and Fehr, 2023) we the weights of the Gaussian and Dirichlet KL divergences be independent of the sentence length n and dimensionality of vectors
Ap = 1 -D= Ao= =
-= _X
C KL Divergence Loss
d
Henderson and Fehr (2023) define the Kullback- Leibler divergence for NVIB with two terms: the Lp for the Dirichlet distribution weights defined by ar; and the T.G for the distribution of vectors generated by the Gaussian components. We set the approximation parameter Ko 1. This gives us the following loss terms for the KL divergence, where I' is the gamma function and &e is the digamma function:
where Xp and Ng are fixed hyperparameters. All combinations of the following hyperparameters were considered in grid search for the respective models:
Ir = {le-4, 11--3
N/4 = {le 5, 4, 1e le 2}
XD [le 2 1c 1, 1}
a = (0, 0.05. ..., 0.45, 0.5}
Ln log [(ag) log [(ag')
where X/ and Xn are the weights on the Gaussian and Dirichlet KL divergences. The A represents the conditional prior parameter. The final models'
+ (a af') (((((a)) + ((a))) (4)
(log[(af) log ((g(())))

---

where, ajj is the sum of all a parameters gener- ated by the NVIB layer. The conditional prior of ag no is controlled by og = and extra pseudo-counts defined by the length and hyperparameter A The KL divergence between two Gaussians (with diagonal covariance with val- ues and weighted by the a parameters) is:
where 1p is row vector of p ones and (a")² (vd + (or)2).
E Visualisations
In Figures 5 to we include additional visualisa- tions of the ceff-attention weights.
n+1
d
(w), - ph)
Lo = 2
(o
A=1
(5)
(oth) )2
(oih)
ih (a))
log
(a
D Denoising attention function
Henderson and Fehr (2023) generalise the set of vectors input to an attention function probabil- ity distribution over vectors, and generalise atten- tion to a function of these probability distributions called denoising attention.
Training function During training, the set of sampled vectors Z Ruxp and their sampled log-probability weights log(T)) Rix= are both output by the NVIB layer, thereby specifying the sampled mixture distribution F. set of query vectors E RRx is projected into key space by the grouped matrices Wº, W E Roxd to u = (u' (W'))(W')))) The keys' dimensionality d is used for scaling. Denoising attention can then be computed as:
Figure 5: Self-attention patterns of the last layers of 6-layer Transformer encoders. Left: Standard self- attention. Right: With NVIB regularisation. Sentence: "I think therefore am." Dark purple and light yellow I for the attention values.
softmax
Vau uz (log(T))
1 2vd
2
Z
F Probing Classifiers
(6)
We used two types of probing classifiers to perform our tasks. First, we employ an attention- basedd probing classifier to operate on top of the set of representations for predicting the specified prop- erty. This would be similar to having learnable [CLS] token which used as the representation of the sequence in BERT-like models. This is also in line with the findings of Pimentel al. (2022) that the way we do the probing should resemble how the model itself would use the information within its architecture. In particular, we first map the rep- resentations into new space with 2 layer ML.. Then, compute the attention with Icarnable query vector. Finally, we linearly project the result- ing vector into the number of classes for each task. We refer to this probe as Attention-basel probe. Sccond, we tried a less complicated and more com- mon type of probing in which we first aggregate
For self-attention, we define the original query vec- tors u' to be the set of vectors input to the NVIB layer, before projecting to DP parameters and sam- pling.
Testing function During test time, do not sample F, but instead use the mean of the posterior distribution. The test-time denoising attention can then be computed as:
(ul
((
1l log( ao
softmax
[o')3
1 14
2) T
1p (log(o))))
(7)
2 o"
(a))
Vd
(1 (
(ar)
(ar)² @

---

Figure 6: Self-attention patterns of the last 3 layers of 6-layer Transformer encoders. Left: Standard self- attention. Right: With NVIB regularisation. Sentence: "Wow, it's abstracting." Dark purple is O and light yellow is for the attention values.
Figure 8: Self-attention patterns of the last layers of 6-layer Transformer encoders. Left: Standard self- attention. Right: With NVIB regularisation. Sentence: "I took the one less travelled by, and that made all the difference." Dark purple is 0 and light yellow is for attention values.
perform the task. We refer to this probe as Aggre- gating probe.
G SentEval Tasks
G.1 Model
We employ the Aggregating probe for perform- ing this task. We froze our models and trained the probes for 10 epochs with a batch-size of 128. The hidden dimension for the probe set to 256. We trained the model with Adam optimizer with a learning rate of 1e 4. We report the test set accuracy for the best best-performing model in terms of validation accuracy.
G.2 Supplementary Results
We report the results in Table 4 on subset of the 10 SentEval tasks as sentence length (SentLen). word content (WC) and semantic odd man out (SOMO) tasks are too challenging for our mod- els when encoding from a character level.
Figure 7: Self-attention patterns of the last 3 layers 6-layer Transformer encoders. Left: Standard self- attention. Right: With NVIB regularisation. Sen- tence: "Thats one small step for a man, giant leap for mankind." Dark purple and light yellow is for the attention values.
H Arxiv Classification Task
Our goal here is to compare the representations and not have ultimate performance in the task, thus we do not fine-tune the models. Hence, we only
the set of representation vectors by mean and then apply a 2 layer MLP with ReLU non-linearity to

---

Layer
CoordInv
ObjNum
TreeDepth
TopConst
BShift
Tense
SubjNum
Chance
0.3
0.5
0.125
0.05
0.5
0.5
0.5
0.5023
0.6498
0,2200
0,2880
0,5006
0.7306
0.6500
2
0.5144
0.7255
0,2350
0.3721
0.4994
0.7891
0.7131
3
0.5190
0.7547
0,2594
0.4261
0,5055
0.8263
0.7297
Transformer
4
0.5196
0.7687
0.2692
0.4368
0.5108
0.8114
0.7545
0.5196
0.7737
0.2736
0.4369
0.5304
0.8320
0.7435
0.5227
0.7756
0.2736
0.4212
0.5465
0.8384
0.7683
0.5037
0.7646
0.2349
0.3323
0.5007
0.8344
0.7285
2
0.5069
0.7859
0,2511
0,4243
0.5108
0.8379
0.7777
3
0.5110
0.7963
0.2589
0.4453
0.5466
0.8606
0.7844
NVIB
4
0.5111
0.7879
0,2655
0.5290
0.5361
0.8481
0.7943
0.5299
0.7660
0.2651
0.5283
0,5571
0.8371
0.7793
0.5523
0.8207
0.2923
0.5766
0.6075
0.8531
0.8038
Table 4: Performance Senteval tasks.
evaluated our models on the large division of the task, i... ArXiv-L which consist of 1000 samples for cach area leading to 20000 samples in total. We employ the Attention-based probe to perform this task as it is quite challenging task which requires the information in the vectors to be bet- ter managed by the Attention mechanism and also more similar to the way the model itself would per- form the task. The hidden dimension of the MLP is set to 256 and the query, key, and value matrices are set to the same dimension as the model dimen- sion, namely, 512. We train the classifier with batch size of 256 for 50 epochs with Adam opti- mizer with learning rate of le 3. Following Hofmann al. (2022) we report the test Fl for the best-performing model terms validation F1.
(P), Recall (R), and F1 measure as
P longest common substring length segment length
(8)
and
R longest common substring length
(9)
word length
We reported the average macro F1, P. R over the validation set of our training data. For the base- line Transformer, as it usually predicts units length one or two which are within single word the P would be high as opposed to its R value.
I Quantification of Word Resemblance
We observe a strong resemblance between words and the vertical bands in the final-layer Attention maps of the NVIB integrated model. Therefore, quantify this similarity follows. First, take the argmax over the Key dimension of an Attention map and extract the contiguous segments from the resulting vector. Then, we compute the intersection between the set of obtained segments and the set of words in the sequence. In particu- lar, for each segment and word, we compute the number of intersecting characters (i.e., the length of the longest common substring) as measure of their overlap. This would lead to rectangular ma- trix of scores. Then, we perform the Hungarian matching algorithm (Kuhn, 1955) to find the best 1- match between the two sets. Afterward, for each matched word and segment, we compute Precision


Reviews:
[{"review_id": "a2ZqTECaJN", "reviewer": "Reviewer_YN7E", "summary": "", "questions": "1. How is the decoding and training efficiency of this model compared with sub-word based language model? Especially for current LLM era where sequence length is extremely important.\n2. How is the expected performance of this method on other languages such as Japanese and Chinese which has a much bigger character size?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer YN7E for taking the time to review our short paper, but it appears that the reviewer does not have an accurate perception of the nature and scope of our claimed contributions. As explained in the **Contributions** paragraph on page 1, we claim to have contributed:\n1. A novel application of NVIB [(Henderson and Fehr, 2023)](https://openreview.net/pdf?id=6QkjC_cs03X) to stacked self-attention layers of a Transformer encoder. \n2. Applying NVIB to self-attention in stacked layers of a Transformer encoder leads to greater abstraction in the encoder - Shown  qualitatively with attention plots and quantitatively in F1 segment overlaps. \n3. Semantically meaningful representations - Shown with both ArXiv Topic Classification and SentEval probing tasks.  \n4. Robustness - Shown with input perturbation plots.\n \nIn light of this, we would like to clarify the following points: \n1. We did not introduce NVIB; we only applied it to stacked self-attention layers of a Transformer encoder. \n2. NVIB is not a regularisation term but a information theoretic regularisation for the entire latent representation of Transformers. \n3. We are not trying to show practical effectiveness on NLP tasks, but to test our scientific hypotheses.\n\nAccording to the [ACL reviewing guidelines](https://2023.aclweb.org/blog/review-form/), Soundness is defined by *\"Does the paper clearly state scientific claims and provide adequate support for them?\"*.  Since Reviewer YN7E does not provide any judgement against our specific claims, we feel the **Soundness score of 1 is unjust.**  Proposing different claims that the reviewer thinks we should have made and then arguing that we did not show them is not relevant for Soundness.\n\nWe shall now address the weaknesses and concerns of the Reviewer:\n\n> *The experimental section is relatively simple.*\n\nYes, the simple experimental design is important and a strength. The experiments are designed to find a minimal model to prove our hypothesis - **Applying NVIB to self-attention in stacked layers of a Transformer encoder leads to greater abstraction in the encoder**. We conducted a controlled experiment to compare the effect of introducing NVIB on Transformer Encoders to Transformer Encoders without NVIB.\n\n> *Comparing their method with more baselines, including sub-word based methods.*\n\nThis paper is not claiming absolute improvements on the various evaluation measures, but instead is investigating an unsupervised method for learning abstract representations.  As such, predefined levels of abstraction such as subword models are not relevant points of comparison. We instead report a controlled experiment to compare the effect of introducing NVIB on Transformer Encoders to Transformer Encoders without NVIB. To the best of our knowledge, a plain Transformer Encoder (the null abstraction) is the only baseline that is relevant to learned abstractions. \n\n**Preliminary Additional Experiment** That said, for the specific case of ArXiv topic classification, reporting results for a subword baseline would provide relevant context.  We conducted a preliminary experiment with BPE-based model. In particular, we trained a standard Trasformer autoencoder baseline with BPEs as inputs. We then performed the same experiment as described in section 4.3.1. Provisionally, we found that a BPE-based model achieved only 25.13% average F1, much worse than the reported results. We suspect this is due to the nature of words in the task is different to that of the tokenizer, which is exactly where our model shows its advantages. These results are provisional and would require more investigation to ensure precise comparability.\n\nWhile it would be possible to use subword models as a base model on which we apply NVIB, it is not clear what level of abstraction we should expect to find.  In contrast, it is relatively clear that the next level of abstraction above the character level is something like morphemes or words.  For this reason, we conducted all our experiments starting from character-level inputs.\n\n> *The paper does not provide sufficient information on the decoding and training efficiency.*\n\nWe agree that a detailed analysis of the computational efficiency would be useful.  In theory, there should be a substantial improvement in the $O(N^2)$ complexity of attention because NVIB substantially reduces an $N$ (by dropping key vectors) at higher layers of the Transformer.  However, given the scope of this short paper, the fact that we submitted to the interpretability track and our claims, we feel this is not a weakness. \n\n**Additional Analysis** For completeness we ran extra experiments with our current implementation to access the computational costs overhead of NVIB. Including the NVIB layer adds 33\\% more parameters per attention layer and only adds **10\\% more parameters** to our whole model. We compared the computational cost of including NVIB layer and attention over standard attention. We used input lengths of 8, 512 and 4096 and found **less than 1\\% increase in computational costs** during: training and inference; across all lengths; and across all metrics (memory, FLOPS and GPU time). Hence the computation is still dominated by $O(N^2)$, for our current implementation. Considering the full model, over a single epoch, NVIB is actually **20\\% faster during training** and **15\\% faster during autoregressive decoding** than the full Transformer baseline. However, due to our KL annealing training schedule and regularisation we found that training with NVIB required more steps to reach a similar validation cross-entropy (Appendix A, Table 2).\n\n> *Only evaluates the proposed model on English text.*\n\nWe agree that it would be interesting to check the effectiveness in capturing the linguistic properties of different languages. Our limitations state *\"In addition, the experiments are only done on English, but we would expect more improvements with more morphologically rich languages\"*.  But additional languages are not required to demonstrate the existence of NVIB models which support our claims.  As stated in the [ACL reviewing guidelines](https://2023.aclweb.org/blog/review-acl23/#1-acl-2023-review-form): *\"Any other extra experiments are in the \"nice-to-have\" category, and belong in the \"suggestions\" section rather than \"reasons to reject.\" This heuristic is particularly damaging for short papers.\"*  Also, according to [Reviewing for EMNLP](https://2020.emnlp.org/blog/2020-05-17-write-good-reviews), this is an invalid bases for rejecting a paper: *\"We care about NLP for any language.\"*\n\n> *How is the expected performance of this method on other languages such as Japanese and Chinese which has a much bigger character size?*\n\nThe fact that these characters (for Japanese and Chinese) represent larger chunks of language makes them less interesting for our experiments for the same reason given above for not using subword inputs, namely that it is less clear what the larger abstract units should be. Also, these languages are not morphologically rich and vocabulary size would not affect the proposed attention regularisation mechanism. They would be interesting languages for any future work which addresses the usefulness of NVIB at higher levels of abstraction or for practical downstream tasks, for example for finding sentence groupings for summarisation or question answering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include the introduction of a novel regularization term for character based language representation model, Nonparametric Variational Information Bottleneck (NVIB), that can learn to compress to different levels of abstraction at different layers of the same model, resulting in a more linguistically informed and robust model. The paper also presents experimental results that demonstrate the effectiveness of NVIB in various NLP tasks, including classification and linguistic probing.", "reasons_to_reject": "The paper has some weaknesses. Firstly, the experimental section is relatively simple, and the authors could benefit from comparing their method with more baselines, including sub-word based methods, to demonstrate the benefits of their approach more convincingly. This would provide a more comprehensive evaluation of the proposed model and its effectiveness in comparison to other state-of-the-art methods. \n\nSecondly, the paper does not provide sufficient information on the decoding and training efficiency of the proposed model compared to sub-word based language models. This is particularly important in the current era of large language models where sequence length is a crucial factor. A more detailed analysis of the computational efficiency of the proposed model would be useful to understand its practicality and scalability in real-world applications.\n\nFinally, the paper only evaluates the proposed model on English text, and it would be interesting to see how it performs on other languages such as Japanese and Chinese, which have a much larger character size. This would provide insights into the generalizability of the proposed model and its effectiveness in capturing the linguistic properties of different languages.", "questions_to_authors": "1. How is the decoding and training efficiency of this model compared with sub-word based language model? Especially for current LLM era where sequence length is extremely important.\n2. How is the expected performance of this method on other languages such as Japanese and Chinese which has a much bigger character size?", "paper_topic_and_main_contributions": "This paper is about introducing a novel language representation model called Nonparametric Variational Information Bottleneck (NVIB) that can learn to compress to different levels of abstraction at different layers of the same model. The paper addresses the problem of the high cost of learning textual embeddings, which are tokenization specific and require different models to be trained for each level of abstraction. The main contribution of this paper is the introduction of NVIB, which can learn to compress to different levels of abstraction at different layers of the same model, resulting in a more linguistically informed and robust model. The paper also presents experimental results that demonstrate the effectiveness of NVIB in various NLP tasks, including classification and linguistic probing.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QYUzLpmTai", "reviewer": "Reviewer_LT26", "summary": "", "questions": "It’s interesting how NVIB often organized latent representations around words/subwords in your experiments. I wonder if you discovered any larger constructions than words. Does the size of the discovered units continue to become larger as you increase the regularization or does it plateau after a certain point?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the time and effort from Reviewer LT26 to review our mathematics and model design. We would like to thank the reviewer for the kind remarks by highlighting our **interesting results** and our outlining our contributions in the reasons to accept. We address the reviewer's concerns and questions below. \n\n> *More justification and discussion of specific design choices*\n\nWe are happy that the reviewer is so interested in the properties of our proposed model, but would like to emphasise that our first priority for this short paper is to demonstrate the existence of a model which proves our hypothesis - **Applying NVIB to self-attention in stacked layers of a Transformer encoder leads to greater abstraction in the encoder.**   Conducting a more thorough ablation study of the range of model design choices would have required a long paper.  When our first result has been accepted, then we will be in a position to explore the range of effective models in future work.  \n\nThat said, to the extent that space permits, we would be happy to add more justification and discussion of specific design choices in future versions of this paper.  In particular, based on our experience with exploring the space of models so far, the very interesting specific questions raised by the reviewer can be addressed, as follows. \n\n> *Does it matter that the skip connection is multiplicative rather than additive?*\n\nYes. The multiplicative skip connection of NVIB Dirichlet pseudo-count parameters $\\boldsymbol{\\alpha}^{(l)}$ from the previous layer $(l-1)$ was included for two reasons: \n1. **Overflow** The pseudo-count parameters $\\boldsymbol{\\alpha}$ control the noise in sampling during training and bias the attention weights. During initial training steps with KL annealing the model is incentivised to make the $\\boldsymbol{\\alpha}$ larger. This is due to low initial regularisation in KL annealing. For overflow and precision the original NVIB implementation kept the $\\boldsymbol{\\alpha}$ in log space, thus we thought it would be better to keep the skip connection in log space. \n2. **Magnitude** The skip connection was put in place to allow communication in attention between layers and guide the abstraction. Since we considered only 3 layers of NVIB a multiplicative skip connection has a greater magnitude than an additive skip connection. Considering an additive skip connection is a valuable suggestion and we suspect that this may be helpful when considering many more connected NVIB layers.\n\n> *Does it matter to choose a strictly increasing strength of regularization as layers get higher?*\n\nYes. During initial exploration of the regularisation we found a uniform regularisation strength across layers did not lead to the same abstraction. Often the lower levels were penalised too strongly. We also considered a regularisation that would double in strength through the layers, which is more in line with the multiplicative skip connection. This regularisation was too strong and lead to the final layer being compressed to a single sentence representation.\n\n> *Does the size of the discovered units continue to become larger as you increase the regularization or does it plateau after a certain point?*\n\nYes. The size of the representations become larger until all units are compressed into a single vector, much like a sentence embedding. This single vector made it challenging to get competitive reconstruction cross-entropy during validation. If you over-regularise, the representations collapse to the uniformed prior representation from NVIB (as is typical of VIB models). The model will then try to solve the task with the decoder alone.\n\n> *It may help readability to give a little focus to the description of the big picture with NVIB in section 2.1.*\n\nWe agree that writing Section 2.1 within the space constraints was very challenging, and welcome suggestions for improvement.  We intend to follow the advice from other reviewers to add more detail to this section, which is currently in the Appendix C and D.  We agree that keeping in mind a focus on the relevant properties of NVIB will be important when expanding this section.  If Reviewer LT26 has a different interpretation of \"focus\" in mind, we would welcome more comments on this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The authors propose adaptations of NVIB to self-attention and to use it for learning increasingly abstract representations at higher layers of a transformer encoder. (NVIB was originally introduced for cross-attention.)\n2) The authors presented interesting results from intrinsic evaluation of the attention distributions of such a model pretrained on Wikitext-2 and from evaluating the quality of the representation in linguistic probing and text classification.\n3) The authors compared the robustness of the NVIB-regularized model with that of a regular transformer and found the former more robust to noisy input.\n", "reasons_to_reject": "1) The specific design choices made in adapting the original NVIB to self-attention could benefit from more justification and discussion. \ne.g. Does it matter that the skip connection is multiplicative rather than additive? Does it matter to choose a strictly increasing strength of regularization as layers get higher? These could possibly be answered with ablation studies.\n", "questions_to_authors": "It’s interesting how NVIB often organized latent representations around words/subwords in your experiments. I wonder if you discovered any larger constructions than words. Does the size of the discovered units continue to become larger as you increase the regularization or does it plateau after a certain point?", "paper_topic_and_main_contributions": "This paper proposes to learn text representations at multiple levels of abstraction by applying a sparsity-inducing attention regularizer (NVIB) to the self-attention layers of a character-input transformer encoder.\n\nList of contributions: NLP engineering experiment, reproduction study, publicly available software (upon publication).\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It may help readability to give a little focus to the description of the big picture with NVIB in section 2.1. "}, "extra_text": {}}, {"review_id": "QqGQRp0R7Z", "reviewer": "Reviewer_gprF", "summary": "", "questions": "Question A: Is there a reason why you did not run experiments on language modeling? I understand that the current experiments control for some models to be trained better than others, but it would be interesting if this method also gestures towards better language models as a whole.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer gprF for their time taken to understand our contributions and their encouraging remarks. We are honoured that they found our presentation: **very well written**, **clearly explained**  and **well described**. Our Results: **encouraging** with a **very interesting finding**.  Our idea: **interesting** and could lead to **better representation learning as a whole**. We address the reviewer's concern below. \n\n> *It would perhaps be interesting to run some more experiments, in particular to see if NVIB regularization improves language modeling.*\n \nWe agree it would be interesting to run experiments on language modeling, which we are in fact currently working on. However, our experience is that language modeling requires a rather different architecture from the encoder-decoder architecture we use here (such as the requirement of a causal mask).  It would not be possible to add such experiments to this short paper.  In addition, we believe our existing experiments more directly support our claims. We appreciate the encouraging suggestion and will explore this in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written. Their idea and contribution is explained clearly, and the NVIB regularizer, alongside the changes made to adapt it to stacked self-attention, are well described.\n\nThe idea is interesting, and the results are encouraging. The fact that the attention maps in higher layers better align with words, for example, is a very interesting finding, and the probing results suggest NVIB could lead to better representation learning as a whole.", "reasons_to_reject": "It would perhaps be interesting to run some more experiments, in particular to see if NVIB regularization improves language modeling.", "questions_to_authors": "Question A: Is there a reason why you did not run experiments on language modeling? I understand that the current experiments control for some models to be trained better than others, but it would be interesting if this method also gestures towards better language models as a whole.", "paper_topic_and_main_contributions": "This paper extends previous work by Henderson and Fehr (2023), which introduced the non-parametric informational bottleneck (NVBI) regularizer, to self-attention layers in generic transformers. This method consists of replacing the traditional self-attention mechanism with a Dirichlet process mixture model, where every vector is associated with a Gaussian mixture component, and whose mixture weights are drawn from a Dirichlet distribution. Crucially, the regularizer is setup to incentivize the sparsity of the mixture weight distribution. They train traditional and NVIB-regularized character-level transformers on Wikitext-2, and find that the attention weights at higher layers of the NVIB-regularized model better align with linguistically meaningful units of text (e.g., word), and in probing experiments they find they are better at capturing topic (on Arxiv topic classification task) and linguistic information (on SemEval), as well as being more robust to perturbations of the input sequence.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YwvIMRdtIq", "reviewer": "Reviewer_2hkB", "summary": "", "questions": "The presentation should be improved to make it easier to follow. For example, the right part in Figure 1 is not easy to understand. Does it mean different attention patterns at three different layers? In Figure 2, which color is “lighter” is a challenging task to tell.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the constructive comments from Reviewer 2hkB, and for taking the time to understand our contributions. We would like to thank the reviewer for the kind remarks by highlighting our **interesting** research topic,  our **qualitative and quantitative experiments**, and **easy-to-follow** presentation.\n\nWe believe the constructive concerns of the Reviewer can be addressed and included in future versions of the paper (or Appendix). \n\n>*Method: why applying NVIB to Transformer is challenging?*\n\nWe believe that one of our contributions is to take the complex theory (and code) from [Henderson and Fehr (2023)](https://openreview.net/pdf?id=6QkjC_cs03X) and frame its application to self-attention in a way which appears to not be challenging. We also make general improvements to NVIB through changing the sampling to implicit reparameterisation gradients [(Figurnov et al. 2018)](https://proceedings.neurips.cc/paper_files/paper/2018/file/92c8c96e4c37100777c7190b76d28233-Paper.pdf)\n.  That said, we agree that, due to space constraints, we were not able to convey the range of alternatives which we had to consider when developing a successful application of NVIB to self-attention in stacked layers (see the next paragraph).  Instead we focused on presenting one model which was successful, which includes the layer-wise communication and a new loss structure to encourage the hierarchical abstraction. Finding this model was in fact challenging, which we will try to convey in future versions of the paper.\n\n>*Experiment: a more detailed explanation or exploration of why only these layers were chosen and how different configurations might impact the results.*\n\n**Hypothesis** Our experiments are designed to demonstrate the existence of a model which proves our hypothesis - **Applying NVIB to self-attention in stacked layers of a Transformer encoder leads to greater abstraction in the encoder.**   We did not include an ablation study of this proposed model simply because doing so would have required making this a long paper, and we believe that a better scientific strategy is to first demonstrate that this kind of architecture is effective and then study its properties in more detail in future work.  \n\n**Design** That said, we would be happy to add more motivations for our design decisions in future versions of the paper.\nFollowing from [Henderson and Fehr (2023)](https://openreview.net/pdf?id=6QkjC_cs03X) and [Lewis et al. (2020)](https://aclanthology.org/2020.acl-main.703.pdf)\n we designed a denoising reconstruction objective with a Transformer encoder-decoder structure. In *Appendix A* we define all the model details and that the encoders 6 layers are inspired by the base model of [Vaswani et al. (2017)](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf). For the Transformer decoder we make the design choice to only use 2 layers such that the decoder is not able to overpower and ignore the encoders embeddings from cross attention. \n\n**Configuration** For the NVIB layers during experimentation we considered: All layer including NVIB; the last 3 layers including NVIB; and only the final layer including NVIB. When all layers were included it was challenging to get both compression and performance as the regularisation was too strong. Only regularising the last layer managed to reduce the number of vectors but often converged to a single sentence vector with lower, non-comparable validation cross-entropy. Finally, we settled on only regularising the last 3 layers as it gave the model enough flexibility in the lower layers and progressive compression in the higher layers. We do not think this was an exhaustive evaluation, but found a minimal configuration that proved our hypothesis.\n\n>  *Baseline: more advanced models*\n\nThis paper is not claiming absolute improvements on the various evaluation measures, but instead is investigating an unsupervised method for learning abstract representations.  As such, the predefined levels of abstraction used in current advanced models, such as subword models and stride-based downsampling, are not relevant points of comparison. We instead report a controlled experiment to compare the effect of introducing NVIB on Transformer Encoders to Transformer Encoders without NVIB. To the best of our knowledge, a plain Transformer Encoder (the null abstraction) is the only baseline that is relevant to learned abstractions. \n\n**Preliminary Additional Experiment** That said, for the specific case of ArXiv topic classification, reporting results for a subword baseline would provide relevant context.  We conducted a preliminary experiment with BPE-based model. In particular, we trained a standard Trasformer autoencoder baseline with BPEs as inputs. We then performed the same experiment as described in section 4.3.1. Provisionally, we found that a BPE-based model achieved only 25.13% average F1, much worse than the reported results. We suspect this is due to the nature of words in the task is different to that of the tokenizer, which is exactly where our model shows its advantages. These results are provisional and would require more investigation to ensure precise comparability.\n\n> *Questions: For example, the right part in Figure 1 is not easy to understand. Does it mean different attention patterns at three different layers? In Figure 2, which color is “lighter” is a challenging task to tell.*\n\nThank you! Your specific and constructive suggestions on Figure 1 and Figure 2 will be addressed.  In Figure 1 we have an illustrative example of how we would expect the attention maps of our NVIB-integrated encoder layers to appear. We shall update Figure 1 by finding an alternative to the attention maps and include in the Figure 2 caption that dark purple is 0 and light Yellow is 1 for the attention maps.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[+] Research topic. The research topic of learning different levels of abstraction in the same model is interesting.\n[+] Experiments and evaluation. The authors have conducted experiments both qualitatively and quantitatively for performance evaluation.\n[+] Presentation. The paper is easy to follow in general. The detailed introduction of the experiment setup is also introduced.", "reasons_to_reject": "[-] Method: Why applying NVIB to Transformer is challenging is not well-discussed. This is important to demonstrate the technical contribution of this paper.\n[-] Experiment: While the paper mentions applying NVIB to the final three layers of the Transformer, it might benefit from a more detailed explanation or exploration of why only these layers were chosen and how different configurations might impact the results.\n[-] Baseline: The method is only compared with Transformer, while there are more advanced models for solving the same task. It would be better if some other models can also be compared.\n", "questions_to_authors": "The presentation should be improved to make it easier to follow. For example, the right part in Figure 1 is not easy to understand. Does it mean different attention patterns at three different layers? In Figure 2, which color is “lighter” is a challenging task to tell.", "paper_topic_and_main_contributions": "The paper introduces a novel method to induce abstract representations of text by adapting the Nonparametric Variational Information Bottleneck (NVIB) regularizer for application to self-attention in the stacked layers of a Transformer encoder. The primary contribution is that the model learns how many vectors are needed at each layer, inducing different levels of abstraction in different layers of the same model. This is achieved without a hard-coded abstraction structure. The paper conducts various experiments, including attention map visualizations, probing analysis, and robustness analysis, to validate the model's capabilities.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jfu9zmntFg", "reviewer": "Reviewer_fPkp", "summary": "", "questions": "1. Have you compared this architecture against the transformer using supervised learning for token/sentence classification or text generation?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that Reviewer fPkp finds our proposals **interesting** and that our paper contributes to the topic of interpretability. We address the reviewer's concerns below. \n\n> *The work does not demonstrate any practical benefits of using their method.*\n \nNot every paper at EMNLP needs to be an engineering contribution, especially not a short paper (see [ACL Reviewing Guidelines](https://2023.aclweb.org/blog/review-acl23/#1-acl-2023-review-form), *The authors should have done [X] instead*).  We do not claim practical benefits, which is one reason we submitted to the Interpretability and Analysis of NLP models track.  However, we do demonstrate several properties which will naturally lead to practical benefits in future work, such as sparsity and robustness.  In particular, this includes supervised learning of a downstream task (ArXiv topic classification), just without fine-tuning. \n\n> *Experiments on downstream tasks, using either fine-tuning or prompting.*\n\n**Additional experiments** To address the reviewer's concern, we ran experiments on the ArXiv topic classification task by fine-tuning our pretrained models. Our results are consistent with the ones reported in (Table 1, pg4) in the paper in that our NVIB-integrated model works better than a standard Transformer. We report the F1 performance on ArXiv-L dataset in the Table below. We have an absolute performance gain of 4.03% over the standard Transformer. \n\n| Task      | Transformer | NVIB     |\n| :---        |    :----:   |          ---: |\n| Computer science      | 39.82       | 44.12   |\n| Mathematics   | 43.93        | 47.58      |\n|Physics| 49.18| 53.32|\n| | | |\n|**Average**| 44.31|**48.34**|\n\n> *The paper is not sufficiently self-contained.*\n\nDue to the limit of 4 pages we needed to be brief regarding the description of NVIB [(Henderson and Fehr, 2023)](https://openreview.net/pdf?id=6QkjC_cs03X).  We think that we conveyed the key properties of our model in Section 2 which are needed to understand the contributions of this paper, and we included more details in Appendix C and D for those who are interested. \nHowever, we would be happy to address the reviewer's concern by moving the most relevant details from the appendix to the main text, given the additional page limit upon acceptance. If the reviewer could be more specific about which sections need additional details then that would help us improve the presentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting idea to use the NVIB regularization to learn \"better representations\" in a transformer model. The representations can be interpreted to learn various levels of abstraction. \n ", "reasons_to_reject": "1. The work does not demonstrate any practical benefits of using their method. The analysis shows interesting trends in the representations, but without any experiments on downstream tasks, using either fine-tuning or prompting.  It is hard to imagine if the method is useful. Even if the authors could not pre-train using their method, they could have applied this architecture to a supervised learning problem. \n\n2. A lot of the important paper details are in the appendix. The paper is not sufficiently self-contained.  ", "questions_to_authors": "1. Have you compared this architecture against the transformer using supervised learning for token/sentence classification or text generation?", "paper_topic_and_main_contributions": "This work applies the Nonparametric Variational Information Bottleneck (NVIB) regularizer to the self-attention layer of a transformer model to compress the representations. They demonstrate through various probes and visualizations that the resulting layers learn increasing levels of abstraction for the text. The  authors demonstrate better robustness to noise compared to a base transformer but only for text reconstruction. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "14x0fCKM77", "reviewer": "Reviewer_bGq2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer bGq2 for their time. We believe that Reviewer bGq2 has understood our claims, despite their confidence, and highlighted multiple valid reasons to accept: **innovative**, **comprehensive** and that it **broadens the scope of NVIB’s utility in NLP**. We will address the reviewer's concerns below.\n\n>*The paper does not provide a brief introduction to NVIB*\n\nDue to the limit of 4 pages we needed to be brief regarding the description of NVIB [(Henderson and Fehr, 2023)](https://openreview.net/pdf?id=6QkjC_cs03X).  We think that we conveyed the key properties of our model in Section 2 which are needed to understand the contributions of this paper, and we included more details in Appendix C and D.  However, there are some details which could provide a deeper understanding of why the model works, and if accepted we would be happy to include them given the increased page limit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper builds upon prior research by seamlessly integrating Nonparametric Variational Information Bottleneck (NVIB) into the self-attention layers of a Transformer encoder. This adaptation broadens the scope of NVIB's utility in natural language processing tasks.\n2. The authors utilize implicit reparameterisation gradients, resulting in more efficient and effective learning processes.\n3. The experiments in the paper are comprehensive, and the analyses are thorough.", "reasons_to_reject": "In my view, reading this article requires prior knowledge of NVIB, which the paper does not provide a brief introduction to. This increases the difficulty for readers who are not familiar with NVIB, but it might not be an issue for experts in the field.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel language representation model that learns to compress information to different levels of linguistic abstraction within a single architecture. By applying the NVIB technique, the model achieves a hierarchy of abstraction in its layers and enhances robustness to adversarial perturbations. The contributions lie in the development of this innovative model, its application of NVIB, insights into the hierarchy of abstraction, and its potential to address challenges in linguistic representation learning and adversarial attacks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is consensus amongst the reviewers after the rebuttal period that the presented work is Sound and moderately exciting. It is clear that the methodology itself is exploring novel perspectives, and can shed further light on the internal workings by understand how abstraction of knowledge is represented in a model. Most of the weaknesses point towards a limited set of experimentation and exploration. However, being a short paper, the experiments necessarily have to be focused and directed towards a concrete hypothesis, which the work successful does. Overall, the paper has _good technical soundness_ and _moderate excitement_.

The following is a summary of the strengths, weaknesses and scores across the six reviews:

**Strengths:**

- Proposed methodology (NVIB applied to transformers) is more linguistically motivated towards robustness (**YN7E**, **LT26**, **gprF**, **2hkB**, **fPkp**, **bGq2**)
- Experimental results back the efficacy of the proposed method well across several NLP tasks (**YN7E**, **LT26**, **2hkB**, **bGq2**)
- The main results and overall contributions are well presented (**gprF**, **2hkB**)

**Weaknesses:**

- Design choices and technical contributions are not well motivated, and sufficient prior-knowledge is not communicated well (**LT26**, **bGq2**)
	- Authors agreed to add more details in the rebuttal, and alluded to the lack of space in a short paper for this.
- Experimental setup is limited (e.g. characters only, single language, small alphabet size, specific task-tuned models) (**YN7E**, **gprF**, **fPkp**)

**Scores in decreasing order of confidence:**

|      | Soundness | Excitement | Reproducibility | Confidence |
|------|-----------|------------|-----------------|------------|
| YN7E | 3         | 2          | 4               | 3          |
| gprF | 4         | 4          | 5               | 3          |
| fPkp | 3         | 2          | 3               | 3          |
| LT26 | 3         | 3          | 5               | 2          |
| 2hkB | 3         | 3          | 4               | 2          |
| bGq2 | 3         | 3          | 3               | 1          |
Decision: Accept-Findings