Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention
Authors: Melika Behjati
James Henderson
Venue: Submitted to ICLR 2023 (2023)
Keywords: Unsupervised representation learning
Morphology induction
Deep learning

Abstract: Characters do not convey meaning, but sequences of characters do.  We propose an unsupervised distributional method to learn the abstract meaning-bearing units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the \textit{objects} in the sequence, extending an architecture for object discovery in images.  We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers.  These experiments show that our model succeeds in discovering units which are similar to those proposed previously in form, content and level of abstraction, and which show promise for capturing meaningful information at a higher level of abstraction.

Under review as a conference paper at ICLR 2023
INDUCING MEANINGFUL UNITS FROM CHARACTER SE- QUENCES WITH DYNAMIC CAPACITY SLOT ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Characters do not convey meaning, but sequences of characters do. We propose unsupervised distributional method to learn the abstract meaning-bearin units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence, extending an architecture for object discovery in images. We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers. These experiments show that our model succeeds in discovering units which are similar those proposed previously in form, content and level of abstraction, and which show promise for capturing meaningful information at a higher level of abstraction.
INTRODUCTION
When we look at a complex scene, we perceive its constituent objects, and their properties such as shape and material. Similarly, what we perceive when we read a piece of text builds on the word-like units it is composed of, namely morphemes, the smallest meaningful units in a language. This paper investigates deep learning models which discover such meaningful units from the distribution of character sequences in natural text.
In recent years, there has been an emerging interest in unsupervised object discovery in vision (Eslami et al., 2016, Greff et al., 2019, EEgelcke et al., 2020). The goal is to segment the scene into its objects without supervision and ideally obtain an object-centric representation of the scene. These representations should lead to better generalization to unknown scenes, and additionally should facilitate abstract reasoning over the image. Locatello et al. (2020) proposed a relatively simple and generic algorithm for discovering objects called Slot Attention, which iteratively finds set of feature vectors (i.e., slots) which can bind to any object in the image through a form of attention.
Inspired by this line of work in vision, goal is to learn set of abstract continuous representations of the objects in text. We adapt the Slot Attention module (Locatello et al.) 2020) for this purpose, extending for discovering the meaningful units in natural language character sequences. This makes our work closely related to unsupervised morphology learning (Creutz) 2003 Narasimhan et al. 2015)] Eskander et al. 2020). However, there are fundamental differences between our work and morphology learning. First, we learn a set of vector representations of text which are not explicitly tied to the text segments. Second, our model learns representations by considering the entire input sentence, rather than individual space-delimited words. These properties of our induced representations makes our method more appropriate for inducing meaningful units as part of deep learning models.
In particular, we integrate our unit discovery method on top of the the encoder in Transformer auto-encoder /Vaswani et al. (017)) as depicted in Figure and train with an unsupervised sentence reconstruction objective. This setting differs from previous work on Slot Attention, which has been tested on synthetic image data with a limited number of objects (Locatello et al. 2020). We propose several extentions to Slot Attention for the domain of real text data. We increase the capacity of the model to learn to distinguish a large number of textual units, and add the ability to learn how many units needed to encode sequences with varying length and complexity. Thus, refer to method as Dynamic Capacity Slot Attention. Additionally, as hand-coded alternative, we propose stride-based models and compare them empirically slot-attention based models.

---

Under review as a conference paper at ICLR 2023
To evaluate the induced representations both qualitatively inspect the model itself and quantita- tively compare it to previously proposed representations. Visualisation of its attention patterns shows that the model has learned representations similar to the contiguous segmentations of traditional tokenization approaches. Trained probing classifiers show that the induced units capture similar abstractions to the units previously proposed in morphological annotations (Morpholeex (Sánchez, Gutiérrez et al,, 2018 Mailhot et al. 2 and tokenization methods (Morfessor (Virpioja et al. 2013), BPE (Sennrich et al. 2016)). We propose do this probing evaluation in both directions, to compare both informativeness and abstractness. These evaluations show promising results in the ability of our models discover units which capture meaningful information higher level of abstraction than characters.
To summarize, our contributions are as fol- lows: (i) We propose novel model for learning meaning-bearinn units from a sequence of char- acters (Section 2). (ii) We propose simple stride- based models which could serve as strong base- lines for evaluating such unsupervised models (Section 2.4). (iii) We analyze the induced units by visualizing the attention maps of the decoder over the slots and observe the desired sparse and contiguous patterns (Section 4.2). (iv) We show that the induced units capture meaningful information at an appropriate level of abstrac- tion by probing their equivalence to previously proposed meaningful units (Section 4.4)
Transformer Decoder
shei
play ed
bresket bail
La Drap
ihe
play nd
backet
ball
Siot Altention
Transforer Encoder
The sketch of our model. First, the Trans- former encoder encodes the sequence and then, Slot Attention computes the slot vectors (high- lighted text). Next, the LoDrop layer dynamically prunes out the unnecessary slots. Finally, the de- coder reconstructs the original sequence.
Figure 1:
2 APPROACH
2.1 PROBLEM FORMULATION
Given sequence of N characters X 2. we want to find set of mreaing--bearing units (slots) M = {m mx), ), which could best represent X in a higher level of abstraction. As an example, consider the sequence "she played basketball". where we expect our slots to represent something like the set of morphemes of the sequence, namely {she, play, -ed, basket, -ball).
2.2 OVERVIEW
We learn our representations through encoding the input sequence into slots and then reconstructing the original sequence from them. Particularly, we use an auto-encoder structure where slots as the bottleneck between the encoder and the decoder. Figure [ shows an overview of our proposed model, Dynamic Capacity Slot Attention. First, we encode the input character sequence by Transformer encoder (Vaswani et al.) 2017), which gives us one vector per character. Then, we apply our higher- capacity version of a Slot Attention module (Locatello et al. (020)) over the encoded sequence, to learn the slots. Intuitively, Slot Attention will learn soft clustering over the input where each cluster (or respectively slot) corresponds to a candidate meaningful unit in the sequence. To select which candidates needed to represent the input, (Zhang et al.) 202T), on top of the slots. Although the maximum number of slots is fixed during the course of training, this layer ensures that the model only uses many slots necessary for the particular input. This stops the model from converging to trivial solutions for short inputs, such as passing every character through separate slot. Finally, the Transformer decoder reconstructs the input sequence autoregressively using attention over the set of slots.
integrate Lo regularizing layer, i.c., Lg Drop layer
2.3 MODEL
Encoder. We use Transformer encoder architecture for encoding our sequence (Vaswani et al. 2017 and obtain the representation X' 2//2/2 r.y from input sequence X.
Slot Attention for text. After encoding the character sequence, we use our extended version of Slot Attention for discovering meaningful units of the input character sequence. Slot Attention is
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Slot Attention module (Locatello et al. 2020). q, ki map the slots and inputs to common dimension D and T denotes the number of iterations.
Require:
inputs R R slots (N,, (iag(()) E R stoos
inputs = (iyneroor(iiputs)
for = to T do
slots_prev slots
slots = LayoeNoom(lllsss
Saw-Sooffxxx33 / hrinpats)-g(sloos)), axis 'slots')
updates Weighheddeann attn + d values u(inputs)
slots GRU(states = slots.prev, input updates)
slots += MM.P((_ayerNrrm(sloss) end for return slots
recent method for unsupervised object representation learning in vision (Locatello et al. 2020
learns set of feature vectors (slots) by using iterative attention based algorithm. Al Algorithm T shows the pseudo code of this method. Abstractly, in every iteration, takes the following steps. First, it computes attention map between the slots and the inputs, by slots acting as queries and inputs as keys. Then, it normalizes the attention map over slots, which makes the slots compete for representing each token of the input. Afterwards, it computes the slots' updates as the (normalized) weighted mean over the attention weights and the input values. Finally, it updates the slots through a Gated Recurrent Unit (GRU) (Cho et al. 2014) followed by residual MLP. This process iterates fixed number of times.
In Locatello et al. (2220), the slots are initialized randomly by sampling from Normal distribution with shared learnable parameters and a. i.e., slot, ~ N (/Ashared; sshared )
In other words, the initial value of slots are independent samples of a single Normal distribution with learnable parameters. We found in our experiments that this initialization method does not lead to good results in the text domain. In contrast with the experimental setting of Locatell et al. (2020), which used artificially generated data with a small number and range of objects, real language requires learning about a very large vocabulary of morphemes, and each example can have large number of morphemes. This suggests that a model for language needs a more informative initialization with more trainable parameters, in order to have the capacity to learn about this large vocabulary and distinguish all the objects in a long sentence.
To investigate this issue, we propose another initialization for adapting Slot Attention to text. We consider separate learnable f per slot and we fix the to a predefined value for all the slots. Namely, the slots are initialized as
slot, ~ N(1 constant)
By assigning a separate μ for each slot, the initialization has many more trainable parameters. This allows the model to learn about different kinds of units, such ones that occur at different positions, or ones that have different types of forms, but we do not make any assumptions about what those differences might be. In addition, the intuition behind fixing the a, is prevent it from collapsing to zero which would in turn, force the slots to compress the information in a meaningful way. In particular, since the number of possible n-grams text is finite but the slots can have any continuous value in the space of Patore the slots tend to learn an arbitrary mapping from n-grams in the input the slots, while turning to zero. Thus, there no need for the slots learn the underlying meaning- bearing units. In other words, by imposing a constant noise on slots through the constant o, we limit the information which can be passed through each slot, from the information theoretic point of view. Experiments showing the greater effectiveness of Equation 2 over Equation l are given in Appendix A. We then obtain the set of slots M (m] ... mx) SlotAttention (X"")
Neural sparsification layer: Lg Drop. The number of units needed to represent sequence varies among different sequences in the data. In contrast to the object discovery work where the data is generated synthetically and thereby, the number of objects in the scene is known beforehand, we do not make any assumptions about the number of units required. Therefore, we consider an bound over the number of required units and prune the extra ones per input sequence. We accomplish this goal by using a neural sparsification layer called L (Zhang et al.) (02I). allows our model to dynamically decide on the number of required units for every input sequence.
3

---

Under review as a conference paper at ICLR 2023
This layer consists of stochastic binary-like gates g 91 9K that for every input m, works
(o,ro = g.
When Di is zero the gate is closed and when it is one the whole input is passed. Each gate is continuous random variable in the [0, 1] interval, sampled from a hard-concrete distribution (Louizos et al. 2018). This distribution assigns most of its probability mass over its endpoints (i.c., 0 and 1) in favour of the sparsification goal. Specifically, 9i ~ HardCCnncrete(a; 3,e) where 3 and e are hyperparameters. a; is predicted as function of the encoder output m, i.e. loga; m,w", where w is a learnalee vector. This allows the model to dynamically decide which inputs to pass and which ones prune. The Lo penalty, which yields the expected number of open gates, is computed
(M = Σ p(g. = 8,c,,
1=1
where the probability gi being exactly 0 is provided closed form in Louizos et al. (018). We follow the same approach Louizos et al. (2018) at evaluation time and consider the expectation of each gate as its value. We refer to the pruned slots after applying the LoDrop layer as M' = mi m-
Decoder. Lastly, we regenerate the input sequence from the set of slots by using a simple, shallow decoder. To this end, attention head over the slots. simple decoder forces the slots to learn representations with a straightforward relationship the input, which we expect be more meaningful. In other words, do not use a powerful decoder because it would be able to decode even low quality representations of the input, which are less meaningful (Bowman al., 2015).
one-layer Transformer decoder Vaswanni al. 2017) with a single
Training objective. We train our model end-to-end by using Gumble trick for sampling HardConcrete variables (Maddison al. 2017 Jang al.) 2017). The training objective
Lree (X, M') + X (M(M) < Eg Elg ((X]M + À (M(M) = ((X).
log (F)(M(X))M)) À (o(M)
which consists of the reconstruction loss from the decoder (Lice) and the Lo penalty for the open gates. Hyperparameter A, the sparsification rate, controls the trade-off between the two losses. In practice, we find that in order to impose enough sparsity in the slots, we should slightly increase during the course of training using scheduling techniques.
2.4 STRIDE-BASED MODELS
We propose a simple hand-crafted alternative model to induced units which gain acceptable results in terms of performance. We design this model by replacing our Dynamic Capacity Slot- Attention module with linear-size bottleneck. In particular, we take out of every & encoder outputs and down project them (R Pinpue - R Patots) ) to obtain the representations. In other words, we only pass certain encoder outputs based their position and drop the rest.
M DownProject((2 #1
where n = | N-1 We can get different alternative models by varying the stride The training objective is the reconstruction loss Lree (Sec(X,1.) (X,M) =-log p X M). The idea of using stride-based
models has also been used in Clark et al. (2022): Tay et al. 2022 in a different context and setup.
3 RELATED WORK
Unsupervised Object Discovery. There is a recent line of research in the image domain for discover- ing objects scene without explicit supervision, and building an object-centric representation of them. Most of this work is built around the idea of compositionality of the scenes. MONet (Burgess) ct al., 2019) and GENESIS (Engelcke et al. 2020) similarly use a recurrent attention network for learning the location masks of the objects. Greff et al. (2016) 2017 2019); Van Steenkiste al. (2018); Emami et al. 2021) model the scene as a spatial Gaussian mixture model. Furthermore, AIR network (Eslami al. 2016) and its variants (Crawford & Pineau 2019 Lin al. 2020) model objects from a geometric perspective by defining three specific latent variables. Lately, Locatello et al.)
(2020) propose iitention-hasse algorithm (namely Slot Attention) to learn object representations
4

---

Under review as a conference paper at ICLR 2023
(slots). In contrast to this line of work in vision, our approach is specifically designed for text. We use additional components (e.g., Drop layer) in our architecture to resolve the requirements of modeling textual data. Furthermore, contrast to these previous models which have only been shown to be effective on synthetic scenes.
model is trained and evaluated on real text datasets, in
Unsupervised morphology learning. This subject has been of interest for many years in the NLP field (Elman 1990; Creutz& & Lagus 2002 Baroni et al. 2002). Morphemes have strong linguistic motivations, and are practically important in many downstream tasks because they are the smallest meaning-bearing units in a language (Can & Manandhar (014). Many approaches have been proposed for discovering the underlying morphemes or morpheme segmentations. Morfessor variants are based on probabilistic machine learning methods (MDL, ML, MAP) for morphological segmentation
(Creutz 2003 Creutz& Lagus 2002, 2005| 2007 Virpioja et al. 2013). Some researchers take
Bayesian approach for modeling ing word formation (Poon et al., 2009; Narasimhan et al. 2015
Bergmanis & Goldwater 2017 Luo et al., (017). Adaptor Grammars are another approach for
modeling morphological inflections (Sirts & Goldwater 2013 Eskander et al. 2016 2019 2020), In
addition, Xu et al. (2018) 2020) built their models upon the notion paradigms, set of morphological
categories that be applied to a set of words. Moreover, Soricut & Och (0015); Ustün & Can 2016)
extract morphemes by considering the semantic relations between words in the continuous embedding
space. Cao & Rei (2016) propose learn word embeddings by applying bi-directional RNN with attention over the character sequence. Furthermore, Ataman et al. (2020) model word formation as latent variables which mimic morphological inflections in the task of machine translation.
Our work differs from the previous work in classical morphology learning in two ways. First, instead of explicitly discovering morphemes, we learn a set of continuous vector representations of the input, which would then need to be processed to extract the morphemes. The model itself has no explicit relation between these unit representations and segments of the input. Second, our model learns representations of an entire input sentence, rather than individual space- -pacc-delimited words. This makes fewer assumptions about morphemes, and considers the context of the words a sentence. Our work is similar to Ataman et al. (2020) in modeling morphology implicitly in the latent space. However, we employ a self-supervised objective for our purpose which is more general compared to their supervised loss, as we do not need labeled data.
Unsupervised character segmentation. Learning to segment a character sequence in unsupervised fashion is another relevant area to work. Chung et al. (2017) propose Hierarchical Multi-scale RNNs for modeling different levels of abstractions in the input sequence. In the language modeling task, they observe that the first layer is roughly segmenting the sequence into words, namely at space boundaries. Sun & Deng (2018) propose Segmental Language Models for Chinese word segmentation. Moreover, in (Kawakami et al., 2019), the authors design model to learn the latent word segments in a whitespace-removed character sequence with a language modeling objective. As we mentioned earlier, we learn continuous vector representations of text which is different from explicitly detecting discrete character segments.
Subword discovery algorithms. This set of algorithms have become a standard component of NLP models in recent years. Pete-Pair-Encoding (BPE) (Sennrich et al. (016) iteratively merges the two consecutive tokens with the highest frequency. Word-piece (Schuster & Nakajimal 2012)). sentence-piece (Kudo & Richardson] 2018) and unigram LM (Kudo) 2018) are other similar subword tokenization algorithms. In contrast to these methods, which mostly use local statistical information of the data, our model is trained over complete sentences to learn abstract sophisticated representation.
4 EXPERIMENTS
We evaluate unsupervised model both by visualizing attention maps (4.2). and by probing the slot vectors (.4).
4.1 EXPERIMENTAL SETUP
We apply our model to languages from different morphological typologies. We select English (EN), German (DE), French (FR), Spanish (ES) and Czech (CS) from the fusional family and Finnish (FI) from the agglutinative typology. For English we use the raw Wikitext2 dataset (Merity et al., 2017).
For the rest we use Multilingual Wikipedia Corpus (MWC) (Kawakami al. 2017).
5

---

Under review as a conference paper at ICLR 2023
(a) epoch 5
(b) epoch 50
(c) epoch 125
Figure 2: Attention of the decoder over slots (x-axis) for generating every target character (y-axis) after different training epochs, for target sequence "the red colour associated with lobsters only appears after cooking.". "
As for the models, dimension 256. The encoder consists of 2 layers with 4 self-attention heads and the decoder consists of layer with self-attention head and attention head over the slots. We feed in the sentences with less than 128 characters to our model and consider the number of slots as 64 (half of the maximum input length). In addition, we take the dimension of slots as 128. We scheduled the parameter the training loss to start with low value and exponentially increase it every 10 epochs until it reaches a certain limit. We obtain this limit manually a way that the final number of open gates roughly equals the average number of BPE tokens in sequence. More details of the settings are available in the Appendix B
standard Transformer architecture (Vaswani et al. 2017) with model
4.2 VISUALIZATION
In order to show some qualitative results of our model, we visualize the attention maps for generating every output, shown in Figure [ In particular, we show the attention of the decoder over slots when generating every output character. Interestingly, although we do not impose any sparsity in the decoder's attention weights, the attention maps are quite sparse. Namely, at each generation step only few slots attended, and each slot is attended while generating only few characters. In addition, although we do not impose any bias towards discovering segments of the input. the characters which are generated while attending bands in Figure 2). We believe that the emergence of contiguous spans is a result of the bottleneck create with our Dynamic Capacity Slot Attention. This means that the model is trying to put correlated information about the input in the same vector, so that it can represent the string more efficiently. The strongest correlations in the character string are local in the input, so each slot tends to represent characters which are next to each other in the input.
given slot contiguous in the string (the vertical
In early steps of training, when the sparsity ratio (A) is small, each slot tends represent bigram of characters (2a) and later on, trigrams (2b). These observations confirm the necessity of the LoDrop layer for converging to better units. In particular, as the ratio increases, the number of active slots reduces and they become more specialized in representing contiguous meaning-bearing units of input. For instance, the word cooking in [2clis represented by two slots cook and ing. That these segments roughly correspond to the morphemes which we want the model to discover, is verified quantitatively the probing experiments in Section
4.3 PROBING METHOD
Since our model is unsupervised and does not use artificially generated data, there is no obvious gold standard for what units it should have learned. To quantitatively evaluate how well performs, we freeze the trained model and train probing classifiers to measure to what extent the discovered units capture the same information as previously proposed meaningful units. We use multiple previously proposed representations which have been shown to be good levels of abstraction for range of NLP applications. If the induced units provide a similar level of abstraction, separated into units in a similar way, then we can expect that they will also be effective in NLP applications.
We assume that two representations provide the same level of abstraction if there is a one-to-one mapping between their respective units such that two aligned units contain the same information,

---

Under review as a conference paper at ICLR 2023
meaning that each unit can be predicted from the other. Predicting the previously proposed unit from our induced unit is a probing task, as used in previous work (Belinkov & Glass 2019), which we specify as forward probing. We refer to the prediction of our induced unit from the previously proposed unit as a reverse probing task, which we believe is a novel form of evaluation. We compare various models on their trade-off between forward and reverse probing.
Probing data. We consider three target representations which either have linguistic or practical evidence of being effective in NLP applications. For the languages for which an in-depth linguistic morphological analysis is available. we compare to these morphemes (i.e., EN and FR) (Zabokrtsky et al.) 2022; Sänchez- Gutiérrez et al., 2018 Mailhot et al. 2020). Alternatively. as linguistically inspired units, we have the Morfessor /Virpioja et al. 2013) outputs, and BPEs (Sennrich et al. 2016) as frequency-based subwords.
Forward probing. This is the common way of probing where we want to measure if our induced units include the information about the target representation. We train a 2 layer MLP as our probing classifier ʃ for this purpose. We apply the classifier with shared parameters to each of our slots individually and obtain a set of predictions {f(mm) fm/)).... fm'x)) }} one per slot. As we are dealing with set, during training we need to find a one-to-one matching between the classifier's predictions and the target tokens. Therefore, we follow locatello et al. (2020) to use the Hungarian matching algorithm (Kuhn) 1955) for finding the match which minimizes the classification loss.
We consider the complete set of slots after applying the Drop layer as the inputs to our classifier. Slots whose Lg gate is closed are simply input as zero vectors. This gives us fixed number of vectors. The two sides of matching should have the same size to obtain one-to-one match, therefore, we add an extra target label (i.e., emply) for representing the pruned slots. Due to the fact that many slots pruned out, considering measure like accuracy could be misleading, since a classifier which outputs empty label will achieve very high accuracy. Therefore, we build a confusion matrix as follows. We consider all non-empty labels positive and the empty ones negative, and we report precision (P), recall (R) and F1 measure, to better reflect what the slots have learned.
Reverse probing. To evaluate whether the induced units capture the same level of abstraction, we also need to evaluate whether the induced units abstract away from the same information as the previously proposed units. We propose measure this by training reverse probing classifiers, which predict each induced unit from its aligned target unit. Because the induced unit is continuous vector, we predict the parameters of dddimensional Gaussian distribution with diagonal covariance, and measure the probability of the induced vector in this distribution. We first pass the tokens into an Embedding layer and then apply a 2 layer MIP on top of each embedding individually to predict the parameters of the distribution. We take the mappings between the tokens and slots from the trained (forward) probing classifier described previously. As the objective, we maximize the log-probability of representations under the predicted distributions. We consider the loss on the test set as our evaluation measure.
4.4 PROBING RESULTS
Given a target set of units and a trained model for finding units, we align the two sets of units and use forward probing to see if the induced units contain as much information as the target units and reverse probing to see if the induced units contain more information than the target units.
Optimizing both measures requires having the same information as each target unit in the associated induced vector, and hence having the same level of abstraction.
The primary mechanism we have to control the amount of information in an induced representation is to control the number of units. More units means that there are more opportunities for the alignment to find a unit which correctly predicts each target unit, but makes harder to predict all the units from the available target units. We take two approaches to controlling the number of induced units. First, we fix the number of units to a target number, which allows an efficient comparison of models with only forward probing. Second, vary the number of units, which allows to evaluate how well different models can match the informativeness trade-off of the target representation.
Targeted informativeness
As less computationally expensive initial evaluation, parameters to set the number of induced units to approximately the same as the number of target BPE tokens. For the stride-based models, simply set the stride according to the average target number
hyper-
7

---

Under review as a conference paper at ICLR 2023
50
-40
59
-50
-00
-10
328
0.750
0.715
D.80
0
0.74
0.78
9.22
6:53
0.75
(a) EN-BPP
(b) EN-Morfessor
(c) EN- Morphemees
(d) FI-BPE
-00
-100
(90)
-30
.20
b/
0.7
08
0.90
0.50
0:59
(c) FR-BPE
(f) FR-Morfessor
(g) FR-Morphemes
(h) FI-Morfessor
Figure 3: 2D graphs showing the informativeness trade-off for slot-attention models (blue points) and the stride-based models (red points). The x-axis shows the Fl measure for the forward probe and the y-axis shows the negative pog-prbbability loss for the reverse probe, so better models are lower and further to the right.
of tokens (i.e., stride= 6). For the slot-attention models, we empirically set hyperparameters so that the number of open gates in the Drop layer is approximately the same as the target number.
Given that all models output ap- proximately the same number of units, we can simply com- pare them with standard (for- ward) probing. We use prob- ing to compare slot attention and stride-based models to an informative baseline representa- tion (untrained), thereby control- ling for the power of the prob- ing classifier learn arbitrary mappings (Conneau et al., 2018 Oord et al., 2018). This baseline is the set of slot vectors output by a randomly-inittalized slot atten- tion model without any training.
Table 1: Forward probing results different languages. Note that
human-annotated morphemes are only available for En and FR.
BPE
Morfessor
Morphemes
lang Model
P
R
F1
P
R
F1
P
R
F1
EN
untrained
0.74
0.10
0.17
0.72
0.14
0.23
stride=6
0.95
0.66
0.76
0.94
0.66
0.76
0.91
0.64
0.74
slot-attn
0.97
0.64
0.76
0.98
0.64
0.76
0.91
0.70
0.78
FI
untrained
0.94
0.09 0.17
0.90
0.06 0.11
stride=6
0.95
0.59 0.72
0.91
0.53 0.66
slot-attn
0.95 0.65 0.76
0.92 0.64 0.74
FR
untrained
0.75 0.08 0.14
0.72
0.11 0.19
stride=6
0.95
0.63 0.75
0.94
0.57 0.70
0.92
0.59
0.71
slot-attn
0.98 0.65 0.77
0.96
0.63 0.76
0.94
0.69 0.78
Table I shows the results of the forward probing tasks on differ- ent languages. As the results show, the trained slots achieve much higher performance both tasks in comparison the ran- dom baselines. Our model achieves very high precision predicting the non-empty labels.
ES
untrained
0.71 0.09 0.17
0.71
0.11 0.19
stride=6
0.95
0.61 0.73
0.94
0.60 0.72
slot-attn
0.96 0.73 0.82
0.95
0.74 0.83
DE
untrained
0.91 0.08 0.16
0.76
0.08 0.15
stride=6
0.96
0.61 0.74
0.94
0.59 0.71
slot-attm
0.99 0.66 0.78
0.96
0.70 0.80
CS
untrained
0.94 0.08 0.16
0.86
0.05 0.10
stride=6
0.95
0.58 0.70
0.91
0.47 0.61
slot-attn
0.98 0.63 0.75
0.96
0.57 0.71
Its performance is weaker on the recall side, but here the improvement over the untrained model is even more pronounced. This seemss be due the imbalance between empty and non-empty labels the training set, where the empty labels comprise around 66% of the data for the probing classifier. For this reason, below we will use Fl as our overall performance measure for forward probing. The comparable stride-based model performs similarly to our slot attention model, but generally not well. The performance gain of our slot-attention models to the stride-based models is more noticable on Morfessor targets (+0.07 average F1), and for the two languages where we have real morpheme annotations, which implies that our att-attention based method is more effective in discovering linguistically meaningful units.
There is no clear difference in our model's performance different targets, and it performs equally well on both the agglutinative language (Finnish) and the fusional languages. We further show some examples from the predictions of the classifiers in the Appendix A.4
8

---

Under review as a conference paper at ICLR 2023
Informativeness trade-off. To vary the number of induced units, use sypee--arameters. For the stride- based models, this simply implies varying the length of the stride. For the slot-attention models, to be able to more easily control the number of units we slightly change the objective of slot-attention models to explicitly control the number of open slots. Thus, we replace the Lo term in 5 with max [o, r), where r indicates the desired number of open slots. In this setup, the model will close Lo gates until it reaches the limit r.
With the results of the probings in the two directions, we can plot two-dimensional graphs compare different models. We train stride-based models with strides=(2, 3, 4, 5, 6] and slot-attention based models with == imput length which would lead both models to have the same number of valid units. stride
Figure plots this informativeness trade-off for English, French and Finnish with different target representations. Overall, the slot-attention based models provide better representations in comparison to the stride-based models, since these models are nearer to the lower right corner of the plot. This is clearer for English and French, with the results for Finnish being more mixed. We believe the stride-based models contain extra information which degrade the performance of the reverse probe and causes them to have higher evaluation loss. These results justify our design choices for the slot-attention based models to achieve the goal of discovering meaningful units which have the same level of abstraction the previously proposed units. In addition, although our prride-based models fall behind the slot-attention based models, they can be utilized as strong baselines for evaluating such unsupervised models and find acceptable abstract units which can omit the need for tokenizing text as a preprocessing step.
5 CONCLUSIONS
In this paper, propose Dynamic Capacity Slot Attention for discovering meaningful units in text in an unsupervised fashion. We use an auto-encoder architecture for encoding a character sequence into set of continuous slot vectors and decoding them to reconstruct the original sequence. We enforced the set of slots to act as a bottleneck in transferring information between the encoder and the decoder by adding constant noise to their vectors and integrating an Lg regularizing layer top of the slots which only retains the necessary vectors. In addition, we propose a set of stride-based models which could serve an alternative to our main model. We evaluate our model by probing the equivalence between the pruned slots and predefined tokens. In particular, we propose to do reverse probing as well as the normal way of probing. Our experiments show that our representations effectively capture meaningful information at a higher level of abstraction.
Limitations We learn a set of abstract continuous vector representations of the input and we do not explicitly discover morphemes or morpheme segments. Although the visualized attention maps show interesting patterns of input segmentation (see Figure 22). the vertical bands are fading near the end-points. More specifically, it is not traiihh-ffowwat to determine boundaries between the bands as the transition is done smoothly due the continuous nature of the attention function. For this reason, we could not obtain good segmentations by employing simple heuristics the attention maps.
Moreover, the purpose of our work is the novel adaptation of an object discovery method to text and developing intrinsic evaluation framework for this line of work. Therefore, evaluating the obtained representations on downstream tasks, is not in the scope of our paper. However, since previous work has shown that the target representations used for our probing analysis are effective ways to capture meaning for downstream tasks, and since these results show that the information captured by our induced units are similar to the information in these representations, there is every reason to believe that the units discovered by the proposed model will also be effective meaning- bearing units for downstream tasks. We leave the empirical verification of this effectiveness to future work. As an initial experiment, we take our pretrained slot vectors and use them as text representations on the challenge set released by Hofmann et al. (2022). The details of this experiment is provided in the Appendix B.5 For the ArXiv-L, on average we get 0.396, 0.409, 0.2732 F1 for slot-attn, stride=6, and character- based models, respectively, on Dev set, and 0.393, 0.394, 0.271 on Test set. These numbers indicate that proposed models work much better than a character- based model on this challenging task and therefore, could be a useful replacement for tokenisation methods in some downstream tasks.

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement We have completely explained the details of our experiments in Appendix B It includes the data we have used and how we have processed it, in addition to the models' parameters and training details. We will release our code upon acceptance.
REFERENCES
Duygu Ataman, Wilker Aziz, and Alexandra Birch. A latent morphology model for open- vocabacayy neural machine translation. In International Conference on Learning Representations (ICLR), 2020. URL htt.ttttt://oooeee
net/ forum? d= SKDH
Marco Baroni, Johannes Matiasek, and Harald Trost. Unsupervised discovery of morphologically related words based on orthographic and semantic similarity. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pp. 48-57. Association for Computational Linguistics. July 2002. doi: 10.3115/118847.1118833 URL https: https::/aclantholoog
org/ hxg/W02-200 0606
Yonatan Belinkov and James Glass. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistiis,, 7249-72,n 04 2019. ISSN 2307-387X. 00254
doi: 10.1 10.1tl2/ 00254.
Toms Bergmanis and Sharon Goldwater. From segmentation analyses: a probabilistic model for unsupervised morphology induction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 337-346, 2017.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio, Generating sentences from a continuous space. arXiv preprint arXiv:1511.06344 2015.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation, 2019.
Burcu Can and Suresh Manandhar. Methods and algorithms for unsupervised learning of morphology. In International Conference on Intelligent Text Processing and Computational Linguistics, pp. 177-205 Springer, 2014.
Kris Cao and Marek Rei. A joint model for word embedding and word morphology. In Proceedings of the 1st Workshop on Representation Learning for NLP, pp. 18-26, 2016.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hicrarchical multiscale recurrent neural networks. In International Conference on Learning Representations (ICLR), 2017.
Jonathan H. Clark, Dan Garrette, Tulia Turc, and John Wieting. Canine: Pre-training effi- cient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73-91, 2022. doi: 10.1162/tacl 00448. URL https
+/aclanthology wwwpanhhhooooooooo//o/00 2ttp://2222.t.ccll...
Alexis Conneau, German Kruszewski, Guillaume Lample. Loic Barrault, and Marco Baroni. What you can cram into single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 126-2113. Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 18.13/ 1/P18-111 1198.
httttpp:///aclaathoooooo org/P18 1198
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
3412-3420, 2019.
Mathias Creutz. Unsupervised segmentation of words using prior distributions of morph length and frequency. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics- Volume I, pp. 280-287. Association for Computational Linguistics, 2003.
10

---

Under review as a conference paper at ICLR 2023
Mathias Creutz and Krista Lagus. Unsupervised discovery of morphemes. In Proceedings of the ACL-02 workshop on Morphological and phonological learning- 6, pp. 21-30. Association for Computational Linguisties, 2002.
Mathias Creutz and Krista Lagus. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Helsinki University of Technology Helsinki, 2005.
Mathias Creutz and Krista Lagus. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing (TSLP), 4(1):1-34, 2007.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2): 211, 1990. URL https
o/on Monelinll tieelibrary. wi ley. ccm/ doi http://ff1...1277711551166001000
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2970-2981. PMLR, 18-24 Jul 2021. URL https: htttps://proceet i www.ing........................m ml m.t.mmlr.perre w.e.s....././..................e /vi 39/ emami 21 a.tmll
Martin Engeleke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representatiios,, In International Conference on Learning Representations (ICLR), 2020. URL http::ss///ppeeeeeeeeeeeee/eeeee net/ forum?id-
BkxfaTVFwi
Ramy Eskander, Owen Rambow, and Tianchun Yang. Extending the use of Adaptor Grammars for unsupervised morphological segmentation of unseen languages. In Proceedings COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 900-910, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL
Ramy Eskander, Judith Klavans, and Smaranda Muresan. Unsupervised morphological segmentation for low-resource polysynthetic languages. In Proceedings of the 16th Workshop on Computational
Research in Phonetics, Phonology, and Morphology, pp. 189-195, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 111555/1/W11--222 URL https://
aclanthology. wttg/19--4/2222
Ramy Eskander, Francesca Callejas, Elizabeth Nichols, Judith Klavans, and Smaranda Muresan. MorphAGram, evaluation and framework for unsupervised morphological segmentation. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 7112-7122, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 URL
https: wwtp:/wwwcclccccc..... www.:aawlwbbb....c http://wwww.aaeee.o.//oo//oaooo/ htt.://w.......aaaooooo///////..
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, 2016.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, volume 29, pp. 4484 4492, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, 2019.
Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 385-393, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1865// v//2022.acl- sch-sh htL.tttss:: http://ccccartoooooorro/////222 acl short. 43.

---

Under review as a conference paper at ICLR 2023
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to create and reuse words in open-
vocabulary neural language modeling. In Proceedings of 55th Annual Meeting of the Association for Computational Linguistics /Volume I: Long Papers), pp. 1492-1502, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/11/11-11113 URL https
7/aclantho ogy. log..org//17-1133
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to discover, ground and use words with segmental neural language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 6429-6441, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//11--1455 URL https: htttts://aaaaatthottt
org/P19-
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2013.
Taku Kudo. Subword regularization: Improving neural network translation models with mul- tiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18660/1/P118-000., URL
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, 2018.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics
quarterly, 2(1-2):83-97, 1955.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng. Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations (ICLR), 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33, 2020.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through (o regularization. In International Conference on Learning Representations (ICLR), 2018.
Jiaming Luo, Karthik Narasimhan, and Regina Barzilay. Unsupervised learning of morphological forests. Transactions of the Association for Computational Lingwistics, 5:353-364, 2017. doi: 10.1 10.1162/tacl. a 00066.
Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.
Hugo Mailhot, Maximiliano A. Wilson, Joël Macoir, S. Hélène Deacon, and Claudia H. Sánchez- Gutiérrez. MorphoLex-Pi A derivational morphological database for 38,840 French words. Behavior Research Methods, 52(3):1008 -005,, June 2020. ISSN 1554-3528. doi: 10.3758/ 133428-019-01297-1
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations (ICLR), 2017.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. An unsupervised method for uncovering morphological chains. Transactions of the Association for Computational Linguistics, 3:157-167, 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018
12

---

Under review as a conference paper at ICLR 2023
Hoifung Poon, Colin Cherry, and Kristina Toutanova. Unsupervised morphological segmentation with log-lineat linear models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 209-217, Boulder, Colorado, June 2009. Association for Computational Linguistics. URL https:: / www.aaaaanntoto.ooo angoo wttpllloooooooo-11o01///
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5149-5152 IEEE, 2012.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1 11111/61//1111111111 URL hwt.h:ttp///// //cl antho logy
org/P16-1162
Kairit Sirts and Sharon Goldwater. Grammars. Transactions of the Association for Computational Linguistiics Linguitits,,ll,iiiiisssississ 1:255- 266, 2013. doi:
morphological segmentation using Adaptor
10.1 00225. httpp::// anthe ogy. wtgp/133rr//1333 1021
Radu Soricut and Franz Och. Unsupervised morphology induction using word embeddings. In
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1627-1637, Denver, Colorado, 1186. URL
May-June 2015. Association for Computational Linguistics. doi:
1ww.nttoooooy.o///l--11111 org/N15-1186
Zhiqing Sun and Zhi Hong Deng. Unsupervised neural word segmentation for Chinese via segmental language modeling. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 4915-4920, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 1..18865/11/133 1531. URL
org/D18-1531
Claudia H. Sanchez-Gutiérres Hugo Mailhot, S. Hélène Deacon, and Maximiliano A. Wilson. MorphoLex: A derivational morphological database for 70,000 English words. Behavior Research Methods,
August 2018. ISSN 1554-3528. doi: 10.3758/s13428-017-00
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-basee based subword tokenization. In International Conference on Learning Representations, 2022. TtL.aa/es//pppeeeeessesse//rr/er
Ahmet Üstün and Burcu Can. Unsupervised morphological segmentation using neural word embed- dings. In Pavel Král and Carlos Martín-Vide (eds.), Statistical Language and Speech Processing, pp. 43-53, Cham, 2016. Springer International Publishing. ISBN 978-3-319-45925-7,
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Sami Virpioja, Peter Smit, Stig-Aree Grönroos, and Mikko Kurimo. Morfessor 2.0: Python implemen- tation and extensions for morfessor baseline. In Aalto University publication series. Department of Signal Processing and Acoustics, Aalto University, 2013.
Hongzhi Xu, Mitchell Marcus, Charles Yang, and Lyle Ungar. Unsupervised morphology learning with statistical paradigms. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 44-54, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL ht tptt:p:// https:////aaaathohoooeee org/ c105
Hongzhi Xu, Jordan Kodner, Mitchell Marcus, and Charles Yang. Modeling morphological typology for unsupervised learning of language morphology. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6672-6681, Online, July 2020. Association
13

---

Under review as a conference paper at ICLR 2023
for Computational Linguistics. doi: 0t0p:3/////2222.ccmmmmmm aclweb. htt..eb...rrr/an.ooooooo/2222200 www.://yy.........a.......a.. maa.c-mmin.559.
URL hetps: Www.
Zdenčk Zabokrtsky, Nyati Bafna, Jan Bodnár, Lukáš Kyjánek, Emil Svoboda, Magda Ševčíková, Jonáš Vidra, Sachi Angle, Ebrahim Ansari, Timofey Arkhangelskly, Khuyagbaatar Batsuren, Gábor Bella, Pier Marco Bertinetto, Olivier Bonami, Chiara Celata, Michael Daniel, Alexei Fedorenko, Matea Filko, Fausto Giunchiglia, Hamid Haghdoost, Nabil Hathout, Irina Khomchenkova, Vic- toria Khurshudyan, Dmitri Levonian. Elconora Litta, Maria Medvedeva, S. N. Muralikrishna, Fiammetta Namer, Mahshid Nikravesh, Sebastian Padó, Marco Passarotti, Vladimir Plungian, Alexey Polyakov, Mihail Potapov, Mishra Pruthwik, Ashwath Rao B, Sergei Rubakov, Husain Samar, Dipti Misra Sharma, Jan Snajder, Krešimir Sojat, Vanja Stefanec, Luigi Talamo, Delphine Tribout, Daniil Vodolazsky, Arseniy Vydrin, Aigul Zakirova, and Britta Zeller. Universal segmen- tations 1.0 (UniSegments 1.0), 2022. URL http:/ //hdl. handla e wwt. net/11234/1-4622 LINDAT/CLARIAH-C. digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Biao Zhang, Ivan Titov, and Rico Sennrich. sparsifying encoder outputs sequence-to-sequenee models. In Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, pp. 2888-2900, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 12202l.indings-aae URL https: /aaa///aamboooooy..r///200101111 fingr//i/2n.....nigggn.g... findings-ace
255
14

---

Under review as a conference paper at ICLR 2023
A SUPPLEMENTARY RESULTS
A.1 SLOT INITIALIZATION ANALYSIS
Table 2: Probing results of different slot initializations for Spanish language.
BPE
Morfessor
Init.
Model
F1
P
R
F1
Eq
untrained
0.71
0.09
0.17
0.71
0.11
0.19
(ours) slot-attn
0.96
0.73
0.82
0.95
0.74
0.83
Eq untrained
0.80
0.10
0.18
0.78
0.12
0.20
slot-attn
0.97
0.51
0.66
0.96
0.51
0.66
We compare our proposed initialization with the original definition of Slot Attention, where slots are randomly initialized from single shared distribution Table shows the probing results of model under different slot initializations. Having a separate for each slot (Eq increases the capacity of the model and thus yields better results comparison to sharing μ between the slots. The model especially achieves higher recall in this case, which implies better recovery of the BPE and Morfessor tokens.
A.2 RECONSTRUCTION RESULTS
In Table we report the reconstruction error among different languages. We compare the slot attention based model with modified objective, i.e., max(Lo, r = 6), with the stride-based models (stride=6) to ensure that both models have the same average number of units. In all languages, slot attention based models are better capable of reconstructing the sequence. This indicates that the slot-attention vectors are better in capturing information about the structure and meaning of the input than the stride-based ones and therefore, providing better signals to the decoder to reconstruct the sequence.
A.3 VISUALIZATION THE ATTENTION MAPS
A.3.1 ATTENTION OF SLOTS OVER THE INPUT
In Figures 4 [ we illustrate the attention of slots over the input as well as the attention of decoder over the slots for the same input. The two attentions show similar patterns, but on the decoder side the attention weights are higher and therefore the patters more visible. For this reason, used the attention decoder over the slots section 4.2 Interestingly, in Figure there are some slots which are only attending to the space boundaries (the horizontal bands).
A.3.2 DECODER ATTENTION FOR THE STRIDE-BASED MODELS
Figure 6 visualizes the attention of decoder over the vectors of different stide--ased models. As expected, the vertical bands often of the same length and much overlapping for larger strides. As a result, the bands do not correspond to meaningful units in the input in contrast to the slot-attention based model (see Figure 2c
A.4 FORWARD PROBING EXAMPLES
Table shows two examples of the probing classifiers' predictions given the learned slots. As explained in 4.4 the model is quite precise predicting non-empty labels.
A.5 PROBING RESULTS' TABLES
Tables 5 15 show the detailed results of the performance of forward and reverse probing tasks visualized in Figure (Reverse reconstruction vs F1). n input denotes the input length which is 128 in our experiments.
15

---

Under review as a conference paper at ICLR 2023
Table Reconstruction error on training and test set among different languages.
slot attention(1=6)
stride=6
language
train
test
train
test
DE
0.0022
0.0039
0.0174
0.0213
EN
0.0014
0.0033
0.0120
0.0154
CS
0.0021
0.0035
0.0089
0.0137
FI
0.0028
0.0045
0.0097
0.0135
FR
0.0043
0.0075
0.0182
0.0229
ES
0.0014
0.0036
0.010
0.0149
0.35
0.30
0.25
0.6
0,20
0.4
0.15
0.10
(a) Attention decoder (y-axis) over slots (x-axis).
Attention of slots (x-axis) over input (y- axis) before the LoDrop.
Figure 4: Illustaatoon of the Attention of Slots over the input the Attention of decoder over the slots for Finnish language.
B SETTINGS
B.1 DATA
As for the datasets, we lowercased the text and retained the characters which occur more than 25 times in the corpus, following Kawakami al. (2017). We replace the low-frequent characters with an unknown placeholder. Table I7] the licenses of each dataset that we used in our experiments. used the same train/validation/test splits as provided the mentioned datasets.
B.2 MAIN MODEL SETTINGS
Table shows the remaining list of hyperparameters in training the main model. We scheduled the parameter in the training loss to start with low value of 2 x 10-5 and exponentially increase it every 10 epochs until it reaches certain limit. In particular, with ratio for English until reaching 6.4e - and 1.5 for the rest of languages. More specifically, we stop the exponential increase after reaching and Czech. We tried the stopping thresholds ranging from x 10 to 6 x 10-4, These scheduling values lead to having roughly as many slots the average number BPE units per sentence. We also tried our model with statistic À in the {10-4 [1,3, 3, 5, 6. 7,8 7) x 10-53 which did not lead to stable results at training time. We tried 16, 32 and slots in our experiments. We chose the number of slots to be half of the maximum sequence length (128) as this is reasonable upperbound which also matches the maximum number BPE or Morfessor units. We tried the transformer encoder with and 6 layers but qualitatively did not find any improvements. We the Slot Attention algorithm for T=1 iterations. We choose T=1 iterations for simplicity and efficiency, and because preliminary experiments showed no improvements with more iterations. We leave the investigation of how get
schedule the À exponentially increase
for German and Spanish and 5e 4 for Finmish
16

---

Under review as a conference paper at ICLR 2023
1.0
0.20
0.6
0.15
0.4
0.2
0.05
(a) Attention decoder (y-axis) over slots (x-axis).
Attention of slots (x-axis) over the input (y- axis) before the LoDrop.
Figure 5: Illustration of the Attention of Slots the input the Attention of decoder over the slots for French language.
0.8
08
0.4
0.3
0.2
0.2
02
0.1
(a) stride=2
(b) stride=4
(c) stride=6
Figure 6: Attention of decoder the stride-basee model vectors (x-axis) while generating every character (y-axis). The target sentence is "the red colour associated with lobsters only appears after cooking.".
improvements from more iterations to future work. Finally, we used Adam optimizer (Kingma & Welling 2013) for training our models with learning rate 10 -4 and train our models for 200 epochs.
B.3 FORWARD PROBE SETTINGS
Our probing classifier consists two fully connected layers with ReLU activation function in between the two layers. The hidden dimension of the classifier is the same as the slots' dimension, which is 128.
We use the same datasets as our main model for training and testing the probes. We train BPE with vocabulary size 5000 for all languages. For Morfessor, we use the pretrained model and consider the set of its outputs on the training data as target representation. As for the morpheme targets, we take the morphemes for the words which were available in the linguistically annotated data (i.c., MorphoLex
et al. 2018 Mailhot et al. 20201) and for the rest of the words take Morfessor outputs as approximation of morphemes.
For training the probing classifiers take batch size of 4, since the Hungarian matching algorithm requires a huge amount of memory. We train our classifiers for 200 epochs with Adam optimizer with learning rate of 1 x 10-3
17

---

Under review as a conference paper at ICLR 2023
Table Input and output pairs from Spanish and German datasets. The predictions are sorted based on their matching target. The empty label is shown as "#' and wrong predictions are shown in red.
input
output
una razón de su auge aparente éxito en tratar enfermos por epidemias infecciosas...
[BPE] una razón su auge fue su aparente éxito en tratar enfermos por epittasii infifocosasón
[Morfessor] una razón de su auge fue su aparente Éxito tratar enfermcio por epidemiasts
außerdem wurde er besten spieler des turniers gewählt.
[BPE] außerdem wurde #urffierers gewähl#
spieler des
[Morfessor] außerdem wurde er zutur besten spieler des turniers gewählt #
Table 5: Stride-based models for English with BPE targets
stride
stride=2
stride=3
stridc=4
stride=6
F1
0.8647
0.7965
0.8065
0.7678
Loss( (Iet))
1.318
1.7084
2.2163
3,4569
P
0,9376
0.9125
0.9236
0,9514
0.8106
0.7335
0.732
0.6667
Acc
0.9114
0.8923
0.8811
0.8596
Reverse reconstruction
15.14
-30.44
.49.04
-105.1
Table Slot Attention based models for English with BPE targets
limit (r)
n_inpul/2
n_input/3
n_input/4
n_input/5
n. n_input/6
i_input/8
F1
0.8067
0.8636
0,8457
0.809
0.7878
0.7677
Loss(le+5)
1.5364
1,2951
1.5757
2,0522
2.3737
2.8781
0.8634
0,924
0.9394
0.9555
0,9635
0.9763
0,7687
0,8213
0.7819
0.7196
0.6863
0.6517
Acc
0.8845
0.9139
0.9058
0.8934
0.8865
0.8798
Reverse reconstruction
-00.56
-58.98
-84.32
-129
-150.1
-204.4
Table 7: Stride-based models for English with Morpheme targets
stride
stride=2
stride=3
stride=4
stride=5
stride=6
Fl
0.8108
0.8086
0,7835
0.792
0.7461
Loss(le+5)
2.6374
2.5115
2.6292
2.7607
3.5556
P
0.8971
0.883
0.8812
0.9083
0,9155
0.7479
0.7567
0.7179
0.7179
0.6498
Acc
0.8973
0.8964
0.8833
0.8897
0.8688
Reverse reconstruction
21.26
-51.9
-60.54
-151
-116
Table 8: Slot attention based models for English with Morpheme targets
limit (r)
n-inpul/2
n_input/3
n-input/4
n-input/5
n.input/6
n_input/8
Fl Loss(le+5)
0.7821
0.8394
0.8136
0.8022
0.7814
0.7705
2,0958
1.7118
2.1506
1,9883
2.3083
2.4825
0.8353
0.8844
0.8952
0.9083
0.9157
0.9331
0,7482
0.8101
0.7577
0.7383
0.7039
0.6789
Acc
0.884
0.9116
0.9017
0.9025
0.896
0.8949
Reverse reconstruction
5.975
-43.17
-84.65
-90.6
-123.7
-122.7
18

---

Under review as a conference paper at ICLR 2023
Table 9: Stride-based models for English with Morfessor targets
stride
stride=2
stride=3
stride=4
stride=6
Fl
0,8435
0.8126
0.8087
0.7622
Loss(le+5)
2.7637
4.3381
2,6835
3.7541
0.9175
0,9012
0,9162
0.9458
0,7905
0.7529
0.7388
0.6606
Acc
0.9026
0.8872
0.8842
0,8595
Reverse reconstruction
15.28
-24.36
-48.44
-07.99
Table 10: Slot attention- based models for English with Morfessor targets
limit (r)
n input/2
input/3
n input/4
input/5
n input/6
F1
0.7662
0.8318
0.7892
0.7735
0.7451
Loss(Ic+5)
2.7095
1.9951
2.7874
2.728
3.1524
0.8533
0.9089
0.9128
0.9323
0,9367
R
0.7091
0.7787
0.7087
0.6793
0.6383
Acc
0.8636
0.8975
0.8777
0.8763
0.8668
Reverse reconstruction
-8.193
-57.02
-89.83
-122.8
-140
Table 11: Stride-based models for French with BPE targets
stride
stride=2
stride=3
stride=4
stride=5
stride=6
F1
0.6239
0.8365
0.8219
0.7911
0.754
Loss(Ie+5)
5,3584
2.1264
2.669
3.6093
4.5231
0,8814
0.921
0,9356
0.9485
0,9584
R
0.4916
0.7756
0,7454
0.6939
0.6399
Acc
0.7767
0.885
0.8769
0.8603
0.8392
Reverse reconstruction
-10.51
-58.4
-93.31
-129
-160.7
Table 12: Slot attention-based models for French with BPE targets
limit (r)
n_input/2
n_input/3
n_input/4
n_input/5
n_input/6
F1
0.7941
0.8422
0,8238
0,8082
0.7829
Loss(le+5)
2.0313
1.8495
2.3189
3,0405
3.7684
0.8694
0.9247
0.934
0.964
0.9786
0.7403
0.7826
0.7488
0.7087
0.665
Acc
0.8594
0.8876
0.8789
0.8725
0.8593
Reverse reconstruction
-26.5
-68.98
-105.4
-164.2
-00477
Table 13: Stride-based models for French with Morpheme targets
stride
stride=2
stride=3
stride=4
stride=5
stride=6
F1
0.6409
0.7872
0.7922
0.7772
0.7184
Loss(le+5)
5,5099
3.1225
3,0613
3.6183
4.8101
P
0.9014
0.8906
0.8936
0.914
0.9281
0.5051
0.714
0.722
0.6885
0,5989
Ace
0.8107
0.872
0.8746
0.8682
0.8438
Reverse reconstruction
-5.61
-85.57
-207.7
-204.3
-168.5
B.4 REVERSE PROBE SETTINGS
We illustrate how forward and reverse probing are related to each other in Figure 7 We first assign a one-hot vector the tokens in the target of units (i.e., RS, where S is the target size). Then,
19

---

Under review as a conference paper at ICLR 2023
Table 14: Slot attention-based models for French with Morpheme targets
limit (r)
n_input/2
n_input/3
n_input/4
n_input/5
n_input/6
F1
0.7277
0.7845
0.7976
0,8048
0.7896
Loss(le+5)
3,5534
2.9845
2.835
2.565
3.0282
0.8403
0.876
0,8889
0.9195
94.05
65.26
0.7195
0.7329
0.7269
0.6921
Acc
0.8406
0.8679
0,8762
0,8829
0.8777
Reverse reconstruction
-8.35
-69.87
-148.4
-190.2
-181.9
Table 15: Stride-basee models for French with Morfessor targets
stride
stride=2
stride=3
stride=4
stride=5
stride=6
F1
0.6794
0.7919
0.7902
0.7632
0.704
Loss(Ie+5)
5.5356
3.6372
3.7099
4.5012
5.8283
0.9041
0.9086
0.9189
0.9368
0.9454
0,5533
0.7117
0,7045
0.6578
0.5738
Acc
0.8078
0.8615
0.8619
0.8481
0.821
Reverse reconstruction
-10.33
-56.74
-90.33
-120.3
-150.1
Table 16: Slot attention- based models for French with Morfessor targets
limit (r)
n_input/2
n_input/3
n_input/4
n_input/5
n_input/6
F1
0.7383
0.7979
0.7877
0.7833
0,7621
Loss(Ie+5)
4,2044
3.1744
3,4661
3,4707
4.229
0.867
0.9016
0.9117
0,9466
0.963
0.6474
0.7256
0.7044
0,681
0,6425
Acc
0.8302
0.8646
0,8602
0,8619
0.8521
Reverse reconstruction
-22.51
-66.06
-107.5
-160.8
-205
we learn an embedding layer to map every one-hot vector into continuous vector with dimension 128. Afterwards, we pass the embedded token into two fully connected layers with ReLU activations in between with the hidden dimension 128. We then predict the mean and the standard deviation of Gaussian distribution. The dimension of the Gaussian distribution d is Datats = 128. We use the same matching between the target units as slots as in the forward probing. Namely, we do not run the matching algorithm for this experiment. As for the training objective we minimize the negative log-likes log-likelihood of the slot vector given this distribution, i.e., (og(((mmg) /Ppredctedd; ppedicted)) which cquivalant to minimizing
Σ 2n pietkaed
(m; - Mpredicted - (og(( We train this model with
Adam optimizer and the learning rate 1 x 10 for 200 epochs. We report the best evaluation loss on
test set.
B.5 ARXIV CHALLENGE SETTINGS
We used the challenge set released by Hofmann et al. (2022) which is supposed to serve as benchmark for PLM tokenizers. The task is classify each ArXiv title into its corresponding sub-area (within a category of 20) in the three subjects of CS, Maths, Physics. The dataset requires challenging generalization from a small number of short training examples with highly complex language (Hofmann et al.) 2022). For this reason, believe it would be good benchmark for evaluating our models.
Since our models output set of vectors the high-level representation of the character sequence, design an attention-bel based classifier to perform the task. In particular, we learn a vector which acts as the keys and values. After applying attention
the query where the sequence representations the sequence representations, we apply a 2 layer MLP with ReLU non-linearity to compute the scores for each sub-area. We freeze the representations get from trained models and only train the
20

---

Under review as a conference paper at ICLR 2023
Table 17: Data licenses
dataset
license
WikiTex12 Merity et al.
2017)
Creative
Commons
Attribution-
ShareAlike 3.0 Unported License (link
dataset)
Multilingual Wikipedia Corpus (MWC)
Kawakami et al.
2017)
https:/ /ttaaaaaaaaaaa.............
P P17-1137/
Morpholeex Sunterez-Gutiérrez et al.
2018 Marloo al.
2020
Creative
Commons
Attribution-
NonCommercial--hareeee
4.0
License
(CC BY-NC-SA
4.0)
(nttps://Lindat mFf
cuni htt.:////gooooooo//trtooo.o um
handLe/1123//74422f
Table 18: Hyperparameters of the main model.
module
parameter
value
batch size
16
learning rate
transformer
model dimension
256
transformer
feedforward layer dimension
4 x 256
transformer
dropout rate
0.1
Lrop
0.66
Lg Drop
<
0.1
Slot Attention
Slot dimension (Datots)
128
Slot Attention
MLP hidden dimension GRU hidden dimension
x Dalots
Slot Attention
Dstots
Slot Attention
8
le
Slot Attention
classifier in order to evaluate the quality of induced units. Following Hofmann et al. (2022) we use FI as our evaluation metric and average across the subjects. We compare our Slot-Attentin model with Stride-based model with stride 6 and a character-based model (stride=1). Note that our numbers are not comparable to the ones reported Hofmann al. (2022) because we use a completely different setup (in terms of both data and architecture), and we could not directly compare our models to BPE and Morfessor because these units are discrete instead of vector-based.
C INFRASTRUCTURE
use PyTorch version 1.2.0 framework and Python version 3.6.9 for implementing our code. Table 19 shows the rest of the libraries we use. We code on single GPU with model GTX1080ti and the operating system Debian10 (Buster) 64-bit. We use the same compute for all of our experiments including training the models and probes. The training time for the main model is five hours and for the probings around 2 days. For reporting each of the results we run our algorithm once, since would be too computationally expensive.
21

---

Under review as a conference paper at ICLR 2023
*"asket
"She"
"bal"
"bssket
"pay""
"bal"
Embedding layer
Matching Algorithm
Matching
slots
she
payy
basket
bail
ahe
play
basket
ball
forward probe
reverse probe
Figure An illustration of forward and reverse probing.
Table 19: List of packages and their versions.
package
version
use
NLTK
3.5
and word tokenization BPE implementaion
youtokentome
1.0.5
polyglot
16.7.4
morfessor implementation
matplotlib
3.3.2
attention maps visualization
scipy
1.2.2
hungarian algorithm implementation
numpy
1.19.1
22


Reviews:
[{"review_id": "VHuStswpGUd", "reviewer": "Reviewer_RoTY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This method proposes dynamic slot attention based approach to discover subword units in a completely unsupervised manner. The character based encoder representations are passed through a bottleneck of K slot noisy vectors (K < length of input) and these slot vectors are then used for reconstruction with a lightweight decoder with attention over the slots. In addition to the reconstruction loss, an L0 drop term (relaxed via Hard-Concrete reparametrization) is also added to the training loss and inference to encourage sparseness of slots that are being used. This approach is empiricqally compared to a random baseline and a reasonable stride based approach is also proposed for comparison with a better baseline. Intrinsic evaluation testing correspondence of slots to sub-lexical units like morphemes and BPE validate the effectiveness of this approach against the proposed baseline.", "summary_of_the_review": "Overall, the proposed approach is reasonable and tackles an interesting problem of discovering subword units in a completely unsupervised manner. However, I have slight concern over the lack of extrinsic evaluation and a deeper motivation for the intrinsic properties.", "strength_and_weaknesses": "Strengths:\n\n-- The proposed approach is reasonable with sensible design choices and loss terms.\n\n-- The proposed approach effectively learns contiguous spans that correspond well with human-interpretable morphemes, and automatic subword extraction methods like BPE.\n\nWeaknesses:\n\n-- Although, it is mentioned in the \"limitations\" section, I believe it is a fundamental drawback of the current work that it doesn't show the approach's usefulness over downstream tasks and the evaluation is mostly intrinsic. I find it difficult to articulate the importance of learning subword units with such an expensive procedure. For example, if the proposed use is better tokenization, then experiments that support this should be performed to justify using this expensive approach for tokenization.\n\n-- Related to above, a deeper analysis that focuses on the discovery of sublexical units will strengthen the paper. While this approach is shown to be correlated with morphemes and BPE tokens, these two subword units are intrinsically very different. Hence, analysis of the proposed approach in context of linguistically meaningful units in contrast with engineering-oriented units like BPE should be performed in greater detail.\n\n-- \"Reverse Probing\" is confusing. I don't understand how these experiments were performed. How were the target subword units incorporated into the input of the classifier? Was the learned encoder over characters used for reverse probing. If yes, then isn't there a concern of using an encoder that was learned jointly with the slots for probing over the slots?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MGD-ek2NNoO", "reviewer": "Reviewer_nzdu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method to induce meaningful text units from character sequences. The method is based on Slot Attention for unsupervised object discovery in computer vision. To adapt it to texts, a modification is introduced by using separate trainable parameters for each slot rather than shared. The full model consists of a character-encoder, the modified slot attention module, an L0-drop module to enforce sparsity and a shallow decoder for recovering input sequences, and the full model is trained with an auto-encoding objective. The evaluation of the proposed model is performed by qualitatively visualizing attention maps as well as quantitative probing experiments to measure the correspondence between the induced slots and linguistic or statistical units.", "summary_of_the_review": "This paper explores an interesting question on the unsupervised induction of meaningful text units and proposes a modified Slot-Attention based model for this purpose. Though some of the results and patterns look interesting, it still remains unclear to me how meaningful the slot representations are and there is a lack of further explorations on the effectiveness of them in actual tasks.", "strength_and_weaknesses": "Strength:\n\nThis work explores an interesting direction of letting the neural models induce meaningful linguistic units in an unsupervised way. This might lead to better segmentation/character-level representation methods, and provide more insights on the inner mechanisms of neural models.\n\nWeakness:\n\nI still have some doubts on how meanful the “induced slots” are and I think the two evaluations are not quite enough to show that these slots indeed represent meaningful units. Though the patterns in the attention maps and the obvious improvements in the probing experiments look interesting, I still don’t feel these are enough to arrive at the conclusion that the induced slots are very meaningful, unless there is a more clear way to show what they actually represent. Attention only shows the information may exist but not necessarily that the slots individually represent meaningful units, so does probing (especially with the matching-based probe training).\n\nTo show the effectiveness of the induced slot representations, applications on real text tasks are lacking (except for an initial experiment where there seems to be no obvious performance difference between slot-attn and simpler stride-based representations).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CgaMglEo7TA", "reviewer": "Reviewer_MN1D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method to infer meaningful groupings of characters in a text. It is based on Slot attention method developed in the visual domain to obtain slots (vector representations), corresponding to meaningful parts of an image (objects). The paper adapts this method to work on a sequence of characters. The Slot attention method is used as a bottleneck module inside an encode-decoder Transformer method that takes as input a sequence of characters and is trained to reproduce it. The proposed Dynamic Capacity Slot Attention method can use a different number of slots for each sequence by using a sparsification layer ($L_0$Drop) and uses different initialization for each slot for better optimization. The method seems to produce a meaningful grouping of characters, that is similar to previous methods.\n", "summary_of_the_review": " The paper proposes a good and sound method for grouping elements in a sequence. It is technically sound and seems to produce meaningful grouping but at the moment the purpose of these groupings is unclear. ", "strength_and_weaknesses": "**Strong Points**:\n\nThe proposed method is sound and is capable of producing meaningful groupings of characters.\n\nThe usage of sparsification layer $L_0$Drop (Zhang et al., 2021) to control the number of slots is a good idea for using a different number of slots for each sample.\n\n\n**Weak Points**:\n\nIt is not clear what is the purpose of the grouping of characters. In vision, objects are central for reasoning over real-world scenes, so it makes more sense that obtaining unsupervised entities that are similar to objects is desirable. Do we have any reason to believe that they would be beneficial for reasoning over text? Does discovering these units have a meaning on its own?\n\nAlthough the authors note that evaluating the obtained representations is outside the scope of this work, it should not be the case. We need a reason to do this kind of grouping, and good performance for a downstream task would be ideal. \n\nRegarding the probing: The untrained baseline is too weak, because before training each slot constraints an average of all input tokens, which are also random at first, thus it is extremely unlikely that these representations could be used in a learning task. Maybe a model that averages all the tokens corresponding to the same word be a better baseline. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "90cxnuks6E0", "reviewer": "Reviewer_aQFF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Slot attention (Locatello et al., 2020) was originally designed to learn object-centric representations of images. The authors adapt slot attention to learn text representations, in the hope that those slots would similarly extract meaningful units from sequences of characters. The authors made minor changes to the original design, including a different parameter initialization, an L0-drop layer from Louizos et al. (2018). Experiments show that the models with slot attention 1) produce non-overlapping continuous attention patterns when decoding, and 2) induce subword units that better predict human-labeled morphemes, segmentations from Morfessor, and segmentations from BPE, than stride-attention baselines.", "summary_of_the_review": "The work has novelty as the first to apply slot attention to text. However, the adaptation seems straightforward and no text-specific design was added. It itself won't be a reason to reject but the experimental designs are flawed and that the results do not answer if slot attention is beneficial to text learning. Stronger, more direct experimental support would improve the strength of the paper. ", "strength_and_weaknesses": "### Strengths\n1. Interesting idea to adapt unsupervised (implicit) segmentation from image to text.\n2. The paper is easy to follow, with major motivation, design, and results are clearly stated. There are few under-supported claims or relatively minor points worth clarifying, e.g. LPs P2, Sec 2.1; P6, Sec 4.2; P7, Sec 4.3; P8, Fig 3, but themselves are unlikely to refute the key contributions.\n\n### Weaknesses\n1. Limited technical contribution. No design specific for the language data. More or less a straightforward application of slot attention. This itself won't be a reason to reject if the experimental results are strong. However,\n2. Experimental evidence not strong enough to show that slot attention is beneficial in learning text representation. The weakness is multifold. Roughly categorized into insufficient experiments (a, b), questionable experiment setting (c, d, f, g), weak evidence (e, h, i).\n    - a. The authors trained their model using reconstructive objectives. The performance of this task can show how good a model is at capturing the key meaning and the structure of a sequence. Reporting metrics on the training and validation set and comparing them against the baselines will make a point. See also LP Sec 4.\n    - b. The authors showed no results from downstream tasks. This will be the most direct evidence that slot attention is at least helpful to certain NLP tasks. For example, pos-tagging can show if it captures word meanings or roles; segmentation can show if it learns morphology. Compare against character-level and stride-attention models would provide evidence.\n    - c. Although the authors claim the encoder-decoder information bottleneck being an important factor to the effectiveness of the method, the actual realization does not seem to enforce any meaningful bottleneck and the input sequence can in theory be memorized verbatim. This raises concern on if the setup really facilitates meaningful learning and if the observed behaviors of the learned slots are legit signs of learning. See LP P6, Sec 4.1.\n    - d. In Sec 4.2, the authors visualize the attention of decoder, i.e. output sequence, over slots. Why not inspect the relation between the slot and the input sequence? The observed relation may be a bias from the encoder-decoder bottleneck and has nothing to do with slot attention. Comparison against other bottleneck baselines can improve the support. See also LP Sec 4.2, observations.\n    - e. Unsupported claims that attribute the emergence of continuous span to effectiveness of their model. Need further evidence to determine if this is an inherent bias or actually aligns with the language data. See also P6, Sec 4.2, \"We believe ...\"\n    - f. The probing task of \"predicting reference subword segmentation from the subwords induced from the current model\" (Sec 4.3) seems unnecessarily convoluted. I am not aware of anything like this in the existing work. The authors cited a survey but I was not able to find any discussion of this specific probing or even similar probing. See LP P7, Sec 4.3, top. The forward probing task, the reverse probing task, the design of how to induce units, all seems convoluted and artificial. If the authors are able to induce subword units from the slots, can they go one step further and induce subword segmentations and directly evaluate on the subword segmentation task? Subword segmentation or morphological segmentation is an established task. Many benchmarks available. The authors can also brand it as a downstream task evaluation.\n    - g. In the probing task (Sec 4.4), slot-attention has a possibly unfair advantage that its number of slots are set dynamically. See LP P8, top.\n    - h. No clear or consistent trend found in Fig 3 between the performance of slot-attention and the performance of stride-attention.\n    - i. Preliminary results (Sec Limitations) showed slot-attention is slightly worse than stride-attention (baseline) on the downstream task of text classification, measured by F-1 on ArXiv-L. See LP P9, Limitations, bottom.\n\n\n### Localized points (LPs)\n1. P2, Sec 2.1, \"... where we expect our slots to represent something like the set of morphemes ...\". Can you be specific what are the exact properties you want from this slot formation?\n2. P3, after Eq (1), \"In other words, the initial value of slots are independent samples of a single Normal distribution with learnable parameters.\" This sentence repeats the sentence directly before it.\n3. Sec 4. What are the numbers from reconstructive training?\n4. P6, Sec 4.1, \"We feed in the sentences with less than 128 characters to our model and consider the number of slots as 64 (half of the maximum input length). In addition, we take the dimension of slots as 128.\" Assuming the characters are ASCII, each character can be represented in 8 bits. 128x8 = 64x16 bit. A 128-d real vector is more than enough to fully store 16-bit information and thus the entire sequence can be stored verbatim. I see no bottleneck here.\n5. P6, Sec 4.2, observations on the slot attention pattern. Why do the authors focus on the attention pattern with respect to the output sequence not the input sequence? It could be the case that an arbitrary bottleneck without any of the slot-attention design gives a similar pattern. \"An arbitrary bottleneck\" could be taking stride attention or simply reduce the number of vectors (e.g. the first k vector, or the vector of some special token) or the dimension of the vectors (e.g. reducing from 128d to 16d) being passed from the encoder to decoder. I will be more convinced that the slot-attention really did something non-trivial if a bottleneck itself is unable to give such a pattern.\n6. P6, Sec 4.2, \"We believe that the emergence of contiguous spans is a result of the bottleneck ...\" You can test whether the emergence of spans is learned from the text or merely an inductive bias of the model. You can train the model over languages with nonconcatenative morphology, e.g. Arabic, and see if the learned slot is still contiguous or aligns with the morphology.\n7. P7, Sec 4.3, top, \"Predicting the previously proposed unit from our induced unit is a probing task, as used in previous work (Belinkov & Glass, 2019), ...\" Can the authors point us to which previous work used \"predicting reference subword segmentation from the subwords induced from the current model\" as a probe task? The cited survey does not particularly discuss any probing where one predicts subword units from latent presentation or induced units.\n8. P8, Fig 3. Why are all the NLL losses negative? Negative log-likelihood should take value from [0, +\\infty).\n9. P8, top, \"the number of open gates in the L0Drop layer is approximately the same as the target number.\" The number of tokens generated from stride-attention (baseline) is fixed but the number of tokens (determined by the number of gates) is set dynamically. This is an unfair advantage.\n10. P9, Limitations, bottom, \"For the ArXiv-L, on average, we get 0.396, 0.409,\n0.2732 F1 for slot-attn, stride=6, and character-based models, respectively, on Dev set, and 0.393, 0.394, 0.271 on Test set.\" It seems slot-attn <= stride-attn from these preliminary results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject