Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention
Authors: Melika Behjati
James Henderson
Venue: Accepted by TMLR ()

Abstract: Characters do not convey meaning, but sequences of characters do.  We propose an unsupervised distributional method to learn the abstract meaning-bearing units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence, extending an architecture for object discovery in images.  We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers.  These experiments show that our model succeeds in discovering units which are similar to those proposed previously in form, content, and level of abstraction, and which show promise for capturing meaningful information at a higher level of abstraction.

Published in Transactions on Machine Learning Research (10/2023)
Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention
Melika Behjati
melika. melika.behjati@@epf..cc
Ecole Polytechnique Fédérale de Lausanne (EPFL)
Idiap Reseorch Institute
James Henderson
jamess.endersoon@hiiie
Idiap Research Institude
Reviewed on OpenReview: https: /ttpp://openwyeee optp//veen..ee..e. net/forun' m8U9rSa6gU
Abstract
Characters do not convey meaning, but sequences of characters do. We propose an unsuper- vised distributional method learn the abstract meaningful units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence, extending an architecture for object discovery in images. We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers. These experiments show that our model succeeds in discovering units which are similar to those proposed previously in form, content and level of abstraction, and which show promise for capturing meaningful information at higher level of abstraction.
1 Introduction
When we look a complex scene, we perceive its constituent objects, and their properties such as shape and material. Similarly, what we perceive when we read piece of text builds on the word-like units is composed of, namely morphemes, the smallest meaning- units in language. This paper investigates deep learning models which discover abstract representations of such meaningful units from the distribution of character sequences in natural text.
In recent years, there has been an emerging interest in unsupervised object discovery in vision Eslami et al. 2016 Greff et al. 2019 Engeleke et al. 2020 Elsayed et al. 2022 Seitzer el al. 2023). The goal is to segment the scene into objects without supervision and obtain an object-centric representation of the scene. These representations should lead better generalization
unknown scenes, and additionally should facilitate abstract reasoning over the image. Locatello et al. (2200) proposed relatively simple and generic algorithm for discovering objects called Slot Atlention, which iteratively finds set of feature vectors (i.e., slots) which can bind to any object in the image through a form of attention.
Inspired by this line of work in vision, our task to learn a set abstract continuous representations of the objects in text. We adapt the Slot Attention module (Locatello al. 2020) for this purpose, extending it for discovering the meaningful units in natural language character sequences. This makes our proposed task closely related to unsupervised morphology learning (Creutz)) 2003 Narasimhan al. 2015 Eskander el. al.) 2020). However, there are fundamental differences between task and morphology learning. First, learn to represent a text with a set of vectors, which are not explicitly tied to the text segments. Second, our model learns its representations by considering the entire input sentence, rather than individual space-delinited words. These properties of our induced representations make our method appropriate for inducing meaningful units as part of deep learning models.
In particular, we integrate our unit discovery method on top of the encoder in Transformer auto-encoder Vaswani et al. 2017), depicted in Figure 1 and train with unsupervised sentence reconstruction

---

Published in Transactions on Machine Learning Research (10/2023)
P
y
8
8
Transformer Decoder
she
play
ed
basket bail
Lo Drop
she
play
ed
basket bail
Slot Attention
Transformer Encoder
8
a
d
Figure 1: Dynamic Capacity Slot, Attention. First, the Transformer encoder encodes the sequence and then, Slot Attention computes the slot vectors (highlighted text). Next, the LoDrop layer dynamically prunes out the unneccssary slots and the decoder finally reconstructs the original sequence.
objective. This setting differs from previous work Slot Attention, which has been tested on synthetic image data with limited number of objects (Locatello et al. 2020 We propose several extentions to Slot Attention for the domain of real text data. We increase the capacity of the model to learn to distinguish large number of textual units, and add the ability to learn how many units are needed to encode sequences with varying length and complexity. Thus, we refer to our method as Dynamic Capacity Slot Atlention. Additionally, as a hand-coded alternative, we propose stride- based models and compare them empirically to our slot attention based models.
To evaluate the induced representations both qualitatively inspect. the model itself and quantitatively compare it to previously proposed representations. Visualisation of its attention patterns shows that the model has learned representations similar to the contiguous segmentations of traditional tokenization approaches.
Trained probing classifiers show that the induced units capture similar abstractions to the units previously proposed in morphological annotations (MorphoLex "SánchezGGutérree, et al.] 2018 Mailhot et al. 2020) and tokenization methods (Morfessor (Virpioja et al. 2013), BPE (Sennrich et al. 2016]). To compare two representations, we propose to do this probing evaluation in both directions, to compare both informativeness and abstractness of the units. These evaluations show promising results in the ability of our models discover units which capture meaningful information at higher level of abstraction than characters.
To summarize, our contributions are follows: (i) We propose novel model for learning meaningful units from a sequence of characters (Section 2p. (ii) We propose simple stride-based models which serve as strong basclines for evaluating such unsupervised models (Section 2.3). (iii) We analyze the induced units by visualizing the attention maps of the decoder over the slots and observe the desired sparse and contiguous patterns (Section 4.3). (iv) We propose bi-directional probing method evaluate such models by comparing the induced units to given units in terms of both their informativeness and their abstractness (Section 2.4 (v) We show that our induced units capture meaningful information at appropriate level of abstraction by probing their equivalence to previously proposed meaningful units (Section 4.4p. (vi) We evaluate the induced units in downstream task to show its potential benefits over character-level and sogmentation-based representations (Section 4.5
2 Approach
The unsupervised induction of abstract meaningful units from sequences of characters is a novel task which requires both novel methods and novel evaluations. It should allow the induction of meaningful units part of deep learning models. In this section we define the task, propose two types of models, and propose a probing- evaluation.
2

---

Published in Transactions on Machine Learning Research (10/2023)
Algorithm Slot Attention module (Locatello et al. 2020). q, k map the slots and inputs to common dimension D and T denotes the number of iterations.
Require: imputs E x RNapur slots ~ (N, ( E RRXx Kx Datats
inputs = TayerNorm(inputs)
for i = 1 to do
slots pooss_rev_____ slots slots LayerNorm(slots)
attn Softmax( J/ k(inputs)_q(sloss)) axis 'slots')
updates = WeighttedMeennweighte = attn + values = (inputs))
slots = GRU(states = slots slots_proy, prev, input updates) )
slots += MLP (LayerNorm(slott))
end for return slots
2.1 Problem Formulation
Given a sequence of N characters X= 22 ...2...... EN, we want to compute a set of vectors M {mi..... m H which best represents X at higher level of abstraction. We expect that each vector (i.e. slot) the set M should represent a treaning-bearigg unit in the text X, so that the induced representation M captures the desired level of abstraction. As example, consider the sequence "she played basketball", where we expect each of our vectors to represent one of the morphemes of the sequence, namely {she, play, -ed, baskel, -ball]. Although the mraning-bearing units are often assumed to come from fixed vocabulary, we do not assume this and compute the vectors dynamically from the full text.
2.2 Dynamic Capacity Slot Attention
We learn our representations through encoding the input sequence into slots and then reconstructing the original sequence from them. In particular, we use an auto-encoder structure where slots act as the bottleneck between the encoder and the decoder. Figure I shows an overview of our proposed model, Dynamic Capacity Slot Attention. First, we encode the input character sequence by a Transformer encoder (Vaswani et al., 2017), which gives one vector per character. Then, we apply higher-capacity version of Slot Attention module (Locatello et al., 2020) over the encoded sequence, to learn the slots. Inluitively, Slot Atlention will learn a soft clustering over the input where each cluster (or respectively slot) corresponds to a candidate mcaningful unit in the sequence. To select which candidates are needed to represent the input, we integrate an Lo regularizing layer, ie., Drop layer (Zhang et al., 2021), on top of the slots. Since the maximum number of slots is fixed during the course of training, this layer ensures that the model only uses as many slots necessary for the particular input. This stops the model from converging to trivial solutions for short inputs, such passing every character through separate slot. Finally, the Transformer decoder reconstructs the input sequence autoregressively using attention over the set of slots.
Encoder. We use the Transformer encoder architecture for encoding our sequence (Vaswani et al. 2017 and obtain the representation X' = z//z2.... from our input sequence X.
Slot Attention for Text. After encoding the character sequence, we use our extended version of Slot Attention for discovering meaningful units of the input character sequence. Slot Attention recent method for unsupervised object representation learning in vision (Locatello et al.) 2020 It learns set of feature vectors (slots) by using iterative attention based algorithm. Algorithm I shows the pseudo code of this method. Abstractly, in every iteration, it takes the following steps. First, it computes an attention map belween the slots and the inputs, with slots acting as queries and inputs as keys. Then, it normalizes the attention map over the slots, which makes the slots compete for representing each token of the input. Afterwards, it computes the slots' updates as the (normalized) weighted mcan over the attention weights and the input values. Finally, updates the slots through a Gated Recurrent Unit (GRU) Choo al. 2014) followed by residual MLP. This process iterates fixed number of times.
3

---

Published in Transactions on Machine Learning Research (10/2023)
In Locatello et al. (2020), the slots are initialized randomly by sampling from Normal distribution with shared learnable parameters and a, L.e,
slot, ~ (Jsshrrdd Sshared)-
(1)
In other words, the initial value of slots are independent samples of a single Normal distribution with learnable parameters. We found in our experiments that this initialization method does not lead to good results in the text domain (see Appendix [..1 for a comparison). In contrast with the experimental setting of Locatello ctt al. (2020); which used artificially generated data with a small number and range of objects, real language requires learning about very large vocabulary of morphemes, and each example can have large number of morphemes. This suggests that model for language needs a more informative initialization with more trainable parameters, in order to have the capacity to learn about this large vocabulary and distinguish all the objects in a long sentence.
To investigate this issue, we propose another initialization for adapting Slot Attention to text. We consider a separate learnable μ per slot and we fix the a to a predefined value for all the slots. Namely, the slots are initialized as
slot, N Fconstant).
(2)
By assigning separate μ for each slot, the initialization has many more trainable parameters. This allows the model to learn about different kinds of units, such as ones that occur at different positions, or ones that have different. lypes of forms, but we do not make any assumptions about what those differences might be. In addition, the intuition behind fixing the a is to prevent it from collapsing to zero. In particular, since the number of possible u-grams in text is finite but the slots can have any continuous value in the space of R Ratoos, the slots tend to learn an arbitrary mapping from n-grams in the input to the slots while turning o to zero. In this case, there is no need for the slots to learn the underlying meaning-bearing units. By fixing the a to a nonzero value, the initialization effectively introduces noise into slot vectors and thereby limits the amount of information which can be passed through each slot, from the information theoretic point of view. This bottleneck forces the slots to compress information in a meaningful way. We then obtain the set of slots M = (m.... mx} = Attenttiio)))(
Neural Sparsification Layer: Drop. The number of units needed to represent a sequence varies among different sequences in the data. In contrast to the object discovery work where the data is generated synthetically and thercby, the number of objects in the scene is known beforchand, we do not make any assumptions about the number of units required. Therefore, of required units and prune the extra ones per input sequence. We accomplish this goal by using a neural sparsification layer called Drop (Zhang et al. 2021]) It allows our model to dynamically decide on the number of required units for every input sequence.
consider upper-bound over the number
This layer consists of stochastic binary-like gates g = ..... that for every input m, works as
LgDrop(m:) = g.m,-
(3)
When 9i is zero the gate is closod and when is one the whole input is passed. Each gate is a continuous random variable in the [0, 1] interval, sampled from hard-concrete distribution (Louizos et al. 2018). This distribution assigns most of its probability mass over its endpoints (i.e. 0 and 1) in favour of the sparsification goal. Specifically, 9i ~ HardConcrete(q;.B3 B,c) where B and are hyperparameters. a; is predicted as a function of the encoder output m,, i.e., loga, = m,w", where w is a learnable vector. This allows the model to dynamically decide which inputs to pass and which ones to prune. The Lo penalty, which yields the expected number of open gates, is computed as:
(o(M) = Σ - 1-pppg = 0la,,
(4)
where the probability of g, being exactly provided in closod form Louizos et al. (2018). We follow the same approach as Louizos et al. (2018) at evaluation time and consider the expectation of each gate as its value. We refer to the pruned slots after applying the LoDrop layer M' ml m....... m'g-
4

---

Published in Transactions on Machine Learning Research (10/2023)
Decoder. Lastly, regenerate the imput sequence from the set of slots by using a simple, shallow decoder. To this end, we use one-layer Transformer decoder (Vaswani et al. 2017] with single attention head over the slots. A simple decoder forces the slots to learn representations with straightforward relationship to the input, which we expect to be more meaningful. In other words, we do not use a powerful decoder because it would be able to decode even low-quality representations of the input, which are less meaningful (Bowman et al. 2015).
Training Objective. We train our model end-to-end by using Gumble trick for sampling HardConcrete variables (Maddison et al. 2017 Jang et al. 2017 The training objective is
Lree (X, Co(M)
llog (Eglo(X])r)) + X (M(M)
(5)
S p( + Lo(M = ((X),
which consists of the reconstruction loss from the decoder (Irec) and the Lo penalty for the open gates. Hyperparameter X, the sparsification level, controls the trade-off between the two losses. In practice we find that in order to impose enough sparsity in the slots, we should slightly increase À during the course of training using schoduling techniques.
Training Objective with Targeted Sparsity, Controlling the level of sparsity by setting can be difficult, so we provide an alternative version of the loss with an explicit target sparsity rate r (Mai & Henderson 2022
We replace the Lo term in Equation with max(Lo, \/rxN), to have
((X) = Eg log (X] À max(Co(M)) 1/rXN)
(6)
where 1/r indicates the proportion of slots that we want to keep open and N the maximum input length. In this sctup, the model will stop closing more gates when Lg reaches the target 1/r N !
2.3 Stride based Models
We propose a simple hand-crafted alternative model to our induced units which can gain acceptable results in terms of performance. We design this model by replacing our Dynamic Capacity Slot Attention module with a linear-size bottleneck. In particular, we take 1 out of every k encoder outputs and down project them PRppt + R Patots) to obtain the representations. In other words, we only pass certain encoder outputs based their position and drop the rest.
M DownProjen((sfel -...+11---.. ...
where n = 1 We can get different alternative models by varying the stride k. The training objective .M log p(X(M) The idea of using stride-based models has also been used Clark et al. (2022 Tay et (2022) in a different context and setup.
is the reconstruction loss Cee (X, (X,
2.4 Bi-directional Probing Evaluation
Since the task is unsupervised and we do not evaluate using artificially generated data, there is no obvious gold standard for what units a method should learn. To quantitatively evaluate how well a model performs, we freeze the trained model and train probing classifiers to measure to what extent the discovered units capture the same information as previously proposed meaningful units. We use multiple previously proposed reprosentations which have been shown to be good levels of abstraction for range of NLP applications. If the induced representation separates the required information into units with similar level of abstraction, then we can expect that they will also be effective in NLP applications.
We propose to mcasure whether two reprosentations provide the same level of abstraction by mcasuring whether there is a one-to-one mapping between their respective units such that two aligned units contain the same information, meaning that each unit can be predicted from the other. Predicting the previously proposed unit from our induced unit is a probing task, as used in previous work (Belinkov & Glass] 2019
1 Note that this would he effective when 1/r K, where K maximum number of slots.
5

---

Published in Transactions on Machine Learning Research (10/2023)
which we specify as forward probing. We refer to the prediction of our induced unit from the previously proposed unit as reverse probing task, which we believe is a novel form of evaluation We compare various models on their trade-off between forward and reverse probing.
Forward probing. This is the common way of probing where we want to measure if our induced units include the information about the target representation. We train a classifier with shared parameters on each of our slots individually and obtain a set of predictions {f(m)), f(mm)))(mm))...... f(mxx) )} one per slot. As we are dealing with a set, need to find a one-to-one matching between the classifier's predictions and the target tokens. Therefore,
follow Locatello et al. (2020) to use the Hungarian matching algorithm (Kuhn 1955 for finding the match which minimizes the classification loss. The same alignment method is applied both at training and at testing time.
We consider the complete set of slots after applying the LoDrop layer as the inputs to our classifier. Slots whose Drorop gate is closod are simply input as zero vectors. This gives us a fixed number of vectors. The two sides of matching should have the same size to obtain a one-to-one match, therefore, we add an extra target label (i.e., empty) for representing the pruned slots. Due to the fact that many slots are pruned out, considering a measure like accuracy could be misleading, since classifier which outputs empty label will achieve very high accuracy. Therefore, we build a confusion matrix as follows. We consider all non-empty labels as positive and the emply ones as negative, and we report precision (P); recall (R) and F1 measure, to better reflect what the slots have learned.
Reverse probing. To evaluate whether the induced units capture the same level of abstraction, we also need to evaluate whether the induced units abstract away from the same information as the previously proposed units. We propose to measure this by training reverse probing classifiers, which predict cach induced unit from its aligned larget unit. Because the induced unit is continuous vector, we predict the parameters of a d-dimensional Gaussian distribution with diagonal covariance, and measure the density function of the induced vector in this distribution. The probe is trained to minimise this negative-log-density function, and this loss on the test set is used as our evaluation measure 3
3 Related Work
Unsupervised Object Discovery. There is a recent line of research in the image domain for discovering objects in scene without explicit supervision, and building an object-centric representation of them. Most. of this work is built around the idea of compositionality of the scenes. MONet (Burgess et al. 2019) and
GENESIS (Engelcke et al.;
2020) similarly use recurrent attention network for learning the location masks
of the objects. Greff et al.
22016
2017 2019); Van Steenkiste et al. 2018); Emami et al. (0021) model the
scene as a spatial Gaussian mixture model. Furthermore, AIR network (Eslami et [al]] and its variants
(Crawford & Pineau 2019 Lin et al. 2020) model objects from a geometric perspective by defining three
specific latent variables. Lately, Locatello et al. (2020) propose an attention-based algorithm (namely Slot Attention) learn object representations (slots) which have been followed by [Singh et al. (2022) ; Sajjadi et al.
(2022): : Seitzer et al. [2023); Singh ct al.) (2023) and Kipf al. (002)); Elsayed et al. (2022) further expanded it to the video domain. In contrast to this line of work in vision, our approach is specifically designed for text. We use additional components (e.g., DoDp layer) in our architecture to resolve the requirements of modeling textual data. Furthermore, our model is trained and evaluated on real text datasets, in contrast to these previous models which have only been shown to be effective on synthetic scenes.
Unsupervised morphology learning. This subject has been of interest for many years in the NLP field Elman 1990: Creutz & Lagus 2002; Baroni et al. 2002). Morphemes have strong linguistic motivations, and are practically important in many downstream tasks because they are the smallest meaning-bearin units in language (Can & Manandhar 2014). Many approaches have been proposed for discovering the underlying morphemes or morpheme segmentations. Morfessor variants are based on probabilistic machine learning
methods (MDL, ML, MAP) for morphological segmentation (Creutz, 2003 Creutz & Lagus 2002 2005 2007:
Virpioja et al. 2013 Some researchers take a Bayesian approach for modeling word formation Poon et al,
2Scc Figure the Appendix for illustration of forward and reverse probing Note that we are modelling& continuous distribution, the density function can have values higher than and therefore, the Sogative-logarithm of it could be negative.
6

---

Published in Transactions on Machine Learning Research (10/2023)
2009
Narasimhan et al. 2015 Bergmanis & Goldwater 2017 Luo et al. another approach for modeling morphological inflections (Sirts & Goldwater 2013 Eskander et al. 2016
2017 Adaptor Grammars are
2019: 2020). In addition, Xu et al. (2018) 2020) built their models upon the notion of paradigms, set of morphological categorics that can be applied to a set of words. Moreover, Soricut & Och (2015) Ustün
& Can 2016) extract morphemes by considering the semantic relations between words the continuous embedding space. [ao Rei (2016) propose to learn word embeddings by applying a bi-directional RNN
with attention over the character sequence. Furthermore, Ataaman al. (2020) model word formation as
latent variables which mimic morphological inflections in the task of machine translation.
Our work differs from the previous work in classical morphology learning in two ways. First, instcad of explicitly discovering morphemes, we learn a set of continuous vector representations of the input, which would then need to be processed to extract the morphemes. The model itself has no explicit relation between these unit representations and segments of the input. Second, our model learns representations of an entire input sentence, rather than individual spaco-delimited words. This makes fewer assumptions about morphemes, and considers the context. of the words in sentence. Our work is similar to Ataman al. (2020) in modeling morphology implicitly in the latent space. However, we employ self-supervised objective for our purpose which is more general compared to their supervised loss, as we do not need labeled data.
Unsupervised character segmentation. Learning to segment a character sequence in unsupervised fashion is another relevant area to our work. Chung et al. (017)) propose Hierarchical Multi-scale RNNs for modeling different levels of abstractions the input sequence. In the language modeling task, they observe that the first layer roughly segmenting the sequence into words, namely at space boundaries. [unn & Deng (0118) propose Segmental Language Models for Chinese word segmentation. Moreover, in (Kawakami et al. 2019)
the authors design a model to learn the latent word segments in a whitesppce-removed character sequence with a language modeling objective. As we mentioned earlier, we learn continuous vector representations of text which is different from explicitly detecting discrete character segments,
Learning intermediate representations from characters. Recently, a line of work has been proposed to model language at the level of characters and then aggregate the characters into subword-like units and have the core Transformer layers on top of them. CANINE Clark al. (2022) learns these units by applying local attention and strided convolutions and proves the usefulness of their model in multilingual tasks. Charformer Tay et al. (2022) learns representation for each character by computing a weighted sum over character k-grams that this character is involved in and then performs mean pooling to downsample the representation size. The goal of our works differs from theirs as we are focusing on learning meaning-bearing units at the level of morphemes and we are not focusing on improving downstream tasks' performance directly.
Unsupervised speech unit discovery. Unsupervised unit discovery has also been studied in the speech domain, where the speech data includes both acoustic and language information. Wav2vec 2.0 Baevski et al. (020 encodes the raw input using a convolutional neural network. Then, it learns units in the representation by discretizing the output of the feature encoder using product quantization Jegou et al. (201)). HuBERT Hsu et al. (2021) follows the same architecture as Wav2vee 2.0 but they discover acoustic units leveraging K-means algorithm. Authors in Tjandra et al. (0200) extract subword units given speech audio by a variational autoencoder with a finite set of discrete latent variables (aka limited codebook) called vector quantized variational autoencoder (VQ-VAE) Van Den Oord ct al. (2017).
Subword discovery algorithms. This set of algorithms have become a standard component of NLP models in recent years. LByte-Pair-Encodding (BPE) (Sennrich et al. 2016) iteratively merges the two consecutive tokens with the highest frequency. Word-picce (Schuster & Nakajima 2012). sentenco-piece (Kudo & Richardson 2018) and unigram LM (Kudo 2018) are other similar subword tokenization algorithms. In contrast to these methods, which mostly use local statistical information of the data, our model is trained over complete sentences to learn an abstract sophisticated representation.
4 Experiments
We train our models as auto-cncoders and evaluate the resulting induced units. In addition considering the ability of the induced representations to reconstruct the input (their supervised objective), we evaluate
7

---

Published in Transactions on Machine Learning Research (10/2023)
Table 1: Reconstruction error on training and test set among different languages.
slot attention((-6)
stride-6
language
train
test
train
test
DE
0.0022
0.0039
0.0174
0.0213
EN
0.0014
0.0033
0.0120
0.0154
CS
0.0021
0.0035
0.0089
0.0137
FI
0.0028
0.0045
0.0097
0.0135
FR 0.0043
0.0075
0.0182
0.0229
ES
0.0014
0.0036
0.010
0.0149
our unsupervised model both by visualizing attention maps to show what characters the slots correspond to (Section 4.3), and by bi-directional probing of the slot vectors (Section 4.4b. In addition, we show the
potential usefulness of our induced slots a downstream task (Section 4.5
4.1
Experimental Setup
We apply our model to languages from different morphological typologies. We select English (EN), German (DE), French (FR), Spanish (ES) and Czech (CS) from the fusional family and Finnish (FI) from the agglutinative typology. For English we use the raw Wikitext2 dataset (Merity et al. use Multilingual Wikipodia Corpus (MWC) (Kawakami et al.) 2017).
2017): For the rest, we
As for the models, use standard Transformer architecture (Vaswani al. 2017) with model dimension 256. The encoder consists of 2 layers with self-attention heads and the decoder consists of layer with 1 self-attention head and 1 attention head over the slots. We feed in the sentences with less than 128 characters to our model and consider the maximum number of slots as 64 (half of the maximum input length). In addition, we take the dimension of slots as 128. We scheduled the À parameter in the training loss to start with a low value and exponentially increase every 10 epochs until reaches a certain limit. We obtain this limit manually in a way that the final number of open gates roughly equals the average number of BPE tokens in sequence. More details of the settings available in the Appendix B
For the forward probing classifier, we train 2 layer MLP target token given the slot vector. For the reverse probing classifier, we first pass the target tokens into an Embedding layer and then apply a 2 layer MLP on top of each embedding individually to predict the parameters of a Gaussian distribution over slot voctors.
probing classifier f, which predicts the
4.2 Reconstruction Results
The models are trained to reconstruct the input character sequence, so the reconstruction error (reported in Table D indicates how effectively the model can compress information about the text in the available units" vectors. We compare the slot attention based model with targeted sparsity objective (Equation (6), with the strido-based models (stride=6) ensure that both models have the same average number of units. In all languages, slot attention based models are better able to reconstruct the sequence. This indicates that the slot attention vectors are better than the stride-basee vectors at capturing information about the structure and meaning of the input, thereby providing better signals to the decoder to reconstruct the sequence.
4.3 Visualization
In order to show some qualitative results of our model, we visualize the attention maps for generating every output, shown in Figure? 2 In particular, show the altention of the decoder over slots when generaling every output character 4 Interestingly, although we do not impose any sparsity in the decoder's attention
weights, the attention maps are quite sparse. Namely, at each generation step only a few slots are attended, and each slot is attended while generating only a fow characters. In addition, although we do not impose
4We observe the same pattern in the attention slats over input and have illustrated some examples Appendix A.2.1
8

---

Published in Transactions on Machine Learning Research (10/2023)
0.8
0.6
(a) epoch 5
(b) epoch 50
(c) epoch 125
Figure Attention of the decoder over slots (x-axis) for generating every target character (y-axis) after different training epochs, for target sequence "the red colour associated with lobsters only appears after cooking.".
any bias towards discovering segments of the input, the characters which are generated while attending to a given slot are contiguous in the string (the vertical bands in Figure 2]. We believe that the emergence of contiguous spans is result of the bottleneck we create with our Dynamic Capacity Slot Attention. This means that the model is trying to put correlated information about the input in the same vector, so that it represent the string more efficiently. The strongest correlations in the character string are local in the inpul, so each slot tends to represent characters which are next to each other in the input.
In early steps of training, when the sparsity ratio (A) is small, each slot tends to represent & bigram of characters (2a] and later on, trigrams (2b). These observations confirm the necessity of the LrDrop layer for converging better units. particular, as the ratio increases, the number of active slots reduces and they become more specialized in representing contiguous greaning-bearing units of input. For instance, the word cooking in 2c is represented by two slots cook and ing. That these segments roughly correspond to the morphemes which want the model to discover, is verified quantitatively in the probing experiments in Section 4.4 We provide attention map visualizations for other languages and stride-based models in
Appendix
4.3.1 Learned Positional Information
We observed similar attention patterns for the slots across different input samples and thereby, we conjecture that there must be a correlation between what the slots have learned and the corresponding positions in the input sequence. To evaluate this, we average the attention maps between all the samples from the test set and visualize them in Figure 3 The averaged attention map verifies our hypothesis that the slots are highly correlated with the position of characters in the sequence. We think that this average is related what the 14 is learning, but that after the Slot Attention iterations the slots carry more information than just positions.
4.4 Probing Results
Given target set of units and trained model for finding units, align the two sets of units, and use forward probing to see whether each induced unit contains as much information as its aligned target unit and use reverse probing to see whether each induced unit contains more information than its aligned target unit. Optimizing both measures requires having the same information as each target unit in its associated induced vector, and hence having the same level of abstraction.
The primary mechanism we have to control the amount information in an induced representation is control the number of units. More units means that there more opportunities for the alignment. to find

---

Published in Transactions on Machine Learning Research (10/2023)
20
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Figure 3: Average attention maps across all the test set samples. x-axis corresponds slots and y-axis the decoder's output position. The later positions (bottom) occur in fewer examples, but almost all slots show clear preference for specific positions.
unit which correctly predicts each target unit, but makes it harder to predict all the units from the available target units. We take two approaches to controlling the number of induced units. First, we fix the number of units to a target number, which allows an efficient comparison of models. Second, we vary the number of units, which allows us to evaluate how well different models can match the informativeness trade-off of the target representation.
Probing data. We consider three target representations which either have linguistic or practical evidence of being effective in NLP applications. For the languages for which an in-depth linguistic morphological analysis is available, we compare to these morphemes (i.c., EN and FR) (Zabokrtský al. 2022; Gunthez-Guttérrez
et al. 2018 Mailhot. et. al.) 2020). Alternatively, as linguistically inspired units, we have the Morfessor (Virpioja et al. 2013) outputs, and BPEs (Sennrich et al. 2016) as trequency-based subwords.
Targeted informativencss. As a less computationally expensive initial evaluation, we use hyperparametors to set the number of induced units to approximately the same as the number of targel BPE tokens. For the stride-based models, we simply the stride according to the average target number of tokens (i.e., stride= ( to be equal to the stride value.
= 5). For the slot attention models, set the in Equation
We use probing to compare slot attention and stride-based models to an uninformative baseline represent representation itio (untrained), thereby controlling for the power of the probing classifier to learn arbitrary mappings (Conneau et al. 2018 Oord et al., 2018). This baseline is the set of slot vectors output by a randomly iiniooly-nnitializeedd slot. attention model without any training.
Table 2 shows the results of the forward and reverse probing tasks on different languages. As the results show, the trained slots achieve much higher performance in all the tasks in comparison to the random baselines. Our model achieves very high precision in predicting the non-empty labels. Its performance is weaker on the recall side, but here the improvement over the untrained model is even more pronounced. This seems to be due to the imbalance between emply and non-emply labels in the training set, where the emply labels comprise around 66% of the data for the probing classifier. For this reason, below we will use F1 as our overall performance measure for forward probing.
For most languages and targets, our slot attention model performs better than the comparable stride-based model. In the cases where the stride-based model has higher F1, its worse for reverse probing suggest.
10

---

Published in Transactions on Machine Learning Research (10/2023)
Table 2: Forward and reverse probing results on different languages, with a target number of units. Note that numan-annotaii morphemes are only available for En and FR.
BPE
Morfessor
Morphemes
lang Model
P
R
Rev
R
F1
Rev
F1
Rev
EN
untrained
0.74
0.10
0.17
-47.12
0.71
0.11
0.19
-36.53
0.69
0.11
0.19
-50.29
stride=
0.95
0.66
0.76
-105.1
0.94
0.66
0.76
-07.99
0.91
0.64
0.74
-116
slot-attu(r 6)
0.96
0.68
0.78
-150.1
0.93
0.63
0.74
-140
0.91
0.70
0.78
-123.7
FI
untrained
0.74
0.09
0.17
-72.03
0.90
0.06
0.11
299.2
stride=
0.94
0.63
0.74
-123.7
0.89
0.43
0.57
-112.6
slot-attn(r = 5)
0.94 0.58
0.71
-104.7
0.89
0.50
-80.62 0.63
FR
untrained
0.75 0.08
0.14
-51.59
0.72
0.11
0.19
-88.05
0.74
0.11
0.19
-68
stride=
0.94 0.69
0.79
-129
0.93
0.65
0.76 120.3
0.91
0.68
0.77
-204.3
slot-attn(r = 5)
0.96 0.70
0.80
-164.2
0.94
0.68
-160.8 0.78
0.91
0.72
0.80
- 190.2
ES
untrained
0.71 0.09
0.17
-37.85
0.71
0.11
0.19
-44.63
stride=
0.94 0.67
0.77
-103.5
0.93
0.65
0.75
-131.6
slot-attn(r== = 5)
0.94 0.69
0.78
-94.83
0.93
0.66
0.76 - 125.7
DE
untrained
0.91 0.08
0.16
446.91
0.76
0.08
0.15 -64.87
stride= slot-attn(r = 5)
0.95 0.66
0.77
-170.1
0.91
0.58
0.69 - 156.1
0.97 0.68
0.79
-190.8
0.93
0.66
0.76 -15..9
CS
untrained
0.94 0.08
0.16
22.98
0.86
0.05
0.10
-3.69
stride= 5
0.94 0.63
0.74
-1045
0.90
0.52
0.65 87.16
slot-attn(r = 5)
0.96
0.62
0.74
-164.5
0.95
0.47
0.62 148.1
that this may just be the result of an informativeness trade-off. For the majority of cases where the slot attention model has a higher F1, its reverse probing score also better. The performance gain of our slot attention models over the stride-based models is particularly noticcable for the two languages where we have real morpheme annotations, which implies that our slot attention based method is more effective in discovering linguistically meaningful units. The agglutinative language (Finnish) is generally harder than the fusional languages, but slot attention does relatively well at capturing the linguistically-mote motivated Morfessor targets compared to the strido-based model. We further show some examples from the predictions of the forward probing classifiers in the Appendix A.3
Informativeness trade-off. The advantage of our two-directional probing evaluation that we can directly compare the informativeness trade-off of different models by plotting their probing results in two-dimensional graphs. To vary this trade-ofT, we train stride-based models with different strides, stride= 3, 4, 5,,,6), and slot attention based models with different r, where = stride, for the same strides. This leads to the two models having the same range of values for their number of valid units.
Figure plots this informativeness trade-off for the same languages and target representations as in Table 2 Better results are ncarer the lower right corner of the plot. Overall, the slot attention based models provide better representations than the stride-based models. About half the plots show clear trend in favour of slot attention based models, and none of the plots show clear trend in favour of the stride-based models. But many of the plots show substantial amount of variability.
These results justify our design choices for the slot altention based models to achieve the goal of discovering meaningful units which have the same level of abstraction as the previously proposed units. In addition, although our sarid models fall behind the slot attention basod models in many cases, they can be utilized as strong basclines for evaluating such unsupervised models and to find acceptable abstract units which omit the need for tokenizing text preprocessing step.
11

---

Published in Transactions on Machine Learning Research (10/2023)
-50
-30
-50
-60
-150
-50
250
<250
-150
0.76
0.80
0.85
0.45
0.55
0.65
0.75
0.6
0.7
0.8
0.85
0.70
0.75
0.80
(a) EN-BPE
(b) F1-BBE
(c) FR-BPE
(d) ES-BPP
-50
-60
-100
-100
-150
-100
-200
-100
0.65
0.70
0.75
0.80
0.55
0.65
0,75
0.750
0.775
0,500
0.826
0.65
0.60
0.55
(e) DE-BPE
(f) CS-BPE
(g) EN-Morfessor
(h) FI-Morfessor
-50
-60
-125
-100
-100
125
-250
-150
260
200
0.85
0.55
0.65
0.75
0.80
0.65
0.70
0.78
(i) FR-Morfessor
(i) MoS-Moresssor
(k) DE-Morfessor
(1) CS-Morfessor
-00
-50
-100
-150
250
(m) EN-Morphemes
(n) FR- FR-Morphemes
Figure 4: 2D graphs showing the informativennss trade-off for slot altentonn models (blue points) and the trride-based models (red points). The x-axis shows the Fl measure for the forward probe and the y-axis shows the negative log-density function loss for the reverse probe, so better models are lower and further to the right.
4.5 Downstream Task Evaluation
The focus of this work is developing the analoguc of visual object discovery methods in the text domain, and developing an intrinsic evaluation framework for this line of work. We showed above that the information captured by our induced units are similar to the information previously proposed representations, and thus there is every reason believe that the demonstrable effectiveness of these predefined representations will also apply to our induced reprcsentations. To confirm the effectiveness of our induced meaningful units in downstream tasks, report here results using our induced representations in downstream task which we believe to be representative of their potential.
As our downstream task, we used the challenge set released by Hofmann et al. 2022)) which they argue serves as a benchmark for pretrained language models' tokenizers. The task to classify each ArXiv title into its corresponding sub-arca (within a category of 20) in the three subjects of CS, Maths, Physics. The dataset requires challenging generalization from small number of short training examples with highly complex language (Hofmann et al. 2022 For this reason, we believe it is good benchmark for evaluating our unit induction models.
In order to evaluate the quality of the induced units, we first. freeze the representations we get from our trained models and then train classifiers for this downstream task. Since our models output a set of vectors as the high-level representation of the character soquence, we design an attention-baset based classifier perform the task. In particular, after applying a shared layer MLP (with ReLU non-linearity) on top of each of the representation's vectors, we apply attention over the resulting keys and values using learnable query
12

---

Published in Transactions on Machine Learning Research (10/2023)
Dev
slot-attn stride-based
Cs
Maths
Physics
Avg
0.45
stride=1
0.3053
0.3315
0.3867
0.3411
stride=6
0.3919
0.3976
0,468
0.4191
0.40
BPE-based
0.3196
0,3383
0.3693
0.3424
0.35
slot-attn(r-6)
0.3911
0.4229
0.4742
0.4294
Test
0.30
Cs
Maths
Physics
Avg
0.25
2
4
6
8
10
stride=1
0.2934
0.3181
0.3839
0.3318
stride=6
0.3833
0.3863
0.4529
0.4075
stride/r
BPE-based
0.3095
0.345
0.3578
0.3374
(a) Results with varying stride/r on the test set.
slot-atln(r=6)
0.397
0.424
0.4682
0.4297
Table 3: Arxiv-L classification results without fine-tuning the models.
vector. Then, we linearly project the resulting vector to compute the score for each sub-area, and apply softmax with cross-entropy loss. Following [ofmman et al. (2022). we use F1 average across the subjects. We compare our Slot Attention model with Stride-based based model with stride 6 and a character-based model (stride=1) in addition to BPE-based model, as shown in Table 3 order to be fair in comparison, we train the Based baseline model under the same setup as character- based model (stride=1) by only modifying the inputs and targets to be BPE tokens of the sentence. These results indicate that the representations induced by our proposed models work much better than a character-based and BPF--based model on this challenging task, confirming that they could be useful replacement for tokenization methods in downstream tasks. The slot attention representations also perform better than the stride-based representations. The BPE-based model does not work well as other models and we believe this is due to the fact that BPE struggles to find good tokenization of the rare technical words found in ArXiv topics and usually splits them into meaningless tokens.
evaluation metric and
Since there is no target representation in this evaluation, we also consider varying the target percentage of vectors in the induced representatious, shown in Figure 3a with induced units which are around the same granularity as those used in the BPE target reprosentation (s.ride=6). For the stride-based models, results gramularity, whereas our slot attention model is more robust to the choice of the target level of granularity. Perhaps this is due to the ability of its DrDrop layer to adjust the number of units depending on the content. of each individual example.
Interestingly, the best performance is achieved
very dependent on finding the right hand-coded level of
5 Conclusions
In this paper, propose Dynamic Capacity Slot Attention for discovering meaningful units in text in unsupervised fashion. We use auto-encoder architecture for encoding character sequence into set of continuous slot vectors and decoding them to reconstruct the original sequence. We enforced the set of slots to act as a bottleneck in transferring information between the encoder and the decoder by adding a constant noise to their vectors and integrating an Lo regularizing layer on top of the slots which only retains the necessary vectors. In addition, we propose a set of stride-based models which could serve as an alternative our main model. We evaluate our model by probing the equivalence between the pruned slots and predefined tokens. In particular, show that our representations effectively capture meaningful information at higher level of abstraction. Moreover, we show the usefulness of our induced units in a challenging classification task.
propose do reverse probing well the normal way of probing. Our experiments
Note that numbers are comparable to the ones reported in Hofmann different setup of both pre-training data and architecture)
2022 because we use completely
13

---

Published in Transactions on Machine Learning Research (10/2023)
This work is first step towards replacing hand-coded abstract units in text with representations induced within deep learning architecture, demonstrating that it is feasible to improve over hand-coded units, even in the text domain. More generally, it addresses the issue of entity induction, which is a fundamental property of cognitive representations but is currently poorly understood. We believe that the proposed models and evaluation method will lead to greater progress on this fundamental problem.
Future work. One of the advantages of our unsupervised approach to inducing abstract units is that the method is not specific to characters, or indeed to any observable level of representation. Thus it can be trained in an end-to-end fashion with different training objectives, it can be used to induce any level of abstraction, and it can be stacked to induce multiple levels of abstraction. This could potentially avoid the need to hand-code different levels of abstraction, such as vectors associated with tokenization and sentence markers. Replacing hand-coded model design choices with choices which are induced within a deep learning architecture is a central objective of this work.
Limitations. We learn a set of abstract continuous vector representations of the input and we do not explicitly discover morphemes morpheme segments. Although the visualized attention maps show interesting patterns of imput segmentation (see Figure 2), the vertical bands are fading near the end-points. More specifically, it is not forward to determine boundaries between the bands as the transition is done smoothly due the continuous nature of the altention function. For this reason, we could not obtain good segmentations by employing simple heuristics on the attention maps.
Reproducibility Statement. We have completely explained the details of our experiments Appendix It includes the data have used and how we have processed it, in addition the models' parameters and training details.
Acknowledgments
We would like to thank Florian Mai, Andreas Marfurt, Andrei Catalin Coman, and Fabio Fehr for their valuable input and discussions at various stages of this project. We would like to extend our thanks to the anonymous reviewers for their constructive feedback and insights, which significantly contributed the improvement of this work. Melika Behjati was funded by the Swiss National Centre of Competence Research (NCCR) under the project Evolving Language, grant number "51NF40 180888".
References
Duygu Ataman, Wilker Aziz, and Alexandra Birch. A latent morphology model for open-vocabulary neural machine translation. In International Conference on Learning Representations (ICLR), 2020. URL
Alexci Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli wav2vec 2.0: framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-12460 2020.
Marco Baroni, Johannes Matiasck, and Harald Trost. Unsupervised discovery of morphologically related words based hothograp and semantic similarity. Tn Proceedings of the ACT-02 Workshop on Morphological and Phomological Learning, pp. 48-57. Association for Computational Linguistics, July 2002. doi: 10.3115/
1118647.1118653 URL
orgooog////. wrrg/w0-00606
Yonatan Belinkov and James Glass. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 7:49 72, 04 2019. ISSN 2307-387X. doi: 10.1162/ /tacl a
00254. URL https://doii httl/dii.ogg/// 10. 1ee1111ttaaae ///////aal__00 a_0___000555. 00254
Toms Bergmanis and Sharon Goldwater. From segmentation to analyses: a probabilistic model for unsupervised morphology induction. In Proceedings of the 15th Conference of the Europeun Chapler of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 337 346, 2017.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowiez, and Samy Bengio. Generating sentences from continuous space. Xio preprint arXin: 151.:051.00644 106649, 2015.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation, 2019.
Burcu Can and Suresh Manandhar. Methods and algorithms for unsupervised learning of morphology. In International Conference on Intelligent Tert Processing and Computational Linguistics, pp. 177 205. Springer, 2014.
Kris Cao and Marek Rei. A joint model for word embedding and word morphology. In Proceedings of the 1sl Workshop on Representation Learning for NLP, pp. 18 26, 2016.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Damitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation. In FMNLP, 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations (ICLR), 2017.
Jonathan H. Clark, Dan Garrette, Iulia Ture, and John Wieting. Canine: Pre-training efficient tokenization- free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73 91, 2022. doi: 10.1162/taci a 00448. URL
org/ 2022/022.... tacl-1.5
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126 2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/ P18-1199 URL httpss//aaclatthologg..org
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3412-3420, 2019.
Mathias Creutz. Unsupervised segmentation of words using prior distributions of morph length and frequency. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics- Volume 1, pp. 280 287. Association for Computational Linguistics, 2003.
Mathias Creutz and Krista Lagus. Unsupervised discovery of morphemes. In Proceedings of the ACL-02 workshop on Morphological and phonological learning- Volume 6, pp. 21-30. Association for Computational Linguistics, 2002.
Mathias Creutz and Krista Lagus. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Helsinki University of Technology Helsinki, 2005.
Mathias Creutz and Krista Lagus. Unsupervised models for morpheme segmentation and morphology learning.
ACM Transactions Speech and Language Processing (TSLP), 4(1):1-34, 2007.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-25) 211, 1990. URL https //
onl mulineilbbrr...m wely.wiless www.wwwwww.cco.. wttpa//./.ommmo/////...... /ttp://......0011151166009900000
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. +avi++: Towards end-to-end object-centric learning from real-world videos. Advances in Newral Information Processing Systems, 35:28940 28954, 2022.
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2970-2981 PMLR, 18-24 24 Jul 2021. URL https::/proceeiings... press/v139/ett e0:3m//mmmmm22111 html
15

---

Published in Transactions on Machine Learning Research (10/2023)
Martin Engeleke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations (ICLR), 2020. URL
Ramy Eskander, Owen Rambow, and Tianchun Yang. Extending the use of Adaptor Grammars for unsupervised morphological segmentation of unseen languages. In Proceedings of COLING 2016, the 26th International Conference on Computational Lingwistics: Technical Pupers, pp. 900 910, Osaka, Japan, December 2016. The COLING 2016 Organizing Commiltee. URL http::///aclaathoooggg www.:/oogoooooooooe/ cr6/116-1086
Ramy Eskander, Judith Klavans, and Smaranda Muresan. Unsupervised morphological segmentation for low-resource polysynthetic languages. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pp- 189- 195,, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 18.11865111////
URL atttps:/aaclanthologg. org/ (erg/W99-4222
Ramy Eskander, Francesca Callejas, Elizabeth Nichols, Judith Klavans, and Smaranda Muresan. MorphA- Gram, evaluation and framework for unsupervised morphological segmentation. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 7112-7122, Marseille, France, May 2020. European Language Resources Association. ISBN 979- 10991095554--4-4 URL aclbbeb aclweb. org/ wwrgganthology/// 2020. 1rec-1.879
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukeuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, 2016.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, volume 29, pp. 4484 4492, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Newral Information Processing Systems, 2017.
Klaus Greff, Raphaäl Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, 2019.
Valentin Hofmann, Hinrich Schuctze, and Janet Pierrchumbert. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 385-393, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.18653/v1 2202:/2222222220rrreee.e URL
attps://aclanthology) wttg/2022.co wwl.:2..cclla... short. 43.
Woi-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supevvsedd speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451- 34660, 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriiation with gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pollern analysis and machine indelligence, 3311):117-,28, 2010.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to create and reuse words in voca-lacab neural language modeling. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1492-1502, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. (0.18633/11/P17-1133 URL http::://ellaathooog...o///////
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to discover, ground and use words with segmental neural language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6429 6441, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. N0.1863/v11/1191666 URL
16

---

Published in Transactions on Machine Learning Research (10/2023)
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013.
Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference on Learning Representations, 2022. URL httpp:://penrrevie nwt.:e/u.ee/fforrrtee id=aD7uesX1GF
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.11853/v11P18--1 1007. URL http:s////ccaattoooooooo////////
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing, 2018.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 (1-2):83-99) 97, 1955.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahm. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations (ICLR), 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33, 2020.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through la regularization. In International Conference on Learning Representations (ICLR), 2018.
Jiaming Luo, Karthik Narasimhan, and Regina Barzilay, Unsupervised learning of morphological forests. Transactions of the Association for Computational Linguistics, 5:353 364, 2017. doi: 10.1162/tad a. 00066.
URL http:////accaattolooooorrrg/////
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (TCLR), 2017.
Florian Mai and James Henderson. Oag-of-vectors autoencoders for unsupervised conditional text generation. In Proceedings of the 2nd Conference of the Asia-Pacijy Chapter of the Association for Computational Linguisties and the 12th International Joinl Conference on Natural Language Processing, pp. 468 488, 2022.
Hugo Mailhot, Maximiliano A. Wilson, Joël Macoir, S. Hélène Deacon, and Claudia H. Sánchez-Gutiérre, MorphoLex-FR: A derivational morphological database for 38,840 French words. Behavior Research Methods, 52(():1008-100 1025, June 2020. ISSN 1554-3528. doi: 10.3788/s133228-019-02927
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. Tn International Conference on Learning Representations (ICLR), 2017.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. An unsupervised method for uncovering morphological chains. Transactions of the Association for Computational Linguistics, 3:137-167, 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018.
Hoifung Poon, Colin Cherry, and Kristina Toutanova. Unsupervised morphological segmentation with log-linear models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North A American Chapter of the Association for Computational Linguistics, pp. 209-217, Boulder, Colorado, June 2009. Association for Computational Linguistics. URL http:p:://elaaattoooooooo///////
17

---

Published in Transactions on Machine Learning Research (10/2023)
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-999 9524, 2022.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5149-5152. IEEE, 2012.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zictlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://opearreveew peet/foorum//dns9tU---.. _G
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeling of the Associalion for Compulational Linguisties (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653 (1/66/3/1/P161116/// URL
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-e learns to compose. In International Conference on Learning Representations, 2022. URL http:///gpppaaooeee..eeeeeerroap
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh International Conference on Learning Representations, 2023. URL. http:p//w/ppppeeeeee...e//////aa
Kairit Sirts and Sharon Goldwater. Mininally-superviieet morphological segmentation using Adaptor Grammars. Transactions of the Association for Computational Linguistics, 1:255 266, 2013. doi: 10.1162/
tacl a 00225. URL
Radu Soricut and Franz Och. Unsupervised morphology induction using word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1627 1637, Denver, Colorado, May June 2015. Association for Computational Linguistics. doi: 10.3115/vl 10.3115/11/115-11666 11115-11 URL httpp://acclaathhooogg... htwpoooyyoro/NN11111111111
Zhiqing Sun and Zhi-Hong Deng. Unsupervised neural word segmentation for Chinese via segmental language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4915-4920, Brusscls, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 00.18653/11/D18-1533 1531. URL httpps///cclaataaooooooo////-1-/
Claudia H. Sanchez-Cutiérree, Hugo Mailhot, S. Hélène Deacon, and Maximiliano A. Wilson. MorphoLex: A derivational morphological database for 70,000 English words. Behavior Research Methods, (0(4):1568-158) August 2018. ISSN 1554-3528. doi: 10.3758 77775//113388-01-008-111
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient- based subword tokenization. In International Conference Learning Representations, 2022. URL
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Transformer vq-vae for unsupervised unit discovery and speech synthesis: Zerospeech 2020 challenge. arXiv preprint arXi::2005.1100 2020.
Ahmet Üstün and Bureu Can. Unsupervised morphological segmentation using neural word embeddings. In Pavel Král and Carlos Martín-Vide (eds.), Statistical Language and Speech Processing, pp. 43-53, Cham, 2016. Springer International Publishing. ISBN 978-3-319-459257.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in newral information processing systems, 30, 2017.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions, 2018.
18

---

Published in Transactions on Machine Learning Research (10/2023)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Iilia Polosukhin. Attention is all you need. In Advances in newral informalion processing systems, pp- 5998 6008, 2017.
Sami Virpioja, Peter Smit, Stig-Arne Grönroos, and Mikko Kurimo. Morfessor 2.0: Python implementation and extensions for morfessor bascline. In Aalto University publication series. Department of Signal Processing and Acoustics, Aalto Universily, 2013.
Hongzhi Xu, Mitchell Marcus, Charles Yang, and Lyle Ungar. Unsupervised morphology learning with statistical paradigms. In Proceedings of the 27th International Conference on Computational Linguistios, pp. 44-54, Santa. Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL
Hongzhi Xu, Jordan Kodner, Mitchell Marcus, and Charles Yang. Modcling morphological typology for unsupervised learning of language morphology. In Proceedings of the 58th Annual Meeling of the Association for Computational Linguistics, pp. 6672 6681, Online, July 2020. Association for Computational Linguistics. URL httpps://ww https://ww..alleb..oorr wwt.w//wwo..//aaoooo//////////// acl-main..cmmmmmm 596
doi:
Zdenčk Zabokrtský, Nyati Bafna, Jan Bodnár, Lukáš Kyjánek, Emil Svoboda, Magda Ševčíková, Jonás Vidra, Sachi Angle, Ebrahim Ansari, Timofey Arkhangelskiy, Khuyagbaatar Batsuren, Gábor Bella, Pier Marco Bertinetto, Olivier Bonami, Chiara Celata, Michael Daniel, Alexei Fedorenko, Matea Filko, Fausto Giunchiglia, Hamid Haghdoost, Nabil Hathout, Irina Khomchenkova, Victoria Khurshudyan, Dmitri Levonian, Elconora Litta, Maria Modvedeva, S. N. Muralikrishna, Fiammetta Namer, Mahshid Nikravesh, Sebastian Padó, Marco Passarolti, Vladimir Plungian, Alexey Polyakov, Mihail Polapov, Mishra Pruthwik, Ashwath Rao B, Sergei Rubakov, Husain Samar, Dipti Misra Sharma, Jan Snajder, Krešimir Šojat, Vanja Stefanec, Luigi Talamo, Delphine Tribout, Daniil Vodolazsky, Arseniy Vydrin, Aigul Zakirova, and Britta Zeller. Universal segmentations 1.0 (UniSegments 1.0), 2022. http:///h//oo handle wwt/1133//ttaaaaaee 4629 HINDAT/CLARIAL
digital library the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Biao Zhang, Tvan Titov, and Rico Sennrich. On sparsifying encoder outputs sequence-to-seqquecc models. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2888-2900, Online, August 2021. Association for Computational Linguisties. doi: htt.1////22211.111eeeeee URL https://acllanthology .rg. 2021 findings-acl..25555
19

---

Published in Transactions on Machine Learning Research (10/2023)
A Supplementary Results
A.1 Slot Initialization Analysis
Table 4: Forward probing results of different slot. initializations for Spanish langnage.
BPE
Morfessor
Init.
Model
P
R
F1
P
R
F1
eq-
untrained
0.71
0.09
0.17
0.71
0.11
0.19
(ours)
slot-attm
0.96
0.73
0.82
0.95
0.74
0.83
eq.
untrained
0.80
0.10
0.18
0.78
0.12
0.20
slot-atin
0.97
0.51
0.66
0.96
0.51
0.66
We compare our proposed initialization with the original definition of Slot Attention, where slots randomly initialized from single shared distribution. Table shows the probing results of our model under different slot initializations. Having separate μ for each slot (Equation 22) increases the capacity of the model and thus yields better results in comparison to sharing between the slots. The model especially achieves higher recall this case, which implics better recovery of the BPE and Morfessor tokens as can accurately model a greater number of units.
A.2
Visualization of the Attention Maps
A.2.1 Attention of Slots over the Input
In Figures 5 and we illustrate the attention of slots over the input as well as the attention of decoder over the slots for the same imput. The two attention maps show similar patterns, but on the decoder side the attention weights are higher and therefore the patters more visible. We believe that at each generation step, the decoder only attends to the slots which contain the information about that specific character and thus, the attention patterns are stronger at decoding time. For these reason, we used the attention of the decoder over the slots in section 4.3 Interestingly, in Figure 6b there are some slots which are only attending the space boundaries (the horizontal bands).
A.2.2 Decoder Attention for the Stride- Based models
Figure 7 visualizes the attention of decoder over the vectors of different stride-based models. As expected, the vertical bands are often of the same length and too much overlapping for larger strides. As a result, the bands do not correspond to meaningful units in the imput in contrast to the slot attention based model (see Figure 2ch.
A.3 Forward Probing Examples
Table shows two examples of the probing classifiers' predictions given the learned slots. As explained in the model is quite precise in predicting non-empty labels.
A.4 Bidirectional Probing Results' Tables
Tables
16 show the detailed results of the performance of forward and reverse probing tasks visualized in
Figure (Reverse reconstruction F1). input denotes the input length which is 128 in experiments.
20

---

Published in Transactions on Machine Learning Research (10/2023)
1.0
0.35
0.30
0.8
0.25
0.6
0.20
0.4
0.15
0.10
0.05
(a) Attention of decoder (y-axis) over slots (x-axis).
(b) Attention of slots (x-axis) over the input (y-axis) before the LrDro.
Figure 5: Illustration of the Attention of Slots over the input vs the Attention of decoder over the slots for Finnish language.
1.0
0.25
0.8
0.20
0.6
0.15
0.4
0.10
0.2
0.05
0.0
(a) Attention of decoder (y-axis) over slots (x-axis).
(b) Attention of slots (X-axis) over the input (y-axis) before the Lrop.
Figure 6: Illustration of the Attention of Slots over the input vs the Attention of decoder over the slots for French language.
B
Settings
B.1
Data
for the datasets, we lowercased the text and retained the characters which occur more than times in the corpus, following Kawakami al. (2017). We replace the low-frequent characters with an unknown placeholder. Table 18 shows the licenses of each dataset that we used in our experiments. We used the same train htt/vvaaaaaiiootteeee /est splits provided in the mentioned datasets.
21

---

Published in Transactions on Machine Learning Research (10/2023)
0.7
0.6
0.5
0.4
0.4
0.4
0.3
0.2
-02
0.1
(a) stride=2
(b) stride=1
(c) stride=6
Figure 7: Attention of decoder over the basede model vectors (x-axis) while generating every character (y-axis). The larget sentence is "the red colour associaled with lobslers only appears after cooking.".
Table 5: Input and output pairs from Spanish and German datasets. The predictions sorted based on their matching target. The empty label is shown as "#' and wrong predictions are shown in red.
input
probing classifier prediction
razón de auge fue aparente éxito tratar enfermos por epidemias infecciosas....
[BPE] una razón de auge fue aparente éxito tratar enfermos por epi#as#f inf#ocossasón
[Morfessor] razón de auge fue su aparente éxito en tratar enfermcio por epidemias#s .rs
außerdem wurde er zum besten spieler des turniers gewählt
[PЕ] außerdem wurde zum sten spieler des #ur gewähl#
[Morfessor] außerden wurde er zulur besten spieler des turniers gewählt #
Table 6: Stride-based models for English with BPE targets
stride
stride=2
stride=3
stride=4
stride=6
F1
0.8647
0.7965
0.8065
0.7678
Loss(le+5)
1.318
1.7084
2.2163
3.4569
P
0.9376
0.9125
0.9236
0.9514
R
0.8106
0.7335
0.732
0.6667
Ace
0.9114
0.8923
0.8811
0.8596
Reverse reconstruction
15.14
-30.44
-49.04
-105.1
Table 7: Slot Attention based models for English with BPE targets
limit (r)
2=2
r=3
r=4
r=5
r=6
r=8
F1
0.8067
0.8636
0.8457
0.809
0.7878
0.7677
Loss (1e+))
1.5364
1.2951
1.5757
2.0522
2.3737
2.8781
P
0.8634
0.924
0.9394
0.9555
0.9635
0.9763
R
0.7687
0.8213
0.7819
0.7196
0.6863
0.6517
Acc
0.8845
0.9139
0.9058
0.8934
0.8865
0.8798
Reverse reconstruction
-10.66
-58.88
-84.22
-129
-150.1
-204.4
22

---

Published in Transactions on Machine Learning Research (10/2023)
Table Stride- based models for English with Morpheme targets
stride
stride-2
stride=3
stride=4
stride=5
stride=6
F1
0.8108
0.8086
0.7835
0.792
0.7461
Loss(1e+5)
2.6374
2.5115
2.6292
2.7607
3.5556
P
0.8971
0.883
0.8812
0.9083
0.9155
0.7479
0.7567
0.7179
0.7179
0.6498
Acc
0.8973
0.8964
0.8833
0.8897
0.8688
Reverse reconstruction
21.26
-51.9
-60.54
-151
-116
Table 9: Slot attention based models for English with Morpheme largets
limit (r)
r=2
r=3
r=4
8
F1
0.7821
0.8394
0.8136
0.8022
0.7814
0.7705
Loss(1e+5)
2.0958
1.7118
2.1506
1.9883
2.3083
2.4825
P
0.8353
0.8844
0.8952
0.9083
0.9157
0.9331
R
0.7482
0.8101
0.7577
0.7383
0.7039
0.6789
Acc
0.884
0.9116
0.9017
0.9025
0.896
0.8949
Reverse reconstruction
5.975
-43.17
-84.65
-90.6
-123.7
-122.7
Table 10: Stride-based models for English with Morfessor targets
stride
stride=2
stride=3
stride=4
stride=6
F1
0.8435
0.8126
0.8087
0.7622
Loss(le+5)
2. 2.7637
4.3381
2.6835
3.7541
0.9175
0.9012
0.9162
0.9458
R
0.7905
0.7529
0.7388
0,6606
Acc
0.9026
0.8872
0.8842
0.8595
Reverse reconstruction
15.28
-24.36
-48.84
-107.9
Table 11: Slot attention-based models for English with Morfessor targets
limit (r)
1=2
r=3
r=4
r=5
r= 6
F1
0.7662
0.8318
0.7892
0.7735
0.7451
Loss(le+5)
2.7095
1.9951
2.7874
2.728
3.1524
P
0.8533
0.9089
0.9128
0.9323
0.9367
0.7091
0.7787
0.7087
0.6793
0.6383
Ace
0.8636
0.8975
0.8777
0.8763
0.8668
Reverse reconstruction
-8.193
-57.02
-89.83
-122.8
-140
Table 12: based models for French with BPE targets
stride
stride=2
stride=3
stride=4
stride=5
stride=6
F1
0.6239
0.8365
0.8219
0.7911
0.754
Loss (1e+5)
5.3584
2.1264
2.669
3.6093
4.5231
P
0.8814
0.921
0.9356
0.9485
0.9584
0.4916
0.7756
0.7454
0.6939
0.6399
Acc
0.7767
0.885
0.8769
0.8603
0.8392
Reverse reconstruction
-10.51
-58.4
-93.31
-129
-160.7
23

---

Published in Transactions on Machine Learning Research (10/2023)
Table 13: Slot attention- based models for French with BPE targets
limit (r)
r=2
r=3
r=4
r=5
r=
F1
0.7941
0.8422
0.8238
0.8082
0.7829
Loss(1e+))
2.0313
1.8495
2.3189
3.0405
3.7684
P
0,8694
0.9247
0.934
0.964
0.9786
0.7403
0.7826
0.7488
0.7087
0.665
Ace
0.8594
0.8876
0.8789
0.8725
0.8593
Reverse reconstruction
-26.5
-68.98
-105.4
-164.2
-204.7
Table 14: Stride-based models for French with Morpheme targets
stride
stride=2
stride=3
stride=4
stride=5
stride=6
F1
0.6409
0.7872
0.7922
0.7772
0.7184
(oss(1e+5)
5.5099
3.1225
3.0613
3.6183
4.8101
P
0.9014
0.8906
0.8936
0.914
0.9281
R.
0.5051
0.714
0.722
0.6885
0.5989
Ace
0.8107
0.872
0.8746
0.8682
0.8438
Reverse reconstruction
-5.61
-85.57
-207.7
-204.3
-168.5
Table 15: Slot attention basedd models for French with Morpheme targets
limit (r)
r=2
r=3
r=
r=
F1
0.7277
0.7845
0.7976
0.8048
0.7896
Loss(le+5)
3,5534
2.9845
2.835
2.565
3.0282
0,8403
0.876
0.8889
0.9195
94.05
R
65.26
0.7195
0.7329
0.7269
0.6921
Acc
0.8406
0.8679
0.8762
0.8829
0.8777
Reverse reconstruction
-8.35
-69.87
-148.4
-190.2
-181.9
Table 16: Stride-based models for French with Morfessor targets
stride
stride-2
stride=3
stride=4
stride=5
stride=6
F1
0.6794
0.7919
0.7902
0.7632
0.704
Loss(le+t)
5.5356
3.6372
3. 3.7999
4.5012
5.8283
P
0.9041
0.9086
0.9189
0.9368
0.9454
R
0.5533
0.7117
0.7045
0.6578
0.5738
Acc
0.8078
0.8615
0.8619
0.8481
0.821
Reverse reconstruction
-10.33
-56.74
-90.33
-120.3
-150.1
Table 17: Slot attention- based models for French with Morfessor targets
limit (r)
r=2
r=3
r=4
r=5
r=6
F1
0.7383
0.7979
0.7877
0.7833
0.7621
Loss(1e+5)
4.2044
3.1744
3.4661
3.4707
4.229
P
0.867
0.9016
0.9117
0.9466
0.963
0.6474
0.7256
0.7044
0.681
0.6425
Acc
0.8302
0.8646
0.8602
0.8619
0.8521
Reverse reconstruction
-22.51
-66.06
-077.5
-160.8
-205
24

---

Published in Transactions on Machine Learning Research (10/2023)
Table 18: Data licenses
dataset
license
WikiText2 (Merity et al. 2017
Creative Commons Attribation-ShareeAiin 3.0
Unported License (link dataset
Multilingual Wikipedia Corpus (MWC) d kami et al.
2017
attps://acll httpss:/acaaaahhlogeeo org///7-/11/3
MerphoLex Saachez-Gutiérree al. 2018 Mailhot et al.
2020
Creative Commonss
Attribution-
NonCommervial--haacee
4.0
Li-
cense (CC BY-NC-SA
4.0)
httpps
=ff cuni.
Table 19: Hyperparameters of the main model.
module
parameter
value
batch size
16
learning rate
1e 4
transformer
model dimension
256
transformer
feedforward layer dimension
256
transformer
dropout rate
0.1
Drop
B
0.66
Drop
0.1
Slot Attention
Slot dimension (Datots)
128
Slot Attention
MLP hidden dimension
2 Dstots
Slot Attention
GRU hidden dimension
Dalots
Slot Attention
5
le - 8
Slot Atlention
T
1
B.2 Main Model Settings
Table
19 shows the remaining list of hyperparameters in training the main model. We scheduled the X parameter in the training loss to start with low value of x 10-3 and exponentially increase it every 10 epochs until it reaches certain limit. In particular, for English until reaching 6.4e - and 1.5 for the rest of languages. More specifically, we stop the exponential increase after reaching 3e 4 for German and Spanish and 5e - 4 for French, Finnish and Czech. We tried the stopping thresholds ranging from 3 x 10-4 to 6 x 10-4. These scheduling values lead to having roughly as many slots as the average number BPE units per sentence. We also tried our model with statistic À in the {10-4, 3,, 6, 10 5} which did not lead to stable results at training time. We tried 16, 32 and 64 slots in experiments. We chose the number of slots to be half of the maximum sequence length (128) this is reasonable upperbound which also matches the maximum number of BPE or Morfessor units. We tried the transformer encoder with 4 and 6 layers but qualitatively did not find any improvements. We run the Slot Attention algorithm for T=1 iterations. We choose T==1 iterations for simplicity and efficiency, and because preliminary experiments showed no improvements with more iterations. We leave the invostigation of how get improvements from more iterations to future work. Finally, we used Adam optimizer (Kingına) & Welling 2013) for training our models with learning rate 14 and train our models for 200 epochs.
schedule the À to exponentially increase with ratio 2
B.3 Forward Probe Settings
We train a probing classifier for mapping slot's vector to the target representation's vocabulary, namely, [ [m() Pater + R where S is the number of tokens of the target representation. We apply the classifier with shared parameters over each of our slots and obtain a set of predictions, i.e., (f({) (m)) As we are dealing with a set, during training we need find onc-to-one matching between the classificr's 's predictions and the target tokens. Therefore, we use the Hungarian matching algorithm Kuhn (1955) for finding the best match in terms of minimizing the classification loss. Consider the best matching as i, j,
}}-
25

---

Published in Transactions on Machine Learning Research (10/2023)
which matches the iyth slot with the jth output (i.c., B4)- We then compute the loss L = (f(mm)) where is the cross-entropy between the predictions and targets.
Our probing classifier consists of two fully connected layers with ReLU activation function in between the two layers. The hidden dimension of the classifier is the same as the slots' dimension, which is 128.
We use the same datasets as our main model for training and testing the probes. We train BPE with a vocabulary size of 5000 for all languages. For Morfessor, we use the pretrained model and consider the set of its outputs on the training data the morphemes for the words which were available in the linguistically annotated data (i.e., MorphoT.ex (Sanchez-Gutiéreee et al. 2018 Mailhot et al. 2020)) and for the rest of the words we take Morfessor outputs as an approximation of morphemes.
target representation. As for the morpheme targets, take
For training the probing classifiers we take a batch size of 4, since the Hungarian matching algorithm requires huge amount of memory. We train our classifiers for 200 epochs with Adam optimizer with learning rate of 10-3
"basker"
"she"
"play""
ball
"bosket"
"she"
"play"
"ball"
Embedding layer
Matching Algorithm
Matching
MU
slots
she
play
ed
basket
she
play
ed
basket
forward probe
reverse probe
Figure 8: An illustration of forward and reverse probing.
B.4 Reverse Probe Settings
We illustrate how forward and reverse probing are related to each other in Figure g We first assign a one-hot vector the tokens in the target set of units (i.e., R where S the target set size). Then, we Icarn an embedding layer to map every onc-hot vector into continuous vector with dimension 128. Afterward, we pass the embedded loken into two fully connected layers with ReLU activations in between with the hidden dimension 128. We then predict the mean and the standard deviation of Gaussian distribution. The dimension of the Gaussian distribution d is Dstots 128. We use the same matching between the target units and slots as in the forward probing. Namely, we do not run the matching algorithm for this experiment. As for the training objective, we minimize the negative log-likelihood of the slot vector given this distribution, i.e., -log(p(m; Mpredicted: predicted)) which is equivalent to minimizing (m; (predicted) log( (rredditted) We train this model with Adam optimizer and the lcarning
E 20 predicted
rate x 10 for 200 epochs. We report the best evaluation loss on test set.
B.5 Arxiv Classifier Settings
The hidden dimension of the MLP and also the query, key and value matrices are to the same dimension the slots, namely, 128. We train the classifier with batch size of 32 for 200 epochs with Adam optimizer
26

---

Published in Transactions on Machine Learning Research (10/2023)
Table 20: List packages and their versions.
package
version
use
NLTK
3.5
sentence and word tokenization
youtokentome
1.0.5
BPE implementaion
polyglot
16.7.4
morfessor implementation
matplotlib
3.3.2
atlention maps visualization
scipy
1.2.2
hungarian algorithm implementation
numpy
1.19.1
with learning rate of le - 4. For the Arxiv-L dalasel each subarea has 1000 samples which would be 20000 samples in total.
C Infrastructure
We use PyTorch version 1.2.0 framework and Python version 3.6.9 for implementing our code. Table 20 shows the rest of the libraries we use. We run our code on a single GPU with model GTX10800i and the operating system Debian10 (Buster) 64-bit. We use the same compute for all of our experiments including training the models and probes. The training time for the main model five hours and for the probings is around days.
For reporting each of the results we run our algorithm once, since it would be too computationally expensive.
27


Reviews:
[{"review_id": "prhOZEnOoL", "reviewer": "Reviewer_e62S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors presented many evaluation methods (forward probing, reverse probing, a downstream task) over several European languages.\n1. The problem studied is an interesting one and I think the method used is also interesting.\n\nWeaknesses:\n1. The presentation of crucial concepts lacks clarity. For example, the problem formulation, what does \"meaning-bearing units\" mean? In the same section, the authors used morphemes as an example but they discourage such comparison elsewhere (in related works section). My first impression is that such *units* would have persistent meanings (like a morpheme) over different inputs but they are not: the slots are computed by a learned neural network for each input (with a learned prior). As stated elsewhere in the paper, each slots computed over an input carries context information (unlike a morpheme).\n1. Despite having many summarizing evaluations, it lacks sufficient granular analysis of the learned slots. Such analysis seems necessary when the slots as proposed represent a novel way to encode the meaning of a piece of text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission studies the problem of finding \"meaningful units\" in text as represented by a sequence of characters in an unsupervised manner. These units are compared to \"objects\" in a visual scene (and their method is built upon prior work in the computer vision domain called slot attention). The proposed approach consists of learning to reconstruct the input sequence of characters by first encoding it into slots (queries in the transformer encoder attention) and then decoding the slots into the original input. The learned slots are evaluated in a few different ways, including a proposed method called reverse probing, over a few European languages. The results are compared primarily to a baseline fixed stride method, two subword tokenization methods from prior works, BPE and Morfessor, and annotated morphemes. The proposed method seems novel.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I think the studied problem is very interesting but also conceptually challenging. It would help to devote more of the paper to explaining the *mathematical* model of meaning that you are pursuing and avoid over-usage of vague terms like \"a higher level of abstraction.\" For example, does the ordering of the slots matter (for decoding)?\n1. Relatedly, besides enhancing formalism of the presentation, it would also help to examine a few learned slots closely in the main text to provide support to some of claims you have about what the slots *can* represent (\"such as ones that occur at different positions, or ones that have different types of forms\" from section 2.2). For example, does the \"-ing\" slot you mentioned persistently activate for different input sentences? How about when \"ing\" appears in text not as a tense modifier, like in \"mingle\"?\n1. A suggestion. You mentioned that vanilla slot attention was studied on synthesized data. Would evaluating on an artificial language help clarify what learned slots represent? What about programming languages?\n1. A typo in section 2.2: \"... by slots acting as queries...\" -> \"... with ...\".\n1. A suggestion. You might want to transpose the plots in Figure 2 so that the text is easier to read.\n", "broader_impact_concerns": "Adequate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxiohgfmIi", "reviewer": "Reviewer_JWZR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe idea to extend Slot Attention from image to text is interesting. The paper proposes several adaptations of the original Slot Attention to cope with text inputs. The method makes sense and is easy to follow. Visualization shows that the method can achieve unsupervised grouping of characters.\n\nWeaknesses\n- My major concern is that there is insufficient evidence for the claim that the proposed method \"could be a useful replacement for tokenization methods in downstream tasks\".\n    - There is no experiment comparing the proposed method with tokenization methods such as BPE in downstream tasks.\n    - The only comparison is done with the stride-based baseline, and it seems unclear whether this is a strong baseline, and whether outperforming this baseline is a meaningful result.\n    - The probing evaluation may provide some evidence, but it is unclear whether the probing score correlates well with downstream performance.\n- For completeness, it would be better to provide more explanation/equation/figure of the forward/reverse probing method. It is a bit confusing why the forward probing is a classification problem, how empty labels are obtained, and what loss is used exactly.\n- Related work could include more recent work on object-centric learning, such as [SLATE](https://arxiv.org/abs/2110.11405) which also uses a Transformer decoder.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension of Slot Attention to the text domain. To cope with the large vocabulary and variable sequence length, the paper proposes a new slot initialization and a method for dynamically choosing the slots. The paper also proposes a bi-directional probing method to evaluate whether the learned representations contain equivalent information as existing tokenization and morphology learning methods (e.g., BPE, Morfessor). The experiments are done across multiple languages, with a stride-based model as the baseline. Results show that the proposed model achieves better probing scores and higher downstream classification accuracy than the baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think it is critical to provide some empirical evidence directly showing the advantage or similar performance of the proposed method over tokenization / morphology learning methods.\n\n---\n\nI thank the authors for adding the BPE baseline, clarifications, and related work. My concerns have been addressed.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w1X6EKgpe9", "reviewer": "Reviewer_dFr6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: * The paper is clearly written and well organized. It addresses interesting task of inducing meaningful text units in an unsupervised fashion.  \n* The approach of adapting slot attention (originally used in vision) for text is interesting and shows promising empirical results. \n* The experiments span multiple languages, and claims are mostly reasonable. \n\nWeakness: * Relate work is weak. They only have one line discussing char former (Tay et al) and canine (Clark et al), but the motivation is fairly similar, and they should be discussed more throughly. Also, HuBERT model (from speech community) is also quite relevant in discovering units from self-supervision. \n\n* Comparison: They report probing results with BPE/Morfessor and Morpheme. But I think comparing to word-piece will be helpful, as they are two different, popular mechanisms, especially given the result of [Bostrom and Durrett EMNLP20] argues workpiece model recovers more morphologically aligned subword units. Wordpiece is also slightly more relevant to proposed approach, as it is derived from learned language model similar to autoencoder here. \n\n* Current downstream task evaluation is pretty weak. See requested change. \n\n* It would be really interesting to train a small scale LM with units discovered from the proposed method and compare their performances (perplexities). As the discovered units are continuous and do not map to character sequence, this is not straightforward, but discussing this will be nice. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a way to induce meaningful units from character sequence by adapting slot attention model. They evaluate their model through a suite of experiment, such as (1) reconstructing the original text, (2) probing with other character unit discovery methods and (3) downstream application. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Strengthen discussion of related work (see weakness).\n* The current downstream evaluation is pretty weak in that it only compares to its own baseline. It should compare to other methods such as BPE. \n* I'm a bit unclear about the efficiency of this model, especially compared to its stride-based baseline. Adding a discussion on this would be helpful. \n* Visualization/analysis of learned slot patterns would be helpful. Is it more likely to assign consecutive characters to the same slots? The current visualization (Figure 2) is on decoder attention to slots, but you can also compute / analyze the attention over slots for each character? \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p4HtNeF0VM", "reviewer": "Reviewer_21YV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This work is novel at replacing explicit language unit segmentation with representations induced within a deep learning model. In previous studies, language unit segmentation is often conducted using a statistical approach to count the word concurrence with a (small) fixed context window. In contrast, this work does not explicitly segment a sentence into units and considers the whole sentence as the context.\n\n2. The evaluation is based on both the probing task and downstream classification task, which verifies the effectiveness of the proposed method. The induced units tend to capture meaningful information at an appropriate level of abstraction.\n\nWeaknesses:\n\n1. Although the authors have discussed the connections and differences with previous morphology learning and tokenization approaches in Section 3, the advantage of the proposed method is not clear. As previous methods will segment a sentence into explicit tokens,  those explicit tokens can serve as the fundamental part of any deep learning model. A more fair baseline is a same-sized Transformer encoder fed with tokens segmented by public tokenization approaches (e.g., SentencePieces [1]):\n\n**Baseline**: sentence -> [SentencePiece] -> subwords -> [Transformer Encoder] -> [Classification]\n\n**The Proposed Model**:  sentence -> [The (Trained) Proposed Encoder] -> [Classification]\n\nThe evaluation can be based on any downstream tasks (e.g., the classification task in the paper), to show that the proposed approach is indeed a more effective alternative to the existing paradigm of \"tokenization + encoding.\"\n\n2. The task formulation may be misleading. \n\n\"The unsupervised induction of abstract meaningful units from sequences of characters is a novel task which requires both novel methods and novel evaluations.\"\n\n- According to the problem formulation in Section 2.1, the task seems to be the same as the standard language units segmentation tasks in existing studies [2]. Besides, as the proposed approach does not explicitly produce the set of morphemes-like units, the example output {she, play, -ed, basket, -ball} might not be accurate for indicating the real output of the model. Maybe it is better to highlight the major difference in \"abstract\".\n\nMinor Comments:\n\nThe position of Section 3 Related Work is weird. Why not place it before introducing the proposed approach?\n\n[1] Kudo, T. and Richardson, J., 2018, November. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 66-71).\n\n[2] Zhang, Z., Zhao, H., Ling, K., Li, J., Li, Z., He, S. and Fu, G., 2019. Effective subword segmentation for text comprehension. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(11), pp.1664-1674.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Dynamic Capacity Slot Attention for discovering meaningful units in text in an unsupervised fashion. The method is based on an auto-encoder architecture for encoding a character sequence into a set of continuous slot vectors and decoding them to reconstruct the original sequence. The authors enforced the set of slots to act as a bottleneck in transferring information between the encoder and the decoder by adding a constant noise to their vectors and integrating an $L_{0}$ regularizing layer on top of the slots which only retains the necessary vectors. The proposed method is evaluated by both the probing task and a downstream classification task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More thorough comparisons are needed, e.g., comparison with existing approaches, and more comprehensive experiments on downstream tasks.", "broader_impact_concerns": "Although I find this work interesting, it does not necessarily demonstrate the advantages over existing studies. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received unanimous "Leaning Accept" recommendations from all four reviewers. They found the main idea of Dynamic Capacity Slot Attention novel and interesting, and the experiments were convincing enough to justify the main claims. The authors have already addressed the major concerns raised by the reviewers, including providing BPE-based baseline results, clarifying the problem formulation, and discussing other related work. Overall, this is a solid submission, and we are pleased to recommend its acceptance.
Decision: Accept (poster)