Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Slot attention (Locatello et al., 2020) was originally designed to learn object-centric representations of images. The authors adapt slot attention to learn text representations, in the hope that those slots would similarly extract meaningful units from sequences of characters. The authors made minor changes to the original design, including a different parameter initialization, an L0-drop layer from Louizos et al. (2018). Experiments show that the models with slot attention 1) produce non-overlapping continuous attention patterns when decoding, and 2) induce subword units that better predict human-labeled morphemes, segmentations from Morfessor, and segmentations from BPE, than stride-attention baselines.
strength_and_weaknesses: ### Strengths
1. Interesting idea to adapt unsupervised (implicit) segmentation from image to text.
2. The paper is easy to follow, with major motivation, design, and results are clearly stated. There are few under-supported claims or relatively minor points worth clarifying, e.g. LPs P2, Sec 2.1; P6, Sec 4.2; P7, Sec 4.3; P8, Fig 3, but themselves are unlikely to refute the key contributions.

### Weaknesses
1. Limited technical contribution. No design specific for the language data. More or less a straightforward application of slot attention. This itself won't be a reason to reject if the experimental results are strong. However,
2. Experimental evidence not strong enough to show that slot attention is beneficial in learning text representation. The weakness is multifold. Roughly categorized into insufficient experiments (a, b), questionable experiment setting (c, d, f, g), weak evidence (e, h, i).
    - a. The authors trained their model using reconstructive objectives. The performance of this task can show how good a model is at capturing the key meaning and the structure of a sequence. Reporting metrics on the training and validation set and comparing them against the baselines will make a point. See also LP Sec 4.
    - b. The authors showed no results from downstream tasks. This will be the most direct evidence that slot attention is at least helpful to certain NLP tasks. For example, pos-tagging can show if it captures word meanings or roles; segmentation can show if it learns morphology. Compare against character-level and stride-attention models would provide evidence.
    - c. Although the authors claim the encoder-decoder information bottleneck being an important factor to the effectiveness of the method, the actual realization does not seem to enforce any meaningful bottleneck and the input sequence can in theory be memorized verbatim. This raises concern on if the setup really facilitates meaningful learning and if the observed behaviors of the learned slots are legit signs of learning. See LP P6, Sec 4.1.
    - d. In Sec 4.2, the authors visualize the attention of decoder, i.e. output sequence, over slots. Why not inspect the relation between the slot and the input sequence? The observed relation may be a bias from the encoder-decoder bottleneck and has nothing to do with slot attention. Comparison against other bottleneck baselines can improve the support. See also LP Sec 4.2, observations.
    - e. Unsupported claims that attribute the emergence of continuous span to effectiveness of their model. Need further evidence to determine if this is an inherent bias or actually aligns with the language data. See also P6, Sec 4.2, "We believe ..."
    - f. The probing task of "predicting reference subword segmentation from the subwords induced from the current model" (Sec 4.3) seems unnecessarily convoluted. I am not aware of anything like this in the existing work. The authors cited a survey but I was not able to find any discussion of this specific probing or even similar probing. See LP P7, Sec 4.3, top. The forward probing task, the reverse probing task, the design of how to induce units, all seems convoluted and artificial. If the authors are able to induce subword units from the slots, can they go one step further and induce subword segmentations and directly evaluate on the subword segmentation task? Subword segmentation or morphological segmentation is an established task. Many benchmarks available. The authors can also brand it as a downstream task evaluation.
    - g. In the probing task (Sec 4.4), slot-attention has a possibly unfair advantage that its number of slots are set dynamically. See LP P8, top.
    - h. No clear or consistent trend found in Fig 3 between the performance of slot-attention and the performance of stride-attention.
    - i. Preliminary results (Sec Limitations) showed slot-attention is slightly worse than stride-attention (baseline) on the downstream task of text classification, measured by F-1 on ArXiv-L. See LP P9, Limitations, bottom.


### Localized points (LPs)
1. P2, Sec 2.1, "... where we expect our slots to represent something like the set of morphemes ...". Can you be specific what are the exact properties you want from this slot formation?
2. P3, after Eq (1), "In other words, the initial value of slots are independent samples of a single Normal distribution with learnable parameters." This sentence repeats the sentence directly before it.
3. Sec 4. What are the numbers from reconstructive training?
4. P6, Sec 4.1, "We feed in the sentences with less than 128 characters to our model and consider the number of slots as 64 (half of the maximum input length). In addition, we take the dimension of slots as 128." Assuming the characters are ASCII, each character can be represented in 8 bits. 128x8 = 64x16 bit. A 128-d real vector is more than enough to fully store 16-bit information and thus the entire sequence can be stored verbatim. I see no bottleneck here.
5. P6, Sec 4.2, observations on the slot attention pattern. Why do the authors focus on the attention pattern with respect to the output sequence not the input sequence? It could be the case that an arbitrary bottleneck without any of the slot-attention design gives a similar pattern. "An arbitrary bottleneck" could be taking stride attention or simply reduce the number of vectors (e.g. the first k vector, or the vector of some special token) or the dimension of the vectors (e.g. reducing from 128d to 16d) being passed from the encoder to decoder. I will be more convinced that the slot-attention really did something non-trivial if a bottleneck itself is unable to give such a pattern.
6. P6, Sec 4.2, "We believe that the emergence of contiguous spans is a result of the bottleneck ..." You can test whether the emergence of spans is learned from the text or merely an inductive bias of the model. You can train the model over languages with nonconcatenative morphology, e.g. Arabic, and see if the learned slot is still contiguous or aligns with the morphology.
7. P7, Sec 4.3, top, "Predicting the previously proposed unit from our induced unit is a probing task, as used in previous work (Belinkov & Glass, 2019), ..." Can the authors point us to which previous work used "predicting reference subword segmentation from the subwords induced from the current model" as a probe task? The cited survey does not particularly discuss any probing where one predicts subword units from latent presentation or induced units.
8. P8, Fig 3. Why are all the NLL losses negative? Negative log-likelihood should take value from [0, +\infty).
9. P8, top, "the number of open gates in the L0Drop layer is approximately the same as the target number." The number of tokens generated from stride-attention (baseline) is fixed but the number of tokens (determined by the number of gates) is set dynamically. This is an unfair advantage.
10. P9, Limitations, bottom, "For the ArXiv-L, on average, we get 0.396, 0.409,
0.2732 F1 for slot-attn, stride=6, and character-based models, respectively, on Dev set, and 0.393, 0.394, 0.271 on Test set." It seems slot-attn <= stride-attn from these preliminary results.

clarity,_quality,_novelty_and_reproducibility: Clarity: Good. The authors successfully conveyed major points.

Quality: Fair. The experiments show some promising results for using slot attention in learning text, but the support is weak due to insufficient experiments (2a, 2b), questionable experiment setting (2c, 2e, 2f), weak evidence (2d, 2g, 2h).

Novelty: Mixed. The authors claim to be the first to apply slot attention to text. However, it seems to be a straightforward application with little modifications and nothing specific to text.

Reproducibility: Good, conditioning on code and data release. The authors provided detailed experimental settings in the appendix.

summary_of_the_review: The work has novelty as the first to apply slot attention to text. However, the adaptation seems straightforward and no text-specific design was added. It itself won't be a reason to reject but the experimental designs are flawed and that the results do not answer if slot attention is beneficial to text learning. Stronger, more direct experimental support would improve the strength of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This method proposes dynamic slot attention based approach to discover subword units in a completely unsupervised manner. The character based encoder representations are passed through a bottleneck of K slot noisy vectors (K < length of input) and these slot vectors are then used for reconstruction with a lightweight decoder with attention over the slots. In addition to the reconstruction loss, an L0 drop term (relaxed via Hard-Concrete reparametrization) is also added to the training loss and inference to encourage sparseness of slots that are being used. This approach is empiricqally compared to a random baseline and a reasonable stride based approach is also proposed for comparison with a better baseline. Intrinsic evaluation testing correspondence of slots to sub-lexical units like morphemes and BPE validate the effectiveness of this approach against the proposed baseline.
strength_and_weaknesses: Strengths:

-- The proposed approach is reasonable with sensible design choices and loss terms.

-- The proposed approach effectively learns contiguous spans that correspond well with human-interpretable morphemes, and automatic subword extraction methods like BPE.

Weaknesses:

-- Although, it is mentioned in the "limitations" section, I believe it is a fundamental drawback of the current work that it doesn't show the approach's usefulness over downstream tasks and the evaluation is mostly intrinsic. I find it difficult to articulate the importance of learning subword units with such an expensive procedure. For example, if the proposed use is better tokenization, then experiments that support this should be performed to justify using this expensive approach for tokenization.

-- Related to above, a deeper analysis that focuses on the discovery of sublexical units will strengthen the paper. While this approach is shown to be correlated with morphemes and BPE tokens, these two subword units are intrinsically very different. Hence, analysis of the proposed approach in context of linguistically meaningful units in contrast with engineering-oriented units like BPE should be performed in greater detail.

-- "Reverse Probing" is confusing. I don't understand how these experiments were performed. How were the target subword units incorporated into the input of the classifier? Was the learned encoder over characters used for reverse probing. If yes, then isn't there a concern of using an encoder that was learned jointly with the slots for probing over the slots?
clarity,_quality,_novelty_and_reproducibility: Please see my review above. Some aspects like reverse probing can be clarifies in greater detail and more analysis will shed light on aspects of the proposed approach.
summary_of_the_review: Overall, the proposed approach is reasonable and tackles an interesting problem of discovering subword units in a completely unsupervised manner. However, I have slight concern over the lack of extrinsic evaluation and a deeper motivation for the intrinsic properties.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method to infer meaningful groupings of characters in a text. It is based on Slot attention method developed in the visual domain to obtain slots (vector representations), corresponding to meaningful parts of an image (objects). The paper adapts this method to work on a sequence of characters. The Slot attention method is used as a bottleneck module inside an encode-decoder Transformer method that takes as input a sequence of characters and is trained to reproduce it. The proposed Dynamic Capacity Slot Attention method can use a different number of slots for each sequence by using a sparsification layer ($L_0$Drop) and uses different initialization for each slot for better optimization. The method seems to produce a meaningful grouping of characters, that is similar to previous methods.

strength_and_weaknesses: **Strong Points**:

The proposed method is sound and is capable of producing meaningful groupings of characters.

The usage of sparsification layer $L_0$Drop (Zhang et al., 2021) to control the number of slots is a good idea for using a different number of slots for each sample.


**Weak Points**:

It is not clear what is the purpose of the grouping of characters. In vision, objects are central for reasoning over real-world scenes, so it makes more sense that obtaining unsupervised entities that are similar to objects is desirable. Do we have any reason to believe that they would be beneficial for reasoning over text? Does discovering these units have a meaning on its own?

Although the authors note that evaluating the obtained representations is outside the scope of this work, it should not be the case. We need a reason to do this kind of grouping, and good performance for a downstream task would be ideal. 

Regarding the probing: The untrained baseline is too weak, because before training each slot constraints an average of all input tokens, which are also random at first, thus it is extremely unlikely that these representations could be used in a learning task. Maybe a model that averages all the tokens corresponding to the same word be a better baseline. 

clarity,_quality,_novelty_and_reproducibility: The paper is mainly clear and easy to reproduce. The idea is original, but the purpose is not clear yet.
summary_of_the_review:  The paper proposes a good and sound method for grouping elements in a sequence. It is technically sound and seems to produce meaningful grouping but at the moment the purpose of these groupings is unclear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method to induce meaningful text units from character sequences. The method is based on Slot Attention for unsupervised object discovery in computer vision. To adapt it to texts, a modification is introduced by using separate trainable parameters for each slot rather than shared. The full model consists of a character-encoder, the modified slot attention module, an L0-drop module to enforce sparsity and a shallow decoder for recovering input sequences, and the full model is trained with an auto-encoding objective. The evaluation of the proposed model is performed by qualitatively visualizing attention maps as well as quantitative probing experiments to measure the correspondence between the induced slots and linguistic or statistical units.
strength_and_weaknesses: Strength:

This work explores an interesting direction of letting the neural models induce meaningful linguistic units in an unsupervised way. This might lead to better segmentation/character-level representation methods, and provide more insights on the inner mechanisms of neural models.

Weakness:

I still have some doubts on how meanful the “induced slots” are and I think the two evaluations are not quite enough to show that these slots indeed represent meaningful units. Though the patterns in the attention maps and the obvious improvements in the probing experiments look interesting, I still don’t feel these are enough to arrive at the conclusion that the induced slots are very meaningful, unless there is a more clear way to show what they actually represent. Attention only shows the information may exist but not necessarily that the slots individually represent meaningful units, so does probing (especially with the matching-based probe training).

To show the effectiveness of the induced slot representations, applications on real text tasks are lacking (except for an initial experiment where there seems to be no obvious performance difference between slot-attn and simpler stride-based representations).

clarity,_quality,_novelty_and_reproducibility: Figure 2 might be readjusted for easier reading. And it would be better if there could be more illustrations for the probing experiments for easier understanding.
summary_of_the_review: This paper explores an interesting question on the unsupervised induction of meaningful text units and proposes a modified Slot-Attention based model for this purpose. Though some of the results and patterns look interesting, it still remains unclear to me how meaningful the slot representations are and there is a lack of further explorations on the effectiveness of them in actual tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a method to infer meaningful groupings of characters in a text. The authors adapt slot attention to learn text representations, hoping that those slots would extract meaningful units from sequences of characters. The authors made some changes to the original design, including a different parameter initialization and an L0-drop layer. Experiments show that the models with slot attention induce subword units better predict human-labeled morphemes, segmentations from Morfessor, and segmentations from BPE, than stride-attention baselines.

The reviewers consider that experimental evidence needs to be stronger to show that slot attention learns something meaningful and benefits text representation. The reviewers suggest that some downstream tasks are required to evaluate the proposed method to demonstrate the effectiveness of the induced slot representation. The authors argue that due to large-scale engineering, the evaluation on  downstream tasks is beyond the scope of this work. However, machine learning-based probing can be misleading, so using probing results to draw conclusions is insufficient. Therefore, we still encourage the authors to evaluate the induced slot representation on some downstream tasks.

Some reviewers had questions about probing, and I believe the authors' feedback answers the questions. 
justification_for_why_not_higher_score: Overall, the proposed method is novel and has potential. But the current version needs more support to show that the learned units are meaningful. So the current version is not acceptable.
Decision: Reject