Are there identifiable structural parts in the sentence embedding whole?

Authors: vivi  nastase
paola  merlo
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-06 11:12:28

Abstract: Sentence embeddings from transformer models encode much linguistic information in a fixed-length vector. We investigate whether structural information -- specifically, information about chunks and their structural and semantic properties -- can be detected in these representations. We use a dataset consisting of sentences with known chunk structure, and two linguistic intelligence datasets, whose solution relies on detecting chunks and their grammatical number, and respectively, their semantic roles. Through an approach involving indirect supervision, and through analyses of the performance on the tasks and of the internal representations built during learning, we show that information about chunks and their properties can be obtained from sentence embeddings.

Main paper: 
 introduction: 
 Transformer architectures compress the information in a sentence -morphological, grammatical, semantic, pragmatic -into a fixed-length onedimensional array of real numbers. Sentence embeddings, usually fine-tuned, have proven useful for a variety of high-level language processing tasks, such as the GLUE tasks (Clark et al., 2020) , or story continuation (Ippolito et al., 2020) ). These results, however, do not shed light on what kind of semantic or structural information is encoded in these representations.Understanding what kind of information is encoded in the sentence embeddings, and how it is encoded, has multiple benefits. It connects internal changes in the model parameters and structure with changes in its outputs. It contributes to verifying the robustness of models and whether or not they rely on shallow or accidental regularities in the data. It narrows down the field of search when a language model produces wrong outputs, and ultimately it may help maximize the use of training data for developing more robust models from smaller textual resources. Investigation, or indeed, usage, of raw (i.e. not fine-tuned) sentence embeddings obtained from a transformer model are rare, possibly because most transformer models do not have a strong supervision signal on the sentence embedding. Using PCA analysis, Nikolaev and Padó (2023c) have shown that the dimensions of BERT sentence embeddings have much correlation and redundancy, and encode more shallow information (length), rather than morphological, syntactic or semantic features. Analysis of information propagation through the transformer layers seem to show that specialized information -e.g. POS, syntactic structure -while quite apparent at lower levels, gets lost towards the highest levels of the models (Rogers et al., 2020) , while there are subnetworks that encode specific linguistic functions (Csordás et al., 2021; Conmy et al., 2023) .While previous work has regarded network nodes or embedding dimensions as the unit of analysis, Elhage et al. (2022) show that superposition -whereby each unit, i.e. neuron or embedding dimension, can be involved in the encoding of multiple features -occurs in artificial neural networks. Such features involving overlapping sets of nodes can be learned from a model using sparse autoencoders (e.g. (Cunningham et al., 2023) ). Starting from a similar hypothesis relative to the dimensions of a sentence embedding, we aim to test whether specific information, in particular chunks -noun, verb and prepositional phrases, that may play different structural and semantic roles -can be detected in the sentence representation. We use an encoderdecoder architecture applied to data with specific properties, and verify that, through indirect supervision, we can distill information about chunks and their task-relevant properties from sentence embeddings from a pre-trained transformer model. Besides being practically useful, as they provide useful shallow structure more easily obtainable than detailed syntactic analysis (Abney, 1991; Buchholz et al., 1999) , chunks have psychological plausibility (Gee and Grosjean, 1983) . This motivated us to test whether they are detectable in sentence embeddings, as they would provide syntactically and semantically useful building blocks for assembling higher level information about a sentence. The code and data are available at https://github. com/CLCL-Geneva/BLM-SNFDisentangling. 
related work: 
 How is the information from a textual input encoded by transformers? There are three main approaches to answer this question: (i) tracing specific information from input to output through the model's various layers and components, (ii) isolating subsets of model parameters that encode specific linguistic functions and (iii) investigating the generated embeddings through probes, using purposefully built data for different types of testing.Tracing information through a transformer Rogers et al. (2020) have shown that from the unstructured textual input, BERT (Devlin et al., 2019) is able to infer POS, structural, entity-related, syntactic and semantic information at successively higher layers of the architecture, mirroring the classical NLP pipeline (Tenney et al., 2019a) . Further studies have shown that the information is not sharply separated, information from higher levels can influence information at lower levels, such as POS in multilingual models (de Vries et al., 2020) , or subject-verb agreement (Jawahar et al., 2019) . Surface syntactic and semantic information seem to be distributed throughout BERT's layers (Niu et al., 2022; Nikolaev and Padó, 2023c) . Attention is part of the process, as it helps encode various types of linguistic information (Rogers et al., 2020; Clark et al., 2019) , syntactic dependencies (Htut et al., 2019) , grammatical structure (Luo, 2021) , and can contribute towards semantic role labeling (Tan et al., 2018; Strubell et al., 2018) .Deep learning models have billions of parameters. This makes them not only incomprehensible, but also expensive to train. The lottery ticket hypothesis (Frankle and Carbin, 2018) posits that large networks can be reduced to subnetworks that encode efficiently the functionality of the entire network. Detecting functional subnetworks can be done a posteriori, over a pre-learned network to investigate the functionality of detected subnetworks (Csordás et al., 2021) , the potential compositionality of the learned model (Lepori et al., 2023) , or where task-specific skills are encoded in a finetuned model (Panigrahi et al., 2023) . Instead of learning a sparse network over a prelearned model, Cao et al. (2021) use a pruning-based approach to finding subnetworks in a pretrained model that performs some linguistic task. Pruning can be done at several levels of granularity: weights, neurons, layers. Their analyses confirm previous investigations of the types of information encoded in different layers of a transformer (Conneau et al., 2018a) . Conmy et al. (2023) introduce the Automatic Circuit DisCovery (ACDC) algorithm, which adapts subnetwork probing and head importance score for pruning to discover circuits that implement specific linguistic functions. The model network need not be separated into disjunct subsets of nodes. Elhage et al. (2022) show that neural network models encode more features than the number of their dimensions, individual nodes contributing to more than one feature. Such features could be learned in an unsupervised manner using Sparse AutoEncoders (Cunningham et al., 2023; Trenton Bricken, 2023; Gao et al., 2024) , and correlated with linguistic patterns or phenomena.Word embeddings were shown to encode sentence-level information (Tenney et al., 2019b) , including syntactic structure (Hewitt and Manning, 2019) , even in multilingual models (Chi et al., 2020) . Predicate embeddings contain information about their semantic roles structure (Conia and Navigli, 2022) , embeddings of nouns encode subjecthood and objecthood (Papadimitriou et al., 2021) . The averaged token embeddings are more commonly used as sentence embeddings (e.g. (Nikolaev and Padó, 2023a) ), or the special token ([CLS]/
) embeddings are fine-tuned for specific tasks such as story continuation (Ippolito et al., 2020) , sentence similarity (Reimers and Gurevych, 2019) , alignment to semantic features (Opitz and Frank, 2022) . Sentence embeddings as averages over token embeddings is justifiable as the learning signal for transformer models is stronger at the token level, with a much weaker objective at the sentence level -e.g. next sentence prediction (Devlin et al., 2018; Liu et al., 2019) , sentence order prediction (Lan et al., 2019) . Electra (Clark et al., 2020) relies on replaced token detection, which uses the sentence context to determine whether a (number of) token(s) in the given sentence were replaced by a generator sample. This training regime leads to sentence embeddings that perform well on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) and Stanford Question Answering (SQuAD) dataset (Rajpurkar et al., 2016) , or detecting verb classes (Yi et al., 2022) . Raw sentence embeddings were shown to capture shallower information (Nikolaev and Padó, 2023c) , but Nastase and Merlo (2023) show that raw sentence embeddings have internal structure that can encode grammatical sentence properties.Probing models Analysis of BERT's inner workings has been done using probing classifiers (Belinkov, 2022), or through clustering based on the representations at the different levels (Jawahar et al., 2019) . Probing has also been used to investigate the representations obtained from a pretrained transformer model (Conneau et al., 2018b) . Elazar et al. (2021) propose amnesic probing to test both whether some information is encoded, and whether it is used. VAE-based methods (Kingma and Welling, 2013; Bowman et al., 2016) have been used to detect or separate specific information from input representations. Mercatali and Freitas (2021) capture discrete properties of sentences encoded with an LSTM (e.g. number and aspect of verbs) on the latent layer. Bao et al. (2019) and Chen et al. (2019) learn to disentangle syntactic and semantic information. Silva De Carvalho et al. (2023) learn to disentangle the semantic roles in natural language definitions from word embeddings. Probing can have issues: learning a classifier for a task does not guarantee that the model uses the targeted information (Hewitt and Liang, 2019; Belinkov, 2022; Lenci, 2023) . Michael et al. (2020) introduce latent subclass learning, where a binary classification task has a pre-classification multi-class logistic regression step that helps probe for emergent information.Data Most approaches use datasets built by selecting, or constructing, sentences with specific structure and properties: definition sentences with annotated roles (Silva De Carvalho et al., 2023) , sentences built according to a given template (Nikolaev and Padó, 2023b), sentences with specific structures for investigating different tasks, in particular SentEval (Conneau and Kiela, 2018) (Jawahar et al., 2019) , example sentences from FrameNet (Conia and Navigli, 2022) , a dataset with multilevel structure inspired by the Raven Progressive Matrices (RPM) visual intelligence tests (An et al., 2023) . 
overview: 
 Our approach is also a kind of probe. It uses indirect supervision, though, to avoid the shallow learning of a classifier and datasets with specific structure to test for structural information in sentence embeddings.Our main object of investigation are chunks, sequence of adjacent words that segment a sentence, as defined initially in Abney (1992) ; Collins (1997)  and then Tjong Kim Sang and Buchholz (2000) . We use two types of data. We use sentences with known chunk patterns (Section 4.1), to determine whether chunks and their grammatical properties are identifiable in sentence embeddings with indirect supervision (Section 5). We also use two datasets with multi-level structure built for linguistic intelligence tests for language models (Merlo, 2023) (Section 4.2), to determine whether a system can detect syntactic and semantic structure and information in sentence embeddings based on the requirements of a task.The data, with its repetitive patterns, and the VAE-based system support an indirect supervision approach: the system is not given the patterns to be discovered explicitly, but it needs to find them based on the contrasting answer sets at both the sentence and task levels. This indirect supervision process, together with lexical and structural variations in the data, helps to avoid, at least partly, the critiques against probes based on classification, which can learn a task based on 'artefacts' of the data, regularities different from what is intended (Belinkov, 2022) . 
data: 
 We use data consisting of stand-alone sentences with specific structure, and data consisting of sentences with specific structure and other attributes in larger contexts, to test whether this regular information can be detected.Sentences are built from a seed file containing noun, verb and prepositional phrases, including singular/plural variations. From these chunks, we built sentences with all (grammatically correct) combinations of np (pp 1 (pp 2 )) vp1 . For each chunk pattern p of the 14 possibilities, all corresponding sentences are collected into a set S p . We generate an instance for each sentence s from the sets S p as a triple (in, out + , Out -), where in = s is the input, out + is the correct output, which is a sentence different from s but having the same chunk pattern. Out -are N negs incorrect outputs, randomly chosen from the sentences that have a chunk pattern different from s. The algorithm for building the data and a sample line and generated sentences are shown in appendix A.1.From the generated instances, we sample uniformly, based on the pattern of the input sentence, approximately 4000 instances, randomly split 80:20 into train:test. The train part is further split 80:20 into train:dev, resulting in a 2576:630:798 split for train:dev:test. We use a French and an English seed file and generate French and English variations of the dataset, with the same statistics.Blackbird Language Matrices (BLMs) (Merlo, 2023) -language versions of the visual Raven Progressive Matrices (RPMs)-are multiple-choice problems, where the input is a sequence of sentences built using specific generating rules, and the answer set consists of a correct answer that continues the input sequence, and several incorrect contrastive options, built by violating the underlying generating rules of the sentences. In a BLM matrix, all sentences share a targeted linguistic phenomenon, but differ in other aspects relevant for the phenomenon in question. Thus, BLMs, like their visual counterpart RPMs, require identifying the entities (the chunks), their relevant attributes (their morphological or semantic properties) and their connecting operators, to find the correct answer.To test the detection of different types of information in different languages, we use two BLM datasets, which encode two different linguistic phenomena, each in a different language: (i) BLM-AgrF -subject verb agreement in French (An et al., 2023) , and (ii) BLM-s/lE -verb alternations in English (Samo et al., 2023) . The structure of these datasets -in terms of the sentence chunks and sequence structure, as well as the answer sets and the erroneous answers and their error types -is shown in Figure 1 . Examples are in appendices A.1, A.2. BLM datasets also have a lexical variation dimension, to explore the impact of lexical variation on detecting relevant structures: type I -minimal lexical variation for sentences within an instance, type II -one word difference across the sentences within an instance, type III -maximal lexical variation within an instance. The BLM-s/lE dataset is used as is. We built a variation of the BLM-AgrF (An et al., 2023) understand better how the BLM tasks are solved. The errors in both BLM tasks allow us to study in more detail the performance and understand where the weaknesses are when solving the task.Datasets statistics Table 1 shows the datasets statistics for the BLM problems. After splitting each subset 90:10 into train:test subsets, we randomly sample 2000 instances as train data. 20% of the train data is used for development. 
experiments: 
 We build upon (Nastase and Merlo, 2023) , and use as sentence representations the embedding of the [CLS] special token from a pretrained Electra model (Clark et al., 2020) 2 reshaped as a twodimensional array. We chose Electra because it has a stronger sentence-level supervision signal as well as strong results on multiple NLU tasks (see Section 2). In Section 5.1.3, we show how it compares to other pretrained models.The BLM tasks have been benchmarked using FFNN and CNN systems which directly predict the correct answer based on the input sequence (An et al., 2023; Samo et al., 2023) . Results improve on both tasks when using a variational encoderdecoder that compresses the input sequence into a very small vector on the latent layer (Nastase and Merlo, 2023) . This previous work, and similarity of the BLM tasks with the visual Raven Progressive Matrices task, have led us to a two-step investigation process: (i) using sentences and a VAE-based system, we test whether we can compress sentences into a smaller representation on the latent layer that captures information about the chunk structure of the sentence (Section 5.1 below); (ii) to see if the system can detect and extract the kind of information relevant to a specific task, we combine the compression of the sentence representation with the BLM problems, where a crucial part of the solution lies in identifying the structures of sentences and their sequence in the input (Section 5.2 below). This two-step approach to solving a BLM problem fits with the way humans solve the visual RPM problems from which the BLMs are inspired: (i) identify the relevant objects and their attributes; (ii) decompose the main problem into subproblems, based on object and attribute identification, in a way that allows detecting the global pattern or underlying rules (Carpenter et al., 1990) .We test whether sentence embeddings contain information about the chunk structure of the corresponding sentences by compressing them into a lower dimensional representation in a VAE-like system.The architecture of the sentence-level VAE is similar to a previously proposed system (Nastase and Merlo, 2023) : the encoder consists of a CNN layer with a 15x15 kernel, which is applied to a 32x24shaped sentence embedding, followed by a linear layer that compresses the output of the CNN into a latent layer of size 5. The decoder mirrors the encoder, and unpacks a sampled latent vector into a 32x24 sentence representation.An instance consists of a triple (in, out + , Out -), where in is an input sentence with embedding e in and chunk structure p, out + is a sentence with embedding e out + with same chunk structure p, and Out -= {s k |k = 1, N negs } is a set of N negs = 7 sentences with embeddings e s k , each with chunk pattern different from p (and different from each other). The input e in is encoded into a latent representation z i , from which we sample a vector zi , which is decoded into the output êin . We enforce that the latent encodes the structure of the input sentence by using a max-margin loss function, to push for a higher cosine similarity score with the sentence that has the same chunk pattern as the input (e out + ) than the ones that do not (E -= {e s k |e s k = embedding(s k ), s k ∈ Out -}). loss sent (e in ) = maxM (ê in , e out + , E -) + + KL(z i ||N (0, 1)) maxM (ê in , e out + , E -) = max(0, 1 -cos(ê in , e out + ) + + es k ∈E -cos(ê in ,es k ) Nnegs )At prediction time, the sentence from the {out + } ∪ Out -options that has the highest score relative to the decoded answer is taken as correct.To assess whether the correct patterns of chunks are detected, we analyze the results for the experiments described in the previous section in two ways: (i) analyze the output of the system, in terms of average F1 score over three runs and confusion matrices; (ii) analyze the latent layer, to determine whether chunk patterns are encoded in the latent vectors (for instance, latent vectors cluster according to the pattern of their corresponding sentences).In a binary evaluation (has the system built a sentence representation that is closest to the one that has the same chunk pattern as the input?), the system achieves an average positive class F1 score (and standard deviation) over three runs of 0.9992 (0.01) for French, and 0.997 (0.0035) for English.The pattern-level evaluation for the French data, presented as a confusion matrix based on the pattern information for out + , Out -at the top of Figure 2 , shows that all patterns are detected with high accuracy (the results for English are in Appendix A.4.2). To understand how chunk information is encoded on the latent layer, we perform latent traversals: for each instance in the test data, after encoding it, we modify the value of each unit in the latent layer with ten values in its min-max range, based on the training data, and decode the answer. To confirm that chunk information is present in the latent layer, we plot the projection of the latent vectors in two dimensions (Figure 3 ). The plot shows a very crisp clustering of latents that correspond to input sentences with the same chunk pattern, despite the fact that some patterns differ by only one attribute (the grammatical number) of one chunk. There are differences in the architectures, training objectives and training data for transformer-based models, which lead to differences in how they encode information. Fine-tuning further changes the landscape of the embeddings, and prioritizes different characteristics of the input sentence, often semantics. We can quantify some of these differences using the setup described above.Experiments on the task of reconstructing a sentence with the same chunk structure on BERT3 (Devlin et al., 2019) and RoBERTa4 (Liu et al., 2019) lead to average F1 score over 3 runs of 0.91 (std=0.0346) for BERT and 0.8926 (std=0.0166) for RoBERTa, confirming that Electra's architecture leads to sentence embeddings that encode more explicitly structure-related information.Two sentence transformer models LaBSE and MPNet5 obtained an average F1 of 0.43 (std=0.0336) and 0.669 (std=0.0407) respectively. We chose LaBSE and MPNet because they are tuned differently -LaBSE is trained with bilingual sentence pairs with high results on a cross-language sentence retrieval task, MPNet is optimized for sentence similarity -and their representations have the same dimensionality (768) as the transformer models we used. The low results on detecting chunk structure in sentence embeddings after this tuning indicates that in the quest of optimizing the representation of the meaning of a sentence, structural information is lost.We test whether including the sentence compression step in a system to solve the BLM tasks leads to latent representations that contain information about chunk properties relevant to the tasks.The BLM problems encode a linguistic phenomenon in a sequence of sentences that have regular and relevant structure, which serves to emphasize and reinforce the encoded phenomenon. (Carpenter et al., 1990) . We model the process of solving a BLM in a manner similar to how humans solve RPM visual tasks, by using the two-level intertwined architecture illustrated in Figure 4 : one level for detecting sentence structure, one for detecting the correct answer based on the sentence structure and their sequence. as input, and an answer set A with one correct answer a c , and several incorrect answers a err j . The sentences in S are passed as input to the sentencelevel VAE, which is the system described in Section 5.1. The latent representations from this VAE are used as the representations of the sentences in S. These representations are passed as input to the BLM-level VAE, in the same order as S. From the compressed layer of the BLM-level VAE, the decoder reconstructs a sentence embedding (e S ), which is compared to the embeddings of the answers.An instance for the sentence-level VAE consists of a triple (s i , out + i , Out - i ).For our two-level system, we must construct this triple on the fly from the input BLM instance: s i ∈ S with embedding e s i , out + i = s i , andOut - i = {s k |s k ∈ S, s k ̸ = s i } with embeddings E - i = {e s k |k = 1, N negs }.The loss combines the loss signal from the two levels: The loss at the sentence level is computed as described in Section 5.1:loss(S) = s i ∈S loss sent (e s i ) + loss task (e S )loss sent (e s i ) = maxM (e s i , e out + i , E - i ) + KL(z i |N (0, 1))The loss at the task level is computed in a similar manner, but relative to the answer set A with the corresponding embeddings set E A , and the correct answer a c , of the task: loss task (e S ) = maxM (e S , e ac , E A \ e ac ) + KL seq (z S |N (0, 1)).We run experiments on the BLMs for agreement (Figure 5 ) and for verb alternation (Figures 6, 7 ), to test a range of syntactic and semantic chunk properties that should be identified. While the information necessary to solve the agreement task is more structural, solving the verb alternation task requires both structural information concerning chunks and semantic information, with syntactically similar chunks playing different roles in a sentence (see Figure 1 ). The results show that the two-level sys- tem leads to better results compared to the onelevel process for these structure-based linguistic problems, thereby providing additional support to our hypothesis that chunks and their attributes are detectable in sentence embeddings.The results in terms of average F1 scores for the agreement task, and the latent representation and analysis of the errors made by the system are shown in Figure 5 , and provide several insights. Detailed results are in the appendix.First, the latent representation analysis (Figure 5 .a) shows that while the sentence representations on the latent layer are not as crisply separated by their chunk pattern as for the experiment in Section 5.1, there is a clear separation in terms of the grammatical number of the subject and the verb. This is not surprising as the focus of the task is subjectverb agreement. However, as shown by the results in term of F1 (Figure 5 .b) and the analysis of the errors made by the system on the task (Figure 5 .c, and more detailed in Figure 12 in Appendix A.5.3), there is enough information in these compressed latent representations to capture the structural regularities imposed by the patterns of chunks in the input sequence.Second, the results in terms of F1 (Figure 5 .b) show that the two-level process generalizes better from simpler data -learning on type I and type II leads to better results on all test data, with the highest improvement when tested on type III data, which has the highest lexical variation. Furthermore, the two-level models learned when training on the lexically simpler data perform better when tested on the type III data than the models learned on type III data itself. This result not only indicates that structure information is more easily detectable when lexical variation is less of a factor, but more importantly, that chunk information is separable from other types of information in the sentence embedding, as the patterns detecting it can be applied successfully for data with additional (lexical) variation. 6 The analysis of the errors made by the system (Figure 5 .c) shows that the two-level system has a lower rate of sequence errors (WNA, WN1, WN2 -see Figure 1 ), which from the point of view of the targeted phenomenon are correct (see Section 4.2). The fact that without the sentence compression step (using the one-level model) the system makes more sequence-based errors, indicates that modeling structural information separately is not only possible, but also beneficial for some tasks.The results on the verb alternation BLMs are shown in Figures 6 and 7 . In this problem, structurally similar chunks -NPs, PPs -play different semantic roles in the verb alternation data, as shown in Figure 1 . The TSNE projection of the latent representations on the sentence level (Figures 6.a, 7.a) and the F1 results on the task (Figures 6.b, 7.b) show that the system is able to detect such syntacticsemantic information in the sentence embeddings. The closest latent representations are two that have the same syntactic pattern: NP VerbPass PP, but differ semantically: NP-Theme VerbPass PP-Agent vs. NP-Loc VerbPass PP-Agent, yet they are still distinguished. Detailed error results are included in Figure 13 in Appendix A.5.3.We performed two types of experiments: (i) use individual sentences, and an indirect supervision signal about the sentence structure, (ii) incorporate a sentence representation compression step in a task-specific setting. We have used two tasks, one which relies on more structural information (subject-verb agreement), and one that also relies on semantic information about the chunks (verb 6 Explanation in Appendix A.5.1 alternation).We investigated each setup by the results on the task -average F1 scores, and analysis of the type of errors made by the system (as described in Figure 1 ) -and by the compressed sentence representations on the latent layer of an encoder-decoder architecture.By this dual analysis, one can conclude not only whether a task is solved correctly, but also whether it is solved using structural, morphological and semantic information from the sentence. We found that information about (varying numbers of) chunks -noun, verb and prepositional phrases -and their task-relevant attributes, morphological or semantic, can be detected in sentence embeddings from a pretrained transformer model. The use of probes has been questioned, as the probe itself may assemble the requested information without detecting or modeling the phenomenon of interest (Hewitt and Liang, 2019; Belinkov, 2022; Lenci, 2023) . To partially address this problem, we have used only indirect supervision -within the system, there is no direct information about what characteristics of the answer (on the sentence or the task level) are relevant. Despite the lack of direct supervision, the system is able to compress the structural information necessary to solve the task onto the latent layer of the sentence encoder. In future work, we will investigate whether this information is "hard-coded" -encoded consistently across languages and tasks -in the embeddings, or it relies on shallower features. 
conclusions: 
 Sentence embeddings obtained from transformer models are compact representations, compressing much knowledge -morphological, grammatical, semantic-, expressed in text fragments of various length, into a vector of real numbers of fixed length. We can separate this representation into different layers using a convolutional neural network and distinguish specific information among these layers.In particular, we have shown that we can detect information about chunks -noun/verb/prepositional phrases -and their task-relevant attributes, without providing direct supervision to the system about the targeted structures. This brings us one step closer to understanding and unpacking transformer-based sentence embeddings. 
limitations: 
 We have performed experiments on datasets containing sentences with specific structure and properties to be able to determine whether the type of information we targeted can be detected in sentence embeddings. We have used this data to avoid directly training a classifier, which may learn the task of distinguishing sentences with different chunk patterns without actually using such information from the sentence embeddings. Despite our analyses, there is no guarantee that the information about chunks and their properties is not assembled on the fly from more fine-grained information in the sentence embedding. In future work we plan to investigate whether this is the case, or whether what is encoded is something more abstract, akin to a rule. A.5 The BLM tasks A.5.1 Discussion of errors on the sentence level, when solving the BLM task It might appear surprising that the two-level approach leads to lower performance on type III data, particularly when lexical variation had not been an issue for the sentence representation analysis (see Section 5.1). 
 
 Appendix: The difference comes from the way the instances were formed, on the fly, for the two-level process. The only input to the system is the input of the task. This input, consisting of a sequence of 7 sentences, is used to generate an instance -i.e. a (in, out + , Out -) triple -for the sentence level process for each of these sentences. Because each sentence has a different pattern, and the input and correct output of the sentence level VAE must have the same pattern, the only possible out + is the input sentence in itself. Out -will consist of all the other sentences in the task input sequence.We hypothesize that the fact that the input and output are identical weakens the (indirect) supervision signal. In the stand-alone sentence analysis experiment, the lexical variation between the input and correct answer for the sentence level forces the system to find deeper shared information between the two, and this is not the case when solving the BLM tasks with the two-level system. For type I and type II data, because a task instance (and thus the input sequence) has very little lexical variation, the incorrect answers for the sentence level are very close lexically to the correct answer, and thus the system is guided to encode on the latent layer other distinctions between the correct and incorrect answers, which are mainly the chunk patterns. For type III data, with its maximal lexical variation, there is no pressure on the system to find something other than shallower differences between the answer candidates.We plan to test this hypothesis in future work using a pre-trained sentence-level VAE.A.5.2 Detailed task results We use BNF notation: pp1 and pp2 may be included or not, pp2 may be included only if pp1 is includedgoogle/electra-base-discriminatorhttps://huggingface.co/google-bert/ bert-base-multilingual-casedhttps://huggingface.co/FacebookAI/ xlm-roberta-basehttps://huggingface.co/sentence-transformers/ LaBSE,https://huggingface.co/ sentence-transformers/all-mpnet-base-v2 


Decision: While reviewers found the research question and the experiments interesting, they unanimously had trouble following some parts of the paper. The authors should incorporate the reviewers' comments to improve the writing and presentation of the paper, and expand the informativeness of figure captions for the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper uses a VAE model to analyse the syntactic structure that can be gleamed from sentence embeddings. Specifically, the authors look at the classification token, "[CLS]" of BERT-style encoder models (as well as a brief look at semantically trained models, e.g. SentenceTransformers). Specifically there are two main tasks:

i) A simple VAE trained on sentences constructed from a specific syntactic recipe NP (PP (PP)) VP. () here meaning optional. (It's not entirely clear what characterized the NPs or VPs (presumably they did not include PPs themselves)).  The model takes a 768 dimensional sentence embedding and re-organises it into a 32x24 matrix and then passes it through a 1 layer CNN into a latent dimension of only 5 after which it is passed to the encoder which upscales it to 32x24 again. The loss was trained to maximise the similarity of the output vector to the input vector while getting as far as possible from other sampled vectors (which crucially didn't have the same syntactic parse). 

The trained VAE is close to the target vector in the vast majority of cases (rather unsurprisingly) but interestingly the latent dimension can be perturbed and systematic (e.g. latent units 2-3 encode grammatical number for subject verb agreement). The latent representations also seem highly separable along syntactic lines, even when differences are subtle (pl vs sg agreement) and the sentences are semantically quite different (fig 3). Most interestingly, sentence transformers which are trained on semantic similarity of sentences rather than BERT style cloze tasks or ELECTRA's pretraining paradigm perform much much worse and separating different syntactic patterns.

ii) The second task uses a similar VAE model to do Blackbird Language Matrices (a syntactic analogy of Raven progressive matrices), here the model must choose the best answer out of a set of possible answers given a context of preceding sentences which match to one sentence's syntactic structure. This model contains two VAEs. The first is akin to (i) and is trained by first taking the latent representations of all sentences in a BLM sequence and then trying to reconstruct the latent representations for each sentence while maximising the difference to the other sentences in the BLM sequence. The second then takes these latent representations for all 7 sentences in the sequence and then tries to reconstruct the answer sentence embedding while keeping it far from the wrong answers in the answer set.

Under this approach, the different syntactic structures are not as easily separable in the latent representation. However, the model still largely accurate at choosing the answer sentence in a selection of BLM tasks and is much better than the single layer VAE (except on "type iii" BLMs where the different sentences have maximal lexical variation rather than largely using the same words).
strength_weakness: strength_weakness: - I'm not completely sold that the VAE method really resolves the problems with probing, I think a bit more argumentation in that regard could be useful for the paper.
- The writing is a little opaque, I think the paper could benefit from a bit of restructuring (i.e. being more specific about the tasks done in the first page or two and perhaps a bit less literature review in favour of expansion of explanation of the two-level model)
- In particular, this lack of clarity is particularly problematic with regard to the metrics used. The writing seems ambiguous for two important points:  

i) For Figure 2 and the F1 scores in section 5.1.3 is the model evaluated on choosing the *input* sentence or a sentence with the same syntactic structure as the input sentence? The writing seems compatible with both possibilities and it's crucial to be clear as to which it is. (Likewise it would be good to explicitly state that the F1 scores in figure 5 are on the BLM tasks described in figure 1)

ii) For the 2x VAE model: I initially read this part very differently, understanding it to be that the sentence-level model was trained like the original VAE model and then the sequence level was trained by contrasting the triple constructed from the example sentence members. I think the writing in this paragraph could be greatly improved by just describing the loss function a bit more clearly: it would be good to explicitly show the triple used for sequence level and sentence level side-by-side.
questions: questions: It would be neat to see the similar results for sentence-transformers you have for the first chunk task for BLM! You should expect them to do much much worse than ELECTRA or BERT right?

- More of a writing comment than anything regarding context, the definition of chunk should probably be moved earlier in the text to help with readability. 
- extra parenthesis line 27 page 1
- missing number in line 948 in the appendix
- The references to section numbers in lines 194 to 203 page 3 are a bit confusing
- space after comma line 344 page 5
paper_summary: paper_summary: The paper examines transformer-based sentence embedding models and whether the models encode structural information such as parts of speech or phrase structures.
strength_weakness: strength_weakness: * Although the research question and the premise for the paper are very interesting and worthy of more research, the paper is quite complex and difficult to follow. 
* The paper uses a lot of abbreviations that are well known for practitioners and researchers working with the specific types of models and tasks. However, in such a paper they should be explained. For example: FFNN, CNN, RPM, VAE etc. 
* The graphs and plots are very small with text that is impossible to read. IT is difficult to understand the purpose and the relevance of each graph as the captions are fairly uninformative. E.g. *"Figure 5: VAE vs 2-level VAE (2xVAE) on the agreement BLM problem"*
* The experimental results are difficult to interpret from the paper. The results are not clearly presented.
* The paper refers to error analysis (e.g. line 529), however there is no real error analysis included in the paper.
questions: questions: Consider making the paper more clear to follow by significantly cutting down on content and simplifying it. Now there are too many graphs and a lot of text that do not really add value to the main argument. Present the results more clearly and consider the role of each graph and visualization.
paper_summary: paper_summary: The paper asks whether structural parts of sentences---that is, chunks---are present in sentence embeddings. The paper uses Blackbird Language Matrices, a clever form of Raven's Progressive Matrices where the features that are patterned are grammatical features rather than visual features.
strength_weakness: strength_weakness: - The approach is somewhat convoluted. If I was interested in the research question posed by the title, my first instinct would be to use methods like PCA and ICA to try to find components in the sentence embedding and see if they correspond to sentence parts. A number of other simpler ideas come to mind. Even regarding the BLM task, the method of training a two-level VAE is not as simple as the first thing I would think of, which would be just to provide a BLM pattern as a prompt for an autoregressive model.
- As a result of the unorthodox methods, the results are hard for me to interpret. 
- The work focuses on BERT-style models which are not so relevant to modern LLMs. The same methods could easily be applied to a GPT, but aren't.
- In principle it doesn't seem that reshaping the sentence embedding should not make information any more or less accessible. This suggests that there is something amiss in the approach.
questions: questions: - “Transformer architectures compress the informa021 tion in a sentence – morphological, grammati022 cal, semantic, pragmatic – into a fixed-length one023 dimensional array of real numbers” -- Should be adjusted. This is only one thing you can do with Transformers. They output a sequence of contextualized embeddings. One thing you can do with these embeddings is compress them into a single vector, for example if you use the <CLS> token as input to a classifier. But this is not what the GPTs do, for example.

- Suggestion: Link the VAE approach to SAEs, which are recently extremely hot as an interpretability technique.

"We hypothesize that different types of information from an input are melded together, and no longer overtly accessible in the sentence embeddings" - should be qualified, what is meant is linear accessibility.