Grammatical information in BERT sentence embeddings as two-dimensional arrays
Year: 2023

Abstract:
Sentence embeddings induced with various transformer architectures encode much semantic and syntactic information in a distributed manner in a one-dimensional array. We investigate whether specific grammatical information can be accessed in these distributed representations. Using data from a task developed to test rule-like generalizations, our experiments on detecting subject-verb agreement yield several promising results. First, we show that while the usual sentence representations encoded as onedimensional arrays do not easily support extraction of rule-like regularities, a two-dimensional reshaping of these vectors allows various learning architectures to access such information. Next, we show that various architectures can detect patterns in these two-dimensional reshaped sentence embeddings and successfully learn a model based on smaller amounts of simpler training data, which performs well on more complex test data. This indicates that current sentence embeddings contain information that is regularly distributed, and which can be captured when the embeddings are reshaped into higher dimensional arrays. Our results cast light on representations produced by language models and help move towards developing fewshot learning approaches.

heading: 1 Introduction
text: Transformer-based models have taken the NLP world, and not only, by storm in recent years. They have even reached super-human performance on standard benchmarks such as SuperGLUE (Wang et al., 2019) and SQuAD (Rajpurkar et al., 2018), and the output of GPT-* and chatGPT are often worryingly difficult to distinguish from humanproduced data (Marcus, 2022; Susnjak, 2022). Considering such performance, the expectations are high that the word and sentence representations produced by transformer-based architectures can also be useful for finer-grained tasks, such as those that target specific grammatical phenomena.
Long-distance agreement, a specific and simple grammatical phenomenon, is often used to test the syntactic abilities of deep neural networks (Linzen et al., 2016; Gulordava et al., 2018; Goldberg, 2019; Linzen and Baroni, 2021). Long-distance agreement tests are usually framed as a prediction task: whether the model gives higher probability to the correct form of the verb, despite intervening attractors – nouns appearing between the subject and the verb. This decision is somehow implicit: the language model gives a prediction based on the data it has been built on.
In this paper, we investigate what happens when we target subject-verb agreement explicitly in BERT sentence embeddings for detecting subjectverb agreement in French sentences. We work with raw BERT sentence embeddings – not fine-tuned for specific tasks – and investigate how this specific information is encoded in the distributed representations. More specifically, we ask: Can we detect subject-verb agreement in sentence embeddings? And can we manipulate the raw sentence embeddings to make this phenomenon more easy to detect? To address these questions, we adopt the framework and task definition for rule-like learning described in Merlo et al. (2021). In this framework, learning a given linguistic property or rule is formulated as a task of implicit rule detection in a multiple-choice setup, based on input sequences that share the target property. For the problem under investigation here, each sequence consists of sentences that share subject-verb agreement, but have different distances between the subject and the verb, different clause structures, and different agreement patterns.
We show that BERT sentence embeddings encoded as a one-dimensional array are only successful at detecting subject-verb agreement when provided with large amounts of data. Reshaping the embeddings to two-dimensional arrays, and combining these with VAE-based architectures, allows
22
a system to detect better the shared patterns in the input sequences, while relying on much smaller amounts of simpler data. These results open up new avenues of exploration for few-shot learning (Fei-Fei et al., 2006; Brown et al., 2020; Schick and Schütze, 2021). They also support further analyses of more disentangled representations, those representations that encode underlying rule-like generalisations, typical of human knowledge representation, but not of neural networks (Sablé-Meyer et al., 2021). The contributions of this paper are:
1. We show that that there are higher-dimension patterns that encode syntactic phenomena in BERT sentence embeddings, beyond the onedimensional array representation that is readily available from the transformer output.
2. We show that, used together with VAE-based architectures, two-dimensional reshapings of these representations facilitate the discovery of patterns that encode specific targeted grammatical phenomena.
3. We show that, through the 2D-ed representations, we get better access to encoded patterns, and can detect better the targeted grammatical phenomenon when training with a smaller amount of simpler data.
The code and the data are available here: https://github.com/CLCL-Geneva/ BLM-SNFDisentangling.
Terminology Sentence embeddings can be read from the output of BERT systems as a 1×N vector (N usually 768 or 1024). This can be viewed as the projection of the sentence into an N-dimensional space. In this paper, we use the word dimensions to refer to the shape of the data structure used to represent the sentence embeddings. In particular, we use one-dimensional array to refer to the 1 × N vector sentence representation obtained directly from BERT, and 2D representations to refer to the 2D reshaped (Rows× Columns) array.

heading: 2 Related work
text: Producing sentence representations is a non-trivial issue, mainly because of the structural grammatical and semantic relations they express and their varying complexity and length (Stevenson and Merlo, 2022). The deep learning framework has allowed for a variety of elegant solutions to explicitly learn
sentence representations or to induce them as a side-effect or modeling of a more complex problem (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017; Peters et al., 2018). Transformer architectures, such as BERT (Devlin et al., 2019), have provided one such solution, where the representation of an input text as a one-dimensional array (usually 1x768 or 1x1024 for the large versions of the model) can be readily obtained from the output of the model. Depending how the system is trained, the sentence embedding can be obtained from the encoding of the [CLS] token, or as a combination of the embeddings of the tokens in the sentence.
Sentence transformers (Reimers and Gurevych, 2019) implement architecture and training regime changes on BERT to optimize sentence embeddings for downstream tasks. Nikolaev and Padó (2023) analyze the relation between specific sentence properties (e.g. the contribution of different POS) and the geometry of the embedding space of sentence transformers.
Whether obtained from sentence transformers or directly from the output of a BERT-based system, sentence embeddings have been shown to capture information about syntactic and semantic properties. For example, Manning et al. (2020) show that attention heads capture information about dependency relations in transformer models, and Thrush et al. (2020) show the BERT representations contain important information about argument structure and the meaning of verbs.
Subject-verb agreement is one of the phenomena used to probe a deep-learning system’s syntactic abilities. While it is a simple word-level phenomenon, it encodes long-distance relations between words and requires knowledge of structural sentence properties to be correctly learned. Goldberg (2019) shows that sentence embeddings capture this property, by testing the language model learned by BERT in predicting the contextually appropriate form of the verb. Linzen and Baroni (2021) include an overview of work that analyzes deep-learning models on this task. While the models tested show high performance in predicting the contextually correct form of a verb, they are guided – and misled – by biases within the corpus on which they were trained, e.g. they pay undue attention to the first noun in the sentence. Linzen and Baroni also include a survey of work that probe deep learning models to understand how grammatical
information is encoded. Giulianelli et al. (2018); Conneau et al. (2018); McCoy et al. (2018) show that specific grammatical information – such as the plurality of the subject, the maximal depth of the parse tree of the sentence, the verb auxiliaries – can be decoded from the sentence encodings (or the hidden state) of the respective systems. Lakretz et al. (2021) analyze the actual architecture of an LSTM language model (Gulordava et al., 2018), and track the impact of each unit on the long-distance agreement performance. They uncover a combination of a sparse mechanism – two units – and a larger distributed circuit that together keep track of number and syntactic structure.
Lasri et al. (2022) focus on how BERT encodes grammatical number in English and how this information is used for performing number agreement. The focus is on word embeddings and quantifying how much number information they encode at various layers of the BERT architecture. Using a combination of probing approaches, they discover that subjects and predicates embeddings do encode number information, but at different layers. Further investigations into where and how the number information is shared reveals that number information is not directly shared, but rather passed through intermediate tokens.
We also target BERT embeddings to investigate the subject-verb agreement property. Rather than looking at properties of word/token embeddings, we analyze sentence embeddings as the embeddings of the special [CLS] token. We investigate how accessible the number agreement is in raw BERT sentence embeddings in several steps:
• test whether the subject-verb agreement rule can be recovered through the sentence representation
• test whether different shapes of the sentence embedding – 1D and various 2D forms – make the targeted rule more easy to find
• test these different shapes of sentence embeddings with several encode-decoder architectures, based on variational autoencoders (Kingma and Welling, 2013).

heading: 3 Sentence representations for detecting subject-verb agreement

heading: 3.1 Data
text: Specific grammatical phenomena are often studied on specifically designed or selected datasets (e.g.
(Nikolaev and Padó, 2023; Linzen et al., 2016)). We use BLM-AgrF (An et al., 2023). The structure of each problem in this task and dataset is inspired from RPM visual pattern tests – Raven Progressive Matrices – where one problem consists of overlapping rules the solver must detect (Raven, 1938; Zhang et al., 2019). A Blackbird Language Matrix (BLM) problem (Merlo et al., 2021) for subjectverb agreement consists of a context set of seven sentences that share the subject-verb agreement phenomenon, but differ in other aspects – e.g. number of intervening attractors between the subject and the verb, different grammatical numbers for these attractors, and different clause structures. An example template is illustrated in Figure 1, and an actual example in Figure 2.
The dataset comprises three subsets, of increasing lexical complexity. Type I data is generated
based on manually provided seeds, and a template that captures the rules mentioned above. Type II data is generated based on Type I data, by introducing lexical variation with the aid of a transformer, by generating alternatives for masked nouns. Type III data is generated by combining sentences from different instances from the Type II data. This will allow us to investigate the impact of lexical variation on the ability of a system to detect grammatical patterns.
Each subset contains an equal number of instances comprising three clause structures (we include complete instances – in French – in Appendix A.1). These structural variations alter the distance and relative depth of the subject and verb to produce a variety of conditions, to allow us to investigate how the subject-verb agreement information is encoded in BERT sentence embeddings.
Each problem is paired with a set of candidate answers. To allow for probing the learned model, apart from the correct answer, the answer sets contain negative examples built by corrupting some of the generating rules. This helps investigate the kind of information and structure learned, and the type of mistakes a system is prone to.
Table 1 shows the data statistics. Each of the three subsets of datasets is split 90:10 into train and test subsets, which are provided with the data. We use 20% of the train data for development.

heading: 3.2 Sentence representations
text: We investigate BERT sentence representations in a series of architectures designed to test whether we can access the relevant information for subjectverb agreement detection. We obtain the sentence embedding from the last layer of BERT, as the embedding of the [CLS] special token.
Figure 3 shows the summary of the architectures explored. Details of the architecture parameters are in Appendix A.2.
To investigate the impact of 2D-ing the sentence embeddings, the input sequences are given as a
stack of 1D or 2D-ed sentence embeddings. This sequence of architectures also allows us to test the impact of additional abstracting steps – through compression into a latent VAE layer – for accessing patterns that encode the desired information.1
FFNN baseline The FFNN baseline is a threelayer feed-forward neural network. It transforms the sequence of the seven context sentence embeddings into a 1D-tensor, which is passed through three fully-connected layers, and outputs a vector that we take to represent the embedding of the answer sentence. This architecture allows the system to find patterns within and across sentences through the nodes in the successive layers.
The learning objective is to maximize the probability of the correct answer from the candidate answer set. Because the incorrect answers in the answer set are specifically designed to be minimally different from the correct answer, we implement the objective through the max-margin loss function. This function combines the scores of the correct and erroneous ones relative to the sentence embedding predicted by the system. We first compute a
1The code is available here: https://github.com/ CLCL-Geneva/BLM-SNFDisentangling
score for the embedding ei of each candidate answer ai in the answer set A with respect to the predicted sentence embedding epred as the cosine of the angle between the respective vectors:
score(ei, epred) = cos(ei, epred)
The loss uses the max-margin between the score for the correct answer ec and for each of the incorrect answers ei: La = ∑
ei
[1−score(ec, epred)+score(ei, epred)]+
For prediction, the answer with the highest score from a candidate set is taken as the correct answer.
CNN baseline The CNN baseline consists of N convolutional steps, followed by a linear layer to compress the output to the desired dimensions. The input consists of a stack of sentence representations. We use two variations of this architecture: (i) Baseline_CNN_1DxSeq: for a stack of 1D sentence representations, there are three 2D convolutional steps, which use kernels of size 3x3; (ii) Baseline_CNN_{NxM}: for a stack of (NxM) 2D-ed sentence representations, there is one 3D convolutional layer, with a kernel size 3x15x15. The size of the kernels was set after preliminary experiments. For both variations, the kernels allow the system to detect patterns within a sentence representation and across the sequence.
This system uses the same max-margin loss function as the FFNN baseline system.
Encoder-decoder This system is essentially a variational autoencoder (VAE) (Kingma and Welling, 2013; Kingma et al., 2015), but it does not reconstruct the input, rather it constructs an answer. For each of the input variations, the encoder consists of a similar architecture as the corresponding CNN baseline2, but the output of the linear layer is the size L of the latent layer (2× L to represent the mean and standard deviation for a vector of length L). A new vector (of size L) is sampled using the output of the encoder as the means and standard deviation of L normal distributions, using the reparametrization trick (Kingma et al., 2015). This vector is then unpacked through a decoder to produce a sentence representation. The
2Because the FFNN baseline performed very well, and this set-up provides a full receptive field, we also had an encoderdecoder variation using FFNN but it performed much worse than the other variations, and we do not report on it here.
architecture of the decoder mirrors as close as possible the architecture of the encoder. It differs in that it outputs a single sentence representation, and not the representation of a sequence of sentences. The two variations are named VAE_1DxSeq and VAE_{MxN} in the upcoming results tables.
The training objective is to maximize the probability of the correct answer, while improving the approximation of the posterior distribution on the latent layer. This is implemented through a loss function that combines the max-margin loss on the constructed answer (as for previous architectures), with an additional factor – the regularization factor on the latent layer, typical of VAE models:
Ll = KL(qenc(z|x) ∥ p(z))
where qenc is the approximate posterior distribution of p, p(z) = N (0, 1) and qenc(z|x) = N (µx, σx), where the [µx;σx] is the latent vector output by the encoder for the input vector x.
The final loss function is:
Lenc−dec = La + β ∗ Ll
The β coefficient is used to push for disentanglement on the latent layer of a VAE (Higgins et al., 2016). For the reported experiments β = 1.
Dual VAE The dual VAE adds a decoder to reconstruct the input to the encoder-decoder system, which mirrors the encoder in architecture and parameters. The two variations are Dual_VAE_1DxSeq and Dual_VAE_{NxM}.
The training objective is to maximize both the probability of the answer and the reconstructed input, and improve the approximation of the posterior distribution on the latent layer. Essentially, we add a factor to the loss function of the encoder-decoder architecture, reflecting the reconstruction loss:
LdV AE = La + α ∗ Lrecon + β ∗ Ll
where α is a coefficient that can control the contribution of the input reconstruction signal. For the experiments reported, α = 0.01, and serves as a scaling factor, to bring the value of the reconstruction loss within the same magnitude as the answer loss La. The reconstruction loss is computed as the mean-square error between the representation of the input sequence (x) and the reconstructed one (x′): Lrecon = MSE(x, x′)

heading: 4 Experiments
text: We report experiments on seven systems – the baseline FFNN, and two variations (the input as a stack of 1D or 2D sentence representations) for each of the other three architectures.
Our aim was to explore raw BERT sentence embeddings, and not variations fine-tuned for specific tasks. The results reported here are based on sentence embeddings obtained using BERTTokenizer and BERTModel from the transformers Python library, using the pretrained "BERT-basemultilingual-cased" model3.4
Preliminary experiments have been used to determine the kernel size for processing the 2D and 3D tensors with the stack of 1D and 2D-ed sentence embeddings respectively. The optimal kernel for 2D tensors was 3x3, and for the 3D tensors was 3x15x15. This large kernel size imposes specific restrictions on the dimensions of the 2D-ed sentence embeddings. Because a sentence embedding is a one-dimensional array of length 768, there are
3https://huggingface.co/ bert-base-multilingual-cased
4We have run preliminary experiments with Frenchspecific sentence embeddings using FlauBERT (Le et al., 2020). The results were lower than when using a multilingual cased BERT language model.
only 4 possible 2D values, where both dimensions are greater than 15 (16x48, 24x32, 32x24, 48x16).
We have also explored the size of the latent layer (5:25, step 5), and chosen the latent size 5. All systems used a learning rate of 0.001 and Adam optimizer, and batch size 100. For the baselines and experiments on the full training set for type II and type III data the training was done for 50 epochs, and for type I data and the sameTrain set-up, the training was done for 120 epochs.
The experiments were run on an HP PAIR Workstation Z4 G4 MT, with an Intel Xeon W-2255 processor, 64G RAM, and a MSI GeForce RTX 3090 VENTUS 3X OC 24G GDDR6X GPU.

heading: 4.1 Results
text: Figures 4 and 5 show the main experimental results. All results represent F1 averages over five runs. More details are provided in Appendix A.3.
Figure 4 shows the difference in F1 between the FFNN baseline (the black lines) and the increasingly complex architectures. Each row corresponds to the data type used for training, and the columns to the data type used for testing. Each panel shows the test results of models trained on the same amount of data, on the left, and training on full data, on the right.
Figure 5 shows the results obtained using different 2D transformations of the one-dimensional tensor BERT sentence embeddings with the dual VAE system, when training on the same amount of data (2073 instances – the amount available for type I data). Overall, the best-performing embedding is the one reshaped in a 48x16 matrix, the setting then used for the results reported in the system study, shown in Figure 4.
Figure 7 shows the impact of the amount of training data on the performance of the models. The results reported are average F1 scores over 5 runs, using the dual-VAE architecture with 48x16 sentence embedding.

heading: 4.2 Discussion
text: Impact of 2D-ed representations and VAE-based architectures In Figure 4, the horizontal black lines represent the performance of the Baseline FFNN system, and the bars show the relative performance of the system variations with 1D and 2D-ed sentence representations.
When using the full training data the results on type II and type III subsets are very high. This is in line with ML theory, as input with more variety leads to better-performing models, when given enough training data. The low results of the baselines and the systems using the 1D representations on the restricted training set-up (2073 instances – the available amount of training+validation data for type I – for all subsets) shows that these systems do not access the most relevant information from the sentence embedding for our targeted phenomenon.
Pairwise comparisons of similar architectures with different types of input show that 2D-ed representations lead to better results in almost all settings, particularly in the harsher training scenario with limited data (the left bar group in each plot).
The progression of architectures – from the CNN
to the dual VAE – also show an increase in results, for both types of input representations. The phenomenon is more evident – and more useful – particularly for the restricted training scenario. It shows that forcing the representations to more compressed and abstract forms is useful for distilling the information useful to detect our targeted grammatical phenomenon.
The most interesting result is a combination of the impact of the 2D-ed representations and the various architectures: as the panels corresponding to training on type I data (first row in Figure 4) show, the combination of the dual VAE architecture with 2D-ed sentence embedding leads to the best results when testing on type II and type III data, which are lexically more complex than type I data. This shows that with a good combination of input representation and system, a model can find robust patterns even in a smaller amount of simple data.

heading: Impact of 2D-ing the sentence representation
text: The results presented in Figure 4 show that the 48x16 2D version of the sentence representation leads to better results than the 1x768 version. Figure 5 shows the impact of various 2D variations, when training the systems on the same amount of training+validation data. The results are obtained with the dual VAE architecture. The (overall) best performing representation from the 4 variations is the 48x16 version. In fact, for the N ×M 2D-ing, the results are better the smaller M becomes. This indicates that information is somehow uniformly distributed in a BERT sentence embedding, within subsequences of length close to 16 – at least the information relevant to our particular subject-verb agreement task.
Impact of training data Figure 4 shows that the 2D-ed sentence representation combined with the dual VAE architecture leads to the best results, par-
ticularly when training the systems on the same amount of training+validation data. We further analyze the learning curves when varying the amount of training+validation data from 50 to 2073 (split 80:20 into training and validation data). Figure 7 shows these results.
The curves corresponding to training on type I
data – in shades of purple – approach the higher performance fastest, showing that they are able to exploit smaller amounts of data better. The lexical variations in type II and type III data seem to obfuscate the targeted patterns, as they require more training data.

heading: 4.3 Error Analyses
text: The error analysis shown in Figure 6 clearly shows that errors are almost always of the same kind — the assignment of the wrong grammatical number to the second attractor, closer to the verb— thus performing a local kind of agreement instead of the correct longer-distance structural agreement. These types of errors are frequent in humans too (Linzen and Leonard, 2018). It can be seen that the dual-VAE corrects these errors to a great degree (and greater than the other models), thus showing that it can detect the non-local patterns better than the other architectures. The combination of 2D-ing the sentence embedding and the large size of the kernel allows the system to find more distant patterns in the sentence embedding, thus connecting more distant tokens.

heading: 5 Conclusion
text: We have proposed and investigated variations of BERT sentence representations for the task of iden-
tifying the rule of subject-verb agreement and some of its properties in a dataset consisting of sequences of sentences that instantiate these underlying rules. BERT sentence embeddings, as 1D arrays, have been successfully used for a variety of tasks, but the information they encode about specific phenomena is distributed over the vector. Reshaping the sentence representation into two-dimensional inputs leads to improved results, and an additional step of abstracting these 2D-ed sentence embeddings leads to further improvements for the task. These more abstracted 2D-ed embeddings also learn a robust model based on a smaller amount of simpler training data, while showing good performance on the more lexically-complex data.
We plan to explore whether BERT sentence embedding may have even higher-dimension patterns, and whether such nD-ed BERT sentence representation can be used to detect other grammatical or semantic phenomena. We also plan to directly distill 2D sentence representation in more compact and disentangled representation, to encode more explicitly some of the distributed information in these embeddings.

heading: Limitations
text: The experiments reported were performed on a dataset of French sentences, with a particular organization: sequences of sentences as input, each with a slightly different structure but sharing the subject-verb agreement rule. All sentences in the input sequence are processed together. In future work we plan to separate the distillation of rules from a sentence representation from the processing of the sequence.
We have investigated only part of the parameters in the proposed architectures. In particular, the β coefficient in the encoder-decoder and the dual VAE architectures was set to 1. Higher values may lead to more disentangled representations on the latent layer.

heading: Ethics Statement
text: To the best of our knowledge, there are no ethics concerns with this paper.

heading: Acknowledgments
text: We gratefully acknowledge the partial support of this work by the Swiss National Science Foundation, through grants #51NF40_180888 (NCCR
Evolving Language) and SNF Advanced grant TMAG-1_209426 to PM.

heading: A.1 Type I instance examples

heading: A.2 System architecture details
text: Baseline_FFNN =================================================================================== Layer (type:depth-idx) Output Shape Param # =================================================================================== BaselineFFNN [100, 768] -- --Linear: 1-1 [100, 1536] 8,259,072 --Linear: 1-2 [100, 1536] 2,360,832 --Linear: 1-3 [100, 768] 1,180,416 =================================================================================== Total params: 11,800,320 Trainable params: 11,800,320 Non-trainable params: 0 Total mult-adds (G): 1.18 =================================================================================== Input size (MB): 2.15 Forward/backward pass size (MB): 3.07 Params size (MB): 47.20 Estimated Total Size (MB): 52.42 ===================================================================================
Baseline_CNN_1DxSeq: CNN with stack of 1D sentence embeddings =================================================================================== Layer (type:depth-idx) Output Shape Param # =================================================================================== BaselineCNN_1DxSeq [100, 768] -- --Conv2d: 1-1 [100, 4, 5, 766] 40 --Conv2d: 1-2 [100, 8, 3, 764] 296 --Conv2d: 1-3 [100, 16, 1, 762] 1,168 --Linear: 1-4 [100, 768] 9,364,224 =================================================================================== Total params: 9,365,728 Trainable params: 9,365,728 Non-trainable params: 0 Total mult-adds (G): 1.11 =================================================================================== Input size (MB): 2.15 Forward/backward pass size (MB): 37.29 Params size (MB): 37.46 Estimated Total Size (MB): 76.91 ===================================================================================
Baseline_CNN_48x16: CNN with 48x16 2D-ed sentence embedding =================================================================================== Layer (type:depth-idx) Output Shape Param # =================================================================================== BaselineCNN [100, 7, 768] -- --Conv3d: 1-1 [100, 32, 5, 34, 2] 21,632 --Linear: 1-2 [100, 768] 8,356,608 =================================================================================== Total params: 8,378,240 Trainable params: 8,378,240 Non-trainable params: 0 Total mult-adds (G): 1.57 =================================================================================== Input size (MB): 2.15 Forward/backward pass size (MB): 9.32 Params size (MB): 33.51 Estimated Total Size (MB): 44.98 ===================================================================================
VAE_1DxSeq: encoder-decoder with stack of 1D sentence embeddings =================================================================================== Layer (type:depth-idx) Output Shape Param # =================================================================================== VariationalAutoencoder [100, 5] -- --Encoder: 1-1 [100, 5] --
--Conv2d: 2-1 [100, 4, 5, 766] 40 --Conv2d: 2-2 [100, 8, 3, 764] 296 --Conv2d: 2-3 [100, 16, 1, 762] 1,168 --Linear: 2-4 [100, 10] 121,930
--simpleSampling: 1-2 [100, 5] -- --Decoder_answer: 1-3 [100, 1, 1, 768] --
--Linear: 2-5 [100, 12192] 73,152 --ConvTranspose2d: 2-6 [100, 8, 1, 764] 392 --ConvTranspose2d: 2-7 [100, 4, 1, 766] 100 --ConvTranspose2d: 2-8 [100, 1, 1, 768] 13
=================================================================================== Total params: 197,091 Trainable params: 197,091 Non-trainable params: 0 Total mult-adds (M): 230.28 =================================================================================== Input size (MB): 2.15 Forward/backward pass size (MB): 54.40 Params size (MB): 0.79 Estimated Total Size (MB): 57.33 ===================================================================================
VAE_48x16: encoder-decoder with 48x16 2D-ed sentence embeddings
=================================================================================== Layer (type:depth-idx) Output Shape Param # =================================================================================== VariationalAutoencoder [100, 5] -- --Encoder: 1-1 [100, 5] --
--Conv3d: 2-1 [100, 32, 5, 34, 2] 21,632 --Linear: 2-2 [100, 10] 108,810
--simpleSampling: 1-2 [100, 5] -- --Decoder_answer: 1-3 [100, 1, 1, 48, 16] --
--Linear: 2-3 [100, 2176] 13,056 --ConvTranspose3d: 2-4 [100, 1, 1, 48, 16] 7,201
=================================================================================== Total params: 150,699 Trainable params: 150,699 Non-trainable params: 0 Total mult-adds (G): 1.30 =================================================================================== Input size (MB): 2.15 Forward/backward pass size (MB): 11.07 Params size (MB): 0.60 Estimated Total Size (MB): 13.82 ===================================================================================
Dual_VAE_1DxSeq: dual VAE with stack of 1D sentence embeddings
=================================================================================== Layer (type:depth-idx) Output Shape Param # =================================================================================== VariationalAutoencoder [100, 5] -- --Encoder: 1-1 [100, 5] --
--Conv2d: 2-1 [100, 4, 5, 766] 40 --Conv2d: 2-2 [100, 8, 3, 764] 296 --Conv2d: 2-3 [100, 16, 1, 762] 1,168 --Linear: 2-4 [100, 10] 121,930
--simpleSampling: 1-2 [100, 5] -- --Decoder_mirror: 1-3 [100, 1, 7, 768] --
--Linear: 2-5 [100, 12192] 73,152 --ConvTranspose2d: 2-6 [100, 8, 3, 764] 1,160 --ConvTranspose2d: 2-7 [100, 4, 5, 766] 292 --ConvTranspose2d: 2-8 [100, 1, 7, 768] 37
--Decoder_answer: 1-4 [100, 1, 1, 768] -- --Linear: 2-9 [100, 12192] 73,152 --ConvTranspose2d: 2-10 [100, 8, 1, 764] 392 --ConvTranspose2d: 2-11 [100, 4, 1, 766] 100 --ConvTranspose2d: 2-12 [100, 1, 1, 768] 13
=================================================================================== Total params: 271,732 Trainable params: 271,732 Non-trainable params: 0 Total mult-adds (M): 635.19 =================================================================================== Input size (MB): 2.15 Forward/backward pass size (MB): 95.37 Params size (MB): 1.09 Estimated Total Size (MB): 98.61 ===================================================================================
Dual_Vae_48x16: dual VAE with 48x16 2D-ed sentence embeddings
=================================================================================== Layer (type:depth-idx) Output Shape Param # =================================================================================== VariationalAutoencoder [100, 5] -- --Encoder: 1-1 [100, 5] --
--Conv3d: 2-1 [100, 32, 5, 34, 2] 21,632 --Linear: 2-2 [100, 10] 108,810
--simpleSampling: 1-2 [100, 5] -- --Decoder_mirror: 1-3 [100, 1, 7, 48, 16] --
--Linear: 2-3 [100, 10880] 65,280 --ConvTranspose3d: 2-4 [100, 1, 7, 48, 16] 21,601
--Decoder_answer: 1-4 [100, 1, 1, 48, 16] -- --Linear: 2-5 [100, 2176] 13,056 --ConvTranspose3d: 2-6 [100, 1, 1, 48, 16] 7,201
=================================================================================== Total params: 237,580 Trainable params: 237,580 Non-trainable params: 0 Total mult-adds (G): 12.92 =================================================================================== Input size (MB): 2.15 Forward/backward pass size (MB): 24.07 Params size (MB): 0.95 Estimated Total Size (MB): 27.17 ===================================================================================

heading: A.3 Detailed experimental results
text: Results on 2D-ing BERt sentence embeddings The detailed version of the results in Figure 5
Results on system analysis The detailed version of the results in Figure 4

heading: TRAIN ON THE SAME AMOUNT OF DATA (2073 INSTANCES: 1658 TRAIN/415 VALIDATION)

heading: TRAIN ON TEST ON BASELINE_CNN_1DXSEQ VAE_1DXSEQ DUAL_VAE_1DXSEQ TRAIN ON FULL TRAINING DATA

heading: Future Work
text: In future work we plan to separate the distillation of rules from a sentence representation from the processing of the sequence. We have investigated only part of the parameters in the proposed architectures. In particular, the β coefficient in the encoder-decoder and the dual VAE architectures was set to 1. Higher values may lead to more disentangled representations on the latent layer