Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks (2024)

Abstract: Disentangled latent spaces usually have better semantic separability and geometrical properties, which leads to better interpretability and more controllable data generation. While this has been well investigated in Computer Vision, in tasks such as image disentanglement, in the NLP domain sentence disentanglement is still comparatively under-investigated. Most previous work have concentrated on disentangling task-specific generative factors, such as sentiment, within the context of style transfer. In this work, we focus on a more general form of sentence disentanglement, targeting the localised modification and control of more general sentence semantic features. To achieve this, we contribute to a novel notion of sentence semantic disentanglement and introduce a flow-based invertible neural network (INN) mechanism integrated with a transformer-based language Autoencoder (AE) in order to deliver latent spaces with better separability properties. Experimental results demonstrate that the model can conform the distributed latent space into a better semantically disentangled sentence space, leading to improved language interpretability and controlled generation when compared to the recent state-of-the-art language VAE models.

heading: 1 Introduction
text: Most previous work on controlled text generation have concentrated on style transfer tasks: modifying sentences with regard to markers of sentiment, formality, affirmation/negation (John et al., 2019; Bao et al., 2019; Hu and Li, 2021; Vasilakes et al., 2022; Gu et al., 2022; Liu et al., 2023; Gu et al., 2023) (Figure 1 top). Disentanglement of language generative factors over Variational Autoencoder (VAE) spaces has been a key mechanism to deliver this type of generative control (John et al., 2019; Bao et al., 2019; Vasilakes et al., 2022). Recently, Zhang et al. (2022) demonstrated that a more general form of semantic control can be achieved in the latent space of Optimus (Li et al., 2020b), the first
standard transformer-based VAE, where a BERT (Devlin et al., 2018) encoder and a GPT2 (Radford et al., 2019) decoder are connected within a VAE bottleneck. Using representations of conceptually dense explanatory sentences (Jansen et al., 2018b), they showed that sentences (e.g. animals require oxygen for survival), can be represented within a space which can be organised around the associations between predicate, arguments and their associated token content: ARG0-animals or VERBrequire, is geometrically resolved to a hypersolid
2113
over the latent space. Nevertheless, the ability to learn and control such separation is still limited as different semantic factors of the sentence are still overlapped and entangled in the latent space (e.g., V-eat and V-require in Figure 1 bottom left), indicating distributional sentence semantics cannot be currently localised and controlled from the perspective of formal semantics (i.e., predicate-argument structures, compositionality) (Marcus, 2003; Nefdt, 2020; Dankers et al., 2022).
This work aims to improve the localisation and semantic control of latent sentence spaces, by delivering a model which can better separate and control syntactic-semantic features (e.g. predicateargument) and their associated lexical semantics content. This type of representation can provide the foundation to shorten the gap between deep latent semantics and formal linguistic representations (Gildea and Jurafsky, 2000; Banarescu et al., 2013; Mitchell, 2023), integrating the flexibility of distributional-neural models with the properties of linguistically grounded representations, facilitating both interpretability and generative control.
To deliver this type of semantic control within the distributional sentence space, following the methodological framework introduced by (Zhang et al., 2022), we target on improving the semantic separability of sentences by focusing on explanatory sentences 1, rather than synthetic or style transfer datasets (Hupkes et al., 2020; Yanaka et al., 2021), in which the semantic structure of sentences can be isolated and controlled. Inspired by the work of Esser et al. (2020), we integrate a flowbased invertible neural network (INN) (Dinh et al., 2014) as a plug-in control component to learn the bijective transformation between the distributional hidden space of the transformer-based language autoencoder (BERT-GPT2) and the smooth Gaussian space of the INN (Figure 2). Specifically, we first pre-train an autoencoder (AE) to learn sentence representations from the transformers’ latent spaces. Then, we freeze the AE weights and train the INN to map the AE representations to a Gaussian space. Since INN models define a bijective transformation, we can control the autoencoder generation by manipulating the INN latent spaces, which is more efficient and significantly less resource intensive than re-training a language AE end-to-end.
More importantly, we propose a supervised train-
1The rationale for choosing explanatory sentences and their semantic details are provided in Appendix A.
ing strategy within the INN setting to learn a latent space with improved semantic separability, namely: the semantic role-content pairs and their associated clusters can be better separated over the latent space modelled by the INN (Section 4.1). In this case, we can improve localised control over the decoding process due to the reduction of overlapping (ambiguous) regions. A more separable and geometrically consistent sentence space can be then operated over to improve the generative control with support of geometric operators, such as interpolation (Bowman et al., 2016) (Section 4.2). The contributions of this work are summarised below:
1. We approach sentence disentanglement and generation control from the point of view of Argument Structure Theory (AST), bridging latent space features with a canonical, linguistics-informed, semantic representation of sentences. 2. We find that integrating a flow-based INN mechanism into a transformer-based language-AE architecture is an effective mechanism for transforming the hidden space of the autoencoder into a smooth Gaussian latent space for representing sentences. 3. We propose a supervised training strategy for INNs to learn a controllable semantic space with higher disentanglement and separability of semantic features, when compared to previous work. 4. Using this mechanism, we systematically employ geometrical data augmentation strategies to assist on sentence representation disentanglement.
Interpreting and controlling sentence generation from the perspective of the geometric manipulation of the latent space is still largely unexplored within NLP. To the best of our knowledge, this is the first work which focuses on the introduction of invertible NN-based mechanisms to support latent spaces with better separated argument structure/semantic features, allowing for a more universal form of sentence generation control.

heading: 2 Preliminaries
text: In this section, we first introduce the formal semantics and define the sentence representation model based on Argument Structure Theory (AST), linking with the associated disentanglement/generative factors and then proceed with the description of the proposed flow-based INN mechanism.
Controllability and interpretation in formal semantics. Formal semantics, which provides a canonical, granular, and rigid representation, have been investigated for thousands of years, with well
established theoretical frameworks such as Montague Semantics (Dowty et al., 2012), Davidsonian Semantics (Davidson, 1967), Neo-Davidsonian Semantics (Lasersohn, 2016), etc. One typical characteristic of these formal semantics is the localisation or composition property. For example, in the sentence, animals require oxygen for survival, the words are functionally combined into sentence semantics:
λx(animals(x) → require(x, oxygen)) (1)
Where x is the variable representing any entity within a logical structure. In this case, we can localise the sentence semantics by replacing x with birds, fishes, etc. This localised process indicates the interpretation in Cognitive Science (Smolensky, 2006; Lees, 1957). Disentanglement (Bengio, 2013) can potentially provide such localisation in the context of distributional latent representations, which has been widely investigated to localise image generation (Esser et al., 2020; Jeon et al., 2019; Liu et al., 2021). Therefore, we link several key notions—disentanglement, formal semantics, and localization—to investigate formal control and interpretability in language models.
Sentence semantic disentanglement. AST (Jackendoff, 1992; Levin, 1993; Rappaport Hovav and Levin, 2008) provides a model for representing sentence structure and meaning of sentences in terms of the interface between the their syntactic structure and the associated semantic roles of the arguments within those sentences. It delineates how verbs define the organisation of their associated arguments and the reflection of this organisation in a sentence’s syntactic realisation. AST abstracts sentences as predicate-argument structures, where the predicate p (associated with the verb) has a set of associated arguments argi, where each argument has an associated positional component i and a thematic/semantic roles ri, the latter categorising the semantic functions of arguments in relation to the verb (e.g. agent, patient, theme, instrument). In the context of this work, the AST predicate-argument representation is associated with a lexical-semantic representation of the content ci of the term ti.
In this work, we simplify and particularise the relationship between the argument structure and the distributional lexical semantic representation as a role-content relation, where the structural syntactic/semantic relationship is defined by its shallow
semantics, i.e. as the composition of the content of the terms, their position in the predicate-argument (PArg) structure (argi) and their semantic roles (SRs) (ri: pred, arg). Therefore, this work uses the notion of sentence semantic disentanglement as the cluster separation of the content under the PArg/SRs structure (the corresponding role in rolecontent), aiming to induce a latent space which geometrically encodes the AST structure, better disentangling and separating role-content clusters.
S
NP
NN
ARG0 (Agent)
animals
VP
VBZ
PRED
require
NP
NP
ARG1 (Patient)
oxygen
PP
ARGM-PRP
PRP
for
PRP
survival
Formally, a sentence s (e.g., see above) consists of a sequence PArgs/SRs and word content associations. Upon encoding in latent space, this can be described as:
sem(s) = t1(c1, r1)︸ ︷︷ ︸ i.e.,ARG0−animals ⊕ · · · ⊕ ti(ci, ri)︸ ︷︷ ︸ PRP−survival
where ti(ci, ri) = ci ⊗ ri represents the semantics of term ti with content ci (i.e., animals) and SRL ri (i.e., ARG0) in context s, ⊗: connects the meanings of words with their roles, using the compositionaldistributional semantics notation of (Smolensky and Legendre, 2006; Clark et al., 2008). ⊕: connects the lexical semantics (word content + structural role) to form the sentence semantics. This work applies distinct symbols aiming to emphasise the disentanglement aspects associated with the AST structure. If the sentence representation can be semantically disentangled under ⊕, the sem(s) can be decomposed into:
sem(s) = {t1(c1, r1)} ⊕ · · · ⊕ {ti(ci, ri)} where each set represents a specific role-content cluster resolved to a hypersolid over the latent space, in this case, given a set of N sentences within the same predicate cluster t(c, r) (i.e., Vrequire) but different sem(s), those sentence vectors can represent t(c, r) features independently of
other features (i.e., ARG0-animals), forming the t(c, r) cluster:
{sem(s1), ..., sem(sN )} = {t(c, r)}×N ⊕ {...} Therefore, we can evaluate the disentanglement (i.e., natural clustering property (Bengio, 2013)) of sentence semantics by evaluating the density within {t(c, r)} set(cluster) (classifier recall) and the separation between different {t(c, r)} set(clusters) (classifier accuracy) with downstream classifiers based on the manifold hypothesis for classification (Rifai et al., 2011), rather than disentanglement metrics, which usually calculate the separation between latent dimensions, commonly used in the image domain (Higgins et al., 2017; Kim and Mnih, 2018; Chen et al., 2018; Ridgeway and Mozer, 2018). Next, we will introduce the INNbased mechanism which is used to learn this semantically disentangled space.
Invertible Neural Networks (INNs). Flowbased INNs (Dinh et al., 2014, 2016) are a class of neural networks that models the bijective mapping between the observation distribution p(x) and the latent distribution p(z). We use T to represent the forward mapping (from p(x) to p(z)) and T ′ to represent the backward mapping (from p(z) to p(x)), respectively. Unlike VAEs that approximate the prior distribution to multivariate Gaussian distributions, INNs exactly use multivariate Gaussian distributions. These are trained by the following objective function: L = −Ex∼p(x) [ T (x) ]2 +log |T ′(x)|
where T (x) learns the transformation from x to z ∼ N(0, 1). |T ′(x)| is the determinant of the Jacobian for T (x), which indicates the extent in which the transformation locally expands or contracts the space. The term log |T ′(x)| ensures the integration of the probability density function to be one. The forward and reversed mapping can be implemented via the coupling layer (Dinh et al., 2014; Kingma and Dhariwal, 2018).
The rationale for choosing flow-based INNs lies on the fact that they learn the bijective transformation between the latent and observed spaces, which can be used to guide the autoencoder generation by manipulating the INN latent space, which is more efficient and has lower computational demand than re-training a language VAE. Besides, flow-based INNs that learn the prior distribution (i.e., Gaussian) exactly, can theoretically prevent the information loss from variational inference (ELBO) where the prior is approximated from posterior P (z|x).

heading: 3 Proposed Approach
text: We encode each sentence x with a frozen autoencoder (e.g., Bert-GPT2) and consider its sentence representation E(x) as the input of INNs (Figure 2). We propose two training strategies to map the hidden representations into the Gaussian space.

heading: 3.1 Training Strategy
text: Unsupervised INN. Firstly, we train the INNbased model unsupervised, which minimises the negative log-likelihood of the marginal distribution of latent representation z = E(x):
Lunsup = −Ex∼p(x) [ T (E(x)) ]2 + log ∣∣T ′(E(x)) ∣∣
As the minimisation leads to a bijective mapping between the distributed representation and the disentangled latent representation (multivariate Gaussian space), it allows for a more semantically consistent representation of the geometric (role-content) clustering properties of its latent space, allowing for a more consistent traversal and interpolation (Li et al., 2020b) over the sentence space (Figure 1).
Cluster-supervised INN. According to the findings of (Zhang et al., 2022), the content of the predicate-argument structure/semantic roles can be disentangled over the latent space approximated to multivariate Gaussian learned using the Language VAE setting. Using the same foundation, we train the INN component to learn the embeddings, by minimising the distance between points in the same role-content regions and maximising the distance between points in different regions, based on the explanation embeddings and their corresponding central point from the language autoencoder model.
For example, given a sentence "animals require food for survival" and its central vector of ARG0animals, the training moves the sentence representation closer to the ARG0-animals region centre in the INN latent space. Specifically, during the calculation of the posterior, we replace the mean and variance of the standard Gaussian distribution by the centre point of its cluster and a hyper-parameter, which should be less than one, respectively. In this case, each role-content cluster in the latent space will be mapped to a space where each cluster will have its embeddings more densely and regularly distributed around its centre. The objective function can be described as follows:
Lsup =− Ex∼pcluster(x)
[ T (E(x))− µcluster ]2
1− σ2 + log ∣∣T ′(E(x)) ∣∣
where T (E(x)) learns the transformation from x to z ∼ N(µcluster, 1− σ2). σ2 is a parameter which can be empirically determined (in this particular context the optimal value was found to be 0.6). Additional details are provided in Appendix A.

heading: 3.2 Geometrical Data Augmentation
text: Data augmentation, which captures and augments a common or distinct feature across different samples, has been considered a common technique to assist disentanglement, such as in Graph (Li et al., 2021) and Image (Liu et al., 2022) representations, but is still limited in the context of sentence generation. In this work, we consider the vector arithmetic and traversal operators as a systematic mechanism to support data augmentation (via semantically controlled sentence generation) for each role-content cluster, described as follows:
(1) v = average(E′(xi), E′(xj)) (2) vneighbour = v[i] ∼ N(0, 1)∀i∈{0,..,size(v)} (3) xnew = D ′(vneighbour)
where xk ∈ S (original corpus), E′ and D′ are the encoder and decoder of Optimus fine-tuned over S. average operation aims to modify the sentence while maintaining the target role-content common to both xi and xj (Zhang et al., 2022). The term v[i] ∼ N(0, 1) is introduced to resample each dimension of v in the latent space (i.e., traverse its neighbour) and xnew = D′(vneighbour) generates a new sentence. Finally, we only keep the sentences holding the target role-content, where the
PArgs/SRs of x are annotated via the AllenNLP (Gardner et al., 2018) semantic role labeller. Table 1 lists randomly selected examples from augmented explanations. Full details and the supporting ablation study are provided in Appendices A and D.

heading: 4 Experiments
text: For the experiments, we start by focusing on the effect of the supervised INN mechanism to examine its impact on the sentence semantic separability of the distributional latent space defined in Section 2 (detailed in Section 4.1). Next, we examine the localised semantic generation control enabled by such semantic separability via latent interpolation (Section 4.2). Further details of the AutoEncoder model and dataset are provided in Appendix A.

heading: 4.1 Disentanglement Encoding Evaluation
text: We examine the latent space separability (i.e., natural clustering property (Bengio, 2013)) of our supervision approach on different predicateargument/semantic roles. In the context of this work, the thematic roles’ labels are not referred to control the generation. Instead, we use the predicate argument position markers, e.g. including ARG0, ARG1, PRED(V), where each category has a) four possible word contents (ci), or b) the same content (i.e., animal) with different argument/roles, including ARG0,1,2. We provide the reconstructed examples of INNs in Table 24.
Disentanglement between ARG0 clusters. For ARG0, we choose human, animal, plant, and something due to having the highest frequency in the original dataset, and evaluate model performance from two directions, including forward and backward mapping. Within forward mapping, we assess the disentanglement of the latent space of the INN model from two perspectives (visualisation
and classification metrics). Figure 3 displays the distributions of four role-content clusters over the latent space. As we can observe, after the clustersupervised training strategy, the embeddings are more concentrated at the center of their cluster, and there is a clear boundary between clusters, indicating a better disentanglement when compared to Optimus and unsupervised INNs.
We then quantitatively evaluate the disentanglement of ARG0-content clusters. We consider classification task metrics (accuracy, precision, recall, f1) as proxies for evaluating region separability, effectively testing cluster membership across different clusters. We choose a non-parametric downstream classifier (i.e., kNN) to quantitatively evaluate the separation of clusters and parametric downstream classifiers, including Naive Bayes (NB) and Support Vector Machine (SVM), to assess both separability and representation capability of latent sentence spaces (Rifai et al., 2011; Conneau et al., 2018). The configuration of the downstream classifiers are detailed in Appendix A.
As shown in table 2, all classifiers trained over supervised latent representations outperformed the unsupervised INN (U) and Optimus (O), indicating that the cluster-supervised approach leads to better disentanglement and representation. Moreover, (O) demonstrates superior performance compared to (U) for the KNN-based evaluation. However, it exhibits lower performance than (U) in NB and SVM. This suggests that the INN-AutoEncoder configuration can more effectively capture sentence semantics (from the point-of-view of AST+distributional content), in the context of a reconstruction task since the VAEs’ training process is prone to experiencing posterior collapse.
As for the evaluation of the backward mapping, we calculate the ratio of generated sentences that
hold the same role-content as the inputs (henceforth called the invertibility ratio). We randomly selected 100 embeddings as inputs and showed the corresponding ratios in Table 3. We can observe that both unsupervised and supervised cases can achieve high invertibility ratios, indicating that the INN mechanism provides stable invertibility with or without cluster supervision.
Disentanglement between PRED clusters. Next, we analyze the disentanglement between predicate (PRED) clusters. As shown in Figure 4, although the disentanglement of PRED clusters is not as high as ARG0, the latent space with cluster supervision still performs better than both the unsupervised case and the Optimus model. In Table 4, the supervised INN model achieves better disentanglement, and both unsupervised and supervised could obtain a higher ratio. We also provide the experimental results of ARG1 disentanglement in Appendix B.
Disentanglement between ARG0,1,2 clusters. The experiments up to this point investigated the separation between the same pred-argument type but different content clusters. Next, we explore the separability of different pred-argument types with
the same content. We thus focus on the animal cluster, and investigate the disentanglement between ARG0-animal, ARG1-animal, and ARG2-animal. As illustrated in Figure 5, the animal clusters with different pred-argument types can be separated after cluster-supervised training, which indicates that the INN model can capture the difference between the same content with different pred-argument type in the case of similar topic, indicating the INNbased approach could jointly learn separable embeddings w.r.t. role-content and content alone.
Table 5 and 15 show the disentanglement metrics and the invertibility ratio, respectively. Similarly to the previous experiment, the supervised case outperforms both the unsupervised and the Optimus models. Both INNs can achieve an invertibility ratio of at least 90%.

heading: 4.2 Disentanglement Decoding Evaluation
text: Finally, we evaluate the disentangled sentence geometry from the perspective of sentence generation. We specifically focus on linear interpolation as it can provide more efficient traversal between sentences and clusters than other traversal approaches (e.g., Ornstein-Uhlenbeck), commonly used in the NLP domain (Li et al., 2020b) and in the evaluation of disentanglement (Bengio, 2013).
Interpolation localisation. Firstly, we evaluate the localisation of latent interpolation that interpolates a path zt = z1 · (1 − t) + z2 · t with t increased from 0 to 1 by a step size of 0.1, where z1 and z2 represent the latent representations of source and target sentences. As a result, 9 sentences are generated on each interpolation step. On a latent space with better token-level role-content separation, given two sentences with the same rolecontent as endpoints, we can observe that the intermediate sentence can hold the same role-content during interpolation.
In terms of qualitative evaluation, Table 6 provides the interpolation paths of cluster-supervised INN and Optimus, as for Optimus, we can observe that the intermediate explanations could transition smoothly from source to target for argument. However, the predicate is more abruptly changed, indicating lower predicate-content disentanglement (e.g., predicate-require and predicate-eat). Instead, the supervised INN can fix the predicate-require during interpolation, indicating better separability between different predicate-content results in better generation control. More examples are provided in Table 22 and 23. We then quantitatively evaluate the localisation of interpolation. We randomly select 200 sentence pairs from the dataset holding the same role-content and report the ratio of intermediate sentences with the same role-content as inputs. As illustrated in Figure 6, the intermediate sentences from the supervised INN can better hold the same role-content as inputs, especially for
predicate which usually has a lower effect on distributional sentence semantics (Zhang et al., 2022), indicating that our supervision can lead to better latent space separability and generation control.
Interpolation smoothness. Moreover, we quantitatively evaluate the latent space geometry via interpolation smoothness metrics (IS, Zhang et al. (2024)), which calculates the ratio between the
ideal semantic distance (i.e., the aligned semantic distance between source and target sentences) and the actual semantic distance (i.e., the sum of semantic distance between adjacent sentences during interpolation). A higher ratio indicates that the actual path aligns better with the ideal path, suggesting better semantic-geometric properties. The metric is defined as:
IS = E(s0,...,sT )∼P δ(align(s0, sT ))∑T
t=0 δ(align(st, st+0.1))
where s0, ..., sT is the sequence of sentences during interpolation, δ and align are sentence similarity and alignment functions, respectively, which are performed via Word Mover’s Distance (Zhao et al., 2019). We choose the standard language VAE baselines (i.e., the prior is the std. Gaussian distribution). Their implementation details are provided in Appendix A. We randomly sample 200 sentence pairs and report the IS metric. As illustrated in Table 7, our model can deliver smoother interpolations comparatively to the baselines, indicating semantic disentanglement can lead to better latent space geometry.

heading: 5 Related Work
text: Sentence representation. Sentence representations are usually trained in supervised (Conneau et al., 2017; Reimers and Gurevych, 2019), constrastive (Giorgi et al., 2021; Yan et al., 2021; Chuang et al., 2022), or generation-oriented (Wang et al., 2021; Wu and Zhao, 2022; Chuang et al., 2022) fashion. Recent work (Huang et al., 2023) explored the compositional sentence representation for improved explainability and generation. However, these works still lack the emphasis on the geometric interpretation and control of the underlying sentence space, which this work focused on.
Sentence disentanglement. In the Natural Language Generation domain, most previous investigations explored the disentanglement between two specific linguistic perspectives, such as sentimentcontent (John et al., 2019; Li et al., 2022), semanticsyntax (Bao et al., 2019), and negation-uncertainty (Vasilakes et al., 2022), or syntactic disentanglement (Mercatali and Freitas, 2021; Felhi et al., 2022). In this work, we provide a formalgeometrical lens, with the support of argument structures as a sentence representation model, for sentence disentanglement targeting for localised semantic control. This work is the first integration of flow-based INN mechanisms to improve disentanglement, separation and semantic control of sentence spaces.
INNs in NLP. Şahin and Gurevych (2020) concentrate on modelling morphological inflection and lemmatization tasks, utilizing INN to learn a bijective transformation between the word surface and its morphemes. Li et al. (2020a) proposed BERT-flow, transforming sentences from a BERT sentence space to a standard Gaussian space. Ding and Gimpel (2021) deployed flow-based INN to enrich VAE prior distribution, while Gu et al. (2023) use flow mechanisms to control attributes in style transfer tasks. This work focused on semantic separability, geometrical operations and control over the distributed representation of sentences. Moreover, this work is the first to explore geometrical data augmentation to support semantic disentanglement.

heading: 6 Conclusions and Future Work
text: This work focused on an INN-based mechanism to support better disentangled and separated latent sentence spaces over language autoencoders. By aligning the predicate-argument structure of sentences to the latent representations, we aimed to build a bridge between the formal and distributional semantics perspectives for sentence representation. We define the sentence semantic disentanglement from the perspective of formal semantics, aligning the predicate-argument structure to disentanglement and cluster separation properties, and exploiting the invertibility and bijection properties of INNs to facilitate such alignment. Experimental results indicate that the invertibility mechanisms can transform the distributed hidden space of an autoencoder into a latent space where AST-level syntactic and semantic transformations can be localised, interpolated and controlled. Secondly, we propose a
supervision approach, which leads to an improved disentangled and separated space. This property can facilitate localised interpolation control. Lastly, we utilise these geometric properties to support a semantically controlled data augmentation to assist the disentanglement process.
Since our work connects distributional and formal semantics via disentanglement, one future direction is to explore the safety and control of the formal semantic properties of Large Language Models. Besides, recent work (Liu et al., 2023) revealed that distinct factors can be composed by modelling the moving of latent vectors via ordinary differential equations, which can be adapted to sentence representations to deliver more complex sentence transformations within the latent space.

heading: 7 Limitations
text: This work focused on the disentangled sentence representations geometry to deliver localised/semantic/formal semantic control. While this work is motivated by providing more localised distributed representations, which can positively impact the safety and coherence of generative models, few scoping observations need to be established: 1. The specific safety guarantees of these models are not fully established. 2. While the language autoencoder with unsupervised INN exhibit a distinct learning pattern with regard to semantic distribution, further understanding is required in terms of information bottleneck properties (Saxe et al., 2018) and on the semantic distribution of unsupervised INNs in language modelling tasks. 3. Furthermore, this study exclusively focused on a corpus of sentences which are conceptually dense ((Dalvi et al., 2021)). The exploration of its performance on other types of sentences, including sentences with complex clausal-phrasal constructions, or sentences with non-compositional idioms, is yet to be undertaken.

heading: Acknowledgements
text: This work was partially funded by the EPSRC grant EP/T026995/1 entitled “EnnCore: End-toEnd Conceptual Guarding of Neural Architectures” under Security for all in an AI enabled society, by the CRUK National Biomarker Centre, and supported by the Manchester Experimental Cancer Medicine Centre and the NIHR Manchester Biomedical Research Centre.

heading: A Experiment setting
text: Dataset. Table 8 displays the statistical information of the datasets used in the experiment. The data of the two datasets partially overlap, so only the unique explanations are selected as the experimental data. Table 9 illustrates the semantic,
structure, and topic information of explanatory sentences over the latent space.
The rationale for choosing explanatory sentences is that they are designed for formal/localised/symbolic semantic inference task in natural language form, which provides a semantically complex and yet controlled experimental setting, containing a both well-scoped and diverse set of target concepts and sentence structures, providing a semantically challenging yet sufficiently well-scoped scenario to evaluate the syntactic and semantic organisation of the space. More details about semantic structure and lexical information are provided in Table 9 and 10.
Data Augmentation. Algorithm 1 illustrates the detailed process of data augmentation. The key aspect of data augmentation is to keep the data distribution unchanged while increasing the size of the dataset. Therefore, during traversal, we only sample the value whose probability density is between 0.495 and 0.505. In other words, for each original explanation, we only traverse its close neighbours over the latent space. We increased the number of explanations in each role-content cluster to 3000 and kept the balance of each role-content category. We provide more qualitative examples in Table 11. Moreover, we visualise latent semantic distribution before and after augmentation in Figure 7. As we can observe, the data augmentation can maintain the semantic distribution unchanged. For example, PRED-is (red colour in the right column) is widely distributed over the latent space before and after augmentation. ARG0-something (purple colour in the left column) is far from other clusters with or without data augmentation in latent space.
Downstream Classifier. In this experiment, we apply three downstream classifiers, including nonparametric classifier: k-nearest neighbours (KNN) and parametric classifiers: Naive Bayes (NB) and
Algorithm 1 Data Augmentation Define: R as the role set (ARG0, PRED, ...). Define: C as the content set (vocabulary). Define: S as the explanation corpus (sentences). Define: s = [(c1, r1), ..., (ci, ri)] ∈ S, ci ∈ C, ri ∈ R as a sentence. Define: (ct, rt) | rt ∈ R, ct ∈ C as the target role-content (e.g., ARG1-animal). Define: St = ∀s ∈ S | ∃(ck, rk) = (ct, rt) as the set of sentences with the target rolecontent. Define: E(s) : S → Rn as encoder (embedding) function. Define: D(vec) : Rn → S as the explanation decoded from Decoder D. Define: L: list for keeping augmented sentences. Define: SRLer(s): semantic role label annotator for s. for all (si, sj) ∈ St, si ̸= sj do
vec = average(E(si), E(sj)) for all vec[i] ∈ vec do
vec[i] = N(0, 1) # neighbour traversal sn = D(vec) # new sentence if sn /∈ L AND R ∈ SRLer(sn) then
put sn in L. end if
end for end for
Support Vector Machine (SVM), to evaluate the separability of latent representation. Those classifiers and classification metrics are implemented based on scikit-learn package (Pedregosa et al., 2011) with default hyper-parameters. We train those classifiers on the training set (×60%) and evaluate them on the test set (×40%). For multiclass classification, we set macro for precision, re-
call, and f1 since macro-averaged metric for each class is calculated independently, and then the average is taken, which ensures that the performance of the model in each class contributes equally to the final metric, regardless of the class size.
Visualizer. In this experiment, we implement tSNE and PCA visualisation based on Yellowbrick library (Bengfort and Bilbro, 2019)2. We empirically set decompose_by = 4 for all cases. However, we found no significant difference between different decompose_by parameters.
Baselines for Interpolation Smoothness. In the experiment, we implement five LSTM-based autoencoders, including denoising AE (Vincent et al. (2008), DAE), β-VAE (Higgins et al., 2016), adversarial AE (Makhzani et al. (2015), AAE), label adversarial AE (Rubenstein et al. (2018), LAAE), and denoising adversarial autoencoder (Shen et al. (2020), DAAE). Their implementation relies on the open-source codebase available at the URL 3. As for transformer-based VAEs, we implement Optimus (Li et al., 2020b), AdaVAE (Tu et al., 2022)4,
2https://www.scikit-yb.org/en/latest/api/text/ tsne.html
3https://github.com/shentianxiao/ text-autoencoders
4https://github.com/ImKeTT/AdaVAE
and Della (Hu et al., 2022)5. All baseline models
5https://github.com/OpenVLG/DELLA
undergo training and evaluation with the hyperparameters provided by their respective sources. A latent dimension of 32 is specified to ensure a uniform and equitable comparative analysis.
Autoencoder. In this work, we employ an autoencoder architecture with the same configuration as described in (Li et al., 2020b)6. The encoder component is based on BERT (Devlin et al., 2018), while the decoder component is based on GPT2 (Radford et al., 2019). The latent space dimension is set to 32 (low-dimension) as Michlo et al. (2023) revealed that strong compression, such as strong KL regularisation term in ELBO, can lead to the phenomenon of disentanglement of images.
To establish the connection between the encoder and decoder, the input sentence x is first encoded by BERT[cls] into the latent space, denoted as N(µ,Σ). The parameters µ and Σ are trainable and determine the mean and covariance of the Gaussian distribution. Next, a sample z ∼ N(µ,Σ) is passed through a multi-layer perceptron called W . This step expands the dimensionality of z to obtain a
6https://github.com/ChunyuanLI/Optimus
fixed-length embedding h ∈ RD×L×H , where D represents the dimensions of the heads, L is the number of heads, and H is the number of hidden layers. The latent space injection can be described as:
Attention(Q,K, V ) = softmax( Q[z;K]T√
d )[z;V ]
INN. The INN consists of 10 invertible blocks. Each is built from three layers, including an affine coupling (Dinh et al., 2016), permutation layer, and ActNorm (Kingma and Dhariwal, 2018). Figure 8 displays one single invertible block. The model was implemented using the FrEIA library (Ardizzone et al., 2018-2022) 7. As for training hyperparameters of INN, firstly, both input and output have the same dimensions as the latent space dimension of the autoencoder. Secondly, inside the affine coupling block, the sub-network is MLP with 512 as the hidden dimension. Thirdly, we use AdamW (Loshchilov and Hutter, 2017) to optimise the model where the learning rate is 5e-04 in the experiment.
The forward process of the affine coupling layer can be described as follows:
xa, xb = split(x)
log s, t = mθ(xb)
s = exp(log s)
ya = s⊙ xa + t yb = xb
y = concat(ya, yb)
(2)
Where mθ is a two-layer neural network. x and y
7https://github.com/VLL-HD/FrEIA
are the input and output. The reversed process is:
ya, yb = split(y)
log s, t = mθ(yb)
s = exp(log s)
xa = (ya − t)/s xb = yb
y = concat(xa, xb)
(3)

heading: B Additional Supervision Results
text: Disentanglement between ARG1 clusters We consider four ARG1 clusters, including ARG1-food, ARG1-oxygen, ARG1-sun, ARG1-water, and evaluate model performance following the same procedure. Figure 9 displays the distributions of four role-content clusters over the latent space. With similar observations as before, the INN clustersupervised training strategy can learn better disentanglement between ARG1 clusters. Table 12
and 13 show the disentanglement metrics and invertibility ratio, respectively. With similar observations as the previous experiment: all classifiers trained over the supervised latent representation outperform both the unsupervised INN model and Optimus, and both unsupervised and supervised cases can achieve higher ratios (at least 0.95).
Invertibility ratio. Table 13, 14, and 15 report the invertibility test for ARG1, PRED, and ARG0,1,2 clusters, respectively. We can observe that INN with both training approaches can perform stable invertibility.
Traversal decoding for Animal clusters. Table 16 shows the decoded explanations traversed around the central point of each cluster in the latent space of cluster-supervised INN.
Traversal decoding for cluster connection. Table 17 displays the decoded middle points between
clusters. It is also observable that there are lowdensity embedding regions at the transition (connection) between two clusters. We decode the middle datapoints between animal and human clusters and list them in Table 17. From those examples, we can observe that such explanations are related to both animal and human. This result implies that the explanations may be geometrically represented in a similar way as they were originally designed in the WorldTree corpus (maximising lexical overlaps for pred-arg alignments within an explanation chain) for supporting multi-hop inference tasks.
Principal component analysis (PCA) visualisation. In addition to the non-linearised t-SNE plot, we also provide linearised visualisation via PCA (Shlens, 2014). Figure 10,11,12, and 13 visualize the separation of ARG0, ARG1, PRED, and animal. Similar to the observation before, cluster supervision can lead to better separation and cluster.
C Statistical Significance Tests for PRED Downstream Classifiers
Statistical significance testing is a standard statistical tool devised to ensure that experimental re-
sults are not coincidental and reliable. Following the work (Dror et al., 2018)8, we provide statistical significance tests to rigorous and quantitatively evaluate the stability of trainable downstream classifiers, which indirectly indicates the representation capability.
Our attention was directed towards PRED clusters due to the comparatively decreased performance of downstream classifiers within this category as PRED usually contains less semantic information (Zhang et al., 2022). We select accuracy metric, set α = 0.05, and choose bootstrap statistical test which was used with a variety of NLP tasks (Ouchi et al., 2017; Wu et al., 2017).
As illustrated in Table 18, (1) the U-C pair consistently yields a diminished significance value, suggesting reliable classification performance resulting from superior representational capabilities facilitated by the AutoEncoder with INN configuration, compared with Optimus. (2) the scores of (O-C) pairs are consistently lower than those of (OU) pairs, indicating our supervision (C) can better represent semantic information than unsupervised INN. We refer (Dror et al., 2018) for an in-depth illustration of statistical significance tests in NLP.

heading: D Ablation of Data Augmentation
text: PRED semantic role. Firstly, we analyse the effect of our supervision approach on PRED semantic role with three lexical contents without data augmentation, including are (×449), cause (×380), and require (×262). The rationale for their selection is that they are less frequent in cor-
8https://github.com/rtmdrr/ testSignificanceNLP/tree/master
pus and partially overlap in latent space. Moreover, the contents under PRED usually have less effect on the contextual semantics (Zhang et al., 2022). Those difficulties allow us to fairly analyse the effect of our supervision approach. Following a similar order, we first visualise the t-SNE and PCA plots in Figure 14. As we can observe, the cluster-supervised approach can better represent the cluster and separation for different contents under PRED semantic role label without data augmentation. Next, we apply downstream classifiers to evaluate cluster separation. As illustrated in Table 19, our cluster-supervised approach results in better classification performance, indicating better disentanglement.
ARG0 semantic role. Next, we provide the same analyse for fewer frequent ARG0 clusters: ARG0animal (×126), ARG0-human (×43), ARG0-plant
(×77), and ARG0-something (×186). As illustrated in Figure 15, cluster supervision can lead to better role-content separation/disentanglement. Moreover, we can observe that cluster-supervision leads to better proxy disentanglement metrics in Table 20.
Furthermore, compared with Table 2, the incorporation of latent representation with data augmentation results in enhanced classification performance. This observation implies that our data augmentation technique can more effectively capture semantic information, thereby aiding downstream classifiers.

heading: E Controlled Interpolation
text: In tables 22 and 23, we provide more controllable interpolation examples. Those examples reveal that the latent space with better role-content separation from supervised INN can provide better interpola-
tion control, indicating better latent space geometry.
F INNs: Explanation Reconstruction
Table 24 shows some reconstructed explanations from AutoEncoder, unsupervised INN, and supervised INN, respectively.

heading: Future Work
text: Since our work connects distributional and formal semantics via disentanglement, one future direction is to explore the safety and control of the formal semantic properties of Large Language Models. Besides, recent work (Liu et al. , 2023) revealed that distinct factors can be composed by modelling the moving of latent vectors via ordinary differential equations, which can be adapted to sentence representations to deliver more complex sentence transformations within the latent space