Disentangling Representations of Text by Masking Transformers
Authors: Xiongyi Zhang
Jan-Willem van de Meent
Byron C Wallace
Venue:  (2021)
Keywords: disentanglement
model pruning
representation learning
transformers

Abstract: Representations in large language models such as BERT encode a range of features into a single vector, which are predictive in the context of a multitude of downstream tasks. In this paper, we explore whether it is possible to learn disentangled representations  by identifying subnetworks in pre-trained models that encode distinct, complementary aspects of the representation. Concretely, we learn binary masks over transformer weights or hidden units to uncover the subset of features that correlate with a specific factor of variation. This sidesteps the need to train a disentangled model from scratch within a particular domain. We evaluate the ability of this method to disentangle representations of syntax and semantics, and sentiment from genre in the context of movie reviews. By combining this method with magnitude pruning we find that we can identify quite sparse subnetworks. Moreover, we find that this disentanglement-via-masking approach performs as well as or better than previously proposed methods based on variational autoencoders and adversarial training. 

Under review as a conference paper at ICLR 2021
DISENTANGLING REPRESENTATIONS OF TEXT BY MASKING TRANSFORMERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Representations from large pretrained models such as BERT encode a range of features in single vector that affords strong predictive accuracy on multitude of downstream tasks. In this paper we explore whether it is possible to learn dis- entangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspect representations. Concretely, we learn binary masks over transformer weights or hidden units to uncover the subset of features that correlate with specific factor of variation; this eliminates the need to train a disentangled model from scratch for a particular domain. We evaluate the ability of this method to disentangle representations of syntax and semantics, and sentiment from genre in the context of movie reviews. By combin- ing masking with magnitude pruning we find that works within BERT that strongly encode particular aspects (e.g., movie sentiment) while only weakly encoding others (movie genre). Moreover, despite only learn- ing masks, we find that
identify sparse subnet-
wewenngleeeetvviamaieeeee performs as well as and often better than previously proposed methods based on variational autoen- coders and adversarial training.
INTRODUCTION AND MOTIVATION
Large-scale pretrained models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2018). and XLNet (Yang al., 2019) have come to dominate in modern natural language processing (NLP). Such models rely on self-supervision over large datasets to learn general-purpose representations of text that achieve competitive predictive performance across a spectrum of downstream tasks (L.iu et al., 2019). A downside of such learned representations is that it is not obvious what information they are encoding, which hinders model robustness and interpretability. The opacity of representa- tions produced by models such as BERT has motivated line of NLP research designing probing tasks as a means of uncovering what properties of input texts are encoded into token- and sentence- level representations (Rogers et al., 2020; Linzen al., 2019; Tenney et al., 2019).
In this paper we investigate whether we can uncover disentangled representations from pretrained models. That rather than mapping inputs onto single vector that captures arbitrary combinations of features, our aim is to extract a representation that factorizes into distinct, complementary proper- ties of the input. An advantage of explicitly factorizing representations is that it aids interpretability, in the sense that it becomes more straightforward to determine what factors of variation inform predictions in downstream tasks. Further, disentangled representations may facilitate increased ro- bustmess under distributional shifts by capturing notion of invariance: If syntactic changes in sentence do not affect the representation of semantic features, and vice versa, then we can hope to learn models that are less sensitive to any incidental correlations between these factors. A general motivation for learning disentangled representations to try and minimize - or least expose model reliance on spurious correlations, i.c., relationships between (potentially sensitive) attributes and labels that exist in the training data but which are not casually linked (Kaushik et al., 2020). This is particularly relevant in the context of large pretrained models like BERT, do not know what the representations produced by such models encode.
To date, most research on disentangled representations has focused applications in computer vi- sion (Locatello et al., 2019b; Kulkarni et al., 2015; Chen et al., 2016; Higgins et al., 2016), where there exist comparatively clear independent factors of variation such size, position, color, and

---

Under review as a conference paper at ICLR 2021
Masked
Weght
Addoloonal
Syerr
Lower
Hodden
Figure 1:
Masking weights and hidden activations in BERT. We show a linear layer with weights W. imputs h. and outputs h' We learn a mask for each disentangled factor, which is either applied the weights W or intermediate representations h.
orientation, which have physical grounding and can be formalized in terms of actions of symmetry subgroups (Higgins et al., 2018). challenge in learning disentangled representations of text that it is more ambiguous what factors of variation should admit invariance. Still, may hope to disentangle particular properties for certain applications (e.g., sentiment, or perhaps protected demo- graphic information (Elazar & Goldberg, 2018)), and there are also general properties of language that we might hope to disentangle, c.g., syntax and semantics (Chen et al., 2019).
In this paper we ask whether complementary factors of variation might already be captured by pretrained models, and whether it is possible to uncover these by identifying appropriate "sub- networks". The intuition for this hypothesis is that generalization across sufficiently large and diverse training set may implicitly necessitate representations that admit some notion of invariance, the many factors of variation in the training data give rise combinatorial explosion of possible inputs. Intriguing prior work (Radford et al., 2017) examining the correlation between sentiment and individual nodes within pretrained networks offers some additional support for this intuition.
To test this hypothesis, we propose to use masking as a mechanism to isolate representations of individual factors. Recent work on lottery tickets (Frankle & Carbin, 2018) suggest that overpa- rameterized networks are redundant, in that a network reduced to a small subset of weights set to "winning" initial values can achieve predictive performance similar the full network. Building on this intuition, we hypothesize that it might be possible to uncover representation for factor of interest by starting with pretrained representation and simply masking out weights or hidden units that correlate with other factors of variation.
We use BERT (Devlin et al., 2018) as an archetypal pretrained transformer to test two variants of this basic idea. In the first variant we learn binary masks for all weight matrices in the model; in the second we derive masks for all hidden units (intermediate representations). To learn these masks minimize triplet loss that encourages the resultant representations for instances that similar with respect to an aspect of interest to be relatively near to one another, independent of other factors. Our approach of uncovering existing subnetworks within pretrained models that yield disentangled representations differs substantially from prior work on disentangling representations in NLP. which have either relied on adversarial debiasing approaches (Elazar Goldberg, 2018; Barrett et al., 2019) or variational auto-encoders (Chen et al., 2019; Esmacili et al., 2019).
We evaluate masking in the context of two tasks. The first is setting in which we aim disen- tangle a representation of features for a target task from that of information encoding a secondary, non-tan non-target attribute (e.g., this might be sensitive information, or simply an unrelated factor). In the second we follow prior work in attempting to induce representations of semantics and syntax, respectively. both settings, our surprising finding is that masking alone often outperforms pre- viously proposed approaches (which learn finetune networks in their entirety). While small amount of masking generally suffices to achieve disentanglement, we can further increase sparsity by combining masking with weight pruning.
The main contributions of this paper are as follows. (1) We propose a novel method of disentan- gling representations in NLP: Masking weights or hidden units within pretrained transformers (here, BERT). (2) We empirically demonstrate that we are indeed able to identify sub-networks within pretrained transformers that yield disentangled representations that outperform existing approaches
2

---

Under review as a conference paper at ICLR 2021
(which finetune all model weights). (3) We show that masking can be combined with weight-pruning techniques to learn sparse subnetworks for each factor of interest.
METHODS
We are interested in learning a disentangled representation that maps inputs I (text) onto vectors 2(4) and 206) that encode two distinct factors of variation. To do so, we will learn two sets of masks M(a) and M that can be applied to either the weights or the intermediate representations in pretrained model (in our case, BERT). We estimate only the mask parameters and do not finetune the weights of the pretrained model.
To learn the masks M(a) and M we assume access to triplets (zo, (1, in which 2o and are similar with respect to aspect but dissimilar with respect to aspect b, whereas Ty and x2 are similar with respect to aspect b but dissimilar with respect to aspect a. In some of our experiments (i.e., when disentangling the sentiment from the genre in movie reviews) we further assume that we have access to class labels y(a) E {0, and y E {0, for the aspects of interest.
2.1 MASKING WEIGHTS AND HIDDEN ACTIVATIONS
Figure illustrates the two forms of masking that consider in approach. We depict single linear layer of the model, h (hla), (()) are the input activations, W are the weights in the pretrained model,' and h' = (h*a), (b) are the output activations. We augment each layer of the original network with two masks M (M(1), (b) applied in two ways:
1. Masking Weights When training BERT with masked weights, masks M(a) and () have the same shape weights W, and output activations computed using the masked tensor of weights
h h (W M).
2. Masking Hidden Activations When training BERT with masked hidden units, the masks (a) and (D) have the same shape as the intermediate (hidden) activations (1))) and (6)), Output activa- tions are computed by applying the original weights W to masked inputs
= (h o M) W.
(2)
In both methods, we follow Zhao et al. (2020) and only mask the last several layers of BERT, leaving the bottom layers (including the embedding layer) unchanged. In practice we find masking the last 9 layers to work the best.
2.2 TRIPLET LOSS
To learn the masks, we assume that we have access to supervision in the form of triplets, as intro- duced above. Passing (oo, x2) through our model yields two representations of each instance:
(z()), =b (b)),, (z1 (b)), (a) ,22 (()), for which we define the loss
L (w)
umplet max (1=00a)
z(a01
|1 (a) -2(0) t.0).
(3)
L unplet max
(1=0%) (b)
22
(b) ||
11=0 - 2101| | +c, c,0).
(4)
Luriplet
2 imilet
+ _
(5)
Here e is hyperparameter specifying margin for the loss, which we set to 2 in all experiments.
2.3 SUPERVISED LOSS
In some settings we may have access to more direct forms of supervision. For example, when learning representations for the genre and sentiment in movie review, we have access class
We omit the hias in the linear layer, for which no masking is performed.
3

---

Under review as a conference paper at ICLR 2021
labels y(() and y for each aspect. To make full of such supervision when available, we add two linear classification layers ((o) and ((O) to our model and compute the classification losses
(g) CrossEntropy (clo) ),y(a))
((t)
CrossEntropy (2 v0))
(6)
1 ( ccs + Iach)
Luks
2
2.4 DISENTANGLEMET LOSS
To ensure that each of the two aspect representations are distinct, we encourage the masks to be mutually exclusive. That is, masks of different layers should overlap as little as possible, We include an additional term in the loss for each layer L to achieve this
4
Trovertap
1
( +M
(8)
L
1=1
2.5 BINARIZATION AND GRADIENT ESTIMATION
The final loss of our model is weighted sum of all the losses
Arriplet
Luriplet + Aoverlap
Lovertap (+>cls (+ Less)
(9)
Where we parenthetically denote the cls loss, which we will only sometimes have. We minimize this loss with respect M (and the classifier parameters), whilst keeping pretrained BERT weights fixed. Because the loss is not differentiable with respect to binary mask, we learn continuous masks M that are binarized during the forward pass by applying a threshold т, which is global hyperparameter.
1 0
if My 2T
Mr
(10)
if My <T
We then use a straight- tharoughhh estimator (Hinton et al., 2012; Bengio et al., 2013) to approximate the derivative, which is to say that evaluate the derivative of the loss with respect to the continuous mask M at the binarized values M
M
aL
M = M -1 0M M=
(11)
3 EXPERIMENTS
3.1 DISENTANGLING SENTIMENT FROM GENRE IN MOVIE REVIEWS
Experimental Setup this experiment we assume a setting in which each data point x has both 'main' label y and a secondary (possibly sensitive) attribute We are interested in evaluating the degree to which explicitly disentangling representations corresponding to these may afford robust- ness to shifts in the conditional distribution of y given z. As a convenient, illustrative dataset with which to investigate this, we use a set of movie reviews from IMDB (Maas et al., 201 1) in which each review has both a binary sentiment label and a genre label.
Here we consider just two genres: Drama and Horror. We exclude reviews corresponding to other genres, as well as the (small) of instances that belongs to both genres. To investigate robustness to shifts in correlations between z and we sampled two subsets of the training set such that in the first sentiment and genre are highly correlated, while in the second they are uncorrelated. We report the correlations between these variables in the two subsets in Table 1. We train models on the correlated subset, and then evaluate them on the uncorrelated set.
We compare the proposed masking approaches to several baselines. Untuned corresponds to a dense classification layer on top of representations from BERT (without finetuning). In the finetuned vari- ant we omit masks and instead minimize the loss with respect to BERT weights. In the adversarial model we adopt adversarial 'debiasing": addition to minimizing loss on the main task, here we also train adversarial classifier to predict the secondary (non-targeted) attribute, and the encoder trained to mitigate the adversaries ability to do so. We implement this via the gradient-reversal method described by Ganin & Lempitsky (2015). We also compare with two variational autoencoder baselines, VAE is a VAE model with multi-task loss and adversarial loss, introduced in John et al. (2019), and VAE+BER' is the same model but using BERT the embedding module.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Conditional prevalences of the respective classes in the correlated (left) and uncorrelated (right) datasets. We control for the total number of positive and negative samples to be equal in both datasets, both datasets are balanced with regard sentiment and with regard to genre. We use the correlated dataset for training, but evaluate on the uncorrelated dataset, to ensure that the prediction of the model reflects the aspect of interest, rather than correlations between aspects.
Training: Correlated
Test: Uncorrelated
Sentiment / Genre
Drama
Horror
Sentiment / Genre
Drama
Horror
Positive
0.85
0.15
Positive
0,5
0.5
Negative
0.15
0.85
Negative
0.5
Unituned
Finattnee
Adversar
We ghit
Mankeed Hiddew
0.9
0.
Aug. (Sent)
Wonst (Bent)
Avg. (Genre)
Wonst (Genre)
Figure 2: Average and worst main task performance across sentiment/genre combinations. Masked variants (proposed this paper) are cross-hatched. Larger gaps between the average performance and the worst group performance indicate that the corresponding model is using the non-targeted attribute when making predictions for the main task.
Leakage the non-targeted attribute. We evaluate the degree to which representations "leak" non-targeted information. Following Elazar & Goldberg (2018), first train the model to predict the main task label on the correlated dataset. Then we fix the encoder and train an attacker (a single layer MIP) the uncorrelated dataset to probe the learned representations for the non-targeted attribute. Because the attacker is both trained and tested on the uncorrelated dataset, it cannot simply learn the main task and exploit the correlation. We show the performance of our proposed masking models and baselines in Table 2. Masking variants perform comparably to baselines with respect to predicting the main task label, but do with notably less leakage than these methods.
Performance on worst groups. addition leakage of the non-targeted attribute, we are inter- ested in how models perform on the main tasks for each subgroup: (Positive, Drama), (Positive, Horror), (Negative, Drama), and (Negative, Horror). Because the distribution of the four groups is not uniform in the training set, expect that models will perform better on combinations that are over-represented in this set, and worse on under-represented attribute combinations. This decline would indicate that model is implicitly exploiting the correlation between these attributes.
Table 2: Performance on sentiment and genre prediction. We report accuracy on the main task (t higher is better) as well as leakage to the non-targeted task (1 lower is better).
Factor: Sentiment
Factor: Genre
Main task
Leakage
Main task t
Leakage 4
VAE
62.1
59.0
65.6
61.13
VAE BERT
67.5
66.3
71.4
70.3
Untuned
82.3
81.5
81.5
82.3
Finetuned
87.5
85.5
87.3
86.0
Adversarial
86.8
80.3
85.0
75.5
Masked Weights
88.0
72.0
87.0
Masked Hidden
88.0
79.0
85.0
79.0
5

---

Under review as a conference paper at ICLR 2021
Masked Weights
Masked Hidden
Adversarial
Finetuned
Senüment
Figure 3: t-SNE projection of sentiment representations and genre representations of different mod- els. Marker colors denote sentiment (blue for positive and yellow for negative); marker shapes denote genre (x for drama and for horror).
Rlspossetation Type:
Representatonn Type: Syrtes
Freduned
Mecbat Horton
+10
-20
TopDon
TopCon
Figure 4: Differences between the performances achieved via BERT embeddings and the disen- tangled model variants considered on semantics-oriented (WC, STS) and syntax-oriented (Depth, TopCon) tasks compared with BERT embeddings. We plot this difference with respect the mantics and syntax embeddings induced by the models the left and right subplots, respectively.
We report both the average performance on the four groups, and the worst performance observed, which is a proxy for lower bound for model when applied to dataset where group composition differs from the training dataset. Figure plots the results. Again we observe that the masking variants realize similar average performance the baselines, but consistently outperform these other models in terms of worst performance. This indicates that the proposed variants rely less on the correlation between the two attributes when predicting the main label, would hope.
Qualitative Evaluation. In Figure 3 we plot t-SNE visualizations (Maaten & Hinton, 2008) of the representations induced by different models. the representations are indeed disentangled as desired, instances with different sentiment will be well separated, while those belonging to different genres within each sentiment will not separated. Similarly, for genre representations, we hope to that instances of the same genre co-locate, but that there is no clustering of examples that reflects sentiment. While no method perfectly realizes these criteria, we observe that the masking approaches have proposed achieve better results here than do the baselines. In both Adversar- ial (Sentiment) and Finetune (Sentiment), instances that have negative sentiment but different genres still separated, indicating that these sentiment representations still carry genre information.
3.2 DISENTANGLING SEMANTICS FROM SYNTAX
Experimental Setup. For the second experiment, we follow prior work in attempting to disentan- gle semantic from syntactic information encoded in learned (BERT) representations of text. Because we have proposed exploiting triplet-loss, we first construct triplets (Zo, 21, X22) such that Zo and ø1 similar semantically but differ in syntax, while 2g and Z2 syntactically similar but encode dif- ferent semantic information. We follow the methods described in prior work on this task (Chen et al., 2019; Ravfogel al., 2019) derive triplets. Specifically, obtain 2o, 21 from the by ParaNMT-

---

Under review as a conference paper at ICLR 2021
50M (Wieting& & Gimpel, 2018) dataset. Here 21 is obtained by applying back-translation to 20, i..,, by translating Zo from English to a different language and then back into English. To derive Dy we keep all function words in 2o, and replace all content words by masking each in turn, running the resultant input forward through BERT, and selecting one of the top predictions (that differs from the original word) as replacement tokens.
We compare our pennntanglemeet-vi--aaase art approaches disentangling syntax and semantics. In particular, compare against VGVAE (Chen et al., 2019), trained on top of BERT-base to ensure fair comparison. Following prior work that has used triplet loss for disentanglement, we also compare against model in which we finetune BERT using the same triplet loss that we use to train our model, but in which we update all model parameters (as opposed to only estimating mask parameters).
strategies against models that represent state-of-the
To evaluate learned representations with respect to the semantic and syntactic information that they encode, we evaluate them on four tasks. Two of these depend predominantly on semantic informa- tion, while the other two depend more heavily on syntax. For the semantics tasks we use: (i) A word content (WC) (Conneau et al., 2018) task in which we probe sentence representations to assess whether the corresponding sentence contains particular word; and (ii) A semantic textual simi- larity (STS) benchmark (Rosenthal et al., 2017), which includes human provided similarity scores between pairs of sentences. The former we evaluate in terms of accuracy; for the latter (a ranking task) we use Spearman correlation. To evaluate whether representations encode syntax, we use: (i) A task in which the aim is predict the longest path in a sentence's 's parse tree from its embedding (Depth) (Conneau et al., 2018); and (ii) A task in which we probe sentence representations for the type of their top constituents immediately below the S node (TopConst)."
In Figure 4 we report the signed difference between the performance achieved by BERT embeddings (we mean-pool over token embeddings) and the two representation types induced by the respective (disentangled) models considered. Idcally, the semantics sentence embeddings would do well on the semantic tasks (WC and STS) and poorly on the syntax-oriented tasks (Depth and TopCon); likewise, syntax embeddings should do well on Depth and TopCon, but poorly on WC and STS. We observe that the proposed masking strategy achieves performance at least equivalent to - and sometimes (as in the case of the syntax embeddings), superior to alternative approaches. We em- phasize that this is achieved only via masking, and without modifying the underlying model weights.
3.3 INCREASING THE SPARSITY OF THE SUB- -UT-NKTORS
Here we set out to determine if we are able to identify sparse subnetworks by combining the pro- posed masking approaches with magnitude praning (Han al., 2015a). Specifically, we use the loss function defined in Equation 9 to finetune BERT for k iterations, and prune weights associated with the m smallest magnitudes after training. We then initialize masks to the sparse sub-networks iden- tified in this way, and continue refining these masks via the training procedure proposed above. We compare the resultant sparse network to networks similarly pruned (but not masked). Specifically, for the latter we consider: Standard magnitude tuning applied to BERT, without additional tuning (Pruned Untuned), and a method in which after magnitude pruning we resumee finetuning of the subnetwork until convergence, using the aforementioned loss function (Pruned Finetuned).
We compare the performance achieved on the semantic and syntactic tasks by the subnetworks iden- tified using the above strategies varying levels of sparsity, namely after pruning: {0. 20%, 40%, 60%, 80%, 85%, 90%, 95%} of weights. We report results in Figure 5. We observe that proposed masking strategy (in concert with magnitude pruning) consistently yields representations of seman- ties (top row) that perform comparatively strongly on the semantics-oriented tasks (STS, WC), even very high levels of sparsity; these semantics representations also perform comparatively poorly on the syntax-oriented tasks (Depth, TopCon), as we might hope. Similarly, syntax representations (bottom) perform poorly on the semantics-oriented tasks (and seem to leak less of this information than other methods), and outperform alternatives on the syntax-orienter tasks.
"ee acknowledge that this is very simplified view of 'semanties' and 'syntax'. "See Conneau et al. (2018) for more details regarding WC, Depth and TopConst. 'Technically, in the Pruned + Masked Weights method, refining the masks may change subnetwork sparsity, but empirically find this to change the sparsity only slightly (>~1 % in all of our experiments).
7

---

Under review as a conference paper at ICLR 2021
WC
Depth 1
TopCan1
0.5
5.5
0.5
0.5
Representation: Sernantic
022
0.
0.2
0.3
Prune Unttuned
0.00
0.55
0.50
0.75
0.00
0.23
0.33
1.00
0.00
0.25
0.50
0.00
0.23
Prune Finetuned
Sparsty
Prune Mieieed Weights
Ungraned Untured
TopCan
0.6
0.6
0.5
Representatton:: Syntax
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
2.00
0.23
0.55
0.75
1.00
0.00
0.25
2.50
0.75
1.00
0.00
0.22
0.50
0.75
1.00
Figure 5: Model performance as a function of the level of pruning. The æ-axis corresponds to the subnetwork sparsities (percent of weights dropped), while the y axes are performance measures accuracy for all tasks except for STS, where we report Pearson's 's correlation. We compare the performance of models trained on the semantic (top) and syntax representations (bottom) learned by the disentangling strategies considered, after pruning to varying levels of sparsity.
RELATED WORK
Disentangled and structured representations of images. The term Disentangled representations has been used refer a range of methods with differing aims. Much of the initial focus in this space was on learning representations of images, in which certain dimensions correspond to interpretable factors of variation (Kulkarni et al., 2015: Higgins al., 2016; Chen al., 2016). In the context of variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014) this motivated work that evaluates to what extent such representations can recover set of ground-truth factors of variation when learned without supervision (Eastwood & Williams, 2018; Kim & Mnih, 2018; Chen et al., 2018), Other work has investigated representations with the explicit motivation of fairness (Locatello et al., 2019a; Creager al., 2019), which disentanglement may help to facilitate.
Disentangling representations in NLP. Compared vision, there has been relatively little work on methods for learning disentangled representations of for natural language data. Much of the prior work on disentanglement for NLP that does exist has focused on using such factored repre- sentations to facilitate controlled generation, e.g., manipulating sentiment (Larsson et al., 2017). A related notion is that of style transfer, for example, separating style from content in language mod- els Shen et al. (2017); Mir et al. (2019). There has also been prior work on learning representations of particular aspects to facilitate domain adaptation (Zhang et al., 2017), and aspect-specific infor- mation retrieval (Jain et al., 20. Fismaili et al. (2019) focused on disentangling user and item representations for product reviews. Moradshahi et (2019) combines BERT with Representations to improve its transferability across different tasks. Recent work which we build has proposed learning distinct vectors coding for semantic and syntactic properties of text (Chen et al., 2019; Ravfogel et al., 2019). These serve baseline models in experiments.
Finally, while not explicitly framed in terms of disentanglement, efforts to 'de-bias' representations of text are related to our aims. Some of this work has adopted adversarial training to attempt to remove sensitive information (Flazar & Goldberg, 2018; Barrett et al., 2019).
Network pruning. A final thread of relevant work concerns selective pruning of neural networks. This has often been done in the interest of model compression Han al. (2015a;b). Receiiintriguinn work has considered pruning from different perspective: Identifying small subnetworks - winning 'lottery tickets' (Frankle Carbin, 2018) that, trained in isolation with the right initialization, can match the performance of the original networks from which they were extracted. Very recent work has demonstrated that winning tickets exist within BERT (Chen et al., 2020).
DISCUSSION
We have presented a novel perspective on learning disentangled representations for natural language processing in which attempt uncover existing subnetworks within pretrained transformers
8

---

Under review as a conference paper at ICLR 2021
(e.g., BERT) that yield disentangled representations of text. We operationalized this intuition via a masking approach, in which we estimate only binary masks over weights or hidden states within BERT, leaving all other parameters unchanged. We demonstrated that somewhat surprisingly we are able to achieve a level of disentanglement that often exceeds existing approaches (e.g.. a varational auto-encoder on top of BERT), which have the benefit of finetuning all model parameters.
Our experiments demonstrate the potential benefits of this approach. In Section 3.1 we showed that disentanglement via masking can yield representations that are comparatively robust to shifts in correlations between (potentially sensitive) attributes and target labels. Aside from increasing robustness, finding sparse subnetworks that induce disentangled representations constitutes new direction to pursue in service of providing at least one type of model interpretability for NLP. Finally, we note that sparse masking (which does not mutate the underlying transformer parameters) may offer efficiency advantages over alternative approaches.
REFERENCES
Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, and Anders Søgaard. Adver- sarial removal of demographic attributes revisited. In Proceedings the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 6331-6336, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: / 308.3432, 2013.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. A multi-task approach for dis- entangling syntax and semanties in sentence representations. arXiv preprint 19Xv::904..01733 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating Sources of Disen- tanglement in Variational Autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2610-2620. Curran Associates, Inc., 2018
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint XXii:2007.112233 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loie Barrault, and Marco Baroni. What you can cram into single $&!#* vector: Probing sentence embeddings for linguistic proper- tics. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- fies (Volume 1: Long Papers). pp. 1126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1198. URL www. aclweb. orrg/
anthology/ /818 1198.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. arXiv preprint arXiv:1906. 190,00990000288, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv: /80.00000 2018.
Cian Eastwood and Christopher K. Williams. A Framework for the Quantitative Evaluation of Disentangled Representations. February 2018.
Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. arXiv preprint arXiv: 060,00660, 2018.
Babak Esmaeili, Hongyi Huang, Byron Wallace, and Jan Willem van Meent. Structured Neural Topic Models for Reviews. In Artificial Intelligence and Statistics, pp. 3439-3439, April 2019.

---

Under review as a conference paper at ICLR 2021
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015,
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:1510.00149, 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances neural information processing systems, pp. 1135-1143, 2015b.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards Definition of Disentangled Representations. arXiv:1812.022330 [cs, stat], December 2018.
Geoffrey Hinton, Nitsh Srivastava, and Kevin Swersky. Neural networks for machine learning. Coursera, video lectures, 264(1), 1), 2012.
Sarthak Jain, Edward Banner, Jan-Willem van de Meent, lain J Marshall, and Byron C Wallace. Learning disentangled representations of texts with application to biomedical abstracts. arXiv preprint arXiv: 1et.00218..7.2... 2018
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 424 434, 2019.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a differ- ence with saanterffacualll--agmeeeett data. In International Conference on Learning Represen- tations, 2020. URL https: oppnreviewwwweeeeeeeeeeeee net/ forum? wrrrumiddsskkssNFFPS.
Hyunjik Kim and Andriy Mnih. Disentangling by Factorising. International Conference on Machine Learming, pp. (649-2658,, July 2018
Diederik P Kingma and Max Welling. arXiv:1312.6114, 14,, 2013.
Auto-encoding variational bayes.
arXiv preprint
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
Maria Larsson, Amanda Nilsson, and Mikael Kågebäck. Disentangled representations for manipu- lation of sentiment in text. arXiv preprint 1rXiv:1712.10000 2017.
Tal Linzen, Grzegorz Chrupala, Yonatan Belinkov, and Dicuwke Hupkes. Proceedings of the 2019 acl workshop blackboxulp: Analyzing and interpreting neural networks for nlp. In Proceedings of the 2019 ACL Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NLP, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In Advances in Neural Informa- fion Processing Systems, pp. 1461 1-6424, 2019 9a.
10

---

Under review as a conference paper at ICLR 2021
Francesco Locatello, Stefan Bauer, Mario Lucic, Gumnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference machine learning, pp- 4114-4124, 2019b.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNov):2579-2605, 2008.
Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. Evaluating style transfer for text. arXiv preprint 1rXiv:1904.020 2019.
Mchrad Moradshahi, Hamid Palangi, Monica S Lam, Paul Smolensky, and Jianfeng Gao. Hubert untangles bert to improve transfer across nip tasks. arXiv preprint arXiv:1910. 12647, 2019.
Matthew H Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:/802.0555 2018.
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv: 2017.
Shauli Ravfogel, Yanai Elazar, Jacob Goldberger, and Yoav Goldberg. Unsupervised distillation of syntactic information from contextualized word representations. 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Eric P. Xing and Tony Jebara (eds.), Pro- ceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1278-1286, Bejing, China, June 2014. PMLR.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in BERTology: What we know about how bert works. arXiv preprint arXiv:2002.12332, 2020.
Sara Rosenthal, Noura Farra, and Preslav Nakov. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pp- 502-518, 2017.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pp. 6830-6841, 2017.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv: 1199905..5500 2019.
John Wieting and Kevin Gimpel. ParaNMT-5OM:: Pushing the limits of paraphrastic sentence em- beddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 451-462 Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/PP (-188111 1042. URL https: / / www. aawwweebb..... org
1042.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753-5763, 2019.
Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. Aspect-augmented adversarial networks for domain adaptation. Transactions of the Association for Computational Linguisties, 5:515-528, 2017.
Mengjie Zhao, Tao Lin, Martin Jaggi, and Hinrich Schütze. Masking efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004. 1..00004224400 12406, 2020.
11


Reviews:
[{"review_id": "r-ps45B8POr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a problem of disentangling representations generated in pretraining models, such as BERT. That is, it is possible to learn disentangled representations that encode distinct, complementary aspect representations. To this end, the authors proposes a method that employs the mask technique on transformer weights or hidden units to find the subset of features correlating with a specific task. The experimental results show that the proposed method can encode particular aspects while weakly encoding others. The main contributions of the paper is the introduction of binary masks to identifying some subnetworks, which may correlate with specific tasks, within pretrained models. Overall, the paper is well written and is easy to follow. \n\nConcerns:\n1. The experimental setup is not convincing. The authors just consider movie reviews corresponding to Drama and Horror from IMDB and exclude reviews corresponding to other genres. It is obvious that considering only two genres is not convincing and more genres should be considered in the experiments. So, the authors should answer the following questions: (1) Why do the authors just selected these two specific genres to conduct the experiments? (2) Do the authors conduct similar experiments on other genres and what about the experimental results?\n2. Figure 3 does not show that proposed method achieves better results than do the two baselines. In fact, the finetuned baseline performs very well according to Figure 3. I suggest that the author adopts some quantitative measures to accurately reflect the differences.\n3. In addition, how to use these disentangling representations in downstream tasks, such as text classification, natural language inference, and semantic similarity? It is better to discuss and conduct experiment to show the advantages of their disentangling representations in downstream tasks.\n\nMinor comments:\n1. In Formula (9), the parentheses are redundant.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iAbUhllI7EC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a way to learn disentangled representations with respect to target attributes of interest by learning to mask weights or activations. A particular piece of text is encoded into distinct vectors that capture different factors of variation in the data. The method involves learning masks for each factor of variation while keeping the pre-trained model parameters fixed. The masks for every layer are trained using a combination of a triplet-loss, attribute classification loss, and one that encourages masks for different factors to be different across all layers. The triplet loss forces representations of examples that are similar with respect to a particular attribute to be closer than one that are similar based on another attribute.\n\nModels are evaluated on a sentiment/genre classification on a dataset sampled in such a way that introduces spurious correlations between genre and sentiment but evaluated on data that does not have any such correlation. The approach is also evaluated on disentangling syntax and semantics.\n\nStrengths\n\nBuilding models that are robust to spurious correlations in data is important for a variety of reasons and learning disentangled representations is a promising way to achieve that. This paper shows good generalization performance on datasets with such characteristics.\n\nThe overall approach is simple and only requires training masks over weights/activations at each layer. The masks are trained with a fairly straightforward choice of training objectives.\n\nThe paper is well written and the overall approach is easy to understand.\n\nWeaknesses\n\nThe triplet loss as formulated in this work seems to make it possible to disentangle only two factors of variation (a) and (b).\n\nThere is still a fair amount of attribute leakage and the probe designed to measure this leak is only a single layer MLP, there might be more leakage with stronger probes.\n\nThe weight masking strategy significantly increases the number of parameters (although the masks are binary, so it just requires a single bit as opposed to 16/32 bit floating point numbers). In this particular work, the number of parameters triples, and it scales linearly with the number of attributes as well.\n\nIt requires running the model forward multiple times to get representations that encode different factors of variation.\n\n\nQuestions & Comments\n\nWhat would performance look like if masks were trained after fine-tuning on sentiment/genre classification? Rather than training masks directly on top of BERT-base. It would be interesting to see if the model is stable to recover from fine-tuning on data with spurious correlations and still produce disentangled representations.\n\nIs every single weight/activation masked at every transformer layer? The paper seems to lack some specifics about exactly what layers/weights are masked. Along these lines, did you experiment with masking only the last few layers? This could save time & parameters\n\nIn Figure 3 is the model training with L_{cls} corresponding to sentiment and then visualized for sentiment and genre? Or is the top trained with the supervised sentiment loss and the bottom for supervised genre loss?\n\nIt would be interesting to explore an L1 penalty on the masks for increasing sparsity, possibly in conjunction with magnitude pruning as well.\n\nThe WC task doesn't feel very representative of sentence \"semantics\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "dWdl7GTuvUv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**:\n\nThe paper proposes a procedure to extract disentangled representations from pretrained BERT models. In particular, the paper proposes learning binary masks over BERT weights (or, as an alternative, over BERT activations) such that the resulting representations correspond to the desired aspect representations. The model requires additional supervision (binary labels or example triplets) and training (for the masks but not the BERT weights). The experiments aim to perform disentangling to ensure that (1) the learned representation does not “leak” a potentially sensitive attribute, and (2) the downstream classifier’s performance is good across all subgroups formed by the attributes. The experiments show that the proposed method outperforms baselines such as unmasked BERT, unmasked-but-finetuned BERT, and unmasked-but-adversarially-finetuned BERT.\n\n**Concerns**:\n\nThe fact that one can uncover disentangled representations from BERT models by masking weights/activations is a nice result and I'm not aware of similar approaches for BERT. However, it's unclear from the paper that this approach outperforms previous alternatives:\n* First, the abstract mentions that the approach is the same or better than variational auto-encoder approaches, and I don't see it mentioned elsewhere in the main text. Am I missing something?\n* Second, the paper does not show improved results on any benchmarks.\nAs a result, I'm not sure whether the paper will be impactful enough for the community.\n\n**Other Questions**:\n\n* The proposed approach involves training binary masks rather than fine-tuning the BERT weights. Given that the mask has the same shape as the weights, it's unclear whether this is a major speedup. Could you discuss this more?\n* Does using the pretrained model (vs. one trained from scratch) help?\n* Have you considered masking a subset of the weights/activations (e.g. only in the last layer)?\n* Do you have any intuition about the learned masks? E.g. are most weights/activations being removed? How much overlap is there between the masks learned for each attribute?\n\nOverall, I like this research direction, but I think it requires more work to be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0DZtJZeDaDY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a masking strategy to identify subnetworks within language models responsible for predicting different text features. This approach requires no fine-tuning of model parameters and still achieves better results compared to previous approaches. Their experimental results on the movie domain show some level of disentanglement is achieved between sentiment and genre. Disentanglement capabilities of their model between sentence semantics and structure, are also tested on four tasks. \n\nPros:\n- Paper is well-written and the idea is explained well.\n- Experiment results are convincing and support the claims.\n- Achieving comparable results to SOTA without the need to train or finetune models is interesting especially from a computational point of view.\n\nCons:\n- I wish the authors performed their first experiment on more domains: books, music, etc. and consider more than two labels.\nFrom current results, it's hard to confidently conclude that this approach is generalizable.\n- Judging based on Figure 4 results I'm not convinced that the proposed approach does better than the *finetuned* (which I believe has a trained classifier on top of BERT) approach especially for Semantic tasks. Perhaps a discussion/ error analysis would be appropriate given better results on Syntax tasks.\n- Also a discussion on the results for masking weights vs. masking hidden units is missing. If I'm not mistaken, mathematically, hidden unit masking is a subset of weight masking, where masking an item in hidden activation is equivalent to masking an entire column in the weight matrix?\n\n\nComments:\n- Although the idea of masking model parameters to achieve untanglment is new, there has been [previous work](https://www.aclweb.org/anthology/P18-1069.pdf) on using dropout to identify sub-parts of the network that contribute more/ less to model predictions framed as a confidence modeling task. Authors may consider adding it to related work.\n- Another missed citation under related work is [HUBERT](https://arxiv.org/pdf/1910.12647.pdf) which examines untanglement of semantics and structure across a wide range of NLP tasks.\n\n\nMinor typos:\n- \"we *measure* evaluate them on four tasks ...\" on page 7 \n- \"Technically, in the Pruned + Masked Weights method, *the* refining the masks ...\" ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores a methodology for learning disentangled representations using a triplet loss to find subnetworks within a transformer.  The authors compare against several other methods and find that their method performs well without needing to train from scratch. The reviewers thought this paper was well written and the authors were very responsive during the review period.  However, there were some questions about the experimental setup and empirical performance of the paper, leaving the reviewers wondering if the performance was convincing.  We agree that there is value in exploring disentangled representations even if they do not necessarily improve performance (as the authors point out), but clearly explaining the reasoning behind all analyses (e.g. specifically choosing domains to introduce a spurious correlation), and justifying differences in performance is particularly important in these cases.
Decision: Reject