Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**:

The paper proposes a procedure to extract disentangled representations from pretrained BERT models. In particular, the paper proposes learning binary masks over BERT weights (or, as an alternative, over BERT activations) such that the resulting representations correspond to the desired aspect representations. The model requires additional supervision (binary labels or example triplets) and training (for the masks but not the BERT weights). The experiments aim to perform disentangling to ensure that (1) the learned representation does not “leak” a potentially sensitive attribute, and (2) the downstream classifier’s performance is good across all subgroups formed by the attributes. The experiments show that the proposed method outperforms baselines such as unmasked BERT, unmasked-but-finetuned BERT, and unmasked-but-adversarially-finetuned BERT.

**Concerns**:

The fact that one can uncover disentangled representations from BERT models by masking weights/activations is a nice result and I'm not aware of similar approaches for BERT. However, it's unclear from the paper that this approach outperforms previous alternatives:
* First, the abstract mentions that the approach is the same or better than variational auto-encoder approaches, and I don't see it mentioned elsewhere in the main text. Am I missing something?
* Second, the paper does not show improved results on any benchmarks.
As a result, I'm not sure whether the paper will be impactful enough for the community.

**Other Questions**:

* The proposed approach involves training binary masks rather than fine-tuning the BERT weights. Given that the mask has the same shape as the weights, it's unclear whether this is a major speedup. Could you discuss this more?
* Does using the pretrained model (vs. one trained from scratch) help?
* Have you considered masking a subset of the weights/activations (e.g. only in the last layer)?
* Do you have any intuition about the learned masks? E.g. are most weights/activations being removed? How much overlap is there between the masks learned for each attribute?

Overall, I like this research direction, but I think it requires more work to be accepted.
review_title: New approach to disentangling representations

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a way to learn disentangled representations with respect to target attributes of interest by learning to mask weights or activations. A particular piece of text is encoded into distinct vectors that capture different factors of variation in the data. The method involves learning masks for each factor of variation while keeping the pre-trained model parameters fixed. The masks for every layer are trained using a combination of a triplet-loss, attribute classification loss, and one that encourages masks for different factors to be different across all layers. The triplet loss forces representations of examples that are similar with respect to a particular attribute to be closer than one that are similar based on another attribute.

Models are evaluated on a sentiment/genre classification on a dataset sampled in such a way that introduces spurious correlations between genre and sentiment but evaluated on data that does not have any such correlation. The approach is also evaluated on disentangling syntax and semantics.

Strengths

Building models that are robust to spurious correlations in data is important for a variety of reasons and learning disentangled representations is a promising way to achieve that. This paper shows good generalization performance on datasets with such characteristics.

The overall approach is simple and only requires training masks over weights/activations at each layer. The masks are trained with a fairly straightforward choice of training objectives.

The paper is well written and the overall approach is easy to understand.

Weaknesses

The triplet loss as formulated in this work seems to make it possible to disentangle only two factors of variation (a) and (b).

There is still a fair amount of attribute leakage and the probe designed to measure this leak is only a single layer MLP, there might be more leakage with stronger probes.

The weight masking strategy significantly increases the number of parameters (although the masks are binary, so it just requires a single bit as opposed to 16/32 bit floating point numbers). In this particular work, the number of parameters triples, and it scales linearly with the number of attributes as well.

It requires running the model forward multiple times to get representations that encode different factors of variation.


Questions & Comments

What would performance look like if masks were trained after fine-tuning on sentiment/genre classification? Rather than training masks directly on top of BERT-base. It would be interesting to see if the model is stable to recover from fine-tuning on data with spurious correlations and still produce disentangled representations.

Is every single weight/activation masked at every transformer layer? The paper seems to lack some specifics about exactly what layers/weights are masked. Along these lines, did you experiment with masking only the last few layers? This could save time & parameters

In Figure 3 is the model training with L_{cls} corresponding to sentiment and then visualized for sentiment and genre? Or is the top trained with the supervised sentiment loss and the bottom for supervised genre loss?

It would be interesting to explore an L1 penalty on the masks for increasing sparsity, possibly in conjunction with magnitude pruning as well.

The WC task doesn't feel very representative of sentence "semantics"
review_title: Official Review - AnonReviewer2

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a masking strategy to identify subnetworks within language models responsible for predicting different text features. This approach requires no fine-tuning of model parameters and still achieves better results compared to previous approaches. Their experimental results on the movie domain show some level of disentanglement is achieved between sentiment and genre. Disentanglement capabilities of their model between sentence semantics and structure, are also tested on four tasks. 

Pros:
- Paper is well-written and the idea is explained well.
- Experiment results are convincing and support the claims.
- Achieving comparable results to SOTA without the need to train or finetune models is interesting especially from a computational point of view.

Cons:
- I wish the authors performed their first experiment on more domains: books, music, etc. and consider more than two labels.
From current results, it's hard to confidently conclude that this approach is generalizable.
- Judging based on Figure 4 results I'm not convinced that the proposed approach does better than the *finetuned* (which I believe has a trained classifier on top of BERT) approach especially for Semantic tasks. Perhaps a discussion/ error analysis would be appropriate given better results on Syntax tasks.
- Also a discussion on the results for masking weights vs. masking hidden units is missing. If I'm not mistaken, mathematically, hidden unit masking is a subset of weight masking, where masking an item in hidden activation is equivalent to masking an entire column in the weight matrix?


Comments:
- Although the idea of masking model parameters to achieve untanglment is new, there has been [previous work](https://www.aclweb.org/anthology/P18-1069.pdf) on using dropout to identify sub-parts of the network that contribute more/ less to model predictions framed as a confidence modeling task. Authors may consider adding it to related work.
- Another missed citation under related work is [HUBERT](https://arxiv.org/pdf/1910.12647.pdf) which examines untanglement of semantics and structure across a wide range of NLP tasks.


Minor typos:
- "we *measure* evaluate them on four tasks ..." on page 7 
- "Technically, in the Pruned + Masked Weights method, *the* refining the masks ..." 
review_title: Light-weight approach to untangle language model representations

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a problem of disentangling representations generated in pretraining models, such as BERT. That is, it is possible to learn disentangled representations that encode distinct, complementary aspect representations. To this end, the authors proposes a method that employs the mask technique on transformer weights or hidden units to find the subset of features correlating with a specific task. The experimental results show that the proposed method can encode particular aspects while weakly encoding others. The main contributions of the paper is the introduction of binary masks to identifying some subnetworks, which may correlate with specific tasks, within pretrained models. Overall, the paper is well written and is easy to follow. 

Concerns:
1. The experimental setup is not convincing. The authors just consider movie reviews corresponding to Drama and Horror from IMDB and exclude reviews corresponding to other genres. It is obvious that considering only two genres is not convincing and more genres should be considered in the experiments. So, the authors should answer the following questions: (1) Why do the authors just selected these two specific genres to conduct the experiments? (2) Do the authors conduct similar experiments on other genres and what about the experimental results?
2. Figure 3 does not show that proposed method achieves better results than do the two baselines. In fact, the finetuned baseline performs very well according to Figure 3. I suggest that the author adopts some quantitative measures to accurately reflect the differences.
3. In addition, how to use these disentangling representations in downstream tasks, such as text classification, natural language inference, and semantic similarity? It is better to discuss and conduct experiment to show the advantages of their disentangling representations in downstream tasks.

Minor comments:
1. In Formula (9), the parentheses are redundant.

review_title: an interesting work on disentangling representations of text

Meta-review: This paper explores a methodology for learning disentangled representations using a triplet loss to find subnetworks within a transformer.  The authors compare against several other methods and find that their method performs well without needing to train from scratch. The reviewers thought this paper was well written and the authors were very responsive during the review period.  However, there were some questions about the experimental setup and empirical performance of the paper, leaving the reviewers wondering if the performance was convincing.  We agree that there is value in exploring disentangled representations even if they do not necessarily improve performance (as the authors point out), but clearly explaining the reasoning behind all analyses (e.g. specifically choosing domains to introduce a spurious correlation), and justifying differences in performance is particularly important in these cases.
Decision: Reject