Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper analyses concepts that BERT can capture. The core idea is to use agglomerative hierarchical clustering method to discover latent concepts (i.e. clusters) with hierarchy. Those clusters were manually assigned labels / meanings. The authors then used these labelled clusters to analyse concepts that BERT captures, across several syntactic and semantics aspects. The labelled clusters were also used to build a concept dataset (namely BERT concept dataset - BCD) for 1M instances (i.e. words in contexts).  
main_review: ### Strengths 

The problem that the paper tackles, analysing latent concepts captured in BERT, is challenging and needed to understand the model. It is well in line with the trend of model and representation interpretation. 

The proposed solution to the problem requires carefully manually annotation which was carried out thoughtfully. The innovation is that there are several syntactic and semantics aspects taken into account, including hierarchy nature of those concepts. 

The analyses presented in the paper do lead to several findings about the knowledge of BERT, thus they back the claims risen in the paper.


### Weaknesses 

The paper doesn't go any further beyond 1000 clusters / concepts, which doesn't seem to have as wide coverage as e.g. wikipedia. Hence, one would question about the used clustering method, especially whether it is good enough to discover a wide range of concepts / clusters. 

The methodology proposed in the paper requires expensive manual annotation. This leads to the problem of replication. Besides, although the BCD dataset has some potential, it is limited to only BERT. It does't seem useful when other models are examined. 

I found the BCD development (section 7.1) problematic. Although clusters were manually annotated, it is unclear how accurate an instance is assigned to a specific cluster. For instance, although annotators intuitively agreed that cluster A with N instances properly present concept "animal", how many instances among the N instances actually belong to concept "animal"?





summary_of_the_review: The paper tackles an important challenge with thoughtful methodology and reasonable analyses. The paper is also well written, presenting several findings about the knowledge captured in BERT. 
The proposed methodology however has problems which limit the contribution of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the latent concept learned in BERT representations. The authors use a simple and effective clustering method to discover latent concepts in an unsupervised way. The contribution is primarily on the empirical side where the learned concepts are labeled and compared with pre-defined concepts. Then a layer-wise comparison is conducted where the results are similar to previous work but is done in an unsupervised way. The authors summarise this paper by organizing the discovered concepts into a BCD dataset. The writing is also clear. While there are many aspects deserving further pursuit, I think this is a solid paper that makes concrete contributions to the understanding of BERT. 
main_review: Generally, although the method is simple, I think it is effective and solid. Also, I appreciate the efforts in the annotation. I think a direction for improvements is a more in-depth analysis since this paper spends a lot of writing on development (which is fair) but less on analysis. I would suggest simplifying the development details (Section 4.2, 4.4, 6.2 can be shortened, and algorithm 2 can be replaced by a simple textual description) to make space for analysis, specifically: 

## Alignments and Differences between discovered and pre-defined concepts

This is a point worth pursuing because the concepts learned in an unsupervised way respect patterns in data much more than human prior definition. I was expecting a detailed analysis about the concepts that are NOT aligned with pre-defined concepts in section 6.2, but there are not and the existing analysis simply reinforces what we have already known. What are these non-aligned concepts? Are they interpretable? Are they more about syntax or semantics? Why they do not align with existing annotation? I strongly encourage the authors to discuss these questions in detail because these are the questions that lead to new insights, rather than reinforcing what we have already familiar with. 

## Why latent concepts emerge within BERT

This is another interesting question that may be worth pursuing. This is essentially a question about how large-scale masked language modeling automatically leads to clusters of representations. My intuition is that this should trace back to the effect of contextualization, i.e., clusters emerge because words within the same cluster share the same context, and different clusters originate from different contexts. There could also be other explanations, and I would like to encourage the authors to give a more detailed mechanism about how latent concepts emerge. Also, see [1] as analysis from a manifold perspective. 

## Minor comments

- Clustering time and memory consumption should be reported in Section 4.2
- Word examples for each cluster should be provided in the appendix (Table 10)
- Percentages should be provided in table 1

## References

[1] Emergence of Separable Manifolds in Deep Language Representations. Jonathan Mamou, Hang Le, Miguel A Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, SueYeon Chung. ICML 2020
summary_of_the_review: I think this is a solid and interesting paper. My major problem is that this paper spends a lot of writing on development details that could be simplified to make place for more in-depth analysis. I am willing to further increase my score if the authors can present a more in-depth analysis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper performs a clustering analysis on the contextual representations of BERT on a subset of the News 2018 WMT corpus. These clusters are manually tagged by 3 annotators to connect these clusters to human concepts. The authors additionally release a dataset with these concepts labeled.
main_review: The paper reads well, and the authors have clearly presented some interesting clusters within BERT. The authors use human intuition to hierarchically label a selection of "concept clusters" discovered in BERT. In this way, concept exploration in BERT is not limited to pre-defined concepts, and the authors promise to make their compiled dataset (BCD) available for additional research. Their results really do reveal some interesting clusters, and the paper's message could be bolstered if they included some tool to easily explore the concept space learned in BERT outside of the results shown in figures in the paper.


**Weaknesses & Feedback:**

1. In the main paper, the authors fail to describe the model that they use for analysis. All they say is "We base our study on BERT" (S1p1). But there are many flavors of BERT trained on different datasets, and the released BERT Concept Dataset (BCD) will only correspond to this model that the authors analyzed. There is not even a proper citation to BERT in the introduction of the paper (though one does appear later).
2. In addition to not disclosing the model architecture used in the main paper, the authors do not disclose the tokenization scheme. The most common tokenization schemes for a case-sensitive BERT would likely not have the long hyphenations (e.g., "75-year-old"/"90-year-old") and all the German words (e.g., "Mecklenburg-Western") as single tokens in the dictionary. How would this technique combine tokens to represent a single concept?
3. Without architecture information, I am left wondering about their statement that "200k vectors each of size 768 require 400GB of CPU memory". Assuming 32bit floating point, this many vectors should only occupy 0.6GB of space. Are they talking about the model memory? Why wouldn't you batch the representations and save them to hard disk to cluster later? I am also confused by this limitation.
4. The annotation tool examples in the Appendix do not make it clear if annotators had access to the original context each token appeared in. For example, the word clouds in Fig 4(a,b) have little information in the cluster itself to inform an annotator that they one refers to percentage and the other to units of money. The annotation tool seems to show 3 or 4 different sentences that a *single token from the cluster* came from. How does that generalize to the other tokens in the cluster?
5. (minor) There is a lot of speculation in the analysis as to how these findings could help, but I do not find the argument compelling. The authors could improve their narrative by elaborating on some of their claims. For example: 
    - Learning about the existence of a cluster where the model captures an era of time "may help the model to learn relation between a particular era and the events occurring in that time period" (S6.1p6).
    - "A clothing recommendation system may use such information to customize suggestions towards a culture or a religion" (S6.1p6). I find this a questionable application that relies on biases learned in the language of a region/demographic.
    - The authors show no summary of the results in  S7.2p9 "Case Study", and most of this section is spent persuading readers that such analysis could be helpful. It also fails to state clearly how their "neuron analysis" could be replicated.
6. The annotation analysis performed in this paper is limited in that it only explores the representations in the final layer in BERT. 
7. I have a few complaints about the word cloud visualization. The size of each token reflects the frequency of that token (footnote 2), but this is strongly biased by the total frequency of that token in the sub-sampled dataset. Perhaps more meaningful would be if the size of the tokens reflected the proximity to the centroid of the cluster.
summary_of_the_review: In summary, I find the main paper lacking in sufficient disclosure on methods, and I do not believe the techniques or findings of the paper to be sufficiently novel, though I can see that the dataset would be useful. I still vote reject for ICLR 2022.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Summary**  

While a large number of previous papers evaluate DNN models on pre-defined concepts, this paper interprets the models (i.e., BERT) on latent concepts that are learned in an unsupervised manner. The analyses in this paper include: (1) Agglomerative hierarchical clustering from ~200k embedded tokens. (2) Comparing the clustered concepts to some hierarchical concept-tagsets. The authors find that lower layers learn many shallow lexical concepts, while higher layers learn more semantic relations. In addition, this paper releases a novel BERT concept dataset (BCD). With cross validation, the authors show that the BCD dataset has high consistency.

**Contributions**

1. Interpret BERT by analyzing latent concepts learned in the network
2. Collect and annotate a hierarchical concept dataset (BCD)
main_review: **Strengths**

- The topic studying "latent concepts" is interesting and potentially impactful.

- This paper involves a lot of empirical effort.
- The dataset would be beneficial for future researches.

**Weaknesses**

(W1) There appears to be a potential confound in annotation. Sec 3 loosely defines "concept" to be "a group of words that are meaningful". The annotation procedure seems to let humans to decide whether the words are meaningful based on the words themselves (instead of the contexts). Ethayarajh (2019) showed that the context matter a lot for contextualized representations (e.g., BERT). This annotation procedure appears to drop some useful information. E.g., what if the words themselves are not "meaningful", but the way they are used in contexts appear meaningful? Do the annotators mark them as "not meaningful"? 

(W2) The generalizability of the finding is limited. The results reported in this paper appear specific to BERT. How would the clustering results compare to a static embedding (e.g., GloVe) and other contextualized representations (e.g., RoBERTa)? As far as I understand, the representation vectors cluster well because the meanings themselves follow hierarchical clusters, and that different DNN models can indeed embed their meanings fairly well -- BERT, especially, is one of the most powerful DNN models, so it is not surprising that it can do well here. Just showing "BERT clusters them well" does not add novel knowledge to what people already know. Comparing to other models might bring in more novel findings.

(W3) The reporting of the results needs to be improved. Page 6 last paragraph: "We compared the resulting clusters with the following pre-defined concepts: ... CCGBank, WordNet, LIWC": Some comparison results are listed in Table 1. What is the percentage of matches? E.g., Lexical:NGram contains 20 matches, but how many concepts are there in total? In addition, Table 1 only shows the "top-level" match counts (& of layer 12). Among the more fine-grained concepts, how many concepts are matched? 

**Questions to the authors**

(Q1) On the choice of concept database. In 4.3 Concept labels: there seem to be some existing databases with hierarchical concepts. E.g., WordNet. Why do you hand-engineer a concept database instead of using WordNet? Compared to WordNet, the hierarchical concepts defined in this paper appears very simplistic -- WordNet contains 155,327 words (let's say each "concept label" contains 100 words, then they have "concept label" to the magnitude of 1k), while the database of this paper contains 183 concept labels.

(Q2) During the clustering, do the optimal number of clusters remain consistent for representations coming from different layers? (Am I right in assuming that Figure 2 only shows that of the final layer representation?)

(Q3) On the limitation of the data size in clustering. How about first assigning each word to a "large cluster" (e.g., by the part-of-speech), which should be very accurate, and then run the cluster algorithm within each "large cluster"? This might help circumvent the computation bottleneck.
summary_of_the_review: Interesting topic and substantial empirical effort. Methodology (clustering, dataset annotation, etc.) has potential problems. Reporting needs improvements. The generalizability and novelty of the overall finding are limited.
review_title: null

Meta-review: This paper analyzes the latent concepts learned in BERT. In contrast to previous work which tries to map embeddings to predefined
linguistic concepts this paper sets out to discover what is inherently learned by BERT. This is however easier said that done, since 
there is no easy way to inspect the embeddings and draw conclusions on what is being learned. The authors adopt a methodology which could be used to inspect the inner workings of other pretrained models. They employ hierarchical clustering to discover latent concepts and then inspect these clusters by manually labeling them. The reviewers raised various issues regarding the number of clusters, and the amount of effort required which de facto renders the approach not very portable. The authors addressed the comments and flagged several difficulties with undertaking such an analysis. I will vote for the paper to be presented as an oral for two reasons a) it is difficult to analyze pretrained models, and although I am not convinced what the authors propose is feasible, it will at least get the discussion going, b) the manually annotated dataset is useful and will go towards allowing us to perform comparisons between models c) the annotation tool will be useful to others if the authors are considering releasing it.
Decision: Accept (Poster)