Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a framework, using concepts from category theory, to analyze the mechanisms through which Transformer based language models perform contextualize representations. In contrast to prior work, which analyzes contextualization of representations through “token-centric” probing, the authors first cast the language representation problem as a functor from an input category (encoding of observed context) to an output category (d-dimensional space). Next, a manifold learning approach is analyzed in the case of a fixed representation model and treating the outputs at each layer as a different representer. Lastly, the authors perform a range of experiments to better understand various aspects in transformer models including how contexts are represented for [CLS] and the change in context representations by layer.
main_review: Strengths:
- The proposed approach is interesting and suggests an alternative viewpoint to understanding contextualization in language models. 

Weaknesses:
- The empirical results do not seem to rely substantially on the methodological contributions. In particular, experiments are performed with local probing (UMAP) over a small number of neighbors (15) to determine manifolds compared with a general functor over manifolds presented in Sections 4-6. This form of analysis is quite similar to that performed in Cai et al. 2021, except over contexts rather than tokens. Additionally, the layerwise analysis is quite similar to that performed by Aken et al. 2019 CIKM in the context of question answering.
- The paper is a bit hard to follow, especially the description of input and output categories in Section 4. Figure 1 is helpful for this.
- The experiments are somewhat limited and, as mentioned previously, similar to those previously conducted in prior works except over contexts rather than the central word.
- For the [CLS] experiments, it seems straightforward that the functional characterization is mixed before fine-tuning and concentrated after finetuning as the [CLS] token is rarely used in masked language modeling (at least for the BERT formulation of this objective). 

summary_of_the_review: Approach for modeling contextualization in language models using tools from category theory and manifold learning. Interesting approach but there is a gap between the methodology and the experimentation. Additionally, experimentation is similar to studies conducted previously with a focus on substitutions in context rather than center token.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper intends to understand “how does a language representation model represent contexts” which is different from a more popular direction of understanding “how does the context affect the representation of the tokens”. The paper formulates the contextualized language representation problem using category theory. Authors claim that previous works that focus on substituting tokens by keeping the context fixed miss a lot of useful information. According to their proposed framework, representation of the context contains information about the representation of tokens that get substituted while keeping the surrounding conditions unchanged as well as information about the transitions (morphisms) in the representations as a result of these substitutions. However, quantifying the above information is challenging, and this paper intends to analyze this information from a functional (task-related projection) and topological (UMAP) perspective.
main_review: This paper intends to understand “how does a language representation model represent contexts” which is different from a more popular direction of understanding “how does the context affect the representation of the tokens”. The paper formulates the contextualized language representation problem using category theory. Authors claim that previous works that focus on substituting tokens by keeping the context fixed miss a lot of useful information. According to their proposed framework, representation of the context contains information about the representation of tokens that get substituted while keeping the surrounding conditions unchanged as well as information about the transitions (morphisms) in the representations as a result of these substitutions. However, quantifying the above information is challenging, and this paper intends to analyze this information from a functional (task-related projection) and topological (UMAP) perspective.

Before you read further, it is important to mention that I do not have a background in category theory or Manifold learning so I have judged this paper purely on the basis of convincingness of the arguments, quality of experiments, and interpretation of the results.

Overall, I think that paper identifies a very niche problem and provides interesting ideas about how we can think about representation of the contexts in language modelling. But in the same light, I do not think that the paper is well-written to be published at this conference. The paper needs serious revamping in terms of story, writing style and presentation. 

In terms of the story: 

- The main takeaway of the paper remains unclear to me for most of the paper as the authors were not precise with communicating their intentions. For instance, the paper’s abstract and various other parts indicate that the work would explain “how a model represents context itself,” whereas, in the motivation, the paper states that the authors are trying to tackle “how a representation model learns to represent contexts”. I believe that these two statements are not identical. The latter claim is about how context emerges in the model when learning happens during training. In contrast, the former is about analyzing how the context is represented once the learning is complete. The authors should state both of these clearly if they are trying to answer both the questions.

- In the abstract, the paper claims that the work would shed light on the improvements in Transformer-based language representation models. However, the paper does not mention how it plans to do so in the main text. Additionally, I do not understand the nature of improvements the authors are referring to.

In terms of the writing style, there are several places where the authors need to rephrase sentences or correct grammatical errors. To list some:
- In early days, word embeddings ~~are~~ were static, i.e., the representation is was solely determined by a word’s identity…
- GPT ~~revolutionize~~ revolutionized this technique
- When compared to static embedding, ~~mechanistic~~  (use a better word here?) studies of the contextual embedding generally lag behind.
- They also point out **that** the embedding space is generally anisotropic. Also what do you mean by anisotropic? A reader without a manifold learning background is likely to phase out after coming across such terms if they are not explained succinctly. 
- On the other hand, several works try to uncover how contexts influence the geometry of embedding space. (Missing references)
- Therefore, very few studies touch (touch upon?) another critical problem, i.e.,.
- untouched fundamental problem in deep language learning. I am reading the term **deep language learning** for the first time. Can you please clarify it a bit more?
- The ~~fine-turning~~ fine-tuning process …
- FFN layers identifies a ~~Game~~ game between…
- hypothesis that they both of them (not sure what you meant here) try to fit an identity morphism that reflects bona fide linguistic associations.
- I believe that the readers will benefit from detailed explanations of the terms presented in the mathematical equations. For instance, equations (6)-(11) have been introduced together in Definition 4.4 without much explanation as to what each of it intends to convey.

In terms of the presentation:
- It seems as if Figure 1 is an important cue for the readers to understand their framework but it lacks descriptive captions or a reference in the main text that could explain the idea succinctly unlike Figure 2 and Figure 3.
- The graphs in Figure 3 should somehow indicate that if Layer i is Feed-forward network (FFN) then Layer (i+1) is Multi-head attention. Make it clear in the caption as well.

The authors test their method on just one of the GLUE classification dataset, SST-2, using Wikitext-2 as a pre-training corpus. I believe that the authors should include more datasets and discussion to make the experiment results more convincing. There are other single sentence classification datasets in GLUE such as CoLA, and other inference based classification tasks such as MNLI (https://openreview.net/pdf?id=rJ4km2R5t7).

After multiple parses through your experiments section, I am unclear if the authors perform token substitution around [CLS] token or they replace [CLS] token itself. If I think about this experiment from Scenario A’s (Section 5.2) and Figure 2 perspective, it suggests that the authors intend to convey the former. However, the authors mention  “when replacing the [CLS] tokens with other tokens, the representation of the whole sentence does not change a lot” which aligns with the later understanding. The authors should clarify what they actually intended to do in this experiment. The results from the experiment suggest that finetuning helps in learning new patterns and pre-training patterns are forgotten which is not that surprising based on our current understanding of fine tuning. 

I found the authors’ observations on the conflict between Multi-head attention and Feedforward network quite interesting. I believe that the authors should analyze this result and conduct more experiments around this idea on more datasets. It would be interesting to see if the same observation holds for other pretraining corpuses as well which extend beyond English as well. It might be even the case that the observations the authors made were a result of the quirks (linguistic properties) of English rather than something being innate to attention-based language modeling. 
summary_of_the_review: It is important to mention that I do not have a background in category theory or manifold learning so I have judged this paper purely on the basis of convincingness of the arguments, quality of experiments, and interpretation of the results.
Overall, I think that paper identifies a very niche problem and provides interesting ideas about how we can think about representation of the contexts in language modelling. But in the same light, I do not think that the paper is well-written and rigorous enough to be published at this conference. The paper needs serious revamping in terms of story, writing style and presentation. I provide detailed explanation of my concerns in the main review.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims for providing a theory for understanding the context representation in pretrained language models. It presents a category theory then some analysis of representations from pretrained LMs. This inspiration is good. 

However, this paper is severely flawed in multiple aspects: the basic notations and understanding of LMs seem to be wrong; the definitions of concepts are unclear and ambiguous; many symbols are used without definition or explanation; equations are not explained; there is barely any connections between the category theory and the language model; important related work is missed, and the experiments are quite simple and insufficient. I think this paper needs significant improvements in terms of basic understanding, writing, explanation, and experiments.
main_review: This paper tries to build a theory about how context is represented in pretrained language models. This is indeed a meaningful direction as current literature, although extensive, does not contain targeted studies about context. However, this paper is severely flawed in the following aspects: 

## Basics of probability and pretrained LMs

Equation 1: 

- a masked language model outputs the probability of p(masked word | all other tokens in the sentence), NOT p(context |  C, w).
- what does the word *context* mean in eq.1? It seems to be a random variable since it has a probability. So is it a discrete random variable, or a continuous random variable, or a sequence/set of discrete/continuous random variables?
- what does *C* mean? The paper says it is the observed unmasked tokens but isn't this part of the context? My understanding is that the *context* of a given masked word is all other words within the sentence (either other words are masked or not, as recall 15% words are masked in BERT).
- what does the symbol ~ mean in Eq.1? Does it read "follows this distribution"? If it is, does it induce that the symbol G(C, w) is a random variable? The paper says G is "a contextual language representation", but what is it? Is it a sequence of output vectors of the last layer of BERT? If they are, they are deterministic (not random variables), so how can they have distribution?

The term "observed condition C"

- If it means "observed unmasked tokens", why re-name it as "observed condition", since the former is apparently more understandable?

Equation 2:

- Again, what does the symbol ~ read? Does it mean "follows the distribution of"? If yes why the right hand side is a set, but not a probability distribution?

A basic rule about writing equations and symbols is to always define them properly before using them. In this case, I strongly encourage the authors to discuss the definitions of symbols in their equations. In addition to the above questions, the authors may also want to discuss 

- G(C, w), is this a function G applied to variable C and w? If yes what is the domain and what is the image?
- Similarly, for P(C), is P a function applied to a variable C? is C continuous or discrete? is C a scalar or a vector?

## The definitions of symbols are unclear and without definition or explanation, equations are not explained

There are a lot of concepts directly used without explanation or reference to the background. To name a few: 

- Definition 4.1, what is the use of the sigma-filed E? Is it the vocabulary? If yes why not directly say vocabulary?
- Definition 4.2, what is X? Is it a word at a certain location? If yes why not directly say it is a word at a certain location? Does a context mean a set of functions takes a sentence (excepted a mask word) as the input, and output a distribution over the vocabulary (because the paper says "A context is ... a set ... each element can be expressed as a ... function")?
- If a context C is a set of functions, as is in definition 4.2, why the symbol C \in E in equation 3? Isn't E the vocabulary?
- Why introduce all these new symbols and terminologies (like "observation condition") without explanation, when we already have perfectly understandable terms (like "words" and "sentences")?
- For definitions 4.3 and 4.4, I get more lost as these are longer equations without definition and explanation. It is possible that it is only me that have difficulty reading these equations, as I am not so familiar with category theory (but I do know combinatorial categorial grammars well, more on this later), but I find it hard to read equations from 5 to 9. There are a lot of ambiguities when I try to understand, for example:
- It seems that "the representation of context" and "context" are two different things, as this paper writes "the representation of context as a functor". Yet with pretrained language models, the word "representation" usually refers to vectors, so how do these two compare?
- Combining definition 4.2, equation 5, and equation 11, I guess the paper is trying to say: the conditional probability of a masked word (def. 4.2) is the same (equality 5) in different pretrained language models (Gm and Gn in Eq.11) if the context is the same. Is this how equation 11 should be interpreted? If yes, does this paper consider different training datasets of different pretrained LMs?

## Lack of connections between the category theory and the language model

Is it possible that this paper gives a one-to-one mapping between all introduced concepts to standard terminologies in NLP? For example:

- What does the term "category" correspond to? A context?
- What does the term "object"  correspond to? A context in a specific sentence?
- What does the term "morphism" correspond to? A mapping between different contexts?

## Important related work is missed

How does the introduced theory compare to classical combinatorial categorial grammars? What if we build a supervised/ unsupervised CCG tagger upon a pretrained language model like Liu et. al. 2019 (In this case each word is also associated with a linguistic category)? 

## Insufficient experiments

Generally, the current experiments are too simple and not well-explained. Specifically,

- for figure 2a, my understanding is that the authors substitute the original token to another token, then use UMAP to get the probability of the edge, but how does this compare to the original masked LM where one can directly get a distribution of possible substitution words? Why bother with UMAP and what does it tell us?
- What new things can we learn from figure 2bc? My understanding of them is the classes are more separated after fine-tuning (if I have not missed anything), but isn't this trivial?

## References

- Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. NAACL 2019
summary_of_the_review: Although the goal of this paper (to analyze the effects of context) is good, it suffers from severe issues as mentioned above. I would recommend a re-writing, especially the basics (like what kind of variable a context is), with more motivations, explanations, and sufficient experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to explore how contexts are represented in a deep language model. It attempts to go beyond “token-centric” understanding in the previous contextualized representation studies and specifically tries to investigate how different contexts are being represented (e.g., when tokens are being replaced) and how the representation changes through different components (e.g., FFN and attention modules) in the language model.
main_review: Pros:
1. The attempt to explain and investigate how contexts are represented in the language model is interesting.
2. The introduction of the category theory to model the connection between contexts and their representations seems novel.

Cons:
1. The serious issue with this paper is its poor presentation. Many details are very vaguely presented which makes it very hard to understand the contribution of the paper. Firstly, as this paper studies how contexts are being represented in a language model, a clear definition of what is "context" is necessary. My guess is that the "context" of a word is all the remaining words in the sequence, but this is not clearly mentioned in the paper. If this is true, what is the difference between "context" and "observed condition $C$" in Section 3? Why not use some notations/formulas to explicitly define what is "context"? Sections 4 and 5 are very difficult to parse even after putting in lots of effort. Some important terminologies should have clear definitions (e.g., morphisms) before they can be used as the paper needs to be self-contained. Also, there should be high-level explanations/concrete examples to facilitate an easy understanding of the purpose of each definition and how they are connected to the mechanisms of contextualized representation learning. Section 7 also fails to clearly explain how the empirical findings validate the theoretical proposals.
2. The empirical results are from only one dataset representing downstream tasks (SST-2) and are largely presented in the form of case studies. More comprehensive studies are necessary to give reliable and convincing results.
3. There are a lot of typos throughout the paper; some examples:
Section 2, "understanding how the vector represent the token" -> "understanding how the vector represents the token"
Section 2, "supports our hypothesis that they both of them try to" -> "supports our hypothesis that both of them try to"
Section 3, "which motivates as to introduce" -> "which motivates us to introduce"
summary_of_the_review: While the paper attempts to study an important and interesting angle in understanding how language models represent semantics, the ideas and results are presented in a very vague manner which prevents clear understanding and judgment of the contributions in the paper. It is still unclear to me how exactly are "context" represented in language models and what insights can be obtained from the paper, even after reading the paper several times.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper reports several empirical findings. 
1. Replacing the [CLS] tokens after the fine-tuning does not change the output representation a lot. 
2. By using UMAP to analyze the patterns of nearest neighbors in different Transformer layers, it finds that the topological structure in layer 1 is most similar to the structure in layer 2 and less similar to that in layer 24. This shows that each Transformer layer would change the topological structure of the context embeddings.
3. Let's denote the embeddings in the Transformer in this way (H1) -> ATT -> (A1) -> FFN -> (H2) -> ATT -> (A2) -> FFN -> (H3) ..., where (Hk) is the hidden states, ATT is the multi-head attention, (Ak) is the output of the attention layer, FFN is the feed-forward network. Figure 3 (c) shows that (Hk) and (Ak) are very different when k is small, but become similar when k is large. Compared to (Hk) and (Ak), (Hk) and (Hk-1) are more similar when k is small. Similarly, (Ak) and (Ak-1) are more similar than (Hk) and (Ak) when k is small.

main_review: This paper is an example of trying to use some complicated theories to describe some very simple observations. 

This paper does not have a significant theoretical contribution because it does not have any theorem or lemma. All of the notations and definitions are just for describing their analysis method in a very abstract way. I spent quite some time understanding what those notations mean, but realize that the complex math does not help me understand the empirical findings at all, which are the main contributions of the paper. 

The paper's first two empirical findings are not very interesting to me and the authors also do not explain the impact of those findings. The third finding in Figure 3 (c) could potentially deepen our understanding of BERT, but the authors need to investigate the underlying reasons to make the finding become useful. 

By the way, I think the authors should also cite this work (https://aclanthology.org/2020.aacl-main.11.pdf) (and possibly other work that also tries to align the embedding space in different layers of Transformers to analyze what language model learns). These papers show that the analysis methods used in this paper are not very novel.

Minor things:
1. ELMO -> ELMo
2. Your notations are not consistent or not clearly explained. For example, What is E? Why e \in E but also C \in E? What is G_{set} = G_{m}?
3. (7) in Equation should be right justified.
4. Your Figure 1 is helpful. But if you really want to define those symbols for some reason, add more symbols into the figure would make your definitions easier to understand.
5. Your scenario A corresponds to m=n in Equation (9), right? If yes, make it more clear (if you really want to define those symbols for some reason).
summary_of_the_review: I vote for rejection because 
1. The findings are not significant. 
2. The analysis method is not novel.
3. The paper presents a few very simple experiment results using a very complicated way, which makes the paper very hard to read.
review_title: null

Meta-review: This paper proposes a theory for understanding the context representation in pretrained language models. The strengths of the paper, as identified by reviewers, are in the importance of an attempt to explain contextualization in language models, and in the novelty of using the category theory to model the connection between contexts and their representations. However, all the reviewers identify several major weaknesses, including flawed/incoherent definitions of concepts in the proposed theory and insufficient experimental results. Although the authors' rebuttal put a great deal of effort to address raised concerns, all five reviewers agree (and provide very detailed justifications along with suggestions for improvements) that the work is not yet ready for publication.
Decision: Reject