Finding One Missing Puzzle of Contextual Word Embedding: Representing Contexts as Manifold
Authors: Hailin Hu
Rong Yao
Cheng LI
Venue: ICLR 2022 Submitted (2022)
Keywords: Contextual Word Embedding
Category Theory
Manifold

Abstract: The current understanding of contextual word embedding interprets the representation by associating each token to a vector that is dynamically modulated by the context. However, this “token-centric” understanding does not explain how a model represents context itself, leading to a lack of characterization from such a perspective. In this work, to establish a rigorous definition of “context representation”, we formalize this intuition using a category theory framework, which indicates the necessity of including the information from both tokens and how transitions happen among different tokens in a given context. As a practical instantiation of our theoretical understanding, we also show how to leverage a manifold learning method to characterize how a representation model (i.e., BERT) encodes different contexts and how a representation of context changes when going through different components such as attention and FFN. We hope this novel theoretic perspective sheds light on the further improvements in Transformer-based language representation models.

Under review as a conference paper at ICLR 2022
FINDING ONE MISSING PUZZLE OF CONTEXTUAL WORD EMBEDDING: REPRESENTING CONTEXTS AS MANIFOLD
Anonymous authors Paper under double-blind review
ABSTRACT
The current understanding of contextualized word embedding interprets the rep- resentation by associating each token to vector that is dynamically modulated by the context. However, this "token-centrie" understanding does not explain how model represents context, leading to lack of characterization from such a perspective. In this work, to establish a rigorous definition of "context repre- sentation", formalize this intuition using category theory framework, which indicates the necessity of including the information from not only individual con- texts but also how transitions happen among the semantic effect of different con- texts. As practical instantiation of our theoretical understanding, we also show how leverage manifold learning methods to characterize how a representation model (i.e., BERT) encodes different contexts and how representation of con- text changes when going through different model components such as attention and FFN. We hope this novel theoretical perspective sheds light on the further understanding of Transformer-basee language representation models.
INTRODUCTION
In modern natural language processing, using vector representation of words in low dimension space is common practice. In early days, word embeddings were static, i.e., the representation is solely determined by word's identity, such as the case of word2vec (Mikolov et al.) 2013) and
GloVe (Pennington et al., 2014). In contrast, contextualized word embeddings, e.g.. EL (Peters) et al. 2018), BERT (Devlin et al.) 2019) and GPT (Brown et al. (020), revolutionized this technique by introducing information from the contexts to the central word.
When compared to static embedding, studies understand the contextual embedding generally lag behind. This particularly unfavoured as the superior performance of pre-trained contextualized representations is gaining more and more attention. To tackle this problem, previous studies have proposed several directions to explore. For example, researchers in Hewitt & Manning (2019); Co- enen al. (2019); Wu et al. (2020) develop several probing techniques relate contextualized word embedding to syntactic features. On the other hand, several works try to uncover how contexts in- fluence the geometry embedding space. Etthayrajh (2019) measures the context-speciffcity of the word embeddings from different layers of several deep representation models and confirms the ma- jority variance is provided by the context but not the token. They also point out that the embedding space is generally anisotropic. Then, results from Cai et al. (2021) indicate that isotropy actually exists in certain isolated clusters of words in the representation space. MMmou et al. (2020) proposes to analyze the manifolds defined by word identity and other linguistic features. Under both "predic- tive" (i.e., with [MASK] tokens) and "contextualized" (i.e., the real sentence) settings, the authors analyze how well the manifolds can be separated from each other.
While these pioneering works offer some useful insight, their starting points are generally "token- centric", i.e., the analysis focuses on how the context affects the representation of tokens. Therefore, very few studies touch upon another critical problem, i.e., for contextualized word embedding meth- ods, how exactly does the model represent the context itself. In this study, to fill in this gap, introduce a novel perspective to analyze the representation of context. To give rigorous discussion, first provide set of mathematical definitions by formulating the language representation prob-

---

Under review as a conference paper at ICLR 2022
lem using category theory. Then, inspired by manifold learning, we provide two analysis scenarios and also methods to probe the representation of contexts. As a realization of these theoretical analy- ses, we further provide empirical results and cases studies, which give more insights into the current contextual representation models. A particularly interesting example can be the re-thinking of the functions of Multi-head attention and Feed Forward Network (FFN) in Transform-based model.
MOTIVATION AND CONTRIBUTIONS
This work tackles untouched fundamental problem in language representation learning, i.c., how a representation model represent different contexts. Therefore, provides a novel perspective to re-think and characterize these models.
In a contextualized representation model, the representation at a specific sequence position is deter- mined by both the token at that place and the surrounding context. However, most of the attention has been paid to one direction, i.e., understanding how the vector represents the token, leaving the context part largely unexplored. Intuitively, we argue that the effect of different context conditions (termed as observed condition below) should be semantically related. This point, however, is not explicitly modeled in the current representation scheme, which motivates us develop this study.
Ideally, representation scheme of context should have the following two features, Firstly, it should be compatible with the word representation mentioned above. Secondly, the effect of the central token and observed condition should be disentangled, which makes the latter focusing on how dif- ferent they are in affecting the central word's semantics. To achieve these goals, we design the following representational scheme. Firstly, for each observed condition, we fix the context, mutate the central token, and record how much is changed about the representation vector of the central token. This generates collection of vectors whose each element corresponds to a contextualized representation. We hope this kind of collection reflects how each observed condition affects the semantics of the different central tokens, and comparing different collections reflect how different contexts affect the semantics. On the other hand, as mentioned above, to disentangle the effect of observed condition and central token, study the nature of the collection, other than its elements.
However, characterizing this kind of information is challenging. Obviously, taking the collection of mutated sequences (and the associated vectors) as a would miss lot of information among them. Therefore, we propose a systematic and rigorous mathematical formulation based on category theory which states clearly the representation of context should reflect both the set of representations of the substituted central tokens as well as how these sets of word representations can transit to each other. In particular, we introduce the concept "morphism" to characterize the mappings between different token substitution sets (corresponding to the semantic effects of different observed condi- tions). Furthermore, with of help of manifold learning, we also develop tool set to analyze the aforementioned information experimentally, from either a functional or topological perspective.
Besides the above contributions, our empirical results, enabled by these theoretical foundations, also generate several interesting findings. First, our analysis of context around [CLS] token suggests that high-confidence correct predictions tend to occur when contexts show robustness modulating the semantics of the central token as well as consistency of the modulating effect across different samples. Last but not least, our dissection of the attention and FFN layers identifies game between them when information gets through deep model, and their final agreement actually supports hypothesis that they both of them try to fit an identity morphism that reflects bona fide linguistic associations.
PRELIMINARY: CONTEXTUAL LANGUAGE REPRESENTATION
Following the distributional hypothesis (Hariss) 1954,, the output vector of representational model (denoted by G) represents the probability of the appearance of certain tokens (i.c., context) around given token (i.e., central token æ), i.e., p(conteat|z)) Note that here context random variable, whose formal definition will be given in the next section.
In contextualized language modeling (Peters et al., 2018), the distribution of context is conditioned not only on the central token but also on what really appears around the central token in each sample (e.g., the unmasked tokens given sentence). In this sense, we introduce observed condition C to
2

---

Under review as a conference paper at ICLR 2022
denote the observed context. That is say, observed condition C corresponds to an actual value of random variable context given a specific sample, which makes the modeling target of contextualized language modeling be pl p(contezt( 2)).
In this work, we propose to represent each observed condition C' by inspecting how the model represents various central tokens given this condition, which gives the following collection of vector, i.e,
{G(C,z)|a E Vocab}
(1)
where Vocab is the of distinct tokens used by he tokenizer. However, the definition of set only provides rough description of the nature of contexts, which motivates as to introduce category for rigorous definition, as described below.
4 CATEGORY THEORY FORMULATION AND NOTATIONS
Mondelli
Mirpfon 1822
Representstion
Movsism
Sample
Representation
Figure I:
Illustration of the proposed representation scheme. For each observed condition, propose to characterize its semantic modulation effect by a collection F sentences with different the central tokens. Then each substituted sequence can be represented by a language model, we can also formulate a category definition for vectors, which be further characterized by manifold analysis. Therefore, we can use the properties of manifold M to represent the properties of collection F.
category theory to formulate the properties of this collection. As
As mentioned in Section 2, in problem formulation, "context" is random variable. So here we first give the encoding of each observed condition (i.e., an actually appearing context) using the formal language of probability theory (Stroock 2010). Specifically, in BERT-style language model, we use a three-dimensional tensor (corresponding to token species, position and type, respectively) define the sample space and take the o-fiedd of the sample space as the event space. Note that to describe the collection of probability distribution of different central tokens, we introduce a discrete random variable set K {k: and the distribution of each k2 image corresponds to the distribution of context given a certain central token (Note that i is also defined as triplet of species, position and type). Concretely, we have:
Definition 4.1. Let K be finite discrete set whose each element K; Ω + E is a random variable that projects from the discrete sample space of tokens Ω E N to its o-field E, where N set to three, corresponding to token species, token type and token position, respectively. The cardinal number K is marked as n.
Therefore, each element k; indicates the presence of certain token (with defined species, type and position) as the central token that accompanies the context. Following the principle of distributional hypothesis, the representation of each central token represents the probability density function of the context, which gives:
Definition 4.2 An observed condition C. is encoded set F(C)) whose each element can be expressed as a probability density function fx. conditioning on the observed condition C, i.e., F(C) {sk, (context elC, context E E)li E 2).
3

---

Under review as a conference paper at ICLR 2022
Ultimately, one expects the representation of observed conditions to reveal comprehensive char- acterization that transfers well to a general scope of downstream applications. In particular, the following two features are considered here, 1) for each observed condition, how it can affect the semantics of the central token, and 2) how different observed conditions modulate the semantics differently. This intuition is reminiscent of category theory that explicitly study the objects and their mappings (i.e.moophisms
Definition 4.3 (Category of observed conditions) To define the objects in a category of observed conditions, we have
ob(F) (F(C)IC E}
which contains the probability density of all possible context events. Naturally, we can define an abstract collection of morphisms that map from one object to another, i.e.
hom (F)
[hhom(FF) F2) F3 E ob(R)),
where
hi, Fi, (Ti F, - P,lTssSeeX = e/Ci, -elie,
e E)),K
Note that through the above definition, introduce several constrains that reflect the nature of language. First, for each set of morphism between two objects, there only exists one morphism, assuming that the model should learn capture one genuine relationship between two observed conditions. Second, this morphism is a bijection defined on F (note the subscript k in Eq (4)) so each probability density of context should be projected token and vice versa (i.e., this morphism would not cause semantic collision when central tokens are different). We will discuss how our experimental results support these constrains later.
unique partner with the same central
After the definition of categories of observed conditions, we move on to its representation. Note that each observed condition corresponds token substitution set (Definition 4.2), and model G is trained to perform injection from each probability density function in the set to representation vector, i.e., G(C) F -> M, where M consists of vectors in a d-dimensional Fuclidean space. Therefore, we can similarly define the category of vectors as the representation of category of ob- served conditions. What different here is we how have to take different representation models into consideration, which gives the following definition:
= Definition 4.4 (Category vector representations) Suppose we have a set of models: Gset [Gm}, the objects of category of vector representations can be defined
ob(M) = (Mc....
(5)
where
Mc.G.. (Vilk E K, Gm E Gset, V =
(6)
The definition of morphism collections also considers the difference in model, which gives:
hom(M)= =
Mcja, E {bb())
(7)
where
hom (M,.G,,MM,,.... D, Mc, + Mc,G..
(8)
Do,a..-9,00. (G C
and C2 and C stand for two different observed conditions. Given the above definitions, we then formulate the representation of context by a functor L from this category M to category F, i...
L(,)(C;)) F(C:),
(10)
L(Dc.Gm-ACGG.) = Túj, DDc.VDD........ -C,G. E homm(Mc..... Mc,..),,
(11)
where Gm, Gn E Gset-
UNDERSTANDING REPRESENTATION OF CONTEXT UNDER THE CATEGORICAL FRAMEWORK
In this section, we will explain how to analyze the representation of category with given by cate- gory MI output by the word representation model G.
4

---

Under review as a conference paper at ICLR 2022
5.1 REPRESENTING OBJECTS OF CATEGORY F
Given a certain observed condition, Eq. (T)) defines a object-to-object mapping of the vector rep- resentations given by G to set F of probability density functions. Therefore, the nature of each object in category F' can be analyzed by converting each sequence in the token substitution set to vector, which is consistent with the common usage of word representation models. Note that the definition of M does not require model G to be determined, so our formulation includes a collection of representations potentially given by different G.
5.2 REPRESENTING MORPHISMS CATEGORY F
Apart from the "token-centric" understanding of contextual language representation, this work em- phasizes the concept that the learned functor L should also represent the morphisms (each mor- phism is corresponds to a transition from one object to another) from category M- to category F, i.e., the mapping of inter-object structure between the two categories. More specifically, the definition given by Eq.(4) and E indicates both the learned model G and observed condition C' play a role in the representation of morphisms. Here, we propose to analyze the two factors side-by-side in the following two scenarios. Scenario A. The representation of different observed conditions when the representation model G is fixed (m n in Eq.( This scenario characterizes how the morphism between any two objects in category M. (i.e., two collections of representation vectors) represent their counterparts in category F, with the help of the learned representation model G. This scenario reflects how the model distinguishes different observed conditions in modulating the semantics of central tokens, by taking into account that this difference should be represented by some relation- ship of the whole collection of representation vectors (not any single vector alone). Scenario B. The representation of a certain observed condition C' given different representation models. This scenario extends the understanding of the representation model from a static view (i.e., the model parameters are already learned), to a dynamic view (i.e., the model can change either in configu- ration or parameter). This gives more implication of our theoretical perspective in dissecting the model components or the learning process. In this work, we select the first angle. By treating the output given by each layer of Transformers model as outputs from different models, we can actually gain more insights into how the representation of context evolves across the different layers of a pre-trained language model. Considering the constraints introduced Definition 4.3, differ- ent models should all attempt to represent the same identity morphism, i.e., id7.F-.F. This makes sense correspondence with the following semantics understanding: The relationship of semantic functions between two observed conditions should be determined by the language itself but not the model. Keeping this mind, we can measure the consistency of morphism representation across cach layer in pre-trained model probe for the model's behavior.
EVALUATING CHANGES IN REPRESENTATION OF CONTEXT: A MANIFOLD PERSPECTIVE
The aforementioned reasoning indicates the necessity of characterizing the representation of mor- phisms as a complement to the characterization of representation vectors alone. Therefore, here we focuses on how morphisms in category M reflect the nature of category F. More specifically, the elements of the manifolds are obtained following the protocol below, in the spirit of distributional hypothesis: Given central word æ appearing with real-world context C, we substitute x with all possible tokens in the vocabulary of the language model (Note that for the sake of computation effi- ciency, this sampling strategy fixes the dimension of position embedding in the real context). The collection of sampled elements denoted as a token substitution set. To facilitate the characteriza- tion of structural information, following the manifold assumption (Joliffe. 2011). we hypothesize that the vector representation of all the probability distributions corresponding to different central tokens lie low-dimensional manifold in the d-dimensional Fuclidean space. With this approx- imation, while the exact solution of morphisms D is still unsolved, we can leverage comparison of manifold properties to study the behavior of D) empirically, i.c..
D Rd R Rax==
(12)
5

---

Under review as a conference paper at ICLR 2022
where D is the change in manifold property that is induced by D, Q is certain characterization of the manifold. In this work, we propose two specific forms of Q: (1) a task- projection of the manifold, and (2) the topology output by manifold learning algorithm.
Functional characterization When comparing two contexts, straightforward way is to project each representation vector task-related score. practice, one can choose linear head that is either tre-trained along with the representation model, or fine-tuned for a specific downstream task. Then we can use statisties of the task-related scores to analyze the behaviour of each manifold.
Topological characterization To further characterize the topology of the proposed manifold, can also leverage the toolset from manifold learning community to reveal the local features. In practice, we perform an inspection of the existence of topological associations in the local region. Particularly, here we borrow the method for topology determination from a prominent manifold learning algorithm, i.c., UMAP (McInnes & Healy 2018). The local topology determined among k-nearest neighbors, and represented by fuzzy set to indicate whether an edge between nodes exists. Please refer the original UMAP paper for a more detailed description. Note that as UMAP does not offer an estimation in global geodesic distances for arbitrary two elements (it focuses on local topology), our analysis also does not include this part. Nevertheless, we show below that the local information alone can provide some interesting insights for language modeling.
7
EXPERIMENTS
7.1 MODELS AND SETTINGS
In this study, to provide insights originating from our new understanding, we analyze the word representation of a widely used contextualized embedding model, i.e., BERT (Devlin et al. 2019). The implementation, feature extraction and pre-trained model ("Bert-base-uncased") are all based on the Transformers package
Our analysis involves two datasets. Wikitext-2 (Merity et al. 2017) is used as a representative of pre-training corpus, while SST-2 (Socher et al. 2013), one of the GLUE dataset (Wang al. 2018). is used as a typical downstream task (i.e., sentiment analysis of movie reviews) dataset. Detailed statistics of data used in the analysis will be provided in the corresponding sections. To fine-tune the model for the SST-2 task, we use a learning rate of 2e-5, batch size of 32, and epoch number of 3.
For the topology analysis using UMAP, we keep the original hyperparameter of kNN as 15.
In the two subsections below, we provide some empirical results corresponding to the two scenarios defined in Section 5.
7.2 UNDERSTANDING DIFFERENT REPRESENTATION OF CONTEXTS FROM A PRE-TRAINED MODEL.
Star graph as a unit of analysis. To give a straightforward impression of the topology of a context manifold, here we first list an example (Figure 2(a)). The example context comes from Wikitext-2. Here, we visualize the connections of the real context to its neighbors, as determined by the local fuzzy set as UMAP, using the output of the BERT final layer. Note that each context here is associated with a set of substitutions of the original token to other tokens. By treating the original sentence as the center, the topological evaluation can conducted by examining the star-graph. Note that while connections), the results Section 7.3 still reveal some interesting insights into the representational model. We leave the exploration of more complicated topology for future work.
analysis of topology only involves this star graph (i.c., omitting other potential
How contexts modulate the semantics: case study of [CLS]. To provide a more task-related perspective, we also study how contexts are represented for the special token [CLS], whose repre- sentation is often used as input for natural language classification. Intuitively, as [CLS] itself has little semantics, the linguistic feature of the whose sentence should all come from each observed condition. In correspondence to our formulation, the token substitution set's representations are

---

Under review as a conference paper at ICLR 2022
RA wetting consdeee crlles greates of HIN,, and F.S.P.ee matee dinoe, poo prespeedd langa langagge-.
<MREv5.45 copey/2.2
chatogee DO
VIVTN 50
(a)
Latel
negathue
500
Label
nagar
DesiNe
D104
CLD8
DIS
600
0.36
0.12
0.10
200
0G
Magative Score
Nogadie Store
(b)
(c)
Figure 2: (a) An example star graph where each element in the token substitution set has a prob- ability of connecting the real context, determined by the local fuzzy set in UMAP. (b) The effect of topology of [CLS] token before fine-tuning, as evaluated by functional characterization. (c) The effect of topology of [CLS] token after fine-tuning, as evaluated by functional characterization. For (b) and (c), the radius of a dot denotes the standard deviation caused by local topology. The larger the radius is, the larger the standard deviation will be.
projected, together with the [CLS] token's 's representation itself, by the linear classification layer that learned during fine-tune tue-ttunng. Given an input (original or substituted), the linear layer serves projection from the last representational layer to a scalar value to indicate the classification result.
In light of this understanding, we plot the classification score (for the negative class) of each sam- ple (denoted by the dots), together with the standard deviation (denoted by the radius of dots) of scores when using the substituted contexts as input. The results from pre-trained (a linear classifier on the top of frozen BERT encoder) and fine-tuned representation model is plot in Figure 2(b) and Figure 2(c), respectively. In this experiment, we leverage two features of a sample to conduct the analysis, i.e., the classification score of the original sentence beginning with [CLS] and the standard deviation of classification scores of all the elements in the token substitution set. In particular, the classification score indicates the reskerel (a binary sentiment analysis) semantics of the sample. And the standard deviation represents the task-related magnitude of semantic shifts across the token substitution set, which simplifies the comparison between different sets the comparison of dif- ferent standard deviation values. We then show how we analyze the representation of objects and morphisms of context category IF using these two features.
The analysis of the representation of objects (i.e., M) involves the inspection of each token substitu- tion set. Intuitively, as [CLS] itself has little semantics, the linguistic feature of the whose sentence should all come from each observed condition. Therefore, better representation of the token sub- stitution set should have smaller standard deviation in classification scores, as all the elements in the set share the same observed condition. This is quite the case in both the pre-trained and the fine- tuned model, as we can see the better classified samples (i.e., score close to or 1) have a smaller standard deviation in classification score.
Based on our intuition discussed in Section 6, we rely on the comparison of functional characteriza- tion results evaluate the properties of morphism representation D. Specifically, in this experiment, the comparison is made between the standard deviation of classification scores across different to- ken substitution sets output by given representation model. Note that each morphism between two objects reflects the difference the two corresponding observed conditions in modulating the semantics of the central tokens. In experiment, this is manifested by the discrepancy in the stan-

---

Under review as a conference paper at ICLR 2022
dard deviation of classification scores between two samples. Interestingly, we find that samples with similar margin to the classification boundary tend to have similar standard deviation (regardless of the specific score value, in contrast to the aforementioned case of object representation). That is to say, our experiment results show that samples with similar prediction confidence are also sim- ilar in quality of morphism representation D, which is decoupled from the specific semantics of each [CLS].
When comparing the pre-trained the fine-tuned setting, we can observe an interesting point for the fine-tuned model. After fine-tuning, samples near the classification boundary are associated with more significantly enlarged standard deviation in confidence score, which means these ob- served conditions fail to generate consistent task-related modulation to different central tokens. This contrary of high confident samples, which shows small standard deviation along with an almost absolute prediction score. Altogether, our results give an intriguing hint that high-confidence correct predictions tend to occur when the samples' representation of context is good in terms of both object (showing a smaller standard deviation) and morphism (consistency in standard deviation change), which validates the linguistic relevance of our propose theoretical perspec- tive. Note that on the y-axis, we plot the sample id, which shows association with the mean or variance, further verifying the effect is a characteristic of [CLS] token but not any specific context.
7.3 EVOLUTION OF CONTEXT REPRESENTATION TIRRUGII A TRANSFORMER MODEL
1.0
3.5
0.
244
0.4
Laye Type
16
Philt-hadd attenton Ponkklemani
25 0.
(a)
(b)
(c)
Figure 3: (a) The disappearance of neighbors in the manifold given by the first layer. (b) The emergence of neighbors in the manifold given by the last layer. The y-axis is the average probability of connection and the band width is the variance. Note that we label the embedding layer as the layer 0, the last FFN layer as the layer 24 and separately count the attention and FFN layers in one Transformer layer. Therefore, layer with odd number index is attention layer and layer with even number index is an FFN layer. (c) The IoU trend between different layers. Each IoU value is calculated with last layer last layer of the same type. The results of tokens with different POS can be found in the Appendix.
In this analysis, our focus is to characterize how the context manifold from each layer of a pre-trained Transformers model differs from each other. As this does not consider any specific downstream tasks, we use the test set of Wikitext-2 dataset, which mimics a pre- tre-iraining corpus of BERT. For the visualization purpose, we sample the 10 most frequent tokens from each of the three parts of speech (i.e., noun, verb and adjective) respectively. For each token as the token of interest, we further sample 10 samples and calculate the topology around each context.
Upper layers overwrite the context manifold topology given by lower layers. Considering the information flow in Transfomer-based model, each token is gradually exposed to more "integrated" information when layers move forward. When considering the set of representation vectors alone, this can be understood as the addition (with attention weights) of other token's 's vector to the vector of the central token. Here, we propose to analyze this change on a context-level. i.e., how the association among the context neighborhoods reflects morphisms between semantics in the discrete language space. Firstly, we examine the trajectories of evolution for the nearest neighbors appearing in the first layer (i.e., the static embedding layer) and the last layer (i.e., the output layer) (Figure 3(a) and 3(b)). This is done by calculating the average probability for the existence of an edge that connects to the real context based how the tokens rank in the first or last layer. Clearly, observe the gradual disappearance (loss of connection) of the neighbors in the embedding layer, as well as the emergence of new, sample-aware neighbors in the last layer. This result further validate
8

---

Under review as a conference paper at ICLR 2022
the relevance of "context manifold" in language modeling. The corresponding results for other layers can be found in Appendix A.1.
Representation learning converges when the game between Multi-head attention and Feed- Forward Networks ends. When inspecting the manifold change for specific samples, we realize an interesting balance between the attention layers and FFN layers. By statistics, we also plot the average results here (Figure 3(c)). In this experiment, we try to reveal how neighbors given by dif- ferent layers are similar with each other, measured by intersection-over-umee line possessing the sequential comparison of all layers, i.e., each layer has IoU calculated with the last layer. Interestingly, while the consistency of layers generally goes up as layers get higher, the IoUs show trend of "zigzag" especially during the first and upper half of layers. This result means the attention and FFN layers tend to give different results and the model tries to get an agreement between these two layer types, i.c., the final representation of context is achieved by co-opetition relationship of the attention and FFN modules. Inspired by this understanding, we also calculate the loUs solely between attention or FFN outputs (the first embedding layer serves as the starting point for both series). These results further reveal some detailed insights among the Transformer layers. At the first several layers, the results between attention and FFN layers pos- sess higher similarity than results between them, verifying the existence of debate. Also, as the attention layers generally have a higher consistency, a natural understanding is they tend to inject some uniform information from the whole input sequence, while the FFN layers tend to eliminate the information they don't It agree (as they contain ReLU non-linearity)- At the upper layers, in con- trast, the overall consistency between attention and FFN layers surpasses the consistency between the layers with the same type, suggesting the discrepancy in the lower layers actually converges to a better agreement. The above observations actually support our previous hypothesis that differ- ent layers in model (treated as different G here) learns to achieve an identity morphism in F. which reflects the genuine linguistic association between the probability distributions of different contexts (This result holds true for tokens from all the three parts of speech we samples as shown in the Appendix A.2). In other words, topological consistency between the attention layers and the FFN layers is necessary condition of model convergence. In addition, we also notice that the last FFN layer seems to decrease the overall consistency lot, which we speculate is due to the direct connection to linear head (Ma et al. 2019).
(loU). First we have
RELATED WORK
The understanding of contextual word embedding is a field gaining more and more attention (Rogers) jet al. 22000 Hao et al. 2020 van Aken et al. (019). However, it is generally regarded as a chal- lenging task considering the complex interplay between different words. One line of researches proposes to train a probing network on the top of pre-trained word embedding, so as to find some relationship between the learned representation and the linguistic features such as syntactic tree (He-) witt & Manning 2019; Coenen et al.) 2019; Miaschi & Dell' Orletta 2020). The process can also be realized in parameter-free fashion (Wu et al. (020)). Another line of research in this field follows the basic idea of information geometry. As each token is associated with a vector, analysis of the clustering of these vectors can generate insightful findings, such as the geometry of polysemous words (Coenen et al. 2019). isotropy in embedding space (Ethayarajh) 2019; Cai et al. 2021: Ra- jaee & Pilehvar 2021). Moreover, there are works that utilize the geometric information for better model performance, possibly through refinement (Chu et al.) 2019 Hasan & Curry 2017). While our method borrows some practice from information geometry, this work is fundamentally different from the above in the way that we try to characterize the nature of contexts, while these studies try to understand how a representation vector reflects the nature of a token.
9 CONCLUSIONS
In this work, we propose a novel perspective to understand the contextualized word embedding, with emphasis the context rather than the central token. With the mathematical formulation, analysis method development, and empirical results, we hope our study can offer the community some new insights into the popular pre-trained language models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In ICLR, 2021.
Yonghe Chu, Hongfei Lin, Liang Yang. Yufeng Diao, Shaowu Zhang, and Xiaochao Fan. Refining word representations by manifold learning. In IJCAI, 2019.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda B. Viégas, and Martin Wattenberg. Visualizing and measuring the geometry of bert. In NeurlPS, 2019.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geom- etry of bert, elmo, and gpt-2 embeddings. ArXiv, abs/1999.00011 2019.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Investigating learning dynamics of bert fine-tuning. In AACL, 2020.
Zellig S. Harris.
Distributional structure. WORD, 10: 146-162, 1954.
Souleiman Hasan and Edward Curry. Word re-embedding via manifold dimensionality retention. In EMNLP, 2017.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax word representa- tions. In NAACL, 2019.
Ian T. Jolliffe. Principal component analysis. In International Encyclopedia of Statistical Science, 2011.
Xiaofei Ma, Zhiguo Wang, Patrick Ng. Ramesh Nallapati, and Bing Xiang. Universal text represen- tation from bert: An empirical study. ArXiv, abs/1910.07973, 2019.
Jonathan Mamou, Hang Le, Miguel Angel del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Yue Chung. Emergence of separable manifolds in deep language representations. ArXiv, Abs/2006.01109, 2020.
Leland McInnes and John Healy. Umap: Uniform manifold approximation and projection for di- mension reduction. ArXiv, abs/1802.03326. 2018
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ArXiv, (bss/600.00843... 2017.
Alessio Miaschi and F. Orletta,. Contextual and non-contextual word embeddings: an in-depth linguistic investigation. In REPLANLP, 2020.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. ICLR, 2013.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark. Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACI, 2018.
10

---

Under review as a conference paper at ICLR 2022
S. Rajace and Mohammad Taher Pilehvar. A cluster-based approach for improving isotropy in contextual embedding space. In ACL/JCNLP, 2021.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment tree- bank. In EMNLP, 2013.
Daniel W. Stroock. Probability theory, an analytic view. 2010.
Betry van Aken, Benjamin Winter, Alexander Löser, and Felix A. Gers. How does bert answer questions?: layer-wise analysis of transformer representations. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. ArXiv, abs/1 0os/1804.07411, 2018.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: farameter-free probing for analyzing and interpreting bert. ACL, 2020.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 THE OVERWRITING OF LOCAL TOPOLOGY IN DIFFERENT TRANSFORMER LAYERS
Figure 4: The overwriting of local topology in different Transformer layers. Note that we treat the embedding layer the layer and separately count the attention and FFN layers in Transformer layer. Therefore, a layer with odd number index is an attention layer and a layer with even number index an FFN layer. We record changing trajectory of the top 10 neighbors of the real context at each layer.
12

---

Under review as a conference paper at ICLR 2022
A.2
THE GAME BETWEEN THE MULTIHEAD ATTENTION LAYERS AND FFN LAYERS
0.8
0.7
0.6
0.6
Layer Type
Layer Type
0.5
layers
layers
Multi-head attention
Multi-heaa attention
Feed forward network
Feed forward network
0.4
10
15
20
25
Layer ID
Loyer ID
(a) All
(b) Adjective
0.7
0.7
8
0.6
0.6
Layer Type
Layer Type
0.5
layers
layors
Multi-hooo hhalt--head attention
Multi attention
0.4
Feed-forwan network
Feed-forward network
20
Layer ID
Layer ID
(c) Noun
(d) Verb
Figure 5: The IoU trend results of different parts of speech. (a) All. (b) Adjective. (C) Noun. (d) Verb. We sample the top most frequent tokens of each POS and get the average score.
13


Reviews:
[{"review_id": "zgag91rhFOt", "reviewer": "Reviewer_BMjD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed approach is interesting and suggests an alternative viewpoint to understanding contextualization in language models. \n\nWeaknesses:\n- The empirical results do not seem to rely substantially on the methodological contributions. In particular, experiments are performed with local probing (UMAP) over a small number of neighbors (15) to determine manifolds compared with a general functor over manifolds presented in Sections 4-6. This form of analysis is quite similar to that performed in Cai et al. 2021, except over contexts rather than tokens. Additionally, the layerwise analysis is quite similar to that performed by Aken et al. 2019 CIKM in the context of question answering.\n- The paper is a bit hard to follow, especially the description of input and output categories in Section 4. Figure 1 is helpful for this.\n- The experiments are somewhat limited and, as mentioned previously, similar to those previously conducted in prior works except over contexts rather than the central word.\n- For the [CLS] experiments, it seems straightforward that the functional characterization is mixed before fine-tuning and concentrated after finetuning as the [CLS] token is rarely used in masked language modeling (at least for the BERT formulation of this objective). \n", "summary_of_the_paper": "This paper proposes a framework, using concepts from category theory, to analyze the mechanisms through which Transformer based language models perform contextualize representations. In contrast to prior work, which analyzes contextualization of representations through “token-centric” probing, the authors first cast the language representation problem as a functor from an input category (encoding of observed context) to an output category (d-dimensional space). Next, a manifold learning approach is analyzed in the case of a fixed representation model and treating the outputs at each layer as a different representer. Lastly, the authors perform a range of experiments to better understand various aspects in transformer models including how contexts are represented for [CLS] and the change in context representations by layer.", "summary_of_the_review": "Approach for modeling contextualization in language models using tools from category theory and manifold learning. Interesting approach but there is a gap between the methodology and the experimentation. Additionally, experimentation is similar to studies conducted previously with a focus on substitutions in context rather than center token. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qrohqIINOan", "reviewer": "Reviewer_3Rbc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper intends to understand “how does a language representation model represent contexts” which is different from a more popular direction of understanding “how does the context affect the representation of the tokens”. The paper formulates the contextualized language representation problem using category theory. Authors claim that previous works that focus on substituting tokens by keeping the context fixed miss a lot of useful information. According to their proposed framework, representation of the context contains information about the representation of tokens that get substituted while keeping the surrounding conditions unchanged as well as information about the transitions (morphisms) in the representations as a result of these substitutions. However, quantifying the above information is challenging, and this paper intends to analyze this information from a functional (task-related projection) and topological (UMAP) perspective.\n\nBefore you read further, it is important to mention that I do not have a background in category theory or Manifold learning so I have judged this paper purely on the basis of convincingness of the arguments, quality of experiments, and interpretation of the results.\n\nOverall, I think that paper identifies a very niche problem and provides interesting ideas about how we can think about representation of the contexts in language modelling. But in the same light, I do not think that the paper is well-written to be published at this conference. The paper needs serious revamping in terms of story, writing style and presentation. \n\nIn terms of the story: \n\n- The main takeaway of the paper remains unclear to me for most of the paper as the authors were not precise with communicating their intentions. For instance, the paper’s abstract and various other parts indicate that the work would explain “how a model represents context itself,” whereas, in the motivation, the paper states that the authors are trying to tackle “how a representation model learns to represent contexts”. I believe that these two statements are not identical. The latter claim is about how context emerges in the model when learning happens during training. In contrast, the former is about analyzing how the context is represented once the learning is complete. The authors should state both of these clearly if they are trying to answer both the questions.\n\n- In the abstract, the paper claims that the work would shed light on the improvements in Transformer-based language representation models. However, the paper does not mention how it plans to do so in the main text. Additionally, I do not understand the nature of improvements the authors are referring to.\n\nIn terms of the writing style, there are several places where the authors need to rephrase sentences or correct grammatical errors. To list some:\n- In early days, word embeddings ~~are~~ were static, i.e., the representation is was solely determined by a word’s identity…\n- GPT ~~revolutionize~~ revolutionized this technique\n- When compared to static embedding, ~~mechanistic~~  (use a better word here?) studies of the contextual embedding generally lag behind.\n- They also point out **that** the embedding space is generally anisotropic. Also what do you mean by anisotropic? A reader without a manifold learning background is likely to phase out after coming across such terms if they are not explained succinctly. \n- On the other hand, several works try to uncover how contexts influence the geometry of embedding space. (Missing references)\n- Therefore, very few studies touch (touch upon?) another critical problem, i.e.,.\n- untouched fundamental problem in deep language learning. I am reading the term **deep language learning** for the first time. Can you please clarify it a bit more?\n- The ~~fine-turning~~ fine-tuning process …\n- FFN layers identifies a ~~Game~~ game between…\n- hypothesis that they both of them (not sure what you meant here) try to fit an identity morphism that reflects bona fide linguistic associations.\n- I believe that the readers will benefit from detailed explanations of the terms presented in the mathematical equations. For instance, equations (6)-(11) have been introduced together in Definition 4.4 without much explanation as to what each of it intends to convey.\n\nIn terms of the presentation:\n- It seems as if Figure 1 is an important cue for the readers to understand their framework but it lacks descriptive captions or a reference in the main text that could explain the idea succinctly unlike Figure 2 and Figure 3.\n- The graphs in Figure 3 should somehow indicate that if Layer i is Feed-forward network (FFN) then Layer (i+1) is Multi-head attention. Make it clear in the caption as well.\n\nThe authors test their method on just one of the GLUE classification dataset, SST-2, using Wikitext-2 as a pre-training corpus. I believe that the authors should include more datasets and discussion to make the experiment results more convincing. There are other single sentence classification datasets in GLUE such as CoLA, and other inference based classification tasks such as MNLI (https://openreview.net/pdf?id=rJ4km2R5t7).\n\nAfter multiple parses through your experiments section, I am unclear if the authors perform token substitution around [CLS] token or they replace [CLS] token itself. If I think about this experiment from Scenario A’s (Section 5.2) and Figure 2 perspective, it suggests that the authors intend to convey the former. However, the authors mention  “when replacing the [CLS] tokens with other tokens, the representation of the whole sentence does not change a lot” which aligns with the later understanding. The authors should clarify what they actually intended to do in this experiment. The results from the experiment suggest that finetuning helps in learning new patterns and pre-training patterns are forgotten which is not that surprising based on our current understanding of fine tuning. \n\nI found the authors’ observations on the conflict between Multi-head attention and Feedforward network quite interesting. I believe that the authors should analyze this result and conduct more experiments around this idea on more datasets. It would be interesting to see if the same observation holds for other pretraining corpuses as well which extend beyond English as well. It might be even the case that the observations the authors made were a result of the quirks (linguistic properties) of English rather than something being innate to attention-based language modeling. ", "summary_of_the_paper": "This paper intends to understand “how does a language representation model represent contexts” which is different from a more popular direction of understanding “how does the context affect the representation of the tokens”. The paper formulates the contextualized language representation problem using category theory. Authors claim that previous works that focus on substituting tokens by keeping the context fixed miss a lot of useful information. According to their proposed framework, representation of the context contains information about the representation of tokens that get substituted while keeping the surrounding conditions unchanged as well as information about the transitions (morphisms) in the representations as a result of these substitutions. However, quantifying the above information is challenging, and this paper intends to analyze this information from a functional (task-related projection) and topological (UMAP) perspective.", "summary_of_the_review": "It is important to mention that I do not have a background in category theory or manifold learning so I have judged this paper purely on the basis of convincingness of the arguments, quality of experiments, and interpretation of the results.\nOverall, I think that paper identifies a very niche problem and provides interesting ideas about how we can think about representation of the contexts in language modelling. But in the same light, I do not think that the paper is well-written and rigorous enough to be published at this conference. The paper needs serious revamping in terms of story, writing style and presentation. I provide detailed explanation of my concerns in the main review. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qLXqNgwKSoQ", "reviewer": "Reviewer_K1ke", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tries to build a theory about how context is represented in pretrained language models. This is indeed a meaningful direction as current literature, although extensive, does not contain targeted studies about context. However, this paper is severely flawed in the following aspects: \n\n## Basics of probability and pretrained LMs\n\nEquation 1: \n\n- a masked language model outputs the probability of p(masked word | all other tokens in the sentence), NOT p(context |  C, w).\n- what does the word *context* mean in eq.1? It seems to be a random variable since it has a probability. So is it a discrete random variable, or a continuous random variable, or a sequence/set of discrete/continuous random variables?\n- what does *C* mean? The paper says it is the observed unmasked tokens but isn't this part of the context? My understanding is that the *context* of a given masked word is all other words within the sentence (either other words are masked or not, as recall 15% words are masked in BERT).\n- what does the symbol ~ mean in Eq.1? Does it read \"follows this distribution\"? If it is, does it induce that the symbol G(C, w) is a random variable? The paper says G is \"a contextual language representation\", but what is it? Is it a sequence of output vectors of the last layer of BERT? If they are, they are deterministic (not random variables), so how can they have distribution?\n\nThe term \"observed condition C\"\n\n- If it means \"observed unmasked tokens\", why re-name it as \"observed condition\", since the former is apparently more understandable?\n\nEquation 2:\n\n- Again, what does the symbol ~ read? Does it mean \"follows the distribution of\"? If yes why the right hand side is a set, but not a probability distribution?\n\nA basic rule about writing equations and symbols is to always define them properly before using them. In this case, I strongly encourage the authors to discuss the definitions of symbols in their equations. In addition to the above questions, the authors may also want to discuss \n\n- G(C, w), is this a function G applied to variable C and w? If yes what is the domain and what is the image?\n- Similarly, for P(C), is P a function applied to a variable C? is C continuous or discrete? is C a scalar or a vector?\n\n## The definitions of symbols are unclear and without definition or explanation, equations are not explained\n\nThere are a lot of concepts directly used without explanation or reference to the background. To name a few: \n\n- Definition 4.1, what is the use of the sigma-filed E? Is it the vocabulary? If yes why not directly say vocabulary?\n- Definition 4.2, what is X? Is it a word at a certain location? If yes why not directly say it is a word at a certain location? Does a context mean a set of functions takes a sentence (excepted a mask word) as the input, and output a distribution over the vocabulary (because the paper says \"A context is ... a set ... each element can be expressed as a ... function\")?\n- If a context C is a set of functions, as is in definition 4.2, why the symbol C \\in E in equation 3? Isn't E the vocabulary?\n- Why introduce all these new symbols and terminologies (like \"observation condition\") without explanation, when we already have perfectly understandable terms (like \"words\" and \"sentences\")?\n- For definitions 4.3 and 4.4, I get more lost as these are longer equations without definition and explanation. It is possible that it is only me that have difficulty reading these equations, as I am not so familiar with category theory (but I do know combinatorial categorial grammars well, more on this later), but I find it hard to read equations from 5 to 9. There are a lot of ambiguities when I try to understand, for example:\n- It seems that \"the representation of context\" and \"context\" are two different things, as this paper writes \"the representation of context as a functor\". Yet with pretrained language models, the word \"representation\" usually refers to vectors, so how do these two compare?\n- Combining definition 4.2, equation 5, and equation 11, I guess the paper is trying to say: the conditional probability of a masked word (def. 4.2) is the same (equality 5) in different pretrained language models (Gm and Gn in Eq.11) if the context is the same. Is this how equation 11 should be interpreted? If yes, does this paper consider different training datasets of different pretrained LMs?\n\n## Lack of connections between the category theory and the language model\n\nIs it possible that this paper gives a one-to-one mapping between all introduced concepts to standard terminologies in NLP? For example:\n\n- What does the term \"category\" correspond to? A context?\n- What does the term \"object\"  correspond to? A context in a specific sentence?\n- What does the term \"morphism\" correspond to? A mapping between different contexts?\n\n## Important related work is missed\n\nHow does the introduced theory compare to classical combinatorial categorial grammars? What if we build a supervised/ unsupervised CCG tagger upon a pretrained language model like Liu et. al. 2019 (In this case each word is also associated with a linguistic category)? \n\n## Insufficient experiments\n\nGenerally, the current experiments are too simple and not well-explained. Specifically,\n\n- for figure 2a, my understanding is that the authors substitute the original token to another token, then use UMAP to get the probability of the edge, but how does this compare to the original masked LM where one can directly get a distribution of possible substitution words? Why bother with UMAP and what does it tell us?\n- What new things can we learn from figure 2bc? My understanding of them is the classes are more separated after fine-tuning (if I have not missed anything), but isn't this trivial?\n\n## References\n\n- Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. NAACL 2019", "summary_of_the_paper": "This paper aims for providing a theory for understanding the context representation in pretrained language models. It presents a category theory then some analysis of representations from pretrained LMs. This inspiration is good. \n\nHowever, this paper is severely flawed in multiple aspects: the basic notations and understanding of LMs seem to be wrong; the definitions of concepts are unclear and ambiguous; many symbols are used without definition or explanation; equations are not explained; there is barely any connections between the category theory and the language model; important related work is missed, and the experiments are quite simple and insufficient. I think this paper needs significant improvements in terms of basic understanding, writing, explanation, and experiments.", "summary_of_the_review": "Although the goal of this paper (to analyze the effects of context) is good, it suffers from severe issues as mentioned above. I would recommend a re-writing, especially the basics (like what kind of variable a context is), with more motivations, explanations, and sufficient experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iZuZyJGakqH", "reviewer": "Reviewer_awgJ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is an example of trying to use some complicated theories to describe some very simple observations. \n\nThis paper does not have a significant theoretical contribution because it does not have any theorem or lemma. All of the notations and definitions are just for describing their analysis method in a very abstract way. I spent quite some time understanding what those notations mean, but realize that the complex math does not help me understand the empirical findings at all, which are the main contributions of the paper. \n\nThe paper's first two empirical findings are not very interesting to me and the authors also do not explain the impact of those findings. The third finding in Figure 3 (c) could potentially deepen our understanding of BERT, but the authors need to investigate the underlying reasons to make the finding become useful. \n\nBy the way, I think the authors should also cite this work (https://aclanthology.org/2020.aacl-main.11.pdf) (and possibly other work that also tries to align the embedding space in different layers of Transformers to analyze what language model learns). These papers show that the analysis methods used in this paper are not very novel.\n\nMinor things:\n1. ELMO -> ELMo\n2. Your notations are not consistent or not clearly explained. For example, What is E? Why e \\in E but also C \\in E? What is G_{set} = G_{m}?\n3. (7) in Equation should be right justified.\n4. Your Figure 1 is helpful. But if you really want to define those symbols for some reason, add more symbols into the figure would make your definitions easier to understand.\n5. Your scenario A corresponds to m=n in Equation (9), right? If yes, make it more clear (if you really want to define those symbols for some reason).", "summary_of_the_paper": "This paper reports several empirical findings. \n1. Replacing the [CLS] tokens after the fine-tuning does not change the output representation a lot. \n2. By using UMAP to analyze the patterns of nearest neighbors in different Transformer layers, it finds that the topological structure in layer 1 is most similar to the structure in layer 2 and less similar to that in layer 24. This shows that each Transformer layer would change the topological structure of the context embeddings.\n3. Let's denote the embeddings in the Transformer in this way (H1) -> ATT -> (A1) -> FFN -> (H2) -> ATT -> (A2) -> FFN -> (H3) ..., where (Hk) is the hidden states, ATT is the multi-head attention, (Ak) is the output of the attention layer, FFN is the feed-forward network. Figure 3 (c) shows that (Hk) and (Ak) are very different when k is small, but become similar when k is large. Compared to (Hk) and (Ak), (Hk) and (Hk-1) are more similar when k is small. Similarly, (Ak) and (Ak-1) are more similar than (Hk) and (Ak) when k is small.\n", "summary_of_the_review": "I vote for rejection because \n1. The findings are not significant. \n2. The analysis method is not novel.\n3. The paper presents a few very simple experiment results using a very complicated way, which makes the paper very hard to read. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "WD5oqb2jx5R", "reviewer": "Reviewer_ptkS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The attempt to explain and investigate how contexts are represented in the language model is interesting.\n2. The introduction of the category theory to model the connection between contexts and their representations seems novel.\n\nCons:\n1. The serious issue with this paper is its poor presentation. Many details are very vaguely presented which makes it very hard to understand the contribution of the paper. Firstly, as this paper studies how contexts are being represented in a language model, a clear definition of what is \"context\" is necessary. My guess is that the \"context\" of a word is all the remaining words in the sequence, but this is not clearly mentioned in the paper. If this is true, what is the difference between \"context\" and \"observed condition $C$\" in Section 3? Why not use some notations/formulas to explicitly define what is \"context\"? Sections 4 and 5 are very difficult to parse even after putting in lots of effort. Some important terminologies should have clear definitions (e.g., morphisms) before they can be used as the paper needs to be self-contained. Also, there should be high-level explanations/concrete examples to facilitate an easy understanding of the purpose of each definition and how they are connected to the mechanisms of contextualized representation learning. Section 7 also fails to clearly explain how the empirical findings validate the theoretical proposals.\n2. The empirical results are from only one dataset representing downstream tasks (SST-2) and are largely presented in the form of case studies. More comprehensive studies are necessary to give reliable and convincing results.\n3. There are a lot of typos throughout the paper; some examples:\nSection 2, \"understanding how the vector represent the token\" -> \"understanding how the vector represents the token\"\nSection 2, \"supports our hypothesis that they both of them try to\" -> \"supports our hypothesis that both of them try to\"\nSection 3, \"which motivates as to introduce\" -> \"which motivates us to introduce\"", "summary_of_the_paper": "This paper aims to explore how contexts are represented in a deep language model. It attempts to go beyond “token-centric” understanding in the previous contextualized representation studies and specifically tries to investigate how different contexts are being represented (e.g., when tokens are being replaced) and how the representation changes through different components (e.g., FFN and attention modules) in the language model.", "summary_of_the_review": "While the paper attempts to study an important and interesting angle in understanding how language models represent semantics, the ideas and results are presented in a very vague manner which prevents clear understanding and judgment of the contributions in the paper. It is still unclear to me how exactly are \"context\" represented in language models and what insights can be obtained from the paper, even after reading the paper several times.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a theory for understanding the context representation in pretrained language models. The strengths of the paper, as identified by reviewers, are in the importance of an attempt to explain contextualization in language models, and in the novelty of using the category theory to model the connection between contexts and their representations. However, all the reviewers identify several major weaknesses, including flawed/incoherent definitions of concepts in the proposed theory and insufficient experimental results. Although the authors' rebuttal put a great deal of effort to address raised concerns, all five reviewers agree (and provide very detailed justifications along with suggestions for improvements) that the work is not yet ready for publication.
Decision: Reject