Title: THROUGH THE LENS OF NEURAL NETWORK: ANALYZING NEURAL QA MODELS VIA QUANTIZED LATENT REPRESENTATION

Abstract: In recent years, deep learning models remain black boxes, where the decisionmaking process is still opaque to humans. In this work, we try to explore the probabilities of understanding how machine thinks when doing question-answering tasks. In general, words are represented by continuous latent representations in the neural-based QA models. Here we train the QA models with discrete latent representations, so each word in the context is also a token in the model. In this way, we can know what a word sequence in the context looks like through the lens of the QA models. We analyze the QA models trained on QuAC (Question Answering in Context) and CoQA (A Conversational Question Answering Challenge) and organize several rules the models obey when dealing with this kind of QA task. We also find that the models maintain much of the original performance after some hidden layers are quantized.

1. INTRODUCTION:

Over the past few years, more and more people have been enchanted at the power of deep neural networks; they try to apply various networks on different fields such as reading comprehension, computer vision, image recognition, speech recognition, etc. Nowadays, there are many outstanding deep learning-based models capable of yielding human-comparable results on some benchmark corpora. For instance, those leading models on SQuAD (Rajpurkar et al., 2018) and CoQA (Reddy et al., 2019) have already passed the baseline of human performance; however, their mechanisms remain black boxes for us since nobody knows how they make a decision. When models make the wrong prediction, we can merely do some fine-tuning or use more training data. Most of the time, we do not know what modifications to make for better performance.

In general, the latent representation in a deep neural network can be regarded as what a machine thinks when solving tasks. People have already begun to elaborate on the meaning of latent representation (Doshi-Velez & Kim, 2017; Montavon et al., 2018) ; however, only those deep learning researchers are able to understand and utilize the results. In this work, we seek the possibility for not only further realizing how the machine thinks but making deep neural networks much more comprehensible for all people.

In recent years, conversational question answering is found to be pretty challenging for existing deep neural networks. Given one context and a series of questions, machines sometimes should output two different answers to the same question because, in these datasets, output answers according to not only the context and question but also those questions that are asked in the previous time steps. Besides, a model should handle the co-reference issue when trying to obtain correct answers. For instance, it should know the relation between those pronouns and people, food, location, etc. In order to do so, a model has to store these kinds of information in its latent representations. We focus on analyzing the neural QA models trained on this kind of task.

In this work, we demonstrate how to visualize the original latent vectors as more meaningful representations, which can be easily understood by humans. Originally, latent representations lie in a continuous high dimensional space, and it is pretty hard for humans to know what they mean. Therefore, to simplify it, we propose a vector quantized method, which can quantize latent representations into discrete vectors. Although, in general, vector quantization is usually applied to computer vision or speech recognition rather than natural language processing, we believe that this architecture is quite suitable for visualizing the latent representations. After converting the hidden representations into discrete vectors, humans can read the articles through the lens of deep neural networks. Generally, humans are more comfortable with these discrete tokens since words and sentences in all languages are in this form.

One of the reasoning results is shown in Figure 1 . Each color represents a codeword. The rare codewords are with the red tone, while the common ones are with the blue tone. Within the same context, a machine uses different colors to represent the same word. This is because even for the same word, it should contain different meanings at different time steps. For instance, "Richie" is colored as orange in Figure 1 (a) and as blue in Figure 1(b) . By contrast, different words can use the same color, such as those dark blue tokens in Figure 1 . Note that the spans inside green boxes are the ground truth, while the ones with underline are model outputs.

After coloring all tokens, we organize several rules which machine follows based on the experiment result. First of all, we have found that for these deep neural networks, they do not need that a massive size of codebook to achieve good performance. On average, a neural network can use only 20 or so vectors to cover almost all information. This striking fact is based on very little performance degradation. In the second place, we have found the meaning of some codewords, some of which are nonsense, whereas some of which preserve a lot of information. However, even for those nonsense words at the current time step, it may be very meaningful at other time steps, depending on the question machine is asked. Apart from this, we successfully identify those tokens that are responsible for marking the boundary of answer spans, which is supposed to be the key element of most QA models.

2. RELATED WORK:

Deep learning for reading comprehension has attracted much attention (Bordes et al., 2015; Kumar et al., 2015; Xiong et al., 2016; Hermann et al., 2015; Shih et al., 2015) . Generally, conventional reading comprehension systems are designed as a cascade of some components (syntactic parser, semantic parser, etc.), while in the case of end-to-end reading comprehension model, a neural network which takes a story and some questions as input and an answer as output is directly learned from training data, making it feasible to jointly learn components of conventional QA system.

In addition, the mechanisms such as hopping and attention are widely used in reading comprehension models so as to model the deduction process (Weston et al., 2014) . In a dynamic memory network (DMN) (Kumar et al., 2015) , questions trigger an iterative attention process which allows the model to condition its attention on the inputs and the result of previous iterations. After reasoning over multiple supporting facts, Neural Reasoner (Peng et al., 2015) is able to find an answer to a given question. The Recurrent Entity Network (EntNet) (Henaff et al., 2017) is the first method to solve all the tasks in the 10k training examples setting of bAbI (Weston et al., 2015) . The Query-Reduction Network (QRN) (Seo et al., 2016b) effectively handles both short-term and long-term sequential dependencies in order to reason over multiple facts. To achieve complex relational reasoning, new models are proposed (Bansal et al., 2017; Santoro et al., 2017; Palm et al., 2018; Pavez et al., 2018) . Recent attention mechanisms and network architectures have been shown to be quite helpful for SQuAD (Xiong et al., 2017; Seo et al., 2016a; Wang et al., 2017; Hu et al., 2017; Huang et al., 2017) . Also, on the SQuAD leaderboard1 , deep learning-based models are competitive with human performance and even surpass it (Yang et al., 2019; Liu et al., 2019; Devlin et al., 2018) . In addition to exploiting only plain context-sensitive features such as character or word embedding, Zhang et al. (2019a) proposes to incorporate explicit contextual semantics in order to promote natural language understanding. SG-Net (Zhang et al., 2019b) is proposed to guide machines by incorporating syntactic constraints into their attention mechanisms.

Interpretation of neural networks sometimes serve as preliminary works to some tasks such as model compression (Voita et al., 2019) or Machine Translation (Voita et al., 2018) . Among all interpretation methodologies, visualization has been the most intuitive and widely accepted in a plethora of such works. Most of them relate components of a neural network such as activations (Kádár et al., 2016; Liu et al., 2018) , weights of neurons (Ding et al., 2017) or attention heads (Bahdanau et al., 2015) to their corresponding positions in the inputs or outputs. Saliency based on some importance measure is quite common as well. Voita et al. (2018) calculated the importance of each attention head and related them to the syntactic role each head is assumed to take by experiment. Feng et al. (2018) assigned each word in an input context an importance calculated from gradient change by removing the word. Nevertheless, most of these interpretation measures yield continuous values and thus are hard to analyze and visualize directly, leaving the reasoning process of deep neural network still remain unexploited.

3. METHODOLOGY:

In this kind of task, inputs should contain a context C and some related questions Q, both of which are in natural language. Here is an example in Figure 1 . For the same context, models will be asked several different yet related questions, such as the first question in Figure 1 (a) and the second one in Figure 1(b) . Models should know what show is in the second question after answering the first question. For the purpose of understanding the reasoning process, we replace one of any layers that should be analyzed with a vector quantized (VQ) layer. In fact, for any methods that can quantize latent space into discrete one should work. Here we simply choose the one used in VQ-VAE (van den Oord et al., 2017) . The idea of our model structure is shown in Figure 2 . VQ layer should have the exact same shape with the layer that we need to analyze, then with an extra codebook, we are able to replace all latent vectors with quantized ones. In addition, replacing different layers with VQ ones will have different influences on performance; we will discuss this phenomenon based on the experiments on SDNet in section 5.1.

As for the gradient issue of VQ layer, we simply copy the gradients from the next layer's input to the previous layer's output. Then, the final loss for this model structure should include 3 terms: QA loss, VQ (Vector Quantization) loss, and commitment loss. While VQ loss helps the embedding vector move toward the previous layer's output, commitment loss makes sure the previous layer commits to an embedding. 

4. QUESTION-ANSWERING MODEL:

In this paper, we use FlowQA (Huang et al., 2018) and SDNet (Zhu et al., 2018) as the example of QA models to be studied. We modify both of them then train and evaluate on QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) separately. These two datasets are quite similar, both are conversational question answering datasets. Given one context and a series of questions, models are asked to answer them chronologically; they have to answer them based on the previous questions being asked. One example is shown in Figure 1 .

We train a bunch of FlowQA models with the same architecture for the purpose of organizing some general reasoning rules. On the other hand, we implement two different versions of SDNet. We put the VQ layer in different positions (SDNet(v1) and SDNet(v2) ) so as to observe the reasoning process respectively. We will elaborate on the differences between these two versions in section 5.1.

The network architecture and all details of FlowQA are in Appendix A.1 and the two different network architectures of SDNet are shown in Appendix A.2 respectively.

5. EXPERIMENTS:



5.1. DENSITY OF CODEBOOKS:

The final performance of two models with different codebook size, K, is listed in Table 1 . Note that we will not use the latent vectors of those models with gray color or do the further discussion and experiment in the following analysis. And, we also try to boost the performance of VQ models. Firstly, we randomly sample 1M vectors from the reproduced model, then we apply the K-means algorithm to it. Then, we use these 16 vectors to initialize our codebook and find that it does help boost the performance. Nonetheless, after comparing all the hidden representations with the original K = 16 model, we can hardly find any differences between them. Consequently, most of our experiments are based on the vanilla VQ model without K-means vector initialization. Moreover, for the sake of analyzing easily, we try to choose K as small as possible.

It is obvious that even for a very small K such as 4, both QA models can still maintain good performance. In the past, most applications of VQ-VAE are about image, audio, and video, few people have ever imagined that it can be applied to QA models, even for such small K. It is worth noting that the performance of SDNet(v2) is still very good when K = 1. K = 1 implies that each context word uses the same codeword, so the only way for the model to achieve such good performance is mainly because of the residual layers designed in vanilla SDNet. In SDNet(v2), the VQ layers are residual layers, whose outputs are combined with their inputs, including word embedding and BERT embedding, leading to information propagating to the next layers. We have found that even for a large K, models do not actually need that many codewords. In average, for these two kinds of QA reasoning tasks, models only need a small capacity to reach a quite good performance. When we say that only 24 codewords are used, it means that to evaluate the whole validation data, the model only utilizes 24 codewords in its codebook. For example, for the K = 512 FlowQA model, it actually utilize 24 codewords when predicting.

For Model 1, Model 2, and Model 3 of FlowQA, it means that we use the same model structure yet train three individual models. Besides, in SDNet(v1) we put one VQ layer at the rear part of the model, while in SDNet(v2) we put two VQ layers in the middle part of the model. As for Layer 1 and Layer 2 of SDNet(v2), it means that we replace two hidden layers in one model with two VQ layers yet sharing the same codebook.

Other implementation details can be found in Appendix A.2. As for other time steps and yet another example, we put them in Appendix B.5.

After observing tons of the colored figures and the statistical analysis, we define the first two high frequency, which cover 65% for all codewords, as nonsense words (Dark Blue and Sky Blue). These two codewords account for a huge portion in all size codebooks; therefore, we assume that it should be more meaningless than other rare codewords.

Also, we can easily notice that the most colorful part is usually the model output span, and this colorful segment is slowly shifted forward over time; the beginning of the context turns into dark blue and sky blue in the last question turns. This is because, for these two datasets we use, the positions of answers have a pretty strong relation with question turns. For instance, the answer to the first question should appear at the beginning of context, whereas the answer to the last question should appear at the end part of the context. We will discuss this phenomenon in the section 5.3.

In Table 3 , "Overall" column represents the percentage of these two codewords, "Model Output" means that the percentage of these two codewords appear within model output spans, and "Subject of Model Output" is the percentage of these nonsense words appearing within the subjects of model output spans. We put the other three tables of SDNet in Appendix B.2. The most interesting thing of these tables is that although model output spans usually do not contain those two nonsense words, their subjects actually do contain numerous nonsense words. We conclude that it is because, at the layer we quantize, subjects do not really matter to models. For FlowQA here, we set the quantized layer at the rear part, and at this layer, the model should have already known the exact answer span. Thus, other grammatical information such as object, location, adjective,...etc, may be more crucial than subject. Most questions in these 2 datasets are about co-reference between different subjects.

As long as models know the correct answer position, subjects do not play a part anymore. In the previous section, we have mentioned that we consider the most two frequent codewords as nonsense words. In this part, we use another viewpoint to analyze it. In general, the higher a codeword's percentage, the less meaning it preserves. In addition, in section 5.2 we have mentioned that the colorful segments which might contain answers will gradually shift forward as time goes on. In Figure 3 , code 0 always stands for the rarest codeword whereas the largest code number is the most common one. We can see that those crucial codewords certainly shift forward as time goes by; on the contrary, nonsense words shift backward. Note that the y-axis is the normalized position of context. For each word in a context, the normalized position stands for the position of that word divided by the length of the context. For example, normalized position= 0, 0.5, 1, means that the word is the first, middle, and the last word of the context. For other K we put them in Appendix B.1.

Figure 3 : Average position of different codewords in FlowQA (K=8, 16).

5.4. PART-OF-SPEECH:

We find that codewords may contain some syntactic and positional information. Such phenomenon is observed in Figure 5 (for the other K, see Appendix B.4). In the case of QuAC, whose answers tend to be sentences, input words with POS tags such as VERB, NOUN, and PROPN (proper noun), are critical for answering questions. For words with such POS tags, if they are unrelated to the current question, the model will assign them nonsense codewords (say, some of the most frequent) for masking, so as not to mislead the later hidden layers. That is why the critical words' tags, VERB, NOUN, and PROPN, dominate almost all columns including nonsense words, code 0, code 1, and code 2.

Moreover, the most important in Figure 4 and Figure 5 is that there are usually one or two codewords possibly representing the start position and the end position for the answer, which may be due to both models' nature as pointer networks. For the example in Figure 4 , code 9 and code 7 take responsibility for this task. In order to extract answers, models always output the probability of start and end positions. Now we may hypothesize that at least in the quantized layer, the model has already tried to mark the boundary of answers.

Figure 4 : Codewords distribution on start/end punctuation of FlowQA (K=16).

Figure 5 : The frequency of POS tags in each codeword of FlowQA (K=16). Note that for all heat maps in this work, the codewords are ordered by their frequecy in the latent representation, from the most frequent to the rarest. For each codeword in hidden layer, we can find the corresponding word token in input layer. Then we use that word to generate POS tags.

6. CONCLUSION:

In this paper, we propose vector quantized layer (VQ layer) on QA models, a novel network architecture for the QA task. Based on this idea, we are able to understand how machine thinks essentially without losing much performance. For the VQ layer we use in this work, it can be put on any layers of any QA models as long as we need to analyze it. After coloring those quantized vectors, it is pretty easy for people to observe the action of different models. Besides, based on the experiments, we have found some interesting phenomena. For the pointer networks we utilize in this paper, we successfully find that models themselves indeed use some codewords to mark the boundary of answer spans. Also, for some keywords such as VERB, NOUN, and PROPN, the models will use some nonsense tokens to mask them so as not to mislead the latter layers. Finally yet importantly, QA models actually do not need that huge capacity to reach high performance. That is, it can merely use 20 or so codewords in a layer to propagate all information.

A IMPLEMENTATION DETAILS

A.1 FLOWQA Our modified FlowQA is shown in Figure 6 . In this task, each context and a series of questions are first encoded with GloVe2 and Elmo (Peters et al., 2018) . Then, we keep both the bidirectional GRU (Chung et al., 2014) layers, which are used to encode context, and unidirectional GRU layers, which are used to propagate information between different questions, intact. Then, we add an extra codebook with size K and use it to quantize the output of the last flow layer. With the help of this quantization mechanism, we are able to know the reasoning process of this model more easily.

Thus, the output shape of the previous layer is

[L C , N Q , H],

where L C stands for the max length of context, N Q is the number of question turns, and H represents the hidden dimension. Then for each H-dimensional vector, we can always find one vector in the codebook, which is composed of K H-dimensional codewords, to replace it, letting it be the input of the next layer.

Finally, as for the value of β in VQ-VAE, we assign 1.0 on it instead of the default setting 0.25. Instead of using another dimensional flow to propagating information between different questions, SDNet here utilizes the power of attention. We keep all layers intact except for the Context Final Representation layer. Then a size K codebook is also added here. Note that for the original model (Zhu et al., 2018) , the author sets the number of the question turns as batch size for each training iteration and we simply follow this setting.

As a result, the output shape of the previous RNN layer should be [L C , N Q , H], where L C stands for the max length of context, N Q stands for the number of question turns, and H represents the hidden dimension.

Afterward, merely do the same operations as we did for FlowQA: finding one vector in the codebook to replace the original one, feeding them into the next layer, then dealing with the gradient issue. Lastly, we also use QA loss, VQ loss, and commitment loss to train this vector quantized SDNet. Note that the value of β is set to 1.0 here as well. In QuAC, models are asked to whether the question is answerable or not, just like those in SQuAD 2.0 (Rajpurkar et al., 2018) . We found that typically, rare codewords' percentages when the question is not answerable is lower than the percentages when the question is answerable. We use Figure 9 to show. The lines represent answerable questions while dotted lines are for not answerable ones.

As for the meaning of the threshold on the horizontal axis, it means that the percentage we define codewords as a meaningful codeword . For instance,1.0 means that all codewords are meaningful. 

:

https://rajpurkar.github.io/SQuAD-explorer/

https://github.com/stanfordnlp/GloVe

Figures and Tables:

Figure 1: Vector quantized latent representations for different question turns.

Figure 2: Vector quantized model.

Figure 1(a), Figure 1(b), and Figure 1(c). In the figures, different colors represent different codewords.

Figure 6: FlowQA with vector quantized layer.

Figure 7: SDNet with vector quantized layer.

Figure 9: Influence of important words on answerability.

Figure 10: Codebooks distribution on start/end punctuation in FlowQA (K=32, 64, 128, 512).

Performance of two vector quantized models. The light cyan row stands for the vanilla model without vector quantization, while those cells with gray color means that they are under the 90% performance of vanilla model. The pink row is the model with K-means initialization.

Table content: <table><tr><td>F1 Score</td></tr></table>

Actually used codewords. The gray cells mean that models use all codewords in their codebooks.

Table content: <table><tr><td/><td/><td>FlowQA</td><td/><td>SDNet</td><td colspan="2">SDNet(v2)</td></tr><tr><td>K</td><td>Model 1</td><td>Model 2</td><td>Model 3</td><td>Model 1</td><td>Model 1</td><td/></tr><tr><td>Layer</td><td/><td/><td/><td/><td>Layer1</td><td>Layer2</td></tr><tr><td>512</td><td>24</td><td>30</td><td>28</td><td>23</td><td>169</td><td>346</td></tr><tr><td>128</td><td>25</td><td>24</td><td>32</td><td>23</td><td>36</td><td>94</td></tr><tr><td>64</td><td>23</td><td>30</td><td>30</td><td>18</td><td>20</td><td>53</td></tr><tr><td>32</td><td>23</td><td>32</td><td>32</td><td>17</td><td>10</td><td>24</td></tr><tr><td>16</td><td>15</td><td>16</td><td>15</td><td>16</td><td>10</td><td>16</td></tr><tr><td>8</td><td>8</td><td>-</td><td>-</td><td>8</td><td>6</td><td>8</td></tr><tr><td colspan="3">5.2 MEANING OF NONSENSE WORDS</td><td/><td/><td/><td/></tr></table>

Nonsense Words in FlowQA.

Table content: <table><tr><td/><td/><td>FlowQA</td><td/></tr><tr><td>K</td><td>Overall</td><td>Model Output</td><td>Subject of Model Output</td></tr><tr><td>512</td><td>0.4312</td><td>0.2396</td><td>0.4893</td></tr><tr><td>128</td><td>0.3960</td><td>0.2336</td><td>0.4454</td></tr><tr><td>64</td><td>0.4003</td><td>0.2271</td><td>0.4622</td></tr><tr><td>32</td><td>0.4084</td><td>0.2405</td><td>0.4812</td></tr><tr><td>16</td><td>0.4893</td><td>0.2898</td><td>0.5448</td></tr><tr><td>8</td><td>0.5763</td><td>0.4156</td><td>0.6261</td></tr><tr><td>Average</td><td>0.4503</td><td>0.2744</td><td>0.5082</td></tr></table>

Nonsense words in SDNet(v1).

Table content: <table><tr><td/><td/><td>SDNet(v1)</td><td/></tr><tr><td>K</td><td>Overall</td><td>Model Output</td><td>Subject of Model Output</td></tr><tr><td>512</td><td>0.6996</td><td>0.5947</td><td>0.9654</td></tr><tr><td>128</td><td>0.5661</td><td>0.5139</td><td>0.5776</td></tr><tr><td>64</td><td>0.7849</td><td>0.5321</td><td>0.7633</td></tr><tr><td>32</td><td>0.6623</td><td>0.4490</td><td>0.6930</td></tr><tr><td>16</td><td>0.6578</td><td>0.5139</td><td>0.7456</td></tr><tr><td>8</td><td>0.8905</td><td>0.5367</td><td>0.8458</td></tr><tr><td>4</td><td>0.9626</td><td>0.5947</td><td>0.9654</td></tr><tr><td>Average</td><td>0.7463</td><td>0.5189</td><td>0.7538</td></tr><tr><td>B.3 ANSWERABILITY</td><td/><td/><td/></tr></table>

Nonsense words in SDNet(v2) layer 1.

Table content: <table><tr><td/><td/><td>SDNet(v2) layer 1</td><td/></tr><tr><td>K</td><td>Overall</td><td>Model Output</td><td>Subject of Model Output</td></tr><tr><td>512</td><td>0.0518</td><td>0.0536</td><td>0.2183</td></tr><tr><td>128</td><td>0.1316</td><td>0.1656</td><td>0.4106</td></tr><tr><td>64</td><td>0.1984</td><td>0.2115</td><td>0.4731</td></tr><tr><td>32</td><td>0.4373</td><td>0.3807</td><td>0.5752</td></tr><tr><td>16</td><td>0.5208</td><td>0.5707</td><td>0.5255</td></tr><tr><td>8</td><td>0.9998</td><td>1.0</td><td>1.0</td></tr><tr><td>4</td><td>0.9999</td><td>0.9999</td><td>0.9994</td></tr><tr><td>Average</td><td>0.4771</td><td>0.4831</td><td>0.6003</td></tr></table>

Nonsense words in SDNet(v2) layer 2.

Table content: <table><tr><td/><td/><td>SDNet(v2) layer 2</td><td/></tr><tr><td>K</td><td>Overall</td><td>Model Output</td><td>Subject of Model Output</td></tr><tr><td>512</td><td>0.0405</td><td>0.0347</td><td>0.1404</td></tr><tr><td>128</td><td>0.0988</td><td>0.0951</td><td>0.1589</td></tr><tr><td>64</td><td>0.1989</td><td>0.1535</td><td>0.2857</td></tr><tr><td>32</td><td>0.3071</td><td>0.2298</td><td>0.3397</td></tr><tr><td>16</td><td>0.6608</td><td>0.3514</td><td>0.6287</td></tr><tr><td>8</td><td>0.7080</td><td>0.5498</td><td>0.7998</td></tr><tr><td>4</td><td>0.8090</td><td>0.8019</td><td>0.7966</td></tr><tr><td>Average</td><td>0.4033</td><td>0.3166</td><td>0.4450</td></tr></table>

Peer review (iclr2020):
The paper focuses on analyzing and visualizing trained QA neural models. The main idea is to quantize latent representations during training, making the model more interpretable eventually. The general idea is understandable, but the details are very unclear and I found it extremely difficult to follow. As far as I understand, one of the layers (in which kind of a network?) is replaced with a quantizing layer. This quantized layer is different than the original layer only in its forward computation; during backprop - the gradients from the next layer are copied backwards. The network is then trained (this is not said explicitly, until this point I thought that the paper discusses analyzing already-trained models) using this new layer, with 3 losses: QA loss, VQ loss and commitment loss. These losses are not discussed nor defined further. Finally, the quantized vectors are clustered using K-means. Overall, I think that this paper should be rejected. It is very unclear, the proposed technique is not compared to any existing technique, and the interpretation of the network - which is the main goal of the paper - does not produce any really useful insights. For another example of vagueness, the paper refers to a figure by *Each color represents a codeword* (Page 2, para2). What is a codeword? What exactly do the authors refer to here? How did they choose the number of colors and how did they color each word? Another example - *a neural network can use only 20 or so vectors to cover almost all information* (page 2, para3). Which vectors? The reader doesn*t even know which model architecture is discussed at this point. The experiments are based on extracting the quantized vectors from the trained model and running K-means with different values of K. What are the metrics used here? Table 1 shows F1 score, but I don*t understand how precision/recall are computed? Are these the original metrics used in the benchmarks? what is exactly the experiment here? Table 2 is not referenced nor explained. I did not understand Section 5.2 - *Meaning of Nonsense words* at all. What are *nonsense words*? How they are chosen? Please define these more accurately. Since I didn*t understand the settings, I did not understand the results in Table 3 (metrics/units are not mentioned there as well). I did not understand Section 5.3 as well (*Position and Portion of Nonsense words*). If I understand correctly, the main point here is that *the higher a codeword’s percentage, the less meaning it preserves*, but I don*t understand how is this observed from Figure 3, and how does the paper measure *preserving of meaning*. Related work: The paper misses a discussion and comparison with recent network-interpretation approaches, for example [1][2][3] below. Additionally, some of the existing discussion is very lacking. For example - Doshi-Velez (2017) and Montavon (2018) - I have not read these papers, but why *only those deep learning researchers are able to understand and utilize the results*? This is a vague remark regarding these papers. Didn*t these papers show do how they interpret their results? Did they release their code? Are these papers too complicated mathematically? More questions to authors: 1. What are codewords and codebooks? These are not defined nor explained. 2. The main idea in the paper is quantizing some vectors in order to interpret them better. But how interpreting discrete vectors is better than interpreting continuous vectors in this case? If the main applied technique is K-means clustering, why can*t we just cluster continuous vectors? Minor: * I personally don*t like the use of phrases that humanize machines like *the machine thinks* and *Models should know*. Treating neural models with human characteristics is inaccurate and misleading. [1] *What Does BERT Look At? An Analysis of BERT’s Attention*, Clark et al, 2019 [2] *Are Sixteen Heads Really Better than One?* Michel et al, 2019 [3] *How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations* Aken et al, 2019