How LSTM Encodes Syntax:
Exploring Context Vectors andSemi-Quantization on Natural Text
Chihiro Shibata (Tokyo University of Technology)
Kei Uchiumi (Denso IT Laboratory)
Daichi Mochihashi (The Institute of Statistical Mathematics)

COLING2020 - 1

Introduction• LSTMs have been used in wide range of NLP tasks:• Machine translation, text generation, etc.
• LSTM Language Model (LSTM-LM) is the most fundamental architecture forthose applications.
• It is not yet completely clear how syntactic information is represented in it.
• What is the purpose of this research?
• Understanding internal representations of LSTM-LMs w.r.t. syntacticinformation.
• Empirical approach: Real data (plain text) + syntactical annotation.
• Details of representations in each internal vector inside LSTM areinvestigated.
• NOT about BERT's representation.
• nor comparison to BERT
COLING2020 - 2

Outline
1. We investigate the distributions of the elements of the internal vectors.
• empirically show that their distributions are approximately quantized(Semi-quantization).
2. Cell-state vectors (c) are investigated using several datasets, some ofwhich are Dyck-languages.
• How the semi-quantization relates to the representation in c.
3. Cell-update vectors (u) are focused and• showed to have important role in representing syntactic information.
COLING2020 - 3

Semi-Quantization of Internal Vectors andStatistics of each Element of them

• Simple but important facts:• Each element is approximately quantized(because LSTM is designed so).
• Internal vectors such as cell-state and output
have characteristic distributions that have different
peaks.

{0, 1}• Datasets and learning methods such as dropout basicallyare independent to the above characteristics.

ct 1

×

cell-state vector (c){ 1, 0, 1}

utft [0,1][0,1]cell-update (u)vectorct+[-1,1]

×forget-gate (f)vectortanh[-1,1]
tanh[-1,1]

×• Right figure: learning results of a single-layered LSTM-LM
using plain texts (WSJ) are shown.

[0,1]

output (h)vector

We look into each distribution thatthe elements of each internal vector has.
COLING2020 - 4htdistribution with initial weights

Distribution ofelements of cell-updatevectors (u)• Distributions dramatically changes
through learning.
• u is semi-quantized into {-1,0,1}.

ut [-1,1]

ft [0,1][0,1][-1,1]
tanhtanh[-1,1]

ct



ct 1

after learning• has important rules for syntactic
representation as shown later.

ht

[0,1]
COLING2
020 - 5

distribution with initial weights

Distribution ofelements offorget-gate vectors (f)• Distributions of elements of f arebinarized into {0,1} values.

after learning

ut [-1,1]ft [0,1][0,1][-1,1]
tanh

tanh[-1,1]

ct



ct 1

ht

[0,1]
COLING2
020 - 6

distribution with initial weights

Distribution ofelements of cell-statevectors (c)• We can observe peaks in the integer
values:• result of accumulating u vectors.

after learning

ut [-1,1]ft [0,1][0,1][-1,1]
tanhtanh[-1,1]





ct



ct 1

ht

[0,1]
COLING2020 - 7

distribution with initial weights

Distribution ofelements of output(=hidden) vectors (h)• Large peak around 0.
• Small peaks at {-1, -0.75, +0.75, +1}.

after learning

ut [-1,1]ft [0,1][0,1][-1,1]
tanh

tanh[-1,1]

ct



ct



ct 1

ht

[0,1]
COLING2020 - 8

Semi-quantization of internal vectors andstatistics of each element of vectors• Through experiments:• We investigate the representation in cand its relation to the nesting depths ofthe phrase structures.
• Models are learned from several types of data.

cell-state vector (c){0, 1}

ct 1

×{ 1, 0, 1}

ut

ft [0,1][0,1]cell-update (u)vectorct+[-1,1]

×• is just a result of activation functions, andthus there is no contribution to encode syntax?
• or has a certain role for learning syntax?

forget-gate (f)vectortanh[-1,1]
tanh[-1,1]

×• Question:
This kind of semi-quantization[0,1]

output (h)vectorCOLING2020 - 9ht

Experiments: Target-dataset to learn• Making Dyck-like data by adding parentheses to texts in PTB-WSJ.
• Four types of data:
1. Paren : ‘(’ and ‘)’ without words,( ( ) ( ) ( ) )2. Paren+W : ‘(’ and ‘)’ + words,( ( a ) ( nonexecutive ) ( director ) )3. Tag : ‘(T’ and ‘T)’ without words, where T is a nonterminal symbol.
(NP (DT NP) (JJ JJ) (NN NN) NP)

4. Tag+W : ‘(T’ and ‘T)’ + words:(NP (DT a NP) (JJ nonexecutive JJ) (NN director NN) NP)

5. Words : plain text.
a nonexecutive director
COLING2020 - 10

Learning results and accuracies

End Of Sentence

Check accuracies for ① the balancing of parentheses and② kinds of tags (implying orders of tags) :①

LSTM-LM predicts EOS with 100% (almost no mistake).

②It predicts kinds of phrases of ")" with >95% (slight mistake).

ParenParen+W

TagTag+W①①②②①①

COLING2020 - 11

Embedding of nesting depth
on Paren and Paren+W data
• In cell-state vector (c):• Elements whose correlation
coefficient is 1.0 with respect
to the nesting depth.
• Both for Paren and Paren+W.

Paren• LSTM counts the nesting depthof the parentheses through suchelements.

Paren+W
COLING2020 - 12

Visualizing of count of nesting depths by asingle element of c
• For Paren data, we can observe a clear lattice for some singleelement.
• As the height of each step of the lattice is 1, we can knowthat c, u, f are completely quantized to natural numbers.

COLING2020 - 13

Embedding of nesting depth of each tag
on Tag and Tag+W data
• There is a single element that has high correlation.

• the highest correlated elements : 0.96 for NP, and 0.85 for VP.

• However, correlation is not perfect.

• Element of c is quantized well but doesn't corresponds to the nesting
depth of VP perfectly.

quantized wellcell-state and the nesting depth of VP (Tag+W)

COLING2020 - 14

Representation in subspace(linear sum of elements)• we can find a good linear sum so that the correlation coefficient
can be almost 1.

COLING2020 - 15

Embedding of nesting depth
using plain text
• Correlation coefficient is high:

Nesting depth of VPs

0.82 for VP using linear sum of c
• It is not possible to obtain acomplete correlation such that allplots are almost on a straight line.

COLING2020 - 16

Summary so far and Further Question
Summary:• For Paren data, in c, there is a completely quantized element thatacts as a counter of the nesting depth of the parentheses.
• For Tag data, a linear sum of the elements of c can act as a counterof the nesting depth.
• For plain text, we cannot find such a clear counter, but find a highly
correlated direction in c.
Question:• For plain text, can we find any clusters that represents triggering thenesting of the phrase structure, which should be POS such as nouns
and verbs?
COLING2020 - 17

Representation ofParts of Speechc• comparing c, u, h :• c has accumulated context information.
• u has delta that triggers contexts.
• h has information to predict a next word.
u should represent POS most clearly.

u• visualizing POS clusters:• Vectors for the same word are averaged.
• Clusters of {VB, VBZ, NN, NNS,CD} are
obtained most clearly in u.
NN, NNZ, VB, VBZ : noun singular, plural, verb sin., plu.
CD : numbers

hCOLING2020 - 18

list of similar words : understanding roles ofinternal vectors• comparison of vector similarities between c and u.
• For u, syntactically similar words tend to be listed.
• For c, co-occurrence words tend to be listed.

syntactically similar(possessive)co-occurrence words

COLING2020 - 19

list of similar words : understanding roles ofinternal vectors• comparison of similar words to "her"using h, c, u, and θ(u) vectors.
• In h, both types of words that have
similar meaning or syntactic function
are gathered.
• In u, words that have similar
syntactic functions are gathered mostwell.

u• Quantizing u to {-1,0,1} (θ(u))
doesn't change the result so much.
COLING2020 - 20-101How the syntactic functions of the word"that" are embedded in u ?
• Word “that” is a representative
ambiguous functional word.
• Different meanings are clusteredalthough they are notcompletely separated.

IN: Preposition or subordinating conjunction, e.g. "if"
DT: determiner, e.g. "this"COLING2020 - 21
WDT: Wh-determiner. e.g. "which"Conclusion
Statistics of internal vectors (c,h,u,f) :• Characteristic semi-quantization is observed for every internal vector.
Analyses of cell-state vector (c):• For Paren data, in c, there is a completely quantized element that acts asa counter of the nesting depth of the parentheses.
• For Tag data, a linear sum of the elements of c can act as a counter of thenesting depth.
Analyses of cell-update vector (u):• POS is best represented in cell-update vector u .
• Syntactic functions the word "that" has can be clustered in u .
COLING2020 - 22

Related work about capability of LSTM-LMs
w.r.t. capturing syntactic information
Empirical analyses :• Synthetic data
• Dyck-1,2 and shuffle of Dyck-1 languages (Suzugun et al. 2019)• SP-k languages (Enes et al. 2017)• Early studies for LSTMs with few dimensions (Prez-Ortiz et al., 2003; Schmidhuber,2015)• Real data
• a lot of studies
e.g. using number agreement to check if it captures syntax when viewed from the prediction
result.(Linzen et al. 2016)
Theoretical analyses :• expression capabilities are investigated: relation to counter machines are found.
(Weiss et al. 2018, Merrill 2019)
COLING2020 - 23

Thank you foryour listening.

COLING2020- 24