SlowGuess/NAACL_2022_OCR huggingface dataset:
#	Tag	Description
1	CC	Coordinating conjunction
2	CD	Cardinal number
3	DT	Determiner
4	EX	Existential there
5	FW	Foreign word
6	IN	Preposition or subordinating conjunction
7	JJ	Adjective
8	JJR	Adjective, comparative
9	JJS	Adjective, superlative
10	LS	List item marker
11	MD	Modal
12	NN	Noun, singular or mass
13	NNS	Noun, plural
14	NNP	Proper noun, singular
15	NNPS	Proper noun, plural
16	PDT	Predeterminer
17	POS	Possessive ending
18	PRP	Personal pronoun
19	PRP$	Possessive pronoun
20	RB	Adverb
21	RBR	Adverb, comparative
22	RBS	Adverb, superlative
23	RP	Particle
24	SYM	Symbol
25	TO	to
26	UH	Interjection
27	VB	Verb, base form
28	VBD	Verb, past tense
29	VBG	Verb, gerund or present participle
30	VBN	Verb, past participle
31	VBP	Verb, non-3rd person singular present
32	VBZ	Verb, 3rd person singular present
33	WDT	Wh-determiner
34	WP	Wh-pronoun
35	WP$	Possessive wh-pronoun
36	WRB	Wh-adverb
37	#	Pound sign
38	$	Dollar sign
39	.	Sentence-final punctuation
40	,	Comma
41	:	Colon, semi-colon
42	(	Left bracket character
43	)	Right bracket character
44	"	Straight double quote
45	'	Left open single quote
46	"	Left open double quote
47	'	Right close single quote
48	"	Right close double quote
Table 2: Penn Treebank POS tags.
correlation between these attributes and word usage. It is a knowledge-based system where words are mapped different high level concepts.
B BERT-based Sequence Tagger
We trained a BERT-based sequence tagger to auto- annotate our training data. We used standard splits for training, development and test data for the 4 linguistic tasks (POS, SEM, Chunking and CCG super tagging) that we used to carry out our analysis on. The splits to preprocess the data are avail
ANA (anaphoric)	MOD (modality)
PRO	anaphoric & deictic pronouns: he, she, I, him	NOT	negation: not, no, neither, without
DEF	definite: the, loIT, derDE	NEC	necessity: must, should, have to
HAS	possessive pronoun: my, her	POS	possibility: might, could, perhaps, alleged, can
REF	reflexive & reciprocal pron.: herself, each other	DSC (discourse)
EMP	emphasizing pronouns: himself	SUB	subordinate relations: that, while, because
ACT (speech act)	COO	coordinate relations: so, {}, {}, and
GRE	greeting & parting: hi, bye	APP	appositional relations: {}, which, {}, —
ITJ	interjections, exclamations: alas, ah	BUT	contrast: but, yet
HES	hesitation: err	NAM (named entity)
QUE	interrogative: who, which, ?	PER	person: Axl Rose, Sherlock Holmes
ATT (attribute)	GPE	geo-political entity: Paris, Japan
QUC	concrete quantity: two, six million, twice	GPO	geo-political origin: Parisian, French
QUV	vague quantity: millions, many, enough	GEO	geographical location: Alps, Nile
COL	colour: red, crimson, light blue, chestnut brown	ORG	organization: IKEA, EU
IST	intersective: open, vegetarian, quickly	ART	artifact: iOS 7
SST	subjective: skillful surgeon, tall kid	HAP	happening: Eurovision 2017
PRI	private: former, fake	UOM	unit of measurement: meter, $, %, degree Celsius
DEG	degree: 2 meters tall, 20 years old	CTC	contact information: 112, info@mail.com
INT	intensifier: very, much, too, rather	URL	URL: http://pmb.let.rug.nl
REL	relation: in, on, 's, of, after	LIT	literal use of names: his name is John
SCO	score: 3-0, grade A	NTH	other names: table 1a, equation (1)
COM (comparative)	EVE (events)
EQU	equative: as tall as John, whales are mammals	EXS	untensed simple: to walk, is eaten, destruction
MOR	comparative positive: better, more	ENS	present simple: we walk, he walks
LES	comparative negative: less, worse	EPS	past simple: ate, went
TOP	superlative positive: most, mostly	EXG	untended progressive: is running
BOT	superlative negative: worst, least	EXT	untended perfect: has eaten
ORD	ordinal: 1st, 3rd, third	TNS (tense & aspect)
UNE (unnamed entity)	NOW	present tense: is skiing, do ski, has skied, now
CON	concept: dog, person	PST	past tense: was baked, had gone, did go
ROL	role: student, brother, prof., victim	FUT	future tense: will, shall
GRP	group: John {}, Mary and Sam gathered, a group of people	PRG	progressive: has been being treated, aan hetNL
DXS (deixis)	PFT	perfect: has been going/done
DXP	place deixis: here, this, above	TIM	temporal entity)
DXT	temporal deixis: just, later, tomorrow	DAT	full date: 27.04.2017, 27/04/17
DXD	discourse deixis: latter, former, above	DOM	day of month: 27th December
LOG (logical)	YOC	year of century: 2017
ALT	alternative & repetitions: another, different, again	DOW	day of week: Thursday
XCL	exclusive: only, just	MOY	month of year: April
NIL	empty semantics: {}, to, of	DEC	decade: 80s, 1990s
DIS	disjunction & exist. quantif.: a, some, any, or	CLO	clocktime: 8:45 pm, 10 o'clock, noon
IMP	implication: if, when, unless
AND	conjunction & univ. quantif.: every, and, who, any
Table 3: Semantic tags.
Task	Train	Dev	Test	Tags	F1
POS	36557	1802	1963	48	96.69
SEM	36928	5301	10600	73	96.22
Chunking	8881	1843	2011	22	96.91
CCG	39101	1908	2404	1272	94.90
Table 4: Data statistics (number of sentences) on training, development and test sets using in the experiments and the number of tags to be predicted
able through git repository released with Liu et al. (2019a). See Table 4 for statistics and classifier accuracy.
$^{9}$ https://github.com/nelson-liu/contextual-repr-analysis
C Clustering details
Algorithm 1 assigns each word to a separate cluster and then iteratively combines them based on Ward's minimum variance criterion that minimizes intra-cluster variance. Distance between two vector representations is calculated with the squared Euclidean distance.
Algorithm 1 Clustering Procedure
Input:  $\overrightarrow{y}^l$ : word representation of words  
Parameter:  $K$ : the total number of clusters
1: for each word  $w_{i}$  do
2: assign  $w_{i}$  to cluster  $c_{i}$
3: end for
4: while number of clusters  $\neq K$  do
5: for each cluster pair  $c_{i}, c_{j'}$  do
6:  $d_{i,i'} =$  inner-cluster difference of combined cluster  $c_{i}$  and  $c_{i'}$
7: end for
8:  $c_{j}, c_{j'} =$  cluster pair with minimum value of  $d$
9: merge clusters  $c_{j}$  and  $c_{j'}$
10: end while
C.1 Selection of the number of Clusters
The Elbow curve did not show any optimum clustering point, with the increase in number of clusters the distortion score kept decreasing, resulting in over-clustering (a large number of clusters consisted of less than 5 words). The over-clustering resulted in high but wrong alignment scores e.g. consider a two word cluster having words "good" and "great". The cluster will have a successful match with "adjective" since more than  $90\%$  of the words in the cluster are adjectives. In this way, a lot of small clusters will have a successful match with many human-defined concepts and the resulting alignment scores will be high. On the other hand, Silhouette resulted in under-clustering, giving the best score at number of clusters  $= 10$ . We handled this empirically by trying several values for the number of clusters i.e., 200 to 1600 with step size 200. We selected 1000 to find a good balance with over and under clustering. We understand that this may not be the best optimal point. We presented the results of 600 and 1000 clusters to show that our findings are not sensitive to the number of clusters parameter.
D Coarse vs. Fine-grained Categories
D.1 Coarse vs. Fine-grained Categories
Our analysis of compositional concepts showed that several fine-grained concepts could be combined to explain an unaligned concept. For example, by combining verb categories of POS to one coarse verb category, we can align the encoded concept present in Figure 5c. To probe this more formally, we collapsed POS and SEM fine-grained concepts into coarser categories (27 POS tags and 15 SEM tags). We then recomputed the alignment with the encoded concepts. For most of the models, the alignment doubled compared to the fine-grained categorizes with at least  $39\%$  and at most  $53\%$  percent match for POS. This reflects that in several cases, models learn the coarse language hierarchy. We further questioned how many encoded concepts can be explained using coarse human-defined concepts. Compared to Table 1, the matches increased by at most 17 points in the case of BERT-uc. The XLM-R showed the highest matching percentage of  $81\%$ . The higher alignment suggests that most of the encoded concepts learned by pre-trained models can be explained using human-defined concepts. (See Appendix D for detailed results).
D.2 Corase POS and SEM labels
Tables 5 and 6 present results for our mapping of fine-grained SEM and POS tags into coarser categories.
Coarse	Fine-grained
ACT	QUE
ANA	DEF, DST, EMP, HAS, PRO, REF
ATT	INT, IST, QUA, REL, SCO
COM	COM, LES, MOR, TOP
DSC	APP, BUT, COO, SUB
DXS	PRX
EVE	EXG, EXS, EXT, EXV
LOG	ALT, AND, DIS, EXC, EXN, IMP, NIL, RLI
MOD	NEC, NOT, POS
NAM	ART, GPE, HAP, LOC, NAT, ORG, PER, UOM
TIM	DEC, DOM, DOW, MOY, TIM, YOC
TNS	EFS, ENG, ENS, ENT, EPG, EPS, EPT, ETG, ETV, FUT, NOW, PST
UNE	CON, ROL
UNK	UNK
Table 5: SEM: Coarse to Fine-grained mapping
Table 7 presents the alignment results of using coarse POS and SEM concepts. We observed that
Coarse	Fine-grained
Adjective	JJ, JJR, JJS
Adverb	RB, RBS, WRB, RBR
Conjunction	CC
Determiner	DT, WDT
Noun	NN, NNS, NNP, NNPS
Number	CD
Preposition	IN, TO
Pronoun	PRP, PRP$, WP, WP$
Verb	VB, VBN, VBZ, VBG, VBP, VBD
No Changes	$, -LRB-, #, FW, -RRB-, LS, POS, "", EX
SYM, :, RP, :, PDT, MD, UH,
the alignment doubles in most of the cases which reflects that in several cases, models learn the coarse language hierarchy. However, they do not strictly adhere to fine-grained categories existed in human-defined concepts. We further extend the alignment of coarse POS and SEM categories to the overall alignment with the human-defined concepts. Table 8 presents the results. We see a match of up to  $81\%$  in the case of XLM-R. The high alignment suggests that many of the encoded concepts can be explained using coarse human-defined concepts.
Table 6: POS: Coarse to Fine-grained mapping
POS	SEM
Fine	Coarse	Fine	Coarse
BERT-cased	13%	42%	7%	15%
BERT-uncased	16%	43%	9%	18%
mBERT	26%	53%	16%	26%
XLM-RoBERTa	24%	47%	11%	21%
RoBERTa	18%	43%	10%	20%
ALBERT	17%	42%	9%	17%
XLNet	17%	39%	10%	18%
E Compositional Coverage
Table 9 shows the amount of coverage we obtain when aligning with the morphological concepts when allowing  $90\%$  of the words in the cluster to be from  $N$  concepts.
Table 7: Alignment of fine-grained human defined concepts compared to coarse categories
Overall alignment	BERT-c 61.5% RoBERTa 62.9%	BERT-uc 63.6% ALBERT 64.0%	mBERT 77.7% XLNet 55.3%	XLM-R 81.0%
Table 8: Coverage of human-defined concepts using coarse POS and SEM labels across all clusters from a given model
F Robustness of Methodology across Datasets and Settings
Figure 8 shows the layer-wise patterns using 600 clusters instead of 1000 as used in the main paper. We observe that the overall trends largely remain the same.
To further demonstrate the robustness of our method with respect to dataset, we sub-sampled another dataset from the News corpus with a different vocabulary by selecting words that appear between 2 to 10 times in the corpus. Note that the selection of vocabulary is due to the memory and computation limitations. Figure 9 shows the results using this selection of data. Compared to Figure 4, we can see that the overall patterns are largely similar and confirms the robustness of our findings. The slight difference in the patterns of WordNet and LIWC are due to the large selection of proper nouns in the second set of the data.
G Layer-wise results
Figure 10 present layer-wise results for all the understudied models.
(c)
Figure 6: Example clusters: (a) ngram:ace, (b) POS:CD, (c) Chunking:B-VP + Suffix:ed
(c)
Figure 7: Example clusters: (a) LIWC:cause, (b) WORDNET:verb.cognition, (c) WORDNET:noun.artifact
Concepts	BERT-c	BERT-uc	mBERT	XLM-R	RoBERTa	ALBERT	XLNet
1	13%	16%	26%	24%	18%	17%	17%
2	11%	12%	20%	23%	13%	13%	12%
3	14%	13%	14%	18%	11%	15%	9%
4	6%	6%	4%	4%	5%	5%	3%
5	2%	1%	1%	1%	2%	1%	1%
6	1%	0%	0%	0%	1%	1%	0%
Table 9: Percentage of alignment when an encoded concept is composed of  $N$  morphological concepts. As can be seen, most concepts are composed of either 1, 2 or 3 morphological concepts, showing that several concepts learned by these models are indeed compositional in nature.
Figure 8: Layer-wise results using 600 clusters.
Figure 9: Layer-wise results on a separately sampled dataset.
BERT-c
(a) Lexical
BERT-uc
(c) Syntactic
mBERT
(d) Linguistic Ontologies
(b) Morphology and Semantics
RoBERTa
(c) Syntactic
(b) Morphology and Semantics
(c) Syntactic
XLM-R
(b) Morphology and Semantics Layer
ALBERT
XLNet
Figure 10: Laver wise results