SlowGuess/ACL_2020_OCR huggingface dataset:
\(\mathcal{N}_{\mathrm{prof}} = \{\text{'accountant},\text{'acquaintance},\text{'actor'},\text{'actress'},\text{'administrator'},\text{'adventurer'},\text{'advocate'},\text{'aide'},\text{'alderman'},\text{'ambassador'},\text{'analyst'},\text{'anthropologist'},\text{'archaeologist'},\text{'archbishop'},\text{'architect'},\text{'artist'},\text{'artiste'},\text{'assassin'},\text{'astronaut'},\text{'astronomer'},\text{'athlete'},\text{'attorney'},\text{'author'},\text{'baker'},\text{'ballerina'},\text{'ballplayer'},\text{'banker'},\text{'barber'},\text{'baron'},\text{'barrister'},\text{'bartender'},\text{'biologist'},\text{'bishop'},\text{'bodyguard'},\text{'bookkeeper'},\text{'boss'},\text{'boxer'},\text{'broadcaster'},\text{'broker'},\text{'bureaucrat'},\text{'businessman'},\text{'businesswoman'},\text{'butcher'},\text{'cabbie'},\text{'cameraman'},\text{'campaigner'},\text{'captain'},\text{'cardiologist'},\text{'caretaker'},\text{'carpenter'},\text{'cartoonist'},\text{'cellist'},\text{'chancellor'},\text{'chaplain'},\text{'character'},\text{'chef'},\text{'chemist'},\text{'choreographer'},\text{'cinematographer'},\text{'citizen'},\text{'cleric'},\text{'clerk'},\text{'coach'},\text{'collector'},\text{'colonel'},\text{'columnist'},\text{'comedian'},\text{'comic'},\text{'commander'},\text{'commentator'},\text{'commissioner'},\text{'composer'},\text{'conductor'},\text{'confesses'},\text{'congressman'},\text{'constable'},\text{'consultant'},\text{'cop'},\text{'correspondent'},\text{'councilman'},\text{'councilor'},\text{'counselor'},\text{‘critic’}, \text{‘crooner’}, \text{‘crusader’}, \text{‘curator’}, \text{‘custodian’}, \text{‘dad’}, \text{‘dancer’}, \text{‘dean’}, \text{‘dentist’}, \text{‘deputy’}, \text{‘dermatologist’}, \text{‘detective’}, \text{‘diplomat’}, \text{‘director’}, \text{‘doctor’}, \text{‘drummer’}, \text{‘economist’}, \text{‘editor’}, \text{‘educator’}, \text{‘electrician’}, \text{‘employee’}, \text{‘entertainer’}, \text{‘entrepreneur’}, \text{‘environmentalist’}, \text{‘envoy’}, \text{‘epidemiologist’}, \text{‘evangelist’}, \text{‘farmer’}, \text{‘filmmaker’}, \text{‘financier’}, \text{‘firebrand’}, \text{‘firefighter’}, \text{‘fireman’}, \text{‘fisherman’}, \text{‘footballer’}, \text{‘foreman’}, \text{‘gangster’}, \text{‘gardener’}, \text{‘geologist’}, \text{‘goalkeeper’}, \text{‘guitarist’}, \text{‘hairdresser’}, \text{‘handyman’}, \text{‘headmaster’}, \text{‘historian’}, \text{‘hitman’}, \text{‘homemaker’}, \text{‘hooker’}, \text{‘housekeeper’}, \text{‘housewife’}, \text{‘illustrator’}, \text{‘industrialist’}, \text{‘infielder’}, \text{‘inspector’}, \text{‘instructor’}, \text{‘inventor’}, \text{‘investigator’}, \text{‘janitor’}, \text{‘jeweler’}, \text{‘journalist’}, \text{‘judge’}, \text{‘jurist’}, \text{‘laborer’}, \text{‘landlord’}, \text{‘lawmaker’}, \text{‘lawyer’}, \text{‘lecturer’}, \text{ ‘legislator’}, \text{ ‘librarian ’, ‘lieutenant ’, 'lifeguard ’,
$^{14}$ https://github.com/huggingface/pytorch-transformers
Figure 3: Layerwise performance of BERT-24 static embeddings for all possible choices of  $f, g$
Figure 4: Layerwise performance of GPT2-12 static embeddings for all possible choices of  $f, g$
'lyricist', 'maestro', 'magician', 'magistrate', 'manager', 'marksman', 'marshall', 'mathematician', 'mechanic', 'mediator', 'medic', 'midfielder', 'minister', 'missionary', 'mobster', 'monk', 'musician', 'nanny', 'narrator', 'naturalist', 'negotiator', 'neurologist', 'neurosurgeon', 'novelist', 'nun', 'nurse', 'observer', 'officer', 'organist', 'painter', 'paralegal', 'parishioner', 'parliamentarian', 'pastor', 'pathologist', 'patrolman', 'pediatrician', 'performer', 'pharmacist', 'philanthropist', 'philosopher', 'photographer', 'photojournalist', 'physician', 'physicist', 'pianist', 'planner', 'playwright', 'plumber', 'poet', 'policeman', 'politician', 'pollster', 'preacher', 'president', 'priest', 'principal', 'prisoner', 'professor', 'programmer', 'promoter', 'proprietor', 'prosecutor', 'protagonist', 'protege', 'protester', 'provost', 'psychiatrist', 'psychologist', 'publicist', 'pundit', 'rabbi', 'radiologist', 'ranger', 'realtor', 'receptionist', 'researcher', 'restaura
teur', 'sailor', 'saint', 'salesman', 'saxophonist', 'scholar', 'scientist', 'screenwriter', 'sculptor', 'secretary', 'senator', 'sergeant', 'servant', 'serviceman', 'shopkeeper', 'singer', 'skipper', 'socialite', 'sociologist', 'soldier', 'solicitor', 'soloist', 'sportsman', 'sportswriter', 'statesman', 'steward', 'stockbroker', 'strategist', 'student', 'stylist', 'substitute', 'superintendent', 'surgeon', 'surveyor', 'teacher', 'technician', 'teenager', 'therapist', 'trader', 'treasureer', 'trooper', 'trucker', 'trumpeter', 'tutor', 'tycoon', 'undersecretary', 'understudy', 'valedictorian', 'violinist', 'vocalist', 'waiter', 'waitress', 'warden', 'warrior', 'welder', 'worker', 'wrestler', 'writer'}
\(\mathcal{N}_{\mathrm{adj}} = \{\text{'disorganized', 'devious', 'impressionable', 'circumspect', 'impassive', 'aimless', 'effeminate', 'unfathomable', 'fickle', 'inoffensive', 'reactive', 'providential', 'resentful', 'bizarre', 'impractical', 'sarcastic', 'misguided', 'imitative',
Figure 5: Layerwise performance of GPT-24 static embeddings for all possible choices of  $f, g$
Model	N	RG65	WS353	SIMLEX999	SIMVERB3500
Word2Vec	-	0.6787	0.6838	0.4420	0.3636
GloVe	-	0.6873	0.6073	0.3705	0.2271
GPT2-12	10000	0.2843 (0)	0.4205 (1)	0.2613 (2)	0.1472 (6)
GPT2-12	50000	0.5000 (2)	0.5815 (1)	0.4378 (2)	0.2607 (2)
GPT2-12	100000	0.5156 (1)	0.6396 (0)	0.4547 (2)	0.3128 (6)
GPT2-24	10000	0.3149 (0)	0.5209 (0)	0.2940 (0)	0.1697 (0)
GPT2-24	50000	0.5362 (2)	0.6486 (0)	0.4350 (0)	0.2721 (0)
GPT2-24	100000	0.5328 (1)	0.6830 (0)	0.4505 (3)	0.3056 (0)
Table 4: Performance of Static Embeddings on Word Similarity and Word Relatedness Tasks.  $f$  and  $g$  are set to mean for all GPT2-models and (#) indicates the layer the embeddings are distilled from. Bold indicates best performing embeddings for a given dataset.
'pedantic', 'venomous', 'erratic', 'insecure', 'resourceful', 'neurotic', 'forgiving', 'profligate', 'whimsical', 'assertive', 'incorruptible', 'individualistic', 'faithless', 'disconcerting', 'barbaric', 'hypnotic', 'vindictive', 'observant', 'dissolute', 'frightening', 'complacent', 'boisterous', 'presentious', 'disobedient', 'tasteless', 'sedentary', 'sophisticated', 'regimental', 'mellow', 'deceitful', 'impulsive', 'playful', 'sociable', 'methodical', 'willful', 'idealistic', 'boyish', 'callous', 'pompous', 'unchanging', 'crafty', 'punctual', 'compassionate', 'intolerant', 'challenging', 'scornful', 'possessive', 'conceited', 'imprudent', 'dutiful', 'lovable', 'disloyal', 'dreamy', 'appreciative', 'forgetful', 'unrestrained', 'forceful', 'submissive', 'predatory', 'fanatical', 'illogical', 'tidy', 'aspiring', 'studious', 'adaptable', 'conciliatory', 'artful', 'thoughtless', 'deceptive', 'frugal', 'reflective', 'insulting', 'unreliable', 'stoic', 'hysterical', 'rustic', 'inhibited', 'outspoken', 'unhealthy', 'ascetic', 'skeptical', 'paintaking', con
templative', 'leisurely', 'sly', 'mannered', 'outrageous', 'lyrical', 'placid', 'cynical', 'irresponsible', 'vulnerable', 'arrogant', 'persuasive', 'perverse', 'steadfast', 'crisp', 'envious', 'naive', 'greedy', 'presumptuous', 'obnoxious', 'irritable', 'dishonest', 'discreet', 'sporting', 'hateful', 'ungrateful', 'frivolous', 'reactionary', 'skillful', 'cowardly', 'sordid', 'adventurous', 'dogmatic', 'intuitive', 'bland', 'indulgent', 'discontented', 'dominating', 'articulate', 'fanciful', 'discouraging', 'treacherous', 'repressed', 'moody', 'sensual', 'unfriendly', 'optimistic', 'clumsy', 'contemptible', 'focused', 'haughty', 'morbid', 'disorderly', 'considerate', 'humorous', 'preoccupied', 'airy', 'impersonal', 'cultured', 'trusting', 'respectful', 'scrupulous', 'scholarly', 'superstitious', 'tolerant', 'realistic', 'malicious', 'irrational', 'sane', 'colorless', 'masculine', 'witty', 'inert', 'prejudiced', 'fraudulent', 'blunt', 'childish', 'brittle', 'disciplined', 'responsive', 'courageous', 'bewildered', 'courteous', 'stubborn', 'aloof', 'sentimental', 'ath
Figure 6: Layerwise performance of RoBERTa-12 static embeddings for all possible choices of  $f, g$
Figure 7: Layerwise performance of RoBERTa-24 static embeddings for all possible choices of  $f, g$
RoBERTa-24
letic, 'extravagant', 'brutal', 'manly', 'cooperative', 'unstable', 'youthful', 'timid', 'amiable', 'retiring', 'fiery', 'confidential', 'relaxed', 'imaginative', 'mystical', 'shrewd', 'conscientious', 'monstrous', 'grim', 'questioning', 'lazy', 'dynamic', 'gloomy', 'troublesome', 'abrupt', 'eloquent', 'dignified', 'hearty', 'gallant', 'benevolent', 'maternal', 'paternal', 'patriotic', 'aggressive', 'competitive', 'elegant', 'flexible', 'gracious', 'energetic', 'tough', 'contradictory', 'shy', 'careless', 'cautious', 'polished', 'sage', 'tense', 'caring', 'suspicious', 'sober', 'neat', 'transparent', 'disturbing', 'passionate', 'obedient', 'crazy', 'restrained', 'fearful', 'daring', 'prudent', 'demanding', 'impatient', 'cerebral', 'calculating', 'amusing', 'honorable', 'casual', 'sharing', 'selfish', 'ruined', 'spontaneous', 'admirable', 'conventional', 'cheerful', 'solitary', 'upright', 'stiff', 'enthusiastic', 'petty', 'dirty', 'subjective', 'heroic'
'stupid', 'modest', 'impressive', 'orderly', 'ambitious', 'protective', 'silly', 'alert', 'destructive', 'exciting', 'crude', 'ridiculous', 'subtle', 'mature', 'creative', 'coarse', 'passive', 'oppressed', 'accessible', 'charming', 'clever', 'decent', 'miserable', 'superficial', 'shallow', 'stern', 'winning', 'balanced', 'emotional', 'rigid', 'invisible', 'desperate', 'cruel', 'romantic', 'agreeable', 'hurried', 'sympathetic', 'solemn', 'systematic', 'vague', 'peaceful', 'humble', 'dull', 'expeditent', 'loyal', 'decisive', 'arbitrary', 'earnest', 'confident', 'conservative', 'foolish', 'moderate', 'helpful', 'delicate', 'gentle', 'dedicated', 'hostile', 'generous', 'reliable', 'dramatic', 'precise', 'calm', 'healthy', 'attractive', 'artificial', 'progressive', 'odd', 'confused', 'rational', 'brilliant', 'intense', 'genuine', 'mistaken', driving, stable, objective, sensitive, neutral, strict, angry, profound, smooth, ignorant, thorough, logical, intelligent, extraordinary,
RoBERTa-12
Model	N	RG65	WS353	SIMLEX999	SIMVERB3500
Word2Vec	-	0.6787	0.6838	0.4420	0.3636
GloVe	-	0.6873	0.6073	0.3705	0.2271
RoBERTa-12	10000	0.5719 (0)	0.6618 (0)	0.4794 (0)	0.3968 (0)
RoBERTa-12	50000	0.6754 (0)	0.6867 (0)	0.501 (0)	0.4123 (0)
RoBERTa-12	100000	0.6597 (0)	0.6915 (0)	0.5098 (0)	0.4206 (0)
RoBERTa-12	500000	0.6675 (0)	0.6979 (0)	0.5268 (5)	0.4311 (0)
RoBERTa-12	1000000	0.6761 (0)	0.7018 (0)	0.5374 (5)	0.4442 (4)
RoBERTa-24	10000	0.5469 (1)	0.6144 (0)	0.4499 (0)	0.3403 (0)
RoBERTa-24	50000	0.6837 (1)	0.6412 (0)	0.4855 (0)	0.371 (0)
RoBERTa-24	100000	0.7087 (7)	0.6563 (6)	0.4959 (0)	0.3802 (0)
RoBERTa-24	500000	0.7557 (8)	0.663 (6)	0.5184 (18)	0.412 (6)
RoBERTa-24	1000000	0.739 (8)	0.6673 (6)	0.5318 (18)	0.4303 (9)
Table 5: Performance of Static Embeddings on Word Similarity and Word Relatedness Tasks.  $f$  and  $g$  are set to mean for all RoBERTa-models and (#) indicates the layer the embeddings are distilled from. Bold indicates best performing embeddings for a given dataset.
Figure 8: Layerwise performance of XLNet-12 static embeddings for all possible choices of  $f, g$
experimental, 'steady', 'formal', 'faithful', 'curious', 'reserved', 'honest', 'busy', 'educated', 'liberal', 'friendly', 'efficient', 'sweet', 'surprising', 'mechanical', 'clean', 'critical', 'criminal', 'soft', 'proud', 'quiet', 'weak', 'anxious', 'solid', 'complex', 'grand', 'warm', 'slow', 'false', 'extreme', 'narrow', 'dependent', 'wise', 'organized', 'pure', 'directed', 'dry', 'obvious', 'popular', 'capable', 'secure', 'active', 'independent', 'ordinary', 'fixed', 'practical', 'serious', 'fair', 'understanding', 'constant', 'cold', 'responsible', 'deep', 'religious', 'private', 'simple', 'physical', 'original', 'working', 'strong', 'modern', 'determined', 'open', 'political', 'difficult', 'knowledge', 'kind'}
$\mathcal{P} = \left\{\left(\text{'she '},\text{'he '}\right),\left(\text{'her '},\text{'his '}\right),\left(\text{'woman '},\right.\right.$  （20 man'),（mary',john'),（'herself',himself'), (daughter',son'),(mother',father'),（gal',
'guy'), ('girl', 'boy'), ('female', 'male')\}
$\mathcal{A}_{\mathrm{male}} = \{\mathrm{he},\mathrm{son},\mathrm{his},\mathrm{him},\mathrm{father},$  'man', boy, himself, male, brother, sons, fathers, men, boys, males, brothers, uncle, uncles, nephew, nephews
$\mathcal{A}_{\mathrm{female}} = \{\text{'she', 'daughter', 'hers', 'her', 'mother', 'woman', 'girl', 'herself', 'female', 'sister', 'daughters', 'mothers', 'women', 'girls', 'femen', 'sisters', 'aunt', 'aunts', 'niece', 'nieces}\}$
$\mathcal{A}_{\mathrm{white}} = \{\text{'harris', 'nelson', 'robinson', 'thompson', 'moore', 'wright', 'anderson', 'clark', 'jackson', 'taylor', 'scott', 'davis', 'allen', 'adams', 'lewis', 'williams', 'jones', 'wilson', 'martin', 'johnson'}\}$
$\mathcal{A}_{\mathrm{hispanic}} = \{\text{'castillo', 'gomez', 'soto', 'gonza-}$
15 We remove 'femen' when using Word2Vec as it is not in the vocabulary of the pretrained embeddings we use.
Figure 9: Layerwise performance of XLNet-24 static embeddings for all possible choices of  $f, g$
Model	N	RG65	WS353	SIMLEX999	SIMVERB3500
Word2Vec	-	0.6787	0.6838	0.4420	0.3636
GloVe	-	0.6873	0.6073	0.3705	0.2271
XLNet-12	10000	0.604 (0)	0.6482 (0)	0.483 (0)	0.3916 (0)
XLNet-12	50000	0.6056 (1)	0.6571 (0)	0.5157 (1)	0.3973 (1)
XLNet-12	100000	0.6239 (1)	0.6629 (0)	0.5185 (1)	0.4044 (3)
XLNet-12	500000	0.6391 (3)	0.6937 (3)	0.5392 (3)	0.4747 (4)
XLNet-12	1000000	0.6728 (3)	0.7018 (3)	0.5447 (4)	0.4918 (4)
XLNet-24	10000	0.6525 (0)	0.6935 (0)	0.5054 (0)	0.4332 (1)
XLNet-24	50000	0.6556 (0)	0.6926 (0)	0.5377 (5)	0.4492 (3)
XLNet-24	100000	0.6522 (3)	0.7021 (3)	0.5503 (6)	0.4545 (3)
XLNet-24	500000	0.66 (0)	0.7378 (6)	0.581 (8)	0.5095 (6)
XLNet-24	1000000	0.7119 (6)	0.7446 (7)	0.5868 (9)	0.525 (6)
Table 6: Performance of Static Embeddings on Word Similarity and Word Relatedness Tasks.  $f$  and  $g$  are set to mean for all XLNet-models and (#) indicates the layer the embeddings are distilled from. Bold indicates best performing embeddings for a given dataset.
lez', 'sanchez', 'rivera', 'martinez', 'torres', 'rodriguez', 'perez', 'lopez', 'medina', 'diaz', 'garcia', 'castro', 'cruz'}
$\mathcal{A}_{\mathrm{asian}} = \{\text{'cho', 'wong', 'tang', 'huang', 'chu', 'chung', 'ng', 'wu', 'liu', 'chen', 'lin', 'yang', 'kim', 'chang', 'shah', 'wang', 'li', 'khan', 'singh', 'hong'}\}$
$\mathcal{A}_{\mathrm{Islam}} = \{\text{'allah', 'ramadan', 'turban', 'emir', 'salaam', 'sunni', 'koran', 'imam', 'sultan', 'prophet', 'veil', 'ayatollah', 'shiite', 'mosque', 'islam', 'sheik', 'muslim', 'muhammad'}$
$\mathcal{A}_{\mathrm{christian}} = \{\text{'baptism', 'messiah', 'catholicism', 'resurrection', 'christianity', 'salvation', 'protestant', 'gospel', 'trinity', 'jesus', 'christ', 'christian', 'cross', 'catholic', 'church'}\}$
C Naming Conventions
Throughout this work, we make use of several naming conventions/substitutions. In the case of models, we use the form 'MODEL-X' where  $X$  indicates the number of layers in the model and consequently the model produces  $X + 1$  representations for any given subword (including the initial layer 0 representation). Table 9 describes the complete correspondence of our shorthand and the full names. In the case of model names, the full form is the name assigned to the pretrained model (that was possibly reimplemented) released by HuggingFace.
Figure 10: Layerwise performance of DistilBERT-6 static embeddings for all possible choices of  $f, g$
Model	N	RG65	WS353	SIMLEX999	SIMVERB3500
Word2Vec	-	0.6787	0.6838	0.4420	0.3636
GloVe	-	0.6873	0.6073	0.3705	0.2271
DistilBERT-6	10000	0.57 (0)	0.6828 (1)	0.4705 (0)	0.2971 (0)
DistilBERT-6	50000	0.7257 (1)	0.6928 (1)	0.5043 (0)	0.3121 (0)
DistilBERT-6	100000	0.7245 (1)	0.7164 (1)	0.5077 (0)	0.3207 (1)
DistilBERT-6	500000	0.7363 (1)	0.7239 (1)	0.5093 (0)	0.3444 (2)
DistilBERT-6	1000000	0.7443 (1)	0.7256 (1)	0.5095 (0)	0.3536 (3)
Table 7: Performance of Static Embeddings on Word Similarity and Word Relatedness Tasks.  $f$  and  $g$  are set to mean for all DistilBERT-models and (#) indicates the layer the embeddings are distilled from. Bold indicates best performing embeddings for a given dataset.
Figure 11: Layerwise bias of BERT-24 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$  Left: Gender, Center: Race, Right: Religion
Figure 12: Layerwise bias of GPT2-12 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$   
Left: Gender, Center: Race, Right: Religion
Figure 13: Layerwise bias of GPT2-24 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$  Left: Gender, Center: Race, Right: Religion
Figure 14: Layerwise bias of RoBERTa-12 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$  Left: Gender, Center: Race, Right: Religion
Figure 15: Layerwise bias of RoBERTa-24 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$  Left: Gender, Center: Race, Right: Religion
Figure 16: Layerwise bias of XLNet-12 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$  Left: Gender, Center: Race, Right: Religion
Figure 17: Layerwise bias of XLNet-24 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$  Left: Gender, Center: Race, Right: Religion
Figure 18: Layerwise bias of DistilBERT-6 static embeddings for  $f =$  mean,  $g =$  mean,  $N = 100000$  Left: Gender, Center: Race, Right: Religion
Gender	Race M	Religion
B, P	GE, P	GC, P	M, P	GE	GC	M	GE	GC	M
Word2Vec	0.0482	0.1656	0.0435	0.1347	0.1247	0.0343	0.1178	0.0661	0.13	0.0434	0.1264
GloVe	0.095	0.2206	0.0403	0.1289	0.2017	0.0355	0.1108	0.0714	0.2341	0.0606	0.0675
BERT-12	0.0506	0.2637	0.0213	0.2684	0.1879	0.0175	0.2569	0.2358	0.8858	0.0365	0.2677
BERT-24	0.0389	0.4405	0.0277	0.199	0.2978	0.0248	0.189	0.1768	0.5505	0.0316	0.212
GPT2-12	0.4631	26.0809	0.0176	0.6126	2.1238	0.0068	0.7101	0.621	4.4775	0.0152	0.7525
GPT2-24	0.6707	40.4664	0.0141	0.8367	2.1771	0.0023	0.89	0.843	8.3889	0.0064	0.9006
RoBERTa-12	0.0381	0.1754	0.005	0.8472	0.1649	0.0046	0.8444	0.8153	0.2608	0.0069	0.8387
RoBERTa-24	0.0248	0.2626	0.0064	0.7647	0.1821	0.0048	0.7562	0.73	0.4492	0.0117	0.7472
XLNet-12	0.0399	0.6265	0.0312	0.2214	0.3354	0.0237	0.2196	0.1911	0.4716	0.0321	0.2549
XLNet-24	0.0468	0.5423	0.025	0.3307	0.2697	0.0153	0.3144	0.2871	0.4318	0.0282	0.3235
DistilBERT-6	0.0353	0.4274	0.0247	0.2825	0.2461	0.0185	0.2824	0.2603	0.6842	0.035	0.2994
Table 8: Social bias within static embeddings from different pretrained models with respect to a set of adjectives,  $\mathcal{N}_{adj}$ . Parameters are set as  $f = \text{mean}$ ,  $g = \text{mean}$ ,  $N = 100000$  and the layer of the pretrained model used in distillation is  $\lfloor \frac{X}{4} \rfloor$ .
Our Shorthand	Full Name
BERT-12	bert-base-uncased
BERT-24	bert-large-uncased
GPT2-12	gpt2
GPT2-24	gpt2-medium
RoBERTa-12	roberta-base
RoBERTa-24	roberta-large
XLNet-12	xlnet-base-cased
XLNet-24	xlnet-base-cased
DistilBERT-6	distilbert-base-uncased
SL999	SIMLEX999
SV3500	SIMVERB3500
B	biasBOLUKBASI
GE	biasGARG-EUC
GC	biasGARG-COS
M	biasMANZINI
Table 9: Naming conventions used throughout this work