SlowGuess/NAACL_2022_OCR huggingface dataset:
Among the masked-language models, BERT-Base surprisingly performed the best (Table 2). BERT-Large achieved the better Increased Log Probabilities, but the FFN classifier still worked better with the vector representations of the Base variant. This suggests that although associations are represented in a more fine-grained manner in BERT-Large, they are more difficult to retrieve due to the size of this model.
Among the masked-language models, GPT-J (which was trained with by far the largest training data) performs best (Table 3). Context-based models generally seem to determine the target given the source ( $P(target \mid source)$ ) more easily than the reverse ( $Psource \mid target$ ). With verbs, on
the other hand, the reverse effect occurs. The GPT models show that the results for sources are better when weighted, while for targets the results are better without weighting.
In general, the SVM performed surprisingly well, even though only a linear kernel was used. But also the KNN method mostly performed better than the similarity measures. However, FFN performs best in all cases, outperforming cosine (worst case) by increases in the interval  $[6\%, 52\%]$  and outperforming the KNN approach (worst classifier) in each case by increases in the interval  $[2\%, 43\%]$ .
5.2 Dataset-related Observations
In terms of rooms, bathroom scores the best, while living room or office usually score the worst. This may be because many bathroom objects are related to specific bathroom activities (e.g., toothbrush, bathtub), while objects that used to be located in the living room are increasingly found in other rooms (e.g., television in the bedroom). This would also explain why the results for kitchen are also better.
On the part dataset, the static models actually performed significantly better than the contextualized models. This relates especially to GloVe and fastText which outperformed almost all contextualized models. Thus, static models are in some cases a good alternative to their contextualized counterparts. However, the more technical the objects become (here mortise lock and dishwasher), the worse the results become.
On the verb dataset, the contextualized models perform minimally better. As mentioned earlier, these models can associate objects with verbs more easily than the other way around. Here, the largest difference in performance is observed in the case of Levy, where the results are almost equal to those of the other models, probably due to the learning strategy based on dependency trees.
In summary, knowledge extraction using language models, whether static or contextualized, is more effective using classifiers than using similarity measures commonly used in the field of bias research: there is potential for this type of knowledge extraction, but at the price of training classifiers – if one uses similarity measures instead, this knowledge is mostly out of reach.
5.3 Relation Observation
All previous evaluations only examined associations between instances and concepts, but not whether the models represent their true relations.
$^{6}$ https://books.google.com/ngrams
Word2Vec	GloVe	Levy	fastText	static-BERT
cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn	cos	dist	kend	knn	svm	ffn
Room	bathroom	0.37	0.37	0.37	0.39	0.62	0.82	0.38	0.39	0.38	0.57	0.93	0.93	0.39	0.40	0.39	0.14	0.34	0.37	0.53	0.53	0.52	0.73	0.67	0.90	0.54	0.50	0.50	0.25	0.66	0.70
bedroom	0.20	0.20	0.20	0.13	0.49	0.70	0.31	0.29	0.30	0.28	0.66	0.45	0.21	0.21	0.21	0.10	0.25	0.11	0.30	0.31	0.32	0.26	0.44	0.59	0.28	0.27	0.27	0.35	0.33	0.35
kitchen	0.35	0.34	0.35	0.20	0.55	0.53	0.37	0.40	0.41	0.52	0.65	0.81	0.17	0.17	0.18	0.09	0.32	0.30	0.38	0.36	0.34	0.41	0.66	0.76	0.40	0.41	0.41	0.45	0.53	0.68
living room	0.23	0.23	0.24	0.06	0.33	0.35	0.30	0.27	0.28	0.10	0.49	0.51	0.24	0.24	0.23	0.40	0.16	0.25	0.25	0.26	0.24	0.09	0.36	0.60	0.19	0.19	0.19	0.00	0.10	0.46
office	0.28	0.28	0.26	0.51	0.51	0.55	0.14	0.31	0.35	0.51	0.59	0.64	0.25	0.27	0.28	0.40	0.36	0.25	0.25	0.30	0.33	0.45	0.32	0.63	0.40	0.44	0.45	0.10	0.21	0.32
CONC	0.23	0.23	0.23	0.22	0.50	0.60	0.27	0.31	0.32	0.37	0.67	0.67	0.16	0.15	0.15	0.15	0.11	0.23	0.30	0.31	0.31	0.40	0.45	0.70	0.31	0.31	0.31	0.18	0.39	0.48
Bed	0.41	0.41	0.40	0.64	0.56	0.56	0.38	0.51	0.51	0.56	0.76	0.84	-	-	-	-	-	-	0.42	0.51	0.52	0.69	0.61	0.67	0.47	0.48	0.46	0.16	0.59	0.54
dishwasher	0.19	0.23	0.23	0.06	0.37	0.27	0.33	0.32	0.30	0.03	0.19	0.32	-	-	-	-	-	0.35	0.33	0.33	0.06	0.13	0.23	0.17	0.17	0.17	0.13	0.28	0.31
door	0.12	0.11	0.11	0.54	0.75	0.75	0.19	0.23	0.22	0.48	0.81	0.85	-	-	-	-	-	0.25	0.27	0.24	0.36	0.55	0.84	0.24	0.25	0.25	0.36	0.73	0.67
mortise lock	0.15	0.16	0.16	0.16	0.50	0.54	0.22	0.26	0.28	0.45	0.74	0.68	-	-	-	-	-	0.11	0.17	0.20	0.68	0.55	0.68	0.20	0.21	0.21	0.14	0.49	0.47
refrigerator	0.44	0.46	0.46	0.51	0.47	0.52	0.53	0.57	0.56	0.55	0.55	0.66	-	-	-	-	-	0.54	0.58	0.58	0.28	0.40	0.55	0.50	0.50	0.50	0.56	0.56	0.53
toilet	0.28	0.28	0.28	0.01	0.49	0.55	0.33	0.33	0.32	0.31	0.63	0.60	-	-	-	-	-	0.37	0.34	0.33	0.55	0.50	0.72	0.24	0.23	0.23	0.34	0.57	0.58
CONC	0.25	0.27	0.26	0.28	0.52	0.53	0.30	0.34	0.34	0.39	0.60	0.65	-	-	-	-	-	0.28	0.33	0.33	0.35	0.43	0.61	0.29	0.29	0.29	0.23	0.54	0.52
eat	0.79	0.79	0.77	0.89	0.89	0.89	0.77	0.86	0.80	0.89	0.89	0.92	0.46	0.45	0.45	0.66	0.87	0.87	0.73	0.80	0.79	0.69	0.89	0.89	0.83	0.84	0.83	0.61	0.89	0.87*
listen to	0.54	0.64	0.56	0.21	0.38	0.46	0.59	0.70	0.65	0.06	0.53	0.49	0.28	0.22	0.23	0.20	0.38	0.52	0.42	0.53	0.63	0.21	0.42	0.40	0.54	0.56	0.53	0.00	0.39	0.50
play	0.64	0.69	0.64	0.60	0.66	0.60	0.65	0.80	0.73	0.43	0.45	0.45	0.44	0.45	0.43	0.41	0.50	0.57	0.63	0.69	0.68	0.28	0.66	0.66	0.56	0.56	0.54	0.00	0.49	0.63*
read	0.43	0.52	0.48	0.38	0.59	0.61	0.51	0.60	0.59	0.48	0.53	0.50	0.31	0.31	0.31	0.49	0.31	0.50	0.54	0.56	0.59	0.42	0.50	0.59	0.48	0.52	0.48	0.00	0.31	0.47
wash with	0.53	0.54	0.53	0.48	0.61	0.63	0.48	0.57	0.53	0.66	0.66	0.62	0.37	0.34	0.35	0.41	0.66	0.62	0.45	0.51	0.49	0.67	0.66	0.66	0.39	0.40	0.40	0.11	0.55	0.61
wear	0.76	0.78	0.76	0.88	0.84	0.88	0.80	0.87	0.84	0.88	0.83	0.85	0.56	0.52	0.50	0.82	0.85	0.85	0.77	0.80	0.79	0.59	0.93	0.92	0.78	0.82	0.80	0.72	0.81	0.84
CONC	0.58	0.60	0.57	0.56	0.64	0.67	0.59	0.68	0.65	0.55	0.65	0.65	0.34	0.32	0.31	0.46	0.59	0.65	0.51	0.58	0.58	0.43	0.66	0.68	0.54	0.55	0.54	0.15	0.56	0.65
Table 2: All results of the contextual masked-language models. cos: Cosine Measure, m-s: Masked-Source Log Score, m-t: Masked-Target Log Score, knn: K-Nearest Neighbors,svm: Support Vector Machine, fnn: Feed-Forward Network. (A gray cell indicates significant values at  $p < 0.01$ )
GPT2	GPT-Neo	GPT-J
cos	p-s	p-s-1	p-t	p-t-1	knn	svm	ffn	cos	p-s	p-s-1	p-t	p-t-1	knn	svm	ffn	cos	p-s	p-s-1	p-t	p-t-1	knn	svm	ffn
Room	bathroom	0.52	0.20	0.38	0.50	0.37	0.31	0.95	0.95	0.30	0.22	0.51	0.36	0.25	0.53	0.89	0.91	0.50	0.26	0.60	0.66	0.48	0.35	0.89	0.92
bedroom	0.26	0.31	0.23	0.47	0.38	0.26	0.61	0.54	0.19	0.33	0.21	0.53	0.48	0.55	0.49	0.57	0.24	0.32	0.23	0.62	0.48	0.33	0.70	0.64
kitchen	0.34	0.41	0.45	0.69	0.60	0.53	0.82	0.83	0.31	0.49	0.67	0.70	0.57	0.38	0.51	0.81	0.33	0.36	0.52	0.83	0.70	0.70	0.82	0.83
living room	0.21	0.43	0.26	0.41	0.33	0.16	0.27	0.46	0.26	0.57	0.39	0.60	0.44	0.28	0.13	0.48	0.21	0.50	0.47	0.67	0.45	0.46	0.40	0.63
office	0.13	0.21	0.43	0.37	0.23	0.31	0.44	0.73	0.33	0.30	0.43	0.46	0.34	0.24	0.53	0.72	0.23	0.36	0.53	0.49	0.39	0.37	0.52	0.69
CONC	0.26	0.23	0.30	0.46	0.35	0.30	0.61	0.72	0.15	0.34	0.44	0.44	0.35	0.40	0.52	0.71	0.23	0.32	0.42	0.56	0.41	0.42	0.66	0.74
Part	bed	0.36	0.30	0.51	0.67	0.45	0.55	0.59	0.70	0.32	0.38	0.46	0.77	0.70	0.66	0.78	0.88	0.46	0.38	0.36	0.81	0.68	0.71	0.83	0.84
dishwasher	0.11	0.34	0.22	0.25	0.23	0.18	0.21	0.28	0.06	0.23	0.30	0.30	0.29	0.15	0.15	0.24	0.09	0.26	0.30	0.44	0.38	0.12	0.15	0.32
door	0.23	0.07	0.14	0.20	0.28	0.20	0.65	0.66	0.27	0.10	0.17	0.35	0.42	0.20	0.44	0.66	0.15	0.13	0.12	0.37	0.41	0.25	0.67	0.77
mortise lock	0.07	0.34	0.43	0.17	0.18	0.27	0.63	0.65	0.11	0.49	0.42	0.30	0.22	0.27	0.49	0.61	0.15	0.43	0.43	0.47	0.31	0.04	0.63	0.66
refrigerator	0.42	0.41	0.24	0.53	0.52	0.47	0.39	0.51	0.29	0.33	0.33	0.47	0.55	0.44	0.47	0.57	0.46	0.51	0.41	0.57	0.63	0.55	0.53	0.63
toilet	0.29	0.36	0.44	0.20	0.17	0.16	0.49	0.54	0.27	0.42	0.50	0.25	0.26	0.23	0.50	0.58	0.32	0.45	0.48	0.32	0.37	0.26	0.53	0.62
CONC	0.14	0.24	0.28	0.28	0.25	0.29	0.47	0.54	0.12	0.30	0.34	0.37	0.36	0.32	0.46	0.57	0.16	0.34	0.32	0.42	0.43	0.33	0.54	0.62
Verb	eat	0.49	0.82	0.65	-	-	0.82	0.87	0.87	0.45	0.86	0.66	-	-	0.76	0.87	0.88	0.48	0.68	0.74	-	-	0.63	0.89	0.89
listen to	0.22	0.57	0.51	-	-	0.29	0.50	0.58	0.22	0.47	0.47	-	-	0.20	0.42	0.55	0.21	0.60	0.56	-	-	0.29	0.52	0.60
play	0.40	0.64	0.62	-	-	0.62	0.61	0.59	0.32	0.66	0.61	-	-	0.20	0.55	0.62	0.34	0.66	0.70	-	-	0.37	0.67	0.67
read	0.32	0.63	0.30	-	-	0.32	0.45	0.45	0.32	0.61	0.34	-	-	0.29	0.52	0.49	0.40	0.63	0.41	-	-	0.20	0.59	0.49
Wash with	0.39	0.77	0.51	-	-	0.57	0.61	0.63	0.28	0.66	0.52	-	-	0.39	0.41	0.60	0.23	0.69	0.52	-	-	0.66	0.61	0.64
wear	0.44	0.38	0.72	-	-	0.76	0.84	0.87	0.16	0.39	0.66	-	-	0.68	0.79	0.85	0.21	0.38	0.62	-	-	0.87	0.84	0.87
CONC	0.31	0.52	0.53	-	-	0.51	0.64	0.66	0.19	0.49	0.52	-	-	0.40	0.59	0.66	0.23	0.50	0.56	-	-	0.48	0.68	0.69
Table 3: All results of the contextual causal-language models. p-s: Predict Source Score, p-s-l: : Predict Source Log Score, p-t: Predict Target Score, p-t-l: Predict Target Log Score. The gap for the verb p-t score is due to the lack of an easily applicable sentence templates in this direction. (A gray cell indicates significant values at  $p < 0.01$ )
Figure 1: Small relation evaluation of BERT-large after the method of Kurita et al. (2019).
To fill this gap, we repeated the experimental setup of Kurita et al. (2019) for the room and part dataset on BERT-large, but this time masked the relation. The results are shown in Figure 1. Our selection of relations does not claim to be exhaustive, but serves as an illustration. It shows that while BERT-large is still very good at assigning objects in rooms, the dominant relation predicted for parts is used by. This suggests that BERT has problems correctly assigning object parts, an observation that could explain its poorer results while being consistent with findings of (Lin et al., 2020) (e.g., regarding counting parts).
As good as the results obtained using classifiers are, they must be viewed with caution. One can attribute their success to the fine-tuning of numerous parameters (and ultimately to overfitting); however, one can also attribute this success to nonlinear structuring of the information encoded in language models. In other words, these models appear to encode object knowledge, but require a sophisticated apparatus to retrieve it. Thus, they should not be considered as an alternative to unsupervised
Another issue is that our experiments do not yet allow for a comparison of model architectures, as the models studied differ significantly in terms of the size of their parameter spaces and training data. Our experiments do suggest that certain smaller models come close to or even outperform the results of larger models. However, a comparison of model architectures would require controlling for these parameters. Nevertheless, the results we have obtained are, in part, promising enough to encourage such research.
Finally, our experiments show that static models can perform better than contextualized models to some extent. This finding is conditioned by our experiments and their context of application. These observations that older models perform better on certain tasks are consistent with other work (e.g. LSTMs on small datasets for intent classification (Ezen-Can, 2020) or definiteness prediction (Kabbara and Cheung, 2021). At this point, a much broader analysis is needed (considering more areas and object relations), which also exploits contextual knowledge represented in contextualized models more than has been done here and in related work. Nevertheless, it is generally difficult to obtain data for such a broader analysis, and our experiments are already broader in scope and consider finer relationships than similar approaches.
We evaluated static and contextualized models as potential resources for object-related knowledge extraction. To this end, we examined three datasets (to identify typical artifacts in rooms, objects of actions, or parts of objects). We also experimented with different similarity measures and classifiers to extract the information contained in the language models. In doing so, we have shown that the models in combination with the measures differ greatly in terms of the amount of knowledge they allow for extracting. There is a weak trend that BERT-Base is the best performer among contextualized models, and GloVe and fastText among static models. Secondly, our results suggest that approaches based on classifiers perform significantly better than similarity measures. Thirdly, we have shown that static models perform almost as well as contextualized models – in some cases even better. This result shows that research on these models needs to be advanced. In future work we will also investigate
how grounded language models perform on such datasets. However, as noted above, this requires a significant expansion of bias research, such as that conducted here to enable knowledge extraction.
The support by the Stiftung Polytechnische Gesellschaft (SPTG) is gratefully acknowledged.
Hossein Azarpanah and Mohsen Farhadloo. 2021. Measuring biases of word embeddings: What similarity measures and descriptive statistics to use? In Proceedings of the First Workshop on Trustworthy Natural Language Processing, pages 8-14.
Yonatan Belinkov and James Glass. 2019. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1-4. Springer.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata.
Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758-4781, Online. Association for Computational Linguistics.
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7456-7463.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack
Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago.
Cuong Xuan Chu, Niket Tandon, and Gerhard Weikum. 2017. Distilling task knowledge from how-to communities. In Proceedings of the 26th International Conference on World Wide Web, pages 805-814.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Jeff Da and Jungo Kasai. 2019. Cracking the contextual commonsense code: Understanding commonsense reasoning aptitude of deep contextual representations. CoRR, abs/1910.01157.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.
Aysu Ezen-Can. 2020. A comparison of LSTM and BERT for small corpus. CoRR, abs/2009.05451.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? In CogSci.
Huan Fu, Bowen Cai, Lin Gao, Lingxiao Zhang, Cao Li, Qixun Zeng, Chengyue Sun, Yiyun Fei, Yu Zheng, Ying Li, Yi Liu, Peng Liu, Lin Ma, Le Weng, Xiaohang Hu, Xin Ma, Qian Qian, Rongfei Jia, Binqiang Zhao, and Hao Zhang. 2020. 3d-front: 3d furnished rooms with layouts and semantics. arXiv preprint arXiv:2011.09127.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027.
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: What works and what doesn't. In Proceedings of the NAACL-HLT SRW, pages 47-54, San Diego, California, June 12-17, 2016. ACL.
Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6):1464.
Prakhar Gupta and Martin Jaggi. 2021. Obtaining better static word embeddings using contextual embedding models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5241-5253, Online. Association for Computational Linguistics.
William G Hayward and Michael J Tarr. 1995. Spatial language and spatial representation. Cognition, 55(1):39-84.
Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1772-1791, Online. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? (extended abstract). In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 5065-5069. International Joint Conferences on Artificial Intelligence Organization. Journal track.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-438.
Jad Kabbara and Jackie Chi Kit Cheung. 2021. Post-editing extractive summaries by definiteness prediction. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3682-3692, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81-93.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Stephen Kokoska and Daniel Zwillinger. 2000. CRC standard probability and statistics tables and formulae. CRC Press.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942.
Barbara Landau and Ray Jackendoff. 1993. "what" and "where" in spatial language and spatial cognition. Behavioral and Brain Sciences, 16(2):217-238.
Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27:2177-2185.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265-5275, Online. Association for Computational Linguistics.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862-6868, Online. Association for Computational Linguistics.