CMCL 2024 (Findings of ACL 2024)*daiasami@udel.edu

What Makes Language Models Good-enough?
*Daiki Asami (University of Delaware)

Paper

Saku Sugawara (National Institute of Informatics)

Background: Heuristic Language Processing
Language Models:§ LMs often adopt heuristics to make predictions.
e.g., BERT predicts entailment for the following pair (McCoy+, 2019):
Premise: The doctor was paid by the actor. Hypothesis: The actor paid the doctor.

Humans:§ Humans occasionally adopt good-enough heuristic language processing rather than detailedalgorithmic language processing (Ferreira, 2003).
e.g., They misinterpret the dog was bitten by the teacher as the teacher was bitten by the dog.
§ Heuristic processing can occur under a cognitively demanding situation (e.g., sentence complexity).

Research Questions and Hypotheses
RQ1: To what extent is LMs’ heuristic processing human-like?

RQ2: What architectural features contribute to their good-enough language processing?
Deep architecture enables LMs to learn a syntactic generalization (Mueller and Linzen, 2023). → The shallow architecture would lead to good-enough models.
Self-attention head mechanism resembles a working-memory system (Ryu and Lewis, 2021). → The small number of heads would contribute to good-enough models.

Good-enough Language Processing (GELP) Dataset for Natural Language Inference Task
7,680 items (80 sentences * 8 construction types * 3 working-memory loads * 2 premise plausibility types * 2 hypothesis types):§ 8 constructions: (1) transitive, (2) passive, (3) double object, (4) dative, (5) benefactive double object, (6) benefactive for, (7) experiencer subject, & (8) experiencer object constructions.
§ We make 80 sentences for each construction.
§ 3 memory loads: low, medium, and high memory load conditions with one, two, and three
propositions in premises, respectively.

§ 2 plausibility types : implausibility caused by swapping animate and inanimate nouns.
§ 2 hypothesis labels: entailment and non-entailment (covering neutral and contradiction).

Human annotation:§ We collect three human labels for each item on Amazon Mechanical Turk.
§ We assign the human label to each item based on the majority label.

Model Evaluation
24 BERT miniatures (6 numbers of layers (L ∈ {2, 4, 6, 8, 10, 12}) * 4 numbers of self-attention heads (A ∈ {2, 4, 8, 12})) (Turc et al., 2019) fine-tuned on NLI task.

Human–model matching score (= the proportion of the match btw
models’ predicted labels and human labels) for 24 BERT models.

Accuracy for BERT with 8 or 12 heads based on three memory loads.

Accuracy for humans and BERT L12–A12 (the best performing model).

Discussion
RQ1: To what extent is LMs’ heuristic processing human-like?

§ Taking 70% human–model matching score as a threshold, the full model as well as models with fewer layers and/or heads exhibit good-enough performance.
RQ2: What architectural features contribute to their good-enough language processing?
§ Increasing layers (i.e., 8 to 12) does not improve the performance considerably. The shallow models exhibit a good-enough performance like their deeper version.
§ The model with the largest number of heads (i.e., 12) shows decreasing accuracy as the memory load increases like humans. This result does not confirm ourhypothesis that the aimed models require a fewer number of heads.
References
Ferreira. 2003. The Misinterpretation of Noncanonical Sentences. In Cognitive Psychology; McCoy et al. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In ACL; Mueller and Linzen.
2023. How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases. In ACL; Ryu and Lewis. 2021. Accounting for Agreement Phenomena in Sentence Comprehension withTransformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention in CMCL; Turc et al. 2021. Well-read Students Learn Better: On the Importance of Pre-training Compact Models. In arXiv.