Analysis Set Results	The results for the major features and minor features are shown in Figures FIGREF26 and FIGREF35 , respectively. For each feature, we measure the MCC of the sentences including that feature. We plot the mean of these results across the different restarts for each model, and error bars mark the mean INLINEFORM0 standard deviation. For the Violations features, MCC is technically undefined because these features only contain unacceptable sentences. We report MCC in these cases by including for each feature a single acceptable example that is correctly classified by all models. Comparison across features reveals that the presence of certain features has a large effect on performance, and we comment on some overall patterns below. Within a given feature, the effect of model type is overwhelmingly stable, and resembles the overall difference in performance. However, we observe several interactions, i.e. specific features where the relative performance of models does not track their overall relative performance. Among the major features (Figure FIGREF26 ), performance is universally highest on the simple sentences, and is higher than each model's overall performance. Though these sentences are simple, we notice that the proportion of ungrammatical ones is on par with the entire dataset.

These four features mark various determiners, including quantifiers, partitives (two of the boys), negative polarity items (I *do/don't have any pie), and comparative constructions. These three features apply only to unacceptable sentences, and only ones which are ungrammatical due to a semantic or morphological violation, or the presence or absence of a single salient word.

Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments	Recent pretrained sentence encoders achieve state of the art results on language understanding tasks, but does this mean they have implicit knowledge of syntactic structures? We introduce a grammatically annotated development set for the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018), which we use to investigate the grammatical knowledge of three pretrained encoders, including the popular OpenAI Transformer (Radford et al., 2018) and BERT (Devlin et al., 2018). We fine-tune these encoders to do acceptability classification over CoLA and compare the models' performance on the annotated analysis set. Some phenomena, e.g. modification by adjuncts, are easy to learn for all models, while others, e.g. long-distance movement, are learned effectively only by models with strong overall performance, and others still, e.g. morphological agreement, are hardly learned by any model.

Length Analysis	For comparison, we analyze the effect of sentence length on acceptability classifier performance. The results are shown in Figure FIGREF39 . The results for the CoLA baseline are inconsistent, but do drop off as sentence length increases. For BERT and GPT, performance decreases very steadily with length. Exceptions are extremely short sentences (length 1-3), which may be challenging due to insufficient information; and extremely long sentences, where we see a small (but somewhat unreliable) boost in BERT's performance. BERT and GPT are generally quite close in performance, except on the longest sentences, where BERT's performance is considerably better.

There are birds singing) are, by definition, non-canonical arguments, and thus are a subset of add arg. However, some added arguments, such as benefactives (Bo baked Mo a cake), are not expletives. Second, some correlations can be attributed to grammatical properties of the relevant constructions. For instance, question and aux are correlated because main-clause questions in English require subject-aux inversion and in many cases the insertion of auxiliary do (Do lions meow?). Third, some correlations may be a consequence of the sources sampled in CoLA and the phenomena they focus on. For instance, the unusually high correlation of Emb-Q and ellipsis/anaphor can be attributed to  , which is an article about the sluicing construction involving ellipsis of an embedded interrogative (e.g. I saw someone, but I don't know who). Finally, two strongest anti-correlations between major features are between simple and the two features related to argument structure, argument types and arg altern. This follows from the definition of simple, which excludes any sentence containing a large number or unusual configuration of arguments.

Our results are purely correlational, and do not mark whether a particular construction is crucial for the acceptability of the sentence. Future experiments following ettinger2018assessing and kann2019verb can semi-automatically generate datasets manipulating, for example, length of long-distance dependencies, inflectional violations, or the presence of interrogatives, while controlling for factors like sentence length and word choice, in order determine the extent to which these features impact the quality of sentence embeddings.

Feature Descriptions	Here we briefly summarize the feature set in order of the major features. Many of these constructions are well-studied in syntax, and further background can be found in textbooks such as adger2003core and sportiche2013introduction. This major feature contains only one minor feature, simple, including sentences with a syntactically simplex subject and predicate. These three features correspond to predicative phrases, including copular constructions, small clauses (I saw Bo jump), and resultatives/depictives (Bo wiped the table clean). These six features mark various kinds of optional modifiers. This includes modifiers of NPs (The boy with blue eyes gasped) or VPs (The cat meowed all morning), and temporal (Bo swam yesterday) or locative (Bo jumped on the bed). These five features identify syntactically selected arguments, differentiating, for example, obliques (I gave a book to Bo), PP arguments of NPs and VPs (Bo voted for Jones), and expletives (It seems that Bo left).

(222) Gerunds/Present participles Ṣtudents studying English reads Conrad's Heart of Darkness while at university. (262) Knowing the country well, he took a short cut. (411) John became deadly afraid of flying. (440) . Subject VPs Ṫo please John is easy. (315) . Nominalized VPs Ẉhat Mary did Bill was give a book. (473) . Excluded ṫo-VPs acting as complements or modifiers of verbs, nouns, or adjectives

Overall CoLA Results	The overall performance of the three sentence encoders is shown in Table TABREF33 . Performance on CoLA is measured using MCC  . We present the best single restart for each encoder, the mean over restarts for an encoder, and the result of ensembling the restarts for a given encoder, i.e. taking the majority classification for a given sentence, or the majority label of acceptable if tied. For BERT results, we exclude 5 out of the 20 restarts because they were degenerate (MCC=0). Across the board, BERT outperforms GPT, which outperforms the CoLA baseline. However, BERT and GPT are much closer in performance than they are to CoLA baseline. While ensemble performance exceeded the average for BERT and GPT, it did not outperform the best single model.

Arguments are usually selected for by the head, and they are generally not optional. In this case, the argument introduced with by is semantically selected for by the verb, but it is syntactically optional. See [p.190]adger2003core and []collins2005smuggling. . Included Ṗassives Ṫed was bitten by the spider. (613) . Subjects of deverbal nouns ṫhe attempt by John to leave surprised me. (1003) Expletives, or “dummy” arguments, are semantically inert arguments. The most common expletives in English are it and there, although not all occurrences of these items are expletives. Arguments are usually selected for by the head, and they are generally not optional. In this case, the expletive occupies a syntactic argument slot, but it is not semantically selected by the verb, and there is often a syntactic variation without the expletive. See [p.170-172]adger2003core and [p.82-83]kim2008syntax. .

Measure nouns İ bought three quarts of wine and two of Clorox. (667) . Possessed relational nouns J̣ohn's mother likes himself. (484) . Excluded Ṅouns with PP modifiers Ṡome people consider dogs in my neighborhood dangerous. (802) Transitive (non-relational) nouns take a VP or CP argument. See [pp.82-83]kim2008syntax. . Included V̇P argument ṫhe attempt by John to leave surprised me. (1003) . CP argument Ẉhich report that John was incompetent did he submit? (69) . QP argument Ṫhat is the reason why he resigned. (313) These are complex NPs, including coordinated nouns and nouns with modifiers (excluding prenominal adjectives). . Included Ṁodified NPs Ṭhe madrigals which Henry plays the lute and sings sound lousy. (84) John bought a book on the table. (233) . NPs with coordination Ṭhe soundly and furry cat slept.

Violations	These are sentences that include a semantic violation, including type mismatches, violations of selectional restrictions, polarity violations, definiteness violations. . Included V̇olation of selectional restrictions ṃany information was provided. (218) *It tries to leave the country. (275) . Aspectual violations J̣ohn is tall on several occasions. (540) . Definiteness violations Ịt is the problem that he is here. (1018) . Polarity violations Ȧny man didn't eat dinner. (388) These are sentences that include a violation in inflectional morphology, including tense-aspect marking, or agreement. . Included Ċase Ụs love they. (46) . Agreement Ṣtudents studying English reads Conrad's Heart of Darkness while at university. (262) . Gender Ṣally kissed himself. (339) . Tense/Aspect Ḳim alienated cats and beating his dog. (429) These are sentences with a violation that can be identified with the presence or absence of a single word. .

Excluding polar questions, sentences with question-like syntax almost always involve extraction of a wh-word, creating a long-distance dependency between the wh-word and its extraction site, which may be difficult for models to recognize. The most challenging features are all related to Violations. Low performance on Infl/Agr Violations, which marks morphological violations (He washed yourself, This is happy), is especially striking because a relatively high proportion (29%) of these sentences are Simple. These models are likely to be deficient in encoding morphological features is that they are word level models, and do not have direct access sub-word information like inflectional endings, which indicates that these features are difficult to learn effectively purely from lexical distributions. Finally, unusual performance on some features is due to small samples, and have a high standard deviation, suggesting the result is unreliable. This includes CP Subj, Frag/Paren, imperative, NPI/FCI, and Comparative. Comparing within-feature performance of the three encoders to their overall performance, we find they have differing strengths and weaknesses. BERT stands out over other models in Deep Embed, which includes challenging sentences with doubly-embedded, as well as in several features involving extraction (i.e.

Excluded Relative clauses, free relatives These are phrasal Wh-phrases, in which the wh-word moves along with other expressions, including prepositions (pied-piping) or nouns in the case of determiner wh-words such as how many and which. . Included Ṗied-piping Ṭhe ship sank, but I don't know with what. (541) . Other phrasal wh-phrases İ know which book Mag read, and which book Bob read my report that you hadn't. (61) How sane is Peter? (88) Relative clauses are noun modifiers appearing with a relativizer (either that or a wh-word) and an associated gap. See [p.223-244]kim2008syntax. . Included Ṫhough he may hate those that criticize Carter, it doesn't matter. (332) *The book what inspired them was very long. (686) Everything you like is on the table. (736) . Excluded Ṭhe more you would want, the less you would eat. (6) This is wh-movement out of an extraction island, or near-island.

Aux (Auxiliaries)	Any occurrence of negation in a sentence, including sentential negation, negative quantifiers, and negative adverbs. . Included Ṡentential İ can't remember the name of somebody who had misgivings. (123) . Quantifier Ṅo writer, and no playwright, meets in Vienna. (124) . Adverb Ṫhey realised that never had Sir Thomas been so offended. (409) Modal verbs (may, might, can, could, will, would, shall, should, must). See [pp.152-155]kim2008syntax. . Included J̇ohn can kick the ball. (280) As a statesman, scarcely could he do anything worth mentioning. (292) . Excluded Ṗseudo-modals Ṡandy was trying to work out which students would be able to solve a certain problem. (600) Auxiliary verbs (e.g. be, have, do). See [pp.149-174]kim2008syntax. . Included Ṫhey love to play golf, but I do not.

Correlations	We wish to emphasize that these features are overlapping and in many cases are correlated, thus not all results from using this analysis set will be independent. We analyzed the pairwise Matthews Correlation Coefficient  of the 63 minor features (giving 1953 pairs), and of the 15 major features (giving 105 pairs). MCC is a special case of Pearson's INLINEFORM0 for Boolean variables. These results are summarized in Table TABREF25 . Regarding the minor features, 60 pairs had a correlation of 0.2 or greater, 17 had a correlation of 0.4 or greater, and 6 had a correlation of 0.6 or greater. None had an anti-correlation of greater magnitude than -0.17. Turning to the major features, 6 pairs had a correlation of 0.2 or greater, and 2 had an anti-correlation of greater magnitude than -0.2. We can see at least three reasons for these observed correlations. First, some correlations can be attributed to overlapping feature definitions. For instance, expletive arguments (e.g.

Pred (Predicates)	These are sentences including the verb be used predicatively. Also, sentences where the object of the verb is itself a predicate, which applies to the subject. Not included are auxiliary uses of be or other predicate phrases that are not linked to a subject by a verb. . Included J̇ohn is eager. (27) He turned into a frog. (150) To please John is easy. (315) . Excluded Ṫhere is a bench to sit on. (309) John broke the geode open. The cake was eaten. These sentences involve predication of a non-subject argument by another non-subject argument, without the presence of a copula. Some of these cases may be analyzed as small clauses.   . Included J̇ohn called the president a fool. (234) John considers himself proud of Mary. (464) They want them arrested. (856) the election of John president surprised me. (1001) Modifiers that act as predicates of an argument. Resultatives express a resulting state of that argument, and depictives describe that argument during the matrix event. See  .

(227) . Relative Clauses NP arguments These are adjuncts of VPs and NPs that specify a time or modify tense or aspect or frequency of an event. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. . Included Ṡhort adverbials (never, today, now, always) Ẉhich hat did Mike quip that she never wore? (95) . PPs Ḟiona might be here by 5 o'clock. (426) . When İ inquired when could we leave. (520) These are adjuncts of VPs and NPs that specify a location of an event or a part of an event, or of an individual. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. . Included Ṡhort adverbials PPs Ṫhe bed was slept in. (298) *Anson demonized up the Khyber (479) Some people consider dogs in my neighborhood dangerous. (802) Mary saw the boy walking toward the railroad station. (73) .

Purpose VPs Ẇe need another run to win. (769) . 0.5em. Excluded ṖP arguments Ṣue gave to Bill a book. (42) Everything you like is on the table. (736) . S-adjuncts J̇ohn lost the race, unfortunately. These are adjuncts modifying noun phrases. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. Single-word prenominal adjectives are excluded, as are relative clauses (this has another category). . Included ṖP-adjuncts Ṭom's dog with one eye attacked Frank's with three legs. (676) They were going to meet sometime on Sunday, but the faculty didn't know when. (565) . Phrasal adjectives Ȧs a statesman, scarcely could he do anything worth mentioning. (292) . Verbal modifiers Ṫhe horse raced past the barn fell. (900) . Excluded Ṗrenominal Adjectives İt was the policeman met that several young students in the park last night.

Included Ṫough-movement Ḍrowning cats, which is against the law, are hard to rescue. (79) . Infinitival relatives F̣ed knows which politician her to vote for. (302) . Purpose ṫhe one with a red cover takes a very long time to read. (352) . Other non-finite VPs with extraction Ȧs a statesman, scarcely could he do anything worth mentioning. (292) These are non-finite VP arguments of nouns and adjectives. . Included Ṙaising adjectives J̇ohn is likely to leave. (370) . Control adjectives Ṫhe administration has issued a statement that it is willing to meet a student group, but I'm not sure which one. (604) . Control nouns Ȧs a teacher, you have to deal simultaneously with the administration's pressure on you to succeed, and the children's to be a nice guy. (673) . Purpose VPs ṫhere is nothing to do. (983) These are miscellaneous non-finite VPs. . Included İ saw that gas can explode.

Binding	These are cases in which a reflexive (non-possessive) pronoun, usually bound by an antecedent. See [p.163-186]sportiche2013introduction and [p.203-226]sag2003syntactic. . Included Ọurselves like ourselves. (742) Which pictures of himself does John like? (386) These are cases in which a non-reflexive pronoun appears along with its antecedent. This includes donkey anaphora, quantificational binding, and bound possessives, among other bound pronouns. See [p.163-186]sportiche2013introduction and [p.203-226]sag2003syntactic. . Included Ḃound possessor Ṫhe children admire their mother. (382) . Quantificational binding Ėverybody gets on well with a certain relative, but often only his therapist knows which one. (562) . Bound pronoun Ẉe gave us to the cause. (747)

N, Adj (Nouns and Adjectives)	These are nouns and adjectives derived from verbs. . Included Ḋeverbal nouns ṭhe election of John president surprised me. (1001) . “Light” verbs Ṫhe birds give the worm a tug. (815) . Gerunds İf only Superman would stop flying planes! (773) . Event-wh Ẇhat the water did to the bottle was fill it. (33) . Deverbal adjectives Ḣis or her least known work. (95) Relational nouns are NPs with an obligatory (or existentially closed) argument. A particular relation holds between the members of the extension of NP and the argument. The argument must be a DP possessor or a PP. See [pp.82-83]kim2008syntax. . Included Ṅouns with of-arguments J̇ohn has a fear of dogs. (353) . Nouns with other PP-arguments Ḣenri wants to buy which books about cooking? (442) .

Argument Types	Oblique arguments of verbs are individual-denoting arguments (DPs or PPs) which act as the third argument of verb, i.e. not a subject or (direct) object. They may or may not be marked by a preposition. Obliques are only found in VPs that have three or more individual arguments. Arguments are selected for by the verb, and they are (generally) not optional, though in some cases they may be omitted where they are understood or implicitly existentially quantified over. See [p.40]kim2008syntax. . Included Ṗrepositional Ṣue gave to Bill a book. (42) Mary has always preferred lemons to limes. (70) *Janet broke Bill on the finger. (141) . Benefactives Ṁartha carved the baby a toy out of wood. (139) . Double object Ṡusan told her a story. (875) Locative arguments Ȧnn may spend her vacation in Italy. (289) . High-arity Passives Ṃary was given by John the book. (626) .

These four features mark VPs with unusual argument structures, including added arguments (I baked Bo a cake) or dropped arguments (Bo knows), and the passive (I was applauded). This contains only one feature for imperative clauses (Stop it!). These are two minor features, one for bound reflexives (Bo loves himself), and one for other bound pronouns (Bo thinks he won). These five features apply to sentences with question-like properties. They mark whether the interrogative is an embedded clause (I know who you are), a matrix clause (Who are you?), or a relative clause (Bo saw the guy who left); whether it contains an island out of which extraction is unacceptable (*What was a picture of hanging on the wall?); or whether there is pied-piping or a multi-word wh-expressions (With whom did you eat?). These six features apply to various complement clauses (CPs), including subject CPs (That Bo won is odd); CP arguments of VPs or NPs/APs (The fact that Bo won); CPs missing a complementizer (I think Bo's crazy); or non-finite CPs (This is ready for you to eat).

Where İ found the place where we can relax. (307) . Excluded Ŀocative arguments Ṣam gave the ball out of the basket. (129) Jessica loaded boxes on the wagon. (164) I went to Rome. These are adjuncts of VPs and NPs not described by some other category (with the exception of (6-7)), i.e. not temporal, locative, or relative clauses. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. . Included Ḃeneficiary Ị know which book José didn't read for class, and which book Lilly did it for him. (58) . Instrument Ŀee saw the student with a telescope. (770) . Comitative J̇oan ate dinner with someone but I don't know who. (544) . VP adjuncts Ẇhich article did Terry file papers without reading? (431) . Purpose Ẇe need another run to win. (769)

Adjunct	Particles are lone prepositions associated with verbs. When they appear with transitive verbs they may immediately follow the verb or the object. Verb-particle pairs may have a non-compositional (idiomatic) meaning. See [pp. 69-70]carnie2013syntax and [pp. 16-17]kim2008syntax. . Included Ṭhe argument was summed by the coach up. (615) Some sentences go on and on and on. (785) *He let the cats which were whining out. (71) Adjuncts modifying verb phrases. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. See  . . Included ṖP-adjuncts, e.g. locative, temporal, instrumental, beneficiary Ṅobody who hates to eat anything should work in a delicatessen. (121) Felicia kicked the ball off the bench. (127) . Adverbs Ṁary beautifully plays the violin. (40) John often meets Mary. (65) .

(871) The love of my life and mother of my children would never do such a thing. (806) Noun-noun compounds are NPs consisting of two constituent nouns. . Included İt was the peasant girl who got it. (320) A felon was elected to the city council. (938) These are adjectives that take an obligatory (or existentially closed) argument. A particular relation holds between the members of the extension of the modified NP and the argument. The argument must be a DP or PP. See [pp.80-82]kim2008syntax. . Included Ȯf-arguments Ṫhe chickens seem fond of the farmer. (254) . Other PP arguments Ṫhis week will be a difficult one for us. (241) John made Bill mad at himself. (1035) A transitive (non-relational) adjective. I.e. an adjectives that takes a VP or CP argument. See [pp.80-82]kim2008syntax. . Included V̇P argument J̇ohn is likely to leave. (370) .

Otherwise we find that a model's performance on sentences of a given feature is on par with or lower than its overall performance, reflecting the fact that features mark the presence of unusual or complex syntactic structure. Performance is also high (and close to overall performance) on sentences with marked argument structures (Argument Types and Arg(ument) Alt(ernation)). While these models are still worse than human (overall) performance on these sentences, this result indicates that argument structure is relatively easy to learn. Comparing different kinds of embedded content, we observe higher performance on sentences with embedded clauses (major feature=Comp Clause) embedded VPs (major feature=to-VP) than on sentences with embedded interrogatives (minor features=Emb-Q, Rel Clause). An exception to this trend is the minor feature No C-izer, which labels complement clauses without a complementizer (e.g. I think that you're crazy). Low performance on these sentences compared to most other features in Comp Clause might indicate that complementizers are an important syntactic cue for these models. As the major feature Question shows, the difficulty of sentences with question-like syntax applies beyond just embedded questions.

Conclusion	Using a new grammatically annotated analysis set, we identify several syntactic phenomena that are predictive of good or bad performance of current state of the art sentence encoders on CoLA. We also use these results to develop hypotheses about why BERT is successful, and why transformer models outperform sequence models. Our findings can guide future work on sentence embeddings. A current weakness of all sentence encoders we investigate, including BERT, is the identification of morphological violations. Future engineering work should investigate whether switching to a character-level model can mitigate this problem. Additionally, transformer models appear to have an advantage over sequence models with long-distance dependencies, but still struggle with these constructions relative to more local phenomena. It stands to reason that this performance gap might be widened by training larger or deeper transformer models, or training on longer or more complex sentences. This analysis set can be used by engineers interested in evaluating the syntactic knowledge of their encoders. Finally, these findings suggest possible controlled experiments that could confirm whether there is a causal relation between the presence of the syntactic features we single out as interesting and model performance.

Furthermore, all models struggle equally with sentences in Violation, indicating that the advantages of the transformer models over the CoLA baseline does not extend to the detection of morphological violations (Infl/Agr Violation) or single word anomalies (Extra/Missing Expr).

long-distance dependencies) such as VP+Extract and Info-Struc. The transformer models show evidence of learning long-distance dependencies better than the CoLA baseline. They outperform the CoLA baseline by an especially wide margin on Bind:Refl, which all involves establishing a dependency between a reflexive and its antecedent (Bo tries to love himself). They also have a large advantage in dislocation, in which expressions are separated from their dependents (Bo practiced on the train an important presentation). The advantage of BERT and GPT may be due in part to their use of the transformer architecture. Unlike the BiLSTM used by the CoLA baseline, the transformer uses a self-attention mechanism that associates all pairs of words regardless of distance. In some cases models showed surprisingly good or bad performance, revealing possible idiosyncrasies of the sentence embeddings they output. For instance, the CoLA baseline performs on par with the others on the major feature adjunct, especially considering the minor feature Particle (Bo looked the word up).

Acknowledgments	We would like to thank Jason Phang and Thibault Févry for sharing GPT and BERT model predictions on CoLA, and Alex Wang for feedback.

Models Evaluated	We train MLP acceptability classifiers for CoLA on top of three sentence encoders: (1) the CoLA baseline encoder with ELMo-style embeddings, (2) OpenAI GPT, and (3) BERT. We use publicly available sentence encoders with pretrained weights.

Simple	These are sentences with transitive or intransitive verbs appearing with their default syntax and argument structure. All arguments are noun phrases (DPs), and there are no modifiers or adjuncts on DPs or the VP. . Included J̇ohn owns the book. (37) Park Square has a festive air. (131) *Herself likes Mary's mother. (456) . Excluded Ḃill has eaten cake. I gave Joe a book.

These four minor features mark the presence of auxiliary or modal verbs (I can win), negation, or “pseudo-auxiliaries” (I have to win). These five features mark various infinitival embedded VPs, including control VPs (Bo wants to win); raising VPs (Bo seemed to fly); VP arguments of NPs or APs (Bo is eager to eat); and VPs with extraction (e.g. This is easy to read ts ). These seven features mark complex NPs and APs, including ones with PP arguments (Bo is fond of Mo), or CP/VP arguments; noun-noun compounds (Bo ate mud pie); modified NPs, and NPs derived from verbs (Baking is fun). These seven features mark various unrelated syntactic constructions, including dislocated phrases (The boy left who was here earlier); movement related to focus or information structure (This I've gotta see this); coordination, subordinate clauses, and ellipsis (I can't); or sentence-level adjuncts (Apparently, it's raining).

Annotation	The sentences were annotated manually by one of the authors, who is a PhD student with extensive training in formal linguistics. The features were developed in a trial stage, in which the annotator performed a similar annotation with different annotation schema for several hundred sentences from CoLA not belonging to the development set.

Analysis Set	We introduce a grammatically annotated version of the entire CoLA development set to facilitate detailed error analysis of acceptability classifiers. These 1043 sentences are expert-labeled for the presence of 63 minor grammatical features organized into 15 major features. Each minor feature belongs to a single major feature. A sentence belongs to a major feature if it belongs to one or more of the relevant minor features. The Appendix includes descriptions of each feature along with examples and the criteria used for annotation. The 63 minor features and 15 major features are illustrated in Table TABREF5 . Considering minor features, an average of 4.31 features is present per sentence (SD=2.59). The average feature is present in 71.3 sentences (SD=54.7). Turning to major features, the average sentence belongs to 3.22 major features (SD=1.66), and the average major feature is present in 224 sentences (SD=112). Every sentence is labeled with at least one feature.

Using this evaluation set, we compare the syntactic knowledge of GPT and BERT in detail, and investigate the strengths of these models over the baseline BiLSTM model published by warstadt2018neural. The analysis set includes expert annotations labeling the entire CoLA development set for the presence of 63 fine-grained syntactic features. We identify many specific syntactic features that make sentences harder to classify, and many that have little effect. For instance, sentences involving unusual or marked argument structures are no harder than the average sentence, while sentences with long distance dependencies are hard to learn. We also find features of sentences that accentuate or minimize the differences between models. Specifically, the transformer models seem to learn long-distance dependencies much better than the recurrent model, yet have no advantage on sentences with morphological violations.

Included Ṁissing word J̣ohn put under the bathtub. (247) *I noticed the. (788) . Extra word Ẹveryone hopes everyone to sleep. (467) *He can will go (510)

Excluded ṖP adjuncts Particles Arguments of deverbal expressions ṭhe putter of books left. (892) . By-phrase Ṫed was bitten by the spider. (613) Prepositional Phrase arguments of NPs or APs are individual-denoting arguments of a noun or adjective which are marked by a proposition. Arguments are selected for by the head, and they are (generally) not optional, though in some cases they may be omitted where they are understood or implicitly existentially quantified over. . Included Ṙelational adjectives Ṁany people were fond of Pat. (936) *I was already aware of fact. (824) . Relational nouns Ẇe admired the pictures of us in the album. (759) They found the book on the atom. (780) . Arguments of deverbal nouns ṭhe putter of books left. (892) Prepositional arguments introduced with by. Usually, this is the (semantic) subject of a passive verb, but in rare cases it may be the subject of a nominalized verb.

. Included Ṙesultative Ṭhe table was wiped by John clean. (625) The horse kicked me black and blue. (898) . Depictive J̇ohn left singing. (971) In which car was the man seen? (398) . Excluded Ḣe turned into a frog. (150)

Excluded Ṅon-DP arguments Ẇe want John to win (28) . 3rd argments where not all three arguments are DPs Ẇe want John to win (28) Prepositional Phrase arguments of VPs are individual-denoting arguments of a verb which are marked by a proposition. They may or may not be obliques. Arguments are selected for by the verb, and they are (generally) not optional, though in some cases they may be omitted where they are understood or implicitly existentially quantified over. . Included Ḋative Ṣue gave to Bill a book. (42) . Conative (at) C̣arla slid at the book. (179) . Idiosyncratic prepositional verbs İ wonder who to place my trust in. (711) She voted for herself. (743) . Locative J̇ohn was found in the office. (283) . PP predicates Ėverything you like is on the table. (736) .

(387) These are comparative constructions. See  . . Included Ċorrelative Ṫhe angrier Mary got, the more she looked at pictures. (9) They may grow as high as bamboo. (337) I know you like the back of my hand. (775)

Determiner	These are quantificational DPs, i.e. the determiner is a quantifier. . Included Q̇uantifiers Ẹvery student, and he wears socks, is a swinger. (118) We need another run to win. (769) . Partitive Ṇeither of students failed. (265) These are quantifiers that take PP arguments, and measure nouns. See [pp.109-118]kim2008syntax. . Included Q̇uantifiers with PP arguments Ṇeither of students failed. (265) . Numerals Ȯne of Korea's most famous poets wrote these lines. (294) . Measure nouns İ bought three quarts of wine and two of Clorox. (667) These are negative polarity items (any, ever, etc.) and free choice items (any). See kadmon1993any. . Included ṄPI Ėverybody around here who ever buys anything on credit talks in his sleep. (122) I didn't have a red cent. (350) . FCI Ȧny owl hunts mice.

(56) Mary likes to tour art galleries, but Bill hates to. (287) . VP Anaphor İ saw Bill while you did so Mary. (472) . NP Ellipsis Ṫom's dog with one eye attacked Fred's. (679) . NP anaphor ṫhe one with a red cover takes a very long time to read. (352) . Sluicing Ṁost columnists claim that a senior White House official has been briefing them, and the newspaper today reveals which one. (557) . Gapping Ḃill ate the peaches, but Harry the grapes. (646) These are adjuncts modifying sentences, sentence-level adverbs, subordinate clauses. . Included Ṡentence-level adverbs Ṡuddenly, there arrived two inspectors from the INS. (447) . Subordinate clauses Ṫhe storm arrived while we ate lunch. (852)

(575) . Or, nor Ṇo writer, nor any playwright, meets in Vienna. (125) . Pseudo-coordination İ want to try and buy some whiskey. (432) . Juxtaposed clauses Ŀights go out at ten. There will be no talking afterwards. (779) This includes subordinate clauses, especially with subordinating conjunctions, and conditionals. . Included Ċonditional İf I can, I will work on it. (56) . Subordinate clause Ẉhat did you leave before they did? (598) *Because Steve's of a spider's eye had been stolen, I borrowed Fred's diagram of a snake's fang. (677) . Correlative Ạs you eat the most, you want the least. (5) This includes VP or NP ellipsis, or anaphora standing for VPs or NPs (not DPs). See [pp.55-61]sportiche2013introduction. . Included V̇P Ellipsis İf I can, I will work on it.

(807) . Clefts İt was a brand new car that he bought. (347) . Pseudo-clefts Ẇhat John promised is to be gentle. (441) . Excluded Ṫhere-insertion Passive These are parentheticals or fragmentary expressions. . Included Ṗarenthetical Ṁary asked me if, in St. Louis, John could rent a house cheap. (704) . Fragments Ṫhe soup cooks, thickens. (448) . Tag question Ġeorge has spent a lot of money, hasn't he? (291) Coordinations and disjunctions are expressions joined with and, but, or, etc. See [pp.61-68]sportiche2013introduction. . Included ḊP coordination Ḋave, Dan, Erin, Jaime, and Alina left. (341) . Right Node Raising K̇im gave a dollar to Bobbie and a dime to Jean. (435) . Clausal coordination Ṡhe talked to Harry, but I don't know who else.

S-Syntax (Sentence-Level Syntax)	These are expressions with non-canonical word order. See, for example, [p.76]sportiche2013introduction. . Includes Ṗarticle shift Ṃickey looked up it. (24) . Preposed modifiers Ȯut of the box jumped a little white rabbit. (215) *Because she's so pleasant, as for Mary I really like her. (331) . Quantifier float Ṫhe men will all leave. (43) . Preposed argument Ẇith no job would John be happy. (333) . Relative clause extraposition Ẇhich book's, author did you meet who you liked? (731) . Misplaced phrases Ṁary was given by John the book. (626) This includes topicalization and focus constructions. See [pp.258-269]kim2008syntax and [pp.68-75]sportiche2013introduction. . Included Ṫopicalization Ṁost elections are quickly forgotten, but the election of 2000, everyone will remember for a long time.

CP argument J̇ohn is aware of it that Bill is here. (1013) . QP argument Ṫhe administration has issued a statement that it is willing to meet a student group, but I'm not sure which one. (604)

Raising Ȧnson believed himself to be handsome. (499) These are VPs with raising predicates, where one argument is a non-finite to-VP without a covert subject co-indexed with an argument of the matrix verb. Unlike control verbs, the coindexed argument is not a semantic argument of the raising predicate. See [pp.260-266]adger2003core, [pp.203-222]sportiche2013introduction, and [pp.125-148]kim2008syntax. . Included Ṡubject raising U̇nder the bed seems to be a fun place to hide. (277) . Object raising Ȧnson believed himself to be handsome. (499) . Raising adjective J̇ohn is likely to leave. (370) These are embedded infinitival VPs containing a (non-subject) gap that is filled by an argument in the upper clause. Examples are purpose-VPs and tough-movement. See [pp.246-252]kim2008syntax. .

to-VP (Infinitival VPs)	These are VPs with control verbs, where one argument is a non-finite to-VP without a covert subject co-indexed with an argument of the matrix verb. See [pp.252,266-291]adger2003core, [pp.203-222]sportiche2013introduction, and [pp.125-148]kim2008syntax. . Included İntransitive subject control Ịt tries to leave the country. (275) . Transitive subject control J̇ohn promised Bill to leave. (977) . Transitive object control İ want her to dance. (379) John considers Bill to be silly. (1040) . Excluded V̇P args of NP/AP Ṫhis violin is difficult to play sonatas on. (114) . Purpose Ṫhere is a bench to sit on. (309) . Subject VPs Ṫo please John is easy. (315) . Argument present participles Ṁedea denied poisoning the phoenix. (490) .

(290) The car was driven. (296) he had spent five thousand dollars. (301) . Excluded Ṗseudo-auxiliaries Ṣally asked if somebody was going to fail math class, but I can't remember who. (589) The cat got bitten. (926) These are predicates acting as near-auxiliary (e.g. get-passive) or near-modals (e.g. willing) . Included Ṅear-auxiliaries Ṃary came to be introduced by the bartender and I also came to be. (55) *Sally asked if somebody was going to fail math class, but I can't remember who. (589) The cat got bitten. (926) . Near-modals Ċlinton is anxious to find out which budget dilemmas Panetta would be willing to tackle in a certain way, but he won't say in which. (593) Sandy was trying to work out which students would be able to solve a certain problem. (600)

with the following syntax: [S ...[ VP ...[ VP ...[ VP ...] ...] ...] ...] . Included Ėmbedded VPs Ṁax seemed to be trying to force Ted to leave the room, and Walt, Ira. (657) . Embedded clauses İ threw away a book that Sandy thought we had read. (713)

Under AP Ṭhe children are fond that they have ice cream. (842) These are complement clauses with a non-finite matrix verb. Often, the complementizer is for, or there is no complementizer. See [pp.252-253,256-260]adger2003core. . Included Ḟor complementizer İ would prefer for John to leave. (990) . No Complementizer Ṁary intended John to go abroad. (48) . Ungrammatical Ḣeidi thinks that Andy to eat salmon flavored candy bars. (363) . V-ing Ȯnly Churchill remembered Churchill giving the Blood, Sweat and Tears speech. (469) These are complement clauses with no overt complementizer. . Included Ċomplement clause İ'm sure we even got these tickets! (325) He announced he would marry the woman he loved most, but none of his relatives could figure out who. (572) . Relative clause Ṫhe Peter we all like was at the party (484) These are sentences with three or nested verbs, where VP is not an aux or modal, i.e.

Comp Clause (Complement Clauses)	These are complement clauses acting as the (syntactic) subject of verbs. See [pp.90-91]kim2008syntax. . Included Ṫhat dogs bark annoys people. (942) The socks are ready for for you to put on to be planned. (112) . Excluded Ėxpletive insertion İt bothers me that John coughs. (314) These are complement clauses acting as (non-subject) arguments of verbs. See [pp.84-90]kim2008syntax. . Included İ can't believe Fred won't, either. (50) I saw that gas can explode. (222) It bothers me that John coughs. (314) Clefts İt was a brand new car that he bought. (347) These are complement clauses acting as an argument of a noun or adjective. See [pp.91-94]kim2008syntax. . Included U̇nder NP Ḋo you believe the claim that somebody was looking for something? (99) .

Islands include, for example, complex NPs, adjuncts, embedded questions, coordination. A near-island is an extraction that closely resembles an island violation, such as extraction out of an embedded clause, or across-the-board extraction. See [pp.323-333]adger2003core and [pp.332-334]carnie2013syntax. . Included Ėmbedded question *What did you ask who Medea gave? (493) . Adjunct Ẉhat did you leave before they did? (598) . Parasitic gaps Ẇhich topic did you choose without getting his approval? (311) . Complex NP Ẇho did you get an accurate description of? (483)

Question	These are sentences in which the matrix clause is interrogative (either a wh- or polar question). See [pp.282-213]adger2003core, [pp.193-222]kim2008syntax, and [p.315-350]carnie2013syntax. . Included Ẇh-question Ẇho always drinks milk? (684) . Polar question Ḋid Athena help us? (486) These are embedded interrogative clauses appearing as arguments of verbs, nouns, and adjectives. Not including relative clauses and free relatives. See [p.297]adger2003core. . Included U̇nder VP İ forgot how good beer tastes. (235) *What did you ask who saw? (508) . Under NP Ṫhat is the reason why he resigned. (313) . Under AP Ṫhey claimed they had settled on something, but it wasn't clear what they had settled on. (529) . Free relative Ẇhat the water did to the bottle was fill it. (33) .

Imperative	The imperative mood is marked by the absence of the a subject and the bare form of the verb, and expresses a command, request, or other directive speech act. . Included Ẉash you! (224) Somebody just left - guess who. (528)

(139) The passive voice is marked by the demotion of the subject (either complete omission or to a by-phrase) and the verb appearing as a past participle. In the stereotypical construction there is an auxiliary be verb, though this may be absent. See [p.175-190]kim2008syntax, collins2005smuggling, and [p.311-333]sag2003syntactic. . Included V̇erbs Ṫhe earth was believed to be round. (157) . Psuedopassive Ṫhe bed was slept in. (298) . Past participle adjuncts Ṫhe horse raced past the barn fell. (900)

(144) . Transitive noun İ sensed his eagerness. (155) . Expletive insertion Ịt loved Sandy. (949) . Excluded Ṫed was bitten by the spider. (613) These are VPs in which a non-canonical argument of the verb has been added. These cases are clearer to identify where the additional argument is a DP. In general, PPs which mark locations, times, beneficiaries, or purposes should be analyzed as adjuncts, while PPs marking causes can be considered arguments. See []pylkkanen2008introducing. . Included Ėxtra argument Ḷinda winked her lip. (202) Sharon fainted from hunger. (204) I shaved myself. (526) . Causative Ị squeaked the door. (207) . Expletive insertion Ṫhere is a monster in Loch Ness. (928) It annoys people that dogs bark. (943) . Benefactive Ṁartha carved the baby a toy out of wood.

(934) [It] bothers [me] [that he is here]. (1009) . Small clause John] considers [Bill] [silly]. (1039) . Excluded Ṙesults, depictives John] broke [the geode] [open]. These are VPs where a canonical argument of the verb is missing. This can be difficult to determine, but in many cases the missing argument is understood with existential quantification or generically, or contextually salient. See [p.106-109]sportiche2013introduction. . Included Ṁiddle voice/causative inchoative Ṭhe problem perceives easily. (66) . Passive Ṫhe car was driven. (296) . Null complement anaphora J̇ean persuaded Robert. (380) Nobody told Susan. (883) . Dropped argument Ḳim put in the box. (253) The guests dined. (835) I wrote to Bill. (1030) . Transitive adjective J̇ohn is eager. (27) We pulled free.

Arg Altern (Argument Alternations)	These are verbs with 3 or more arguments of any kind. Arity refers to the number of arguments that a head (or function) selects for. Arguments are usually selected for by the head, and they are generally not optional. They may be DPs, PPs, CPs, VPs, APs or other categories. . Included Ḋitransitive [̣Sue] gave [to Bill] [a book]. (42) [Martha] carved [the baby] [a toy] out of wood. (139) . VP arguments [̣We] believed [John] [to be a fountain in the park]. (274) [We] made [them] [be rude]. (260) . Particles He] let [the cats which were whining] [out]. (71) . Passives with by-phrase [̣A good friend] is remained [to me] [by him]. (237) . Expletives [̣We] expect [there] [to will rain]. (282) [There] is [a seat] [available].

Included Ṫhere—inserted, existential Ṭhere loved Sandy. (939) There is a nurse available. (466) . It—cleft, inserted İt was a brand new car that he bought. (347) It bothers me that John coughs. (314) It is nice to go abroad. (47) . Environmental it K̇erry remarked it was late. (821) Poor Bill, it had started to rain and he had no umbrella. (116) You've really lived it up. (160) . Excluded J̇ohn counted on Bill to get there on time. (996) I bought it to read. (1026)

Introduction	The effectiveness and ubiquity of pretrained sentence embeddings for natural language understanding has grown dramatically in recent years. Recent sentence encoders like OpenAI's Generative Pretrained Transformer  and BERT  achieve the state of the art on the GLUE benchmark  . Among the GLUE tasks, these state-of-the-art systems make their greatest gains on the acceptability task with the Corpus of Linguistic Acceptability  . CoLA contains example sentences from linguistics publications labeled by experts for grammatical acceptability, and written to show subtle grammatical features. Because minimal syntactic differences can separate acceptable sentences from unacceptable ones (What did Bo write a book about? / *What was a book about written by Bo?), and acceptability classifiers are more reliable when trained on GPT and BERT than on recurrent models, it stands to reason that GPT and BERT have better implicit knowledge of syntactic features relevant to acceptability. Our goal in this paper is to develop an evaluation dataset that can locate which syntactic features that a model successfully learns by identifying the syntactic domains of CoLA in which it performs the best.

Question: Do they report results only on English data?
Answer: Yes