Title: Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: templates

compositionality

language models
Abstract: For an LLM to correctly respond to an instruction it must understand both the semantics and the domain (i.e., subject area) of a given task-instruction pair. However, syntax can also convey implicit information. Recent work shows that \textit{syntactic templates}---frequent sequences of Part-of-Speech (PoS) tags---are prevalent in training data and often appear in model outputs. In this work we characterize syntactic templates, domain, and semantics in task-instruction pairs. We identify cases of spurious correlations between syntax and domain, where models learn to associate a domain with syntax during training; this can sometimes override prompt semantics. Using a synthetic training dataset, we find that the syntactic-domain correlation can lower performance (mean 0.51 +/- 0.06) on entity knowledge tasks in OLMo-2 models (1B-13B). We introduce an evaluation framework to detect this phenomenon in trained models, and show that it occurs on a subset of the FlanV2 dataset in open (OLMo-2-7B; Llama-4-Maverick),  and closed (GPT-4o) models. Finally, we present a case study on the implications for LLM security, showing that unintended syntactic-domain correlations can be used to bypass refusals in OLMo-2-7B Instruct and GPT-4o. Our findings highlight two needs: (1) to explicitly test for syntactic-domain correlations, and (2) to ensure \textit{syntactic} diversity in training data, specifically within domains, to prevent such spurious correlations.


Peer reviews:
comments: SUMMARY:
In this paper, the authors investigate syntactic-domain spurious correlations in LLMs and show that current models rely on them to achieve high performance. This reliance leads to previously unreported vulnerabilities, such as bypassing refusals, and may also contribute to hallucination behaviors. The authors provide both empirical and mathematical evidence, along with a benchmarking method to measure such reliance.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. The paper identifies and investigates an underexplored yet important issue—syntactic-domain spurious correlations in LLMs.
2. The authors propose a controlled benchmarking method to isolate and evaluate this reliance across domains and syntactic structures.
3. The study covers a wide range of models, from open-weight (e.g., OLMo) to closed models (e.g., GPT-4o and LLaMA-4), suggesting the phenomenon is general.

**Weaknesses**
1. Most experiments are conducted on synthetic or constructed datasets, which may not fully reflect real-world model behavior in open-domain tasks.
2. The conclusions regarding GPT-4o and LLaMA-4 are based on assumptions about training data (e.g., FlanV2), which the authors cannot confirm.
3. While the authors claim a security implication (e.g., refusal bypass), the examples shown are relatively mild and lack broader coverage.

QUESTIONS:
1. To what extent does this reliance on syntactic-domain correlation persist in instruction-tuned or chain-of-thought models?
2. Could the authors provide more impactful or adversarial examples of how refusal mechanisms can be bypassed using this syntactic trick?
3. Is it possible that the observed behavior is more due to prompt-style overfitting rather than syntactic-domain correlation per se?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper argues that LLMs learn spurious correlations between syntax and domain. In other words, during pretraining or finetuning, if some specific syntactic templates (sequences of Part-of-Speech tags) are commonly found in some specific domain, the LLM will learn to map those syntactic templates to that domain. This results in the models taking unwanted shortcuts to solve tasks in the domain. For example, the authors found that in their synthetic setting, prompting “Where is Paris undefined?” or “Quickly sit Paris clouded?” results in unwanted response “France”, as the two examples follow the exact same syntactic template the authors train Olmo to learn location relationships. Further extending from this synthetic setting, they also were able to come up with a similar conclusion on off-the-shelf models using the templates of the Flan dataset. Finally, the authors also found that in some cases leveraging such spurious correlation can bypass answer refusals of LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper proposes an unique perspective. Specifically, the authors argue that specific “syntax” associated with a domain can also affect LLMs’ generalizability across domains.\n\n- The synthetic dataset experiment results are interesting. The results of Synonym and Antonym suggests that the LLM isn’t necessarily overfit to the template, and can generalize within the domain-syntactic template pair, but a general degradation in Cross Domain section suggests that they cannot generalize across the domain-syntactic pairs.\n\n**Weaknesses**\n\n- The paper is hard to read with ambiguities and writeup issues, and I had to go over the paper multiple times to understand the experiment settings. Some parts of the paper are lacking in core details and the details had to be found by looking at the experiment codes in the supplementary materials.\n    - For the synthetic dataset, I did not know that every Exact templates (regardless of the domain) are semantically “A is related to B” style, before looking at the code myself. The paper does not mention this anywhere. The paper was very confusing before seeing the actual template examples.\n    - Please elaborate on how “Step 3: Measuring Syntactic-Domain Correlation Reliance” is done for the FlanV2 dataset. Is it on an instance-level? How many instances are left after the filtration in Footnote 7? \n    - I have put all the other writeup issues under the “Questions”.\n\n- I’m not sure if the cross-domain experiment in Section 5 is fair. The authors (if I understood correctly) split the FlanV2 templates in-domain and cross-domain on an instance-level, by whether the model got the answer correct or not. However, FlanV2 has good templates and bad templates. The bad templates are ambiguous and distant from the chat styles of modern LLMs. Some examples of bad ones are the ones with “Denny” and “Leo” in the cot_esnli set. The model likely got the answer correct when the question is presented with the good ones and wrong when presented with the bad ones, especially if a model was never trained on FlanV2. Therefore perturbations (Synonym, Antonym, etc) of the good ones (In-Domain) are still likely to be better than the perturbations of the bad ones (Cross-Domain).\n\n- Section 6 presents some interesting qualitative analysis, but it's unclear how consistent this finding is as the authors only present a couple of cases. A more general quantitative evaluation would've been nice.\n\nQUESTIONS:\n- How was the synthetic dataset templates, and the perturbation of the templates (for both synthetic and FlanV2) done? Did you do it manually or with an LLM?\n\n**Writeup/clarity issues and suggestions**\n- The authors used the 2024 NeurIPS template, which doesn’t have the Checklist item 16. Declaration of LLM usage.\n- Also the authors were supposed to put 1 or 2 lines of Justifications for all the Checklist items, but some of them are just left blank.\n- Are the math notations in page 3 and 4 necessary? They rarely get mentioned past those pages and for me they do little to clarify things up.\n- Table 1 was never mentioned in the text.\n- Please move Table 6 and 8 to the main sections if this paper gets accepted. Those tables are part of the main results; they should not be in the Appendix. \n- L3: missing period between sentences\n- L:10 different font between 0.51 and 0.06\n- L:50 no space after parentheses\n- L:86-87 “define five classes of prompt perturbations” then the authors again “define three classes of prompt perturbations”\n- L:130 I can’t find the full set in Appendix A\n- L:155 Why did the authors talk about the training dataset details here? It’s confusing because the authors were talking about the test set construction.\n- L:140 Table 3 is something else. I think the authors were trying to say Table 5.\n- “GPT-4o” and “GPT4o” are used both throughout the paper. Please stick to one.\n- L:786 There’s only three domains here. It should be four right?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about determining how sensitive a LLM is to the syntax of the input prompt in determining how to generate output text, where syntax is specified in terms of the part of speech tag sequence (\"syntactic template\") corresponding to the input prompt text.  The paper has a formalization of the problem by specifying each dataset example prompt in terms of a tuple (domain, template, entities) (\"template\" being the syntactic template of the input prompt, and \"entities\" being the expected (correct) output of the LLM), and then specifying perturbations of the input prompt in terms of modifying its syntax or semantics.  With such a formalization, the paper goes through one set of experiments with fine-tuning LLMs on synthetic data with different specific properties, showing generally that it causes the LLM to rely too much on syntax.  The paper goes through another set of experiments on non-fine-tuned LLMs that show that there too, these kinds of LLMs generally rely too much on syntax.\n\nSTRENGTHS AND WEAKNESSES:\nA strength of the paper is that to my knowledge, these kinds of experiments that show how syntax influences LLM outputs are novel.  Another strength is that these experiments are thorough, for example testing on fine-tuned and non-fine-tuned LLMs, testing on different kinds of LLMs, and testing on different LLM tasks.  Another strength of the paper is that the experimental results suggest a way, though not fully formed, of how to get around the limitation of the LLM being influenced so much by syntax--by having syntactially ill-formed input queries.\n\nA weakness of the paper is that although these kinds of experiments showing the dependence of LLM on syntax is novel, the fact that LLM has some sort of dependence on syntax was already known.  Another weakness is although the paper generally shows LLMs being influenced by syntax, there are non-explained outliers, such as the fact that in-domain Llama-4-Maverick on the Sentiment140 dataset would performs so poorly in the SYNONYM condition.\n\nAnother weakness of the paper is that there are some points in the organization of the paper that make it harder to read, as well as the fact that that there are errors in how the paper refers to tables and figures.\n\nQUESTIONS:\nThe \"model refusal\" on line 56 was confusing without the paper defining exactly what it is.\n\nLine 69 says x = (d, t, e) but then on line 73 it says f(x) = (d, t, e) which makes f(x) look like like the identity function, which can't be correct.\n\nOn lines 99-100, M(e|p^-) is defined twice in two different ways, which can't be right.\n\nOn line 131 where it says \"we define the template set for the domain T_d with a single unique syntactic template t to control structural variation across different relation types\" does that mean there is just one template per domain T_d, or does it mean there is just one template per answer type in the domain T_d ?\n\nLiine 140 refers to Table 3, but I can't find a Table 3.\n\nOn line 144, it says \"we create four template settings to measure model responses\" but then it goes on to just talk about two of temp (SYNONYM and ANTONYM).\n\nLine 158 talks about Table 6 and Figure 8a, but I can't find a Table 6 or Figure 8a.\n\nLine 197 says \"input/ouput pairs\" but it should be \"input/output pairs\"\n\nOn line 201, where it says \"EXACT templates \\in [1, 2, 4]\" what does \"[1, 2, 4]\" mean?\n\nInstead of Section 4 and Section 5, it would be clearer if there was just one new Section 4 which is a concatenation of old Section 4 (maybe new Section 4.1)) and old Section 5 (maybe new Section 4.2)) since old Section 4 and old Section 5 are referring to one experiment.\n\nIf these issues are cleared up, I might assign a clarity \"good\" instead of a clarity \"fair\".\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this papers, the authors propose to investigate the syntatctic domain spurious correlation by testing performance shift across domains on performaning instructions with perturbations. Specifically, the authors propose five instruction perturbation methods, and tested whether the language model are robust towards these perturb instructions. A huge robustness drop between domains can indicatate that LM utilize the syntatic information in the prompts to map to certain domain knowledge. Experiments on OLMo-2 models (1B-13B) shows that syntatic spurtious correlation can hurt the model performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper is well written and easy to follow.\n2. The proposed evaluation of syntatic-domain spurious correlation is novel. While there've already been works evaluating instruction following and spurious patterns, the syntatic and domain correlation isn't well discussed yet.\n3. The proposed method for quantifying syntax–domain reliance make sense. \n4. The Case Study: Bypassing Answer Refusals section is very interesting. The authors manage to apply the proposed method to observe syntax-reliance in open-sourced LMs, and provide qualitative examples of bypassing LLMs answer refusal.\n\n\n## Weaknesses\n1. In the related work, the authors should talk about prior evaluation on instruction following and spurious patterns. For example in [1,2,3], there're relevant findings that model can adhere to spurious correlation in prompt and achieve high performance after instruction tuning.\n\n[1] Do prompt-based models really understand the meaning of their prompts\n\n[2] Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning\n\n[3] Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning\n\nQUESTIONS:\n# Comments:\n1. Typo: Missing period in abstract after \"implicit information\" and before \"Recent\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}