Venue: NAACL 2024

# Deceptive Semantic Shortcuts on Reasoning Chains:

How Far Can Models Go without Hallucination?

Bangzheng Li\({}^{1}\) Ben Zhou\({}^{2}\) Fei Wang\({}^{3}\) Xingyu Fu\({}^{2}\) Dan Roth\({}^{2}\) Muhao Chen\({}^{1}\)

\({}^{1}\)University of California, Davis \({}^{2}\)University of Pennsylvania

\({}^{3}\)University of Southern California

bzhli@ucdavis.edu

###### Abstract

Despite the high performances of large language models (LLMs) across numerous benchmarks, recent research has unveiled their suffering from hallucinations and unfaithful reasoning. This work studies a type of hallucination induced by semantic associations. We investigate to what extent LLMs take shortcuts from certain keyword/entity biases in the prompt instead of following correct reasoning paths. To quantify this phenomenon, we propose a novel probing method and benchmark called EureQA. EureQA is an entity-searching task where a model finds a missing entity based on described multi-hop relations with other entities. These deliberately designed multi-hop relations create deceptive semantic associations, and models must stick to the correct reasoning path instead of incorrect shortcuts to find the correct answer. Experiments show that existing LLMs cannot follow correct reasoning paths and resist the attempt of greedy shrotrcuts, with GPT-4 only achieving 62% accuracy. Analyses provide further evidence that LLMs rely on semantic biases to solve the task instead of proper reasoning, questioning the validity and generalizability of current LLMs' high performances.1

Footnote 1: Code and data are available at [https://github.com/VincentLeebang/eureqa](https://github.com/VincentLeebang/eureqa)

_Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7675–7688

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Recent progress of large language models (LLMs) has showcased their emergent abilities in a wide range of reasoning tasks, encompassing competence in program synthesis Kuznia et al. (2022), mathematical reasoning Mishra et al. (2022), symbolic logic reasoning Gaur and Saunshi (2023), and common-sense reasoning Feng et al. (2023). Innovations in inference methods have further improved language models' reasoning capabilities, either by generating intermediate steps toward the ultimate solution Wei et al. (2022) or by decomposing complex inquiries into manageable subproblems Zhou et al. (2022); Yao et al. (2023). Research has further demonstrated that when incorporated with these techniques, LLMs have achieved exceptional results on multi-hop and complex QA benchmarks Geva et al. (2021).

Nonetheless, it remains under-investigated whether the effectiveness of LLMs' reasoning pattern OpenAI (2023); Team (2023) is genuinely based on a sensible reasoning path or if these models predominantly depend on semantic associations (i.e., word distributions from the pre-training data) to generate a plausible answer. Fig. 1 shows how

Figure 1: LLMs make errors when correct surface-level semantic cues – entities – are recursively replaced with descriptions, and the errors are likely related to _token similarity_. GPT-3.5-turbo is used for this example.



ChatGPT2 fails to find the missing entity as we gradually remove relevant semantic cues as the reasoning depth grows. The original question (labeled as "1 Layer" in Fig. 1) asks models to find an individual that satisfies two distinct conditions. It can be resolved by finding the set of people who are producers of the film "Gardener of Eden" and the set of cast members of "J. Edgar", and checking for the overlapping entity. However, language models such as ChatGPT may not necessarily perform this sensible entity-seeking process. This is because the correct answer can simply (but incorrectly) be inferred by the high occurrence of "Leonardo DiCaprio" in the model's training data with "Gardener of Eden" and "J. Edgar".3 In such a scenario, ChatGPT may generate the correct response through this shortcut of semantic association. Using semantic shortcuts such as co-occurring entities works well on common test cases. Still, they limit systems' generalizability and robustness on cases not aligning with pre-training distributions. This is evident if we remove entities directly relevant to the answer, such as "J. Edgar", and replace them with recursive entity-seeking problems where models must first identify them, shown as "3 Layers" in Fig. 1. We observe that ChatGPT no longer produces the right answer without enough semantic hints and hallucinates "Adam Sandler", which may be because "Eden" is related "Adam", and at the same time, both the writer and main character of "Gardener of Eden" is named "Adam." In addition, ChatGPT almost always comes up with "Adam Sandler" if asked for actors named "Adam."

Footnote 2: In this paper, we consistently utilize the gpt-3.5-turbo-0301 checkpoint for ChatGPT.

To investigate and quantify whether LLMs can follow a correct reasoning path and resist the attempt to take "greedy shortcuts" during inference, we introduce EureQA (Extending Underlying Reasoning Chains in QA), a sophisticated multi-hop question answering dataset, meticulously designed for diminishing semantic associations and gauging the capability of LLMs in undertaking extensively chained reasoning processes. To create EuroQA, we design a method that removes semantic shortcuts that will lead to the correct answer while adding "deceptive semantic cues" that are distracting and irrelevant to the correct answer. We also guarantee a simple and consistent reasoning path and only use common facts that should be memorized by the models. In this way, if an LLM fails on such tasks, it directly suggests that the LLM is taking incorrect shortcuts during inference. For instance, we modify the original question in Fig. 1 by substituting "J. Edgar" with a placeholder dubbed "Film A" and supplementing a descriptive sentence about the film. Our intent with this strategy is to lessen the model's reliance on direct information from the film name. We can automatically extend the depth of a reasoning chain by further replacing entities in the descriptive sentence about "Film A"

Figure 2: Even given all the required information needed for the question (selected information shown in the figure), GPT-4 still makes mistakes starting early layers (highlighted in grey). We only show partial output here. Notice that we give GPT few-shot prompts.

 

with corresponding abstract names. In particular, this generation method is developed on a knowledge base, ensuring that the information employed predates the training data of the examined LLMs.

We then evaluate model performances on the EureQA benchmark. We show that state-of-the-art LLMs,4 are incapable of proper reasoning for instances in EureQA, with GPT-4 only achieving 62% to 64% accuracy in identifying common entities in Wikipedia and ChatGPT less than 40%. On the contrary, humans achieve near-perfect performances without much effort because of the simplicity and consistency of the gold reasoning path. We also show that GPT-4 performance strongly correlates with the semantic similarities between the gold answer and other entities mentioned in the question. These findings combined suggest that even the best language models, with detailed in-context-learning (ICL) processes, still fall for deceptive semantic shortcuts and hence hallucinate and fail on EureQA.

Footnote 4: In this context, ChatGPT and GPT-4.

To summarize, the contribution of our work is three-fold. First, we propose a novel method for generating question-answering data with extended reasoning chains, which allows us to create deceptive semantic shortcuts. Second, we propose EuReQA, a QA dataset specifically designed for evaluating LLMs in scenarios with reduced or deceptive semantic associations. Finally, our extensive experimental findings suggest that contemporary LLMs predominantly depend on semantic associations in question answering. As entities in question are replaced with alternate inquiries, LLMs cannot adhere to an accurate reasoning process necessary for problem resolution.

## 2EureQa

In this section, we introduce the dataset EureQA for evaluating LLMs on extended reasoning chains. We start with the definition and generation approach of a reasoning chain (SS2.1), followed by the processes for question generation (SS2.2) and refinement (SS2.3).

### Reasoning Chain

In EureQA, every question is constructed through an implicit reasoning chain, as depicted in Fig. 3 (1). This chain is structured in layers, each of which comprises three components: an **entity**\(e_{i}\), an associated **fact**\(f_{i}\) about \(e_{i}\), and a relation \(r_{i}\) which connects \(e_{i}\) and the entity in the succeeding layer of the chain, referred to as \(e_{i+1}\). The identity of \(e_{i}\) is constrained by \(f_{i}\), \(e_{i+1}\) and \(r_{i}\) together. In most cases, neither \(f_{i}\) nor \(e_{i+1}/r_{i}\) can lead to \(e_{i}\), but we guarantee the uniqueness of \(e_{i}\) when subjecting to both constraints. This design allows us to extend the reasoning chain by replacing \(e_{i}\) with \(f_{i}\), \(r_{i}\), and \(e_{i+1}\), until we provide the actual entity name of \(e_{N}\). A valid reasoning path would be to take the provided \(e_{N}\) and resolve \(e_{N-1}\) based on \(f_{N-1}\) and \(r_{N-1}\) until we identify \(e_{1}\), which is the answer to the overall question.

To automatically collect such data, we use a structured knowledge base DBpedia Auer et al. (2007), because it provides accurate facts and relations between entities, and at the same time, all these facts and relations should have been memorized by most LLMs since Wikipedia is seen during pre-training. We use the _2021-09_ DBPedia snapshot to guarantee that the information of entities was accessible to LLMs during their training phase. The criteria for choosing valid \(e_{i}\), \(r_{i}\), and \(f_{i}\) to form chains are the following:5

Footnote 5: Being “valid” here refers to satisfying the following criteria for data extraction.

**Entity.** Each entity \(e_{i}\) must exist in the knowledge base as an "entity."

**Relation.** Each relation \(r_{i}\) should connect \(e_{i}\) with more than one potential candidate entities \(e_{i+1}\). This ensures no direct semantic shortcut between \(e_{i}\) and \(e_{i+1}\). For example, the relation "award" is deemed valid for the entity "Tiger Woods" because he has won multiple awards. In contrast, the relation "college" is invalid because DBPedia only has one corresponding entity, which is _Stanford University_.

**Fact.** Facts are generated based on DBpedia relations and destination entities for each \(e_{i}\). Specifically, a valid fact \(f_{i}\) consists of a relation \(r^{f}_{i}\) and a destination entity \(e^{f}_{i}\). Note that \(r^{f}_{i}\) is different from \(r_{i}\), because it is only used to generate a single factual statement regarding \(e_{i}\), and not being used to extend the reasoning chain. We adopt two criteria for fact selection, namely easy and hard. The easy criterion selects (\(r^{f}_{i}\), \(e^{f}_{i}\)) pairs where \(e_{i}\) is the only entity in the database that corresponds to the triplet (_?entity_, \(r^{f}_{i}\), \(e^{f}_{i}\)). For example, _Jason Connery_ is the only entity that fits into (_?entity_, father, _Sean Connery_) because _Jason Connery_ is the only child of _Sean Connery_. In this way, \(f_{i}\) itself is 

sufficient for identifying \(e_{i}\). Contrarily, the hard selection ensures that not only \(e_{i}\) satisfies (_?entity_, \(r^{f}_{i}\), \(e^{f}_{i}\)). For instance, (parentCompany, _Comcast_) is a hard fact for "_NBCUniversal_" since "_Comcast_" is also the parent company of "_Xfinity_." The hard criterion ensures that \(f_{i}\) cannot uniquely identify \(e_{i}\). We apply easy and hard criteria to fact generation respectively for EureQA, resulting in two distinct sets of data: EureQA easy and EureQA hard.

**Chain Construction.** Following the previously established criteria, we employ a random-walk algorithm to fabricate chains of reasoning. The procedure initiates with a designated seed entity \(e_{1}\) in the knowledge base. Subsequently, a relation \(r_{1}\), an associated fact \(f_{1}\), and a subsequent entity \(e_{2}\) are chosen randomly to construct a valid layer of the reasoning chain. This operation is continuously executed until no more valid layer is found or the cumulative number of layers equals \(N_{max}\). We set \(N_{max}=5\) in this work. We perform at most 50 random walks starting from each seed entity to optimize efficiency and remove duplicated chains. Seed entities are sourced via two methods, either through prompting ChatGPT or by data extraction from [https://today.yougov.com/](https://today.yougov.com/).

This approach ensures that reasoning chain generation for EureQA satisfies the following three properties of each chain:

**Reasoning-dependent.** Every intermediate layer relies on information from the subsequent layer in the chain for resolution. Consequently, the model must commence from the terminal layer and engage in sequential reasoning throughout the chain.

**Length-flexible.** The extent of the reasoning chain can be conveniently adjusted by adding or removing layers, thereby enabling an evaluation of the depth of reasoning.

**Determinism-adjustable.** Determinism of the reasoning chain can be altered by omitting fact \(f_{i}\) in a layer, facilitating an evaluation of the model in addressing questions with multiple potential answers, which is not considered in this work.

### Question Generation

In EureQA, each question, denoted as \(q\), is a natural language articulation of the above reasoning chains. A layer-by-layer procedure translates this structured chain into a human-readable text. Here we denote \(q=q_{0}+q_{1}+...+q_{n}+m_{n+1}\) where \(\{q_{i}|1<=i<=n\}\) indicates the sub-question for the \(i\)-th layer and "\(+\)" stands for concatenation. \(q_{0}\) and \(m_{n+1}\) will be introduced later in this subsection. Fig. 3 (2) provides a tangible example of this translation process. Every layer comprises an entity \(e_{i}\), a relation \(r_{i}\), a succeeding entity \(e_{i+1}\), and an associated fact \(f_{i}\), which includes the corresponding \(r^{f}_{i}\) and \(e^{f}_{i}\). The initial step involves translating the triplets, (\(e_{i}\), \(r_{i}\)\(e_{i+1}\)) and (\(e_{i}\), \(r^{f}_{i}\), \(e^{f}_{i}\)) into two statements, \(s^{relation}_{i}\) and \(s^{fact}_{i}\), respectively. This is done by few-shot prompting ChatGPT, which we detail in SS8.

The subsequent step involves substituting each entity \(e\) in every statement with a placeholder, as illustrated in Fig. 3 (3). In particular, each layer now associates with a pair of statements, namely \(s^{relation}_{i}\) and \(s^{fact}_{i}\), which derive from the translation of the relational triplet (\(e_{i}\), \(r_{i}\)\(e_{i+1}\)) and the factual triplet (\(e_{i}\), \(r^{f}_{i}\), \(e^{f}_{i}\)) correspondingly. In this context, we proceed to solely obfuscate \(e_{i}\) and \(e_{i+1}\) in \(s^{relation}_{i}\) and \(s^{fact}_{i}\) by substituting them with their respective hyperpyms, \(h_{i}\) and \(h_{i+1}\), whilst preserving the identity of \(e^{f}_{i}\). Hypernym details are extracted from DBpedia if available and sought from the LLM otherwise. ChatGPT is employed in our approach with two few-shot examples, listed in SS8, obtained from DBpedia, to ensure a consistent level of granularity of hypernym. Each hypernym is subsequently appended with a specific label that is assigned in alphabetical order. For example, the third layer of a reasoning chain could hold "Actor C stars in TV Series D" as \(s^{relation}_{3}\) while \(s^{fact}_{3}\) could

Figure 3: The data generation process of EureQA.

 

be "Actor C is the artist for _Along on Christmas Day._" Here, \(e_{3}\) is substituted by "Actor C" and \(e_{4}\), which also appears in the fourth layer, is replaced by "TV Series D." The resulting statements with masked entities are denoted as \(m^{relation}_{i}\) and \(m^{fact}_{i}\) respectively.

The final step involves generating an interrogative question, represented as \(q_{0}\), and an entity information statement, denoted as \(m_{n+1}\), pertaining to \(q_{n}\). The selection of the initial interrogative question is contingent upon the hypernym of the entity \(e_{1}\) in the first layer. Specifically, \(q_{0}\) starts with "Who is ..." if the hypernym \(h_{1}\) corresponds to a person or "What is ..." in all other cases. In the example shown in Fig. 3, "Wes Anderson" has the hypernym "director." The generation process for \(m_{n+1}\) is essentially uncomplicated. Given that \(m^{relation}_{i}\) in the terminal layer exhibits the relation between \(e_{n}\) and \(e_{n+1}\), and no additional layers exist to identify \(e_{n+1}\), we generate \(m_{n+1}\) utilizing the template "\(h_{n+1}\) is \(e_{n+1}\).", which is exemplified as "Film F is _You So Crazy._" in Fig. 3.

### Question Refinement

We perform **viability filtering** to check if a reasoning chain can be correctly followed to derive the expected answer. As mentioned above, each layer of a specific chain specifies \(e_{i}\) by both a relational triplet (\(e_{i}\), \(r_{i}\,e_{i+1}\)) and a factual triplet (\(e_{i}\), \(r^{f}_{i}\), \(e^{f}_{i}\)). By excluding \(e_{i}\) from these triplets, we query our knowledge base with both (?variable, \(r_{i}\,e_{i+1}\)) and (?variable, \(r^{f}_{i}\), \(e^{f}_{i}\)). The layer is deemed viable if \(e_{i}\) is a unique solution of ?variable. We only retain those reasoning chains where each layer has passed the viability filtering.

This work aims to analyze models' reasoning capabilities. As such, we employ a **knowledge filtering** procedure to ensure that most LLMs have sufficient world knowledge to answer our questions. To illustrate this, we consider a question, \(q=q_{0}+q_{1}+...+q_{n}+m_{n+1}\), and verify both \(s^{relation}_{i}\) and \(s^{relation}_{i}\), where \(1<=i<=n\). In more explicable terms, we evaluate knowledge by presenting ChatGPT with the prompt "_Is this statement correct: [\(s_{i}\)] Yes or No ?_" A statement is considered as memorized if the LLM response includes a "Yes.76 A \(q_{i}\) is deemed to have satisfied the examination criteria if and only if _both_ of \(s^{relation}_{i}\) and \(s^{relation}_{i}\) are categorized as memorized.

Footnote 76: we applied self-consistency strategy Wang et al. (2022b) with three runs for a majority vote.

### Data Statistics

As indicated in Tab. 1, EureQA encompasses a total of 2,991 questions. The dataset is split into two levels of difficulty: hard and easy according to the criteria discussed in SS2.2. Specifically, the easy set comprises 428 five-layer questions while the hard set includes a larger set of 1,363 five-layer questions. This easy version enables manifesting LLM behavior towards questions with sufficient semantic shortcuts. From this hard set, we randomly select 300 five-layered questions. By removing layers sequentially, the hard set also yields 300 questions in each number of layers from four to one respectively. These hard questions with varying layers allow for a comprehensive assessment of LLMs in terms of their reasoning capabilities across various depths of reasoning. The distribution of categories of seed entities for the questions in EureQA is shown in Fig. 4, showcasing a broad spectrum of topics encompassed by these entities, effectively reducing any potential bias arising from specific entity categories.

Figure 4: Categorical distribution of seed entities in questions of EureQA.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline \hline Difficulty & easy & hard & hard & hard & hard & hard \\ \hline \#Layers & 5 & 5 & 4 & 3 & 2 & 1 \\ \hline Count & 428 & 1363 & 300 & 300 & 300 & 300 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of EureQA.

 



## 3 Experiment Setup

This section presents the experiment configurations employed for assessing the long-chain reasoning capabilities of LLMs through EureQA. This section begins with a discussion on model configurations (SS3.1), subsequently introducing the adopted prompting methodologies (SS3.2), and finally, it scrutinizes the evaluation configurations (SS3.3).

### Model Configuration

We consider state-of-the-art LLMs to evaluate on EureQA: ChatGPT (gpt-3.5-turbo-0301), Gemini-Pro (Gemini 1.0 Pro) and GPT-4 (gpt-4-0314). All models run with a temperature \(\tau\) of 0.8.

### Prompting Methods

We employ two prompting methods in our experiment: namely, _direct_ and _icl_. The _direct_ technique presents raw questions to the model without any supplementary context. On the other hand, _icl_ method utilizes a few-shot approach, preceding each query with two context-specific examples. Each example is characterized by a sample problem, succeeded by the statement "Let's solve this question step by step" and a step-by-step solution written by humans. In these solutions, the problem is analyzed from the final layer through the initial layer to give the correct answer, thereby obviating any necessity for backtracking during the problem-solving procedure. An example is detailed in SS8.

### Evaluation Protocol

We evaluate the **accuracy** of LLMs towards the correct answer. Considering that EureQA functions as a free-form QA benchmark and our observation indicates that LLMs typically respond with comprehensive reasoning processes, following prior studies Agrawal et al. (2015); Ossowski and Hu (2023), we employ a string-match criterion: If the correct answer, identifiable as an entity, is present in the LLM response, we deem such a response as correct. We use a self-consistency of five runs with ChatGPT and Gemini-Pro, but we do not apply self-consistency to GPT-4 due to cost issues.

## 4 Results

The results of the state-of-the-art LLMs on EureQA, as reported in Tab. 2, illustrate the variations in performance across different depths of reasoning and levels of difficulty. In general, with the entities recursively substituted by the descriptions of reasoning chaining layers, and therefore eliminating surface-level semantic cues, these models generate more incorrect answers. When the reasoning depth increases from one to five on hard questions, there is a notable decline in performance for all models, with an average accuracy decrease of 14.2% when using the _icl_ prompt and 14.3% when using the _direct_ prompt. The performance of LLMs is significantly higher on the easy set compared to the hard set. This is evidenced by a marked increase in accuracy, averaging 7.0% for ChatGPT, 15.4% for Gemini-Pro, and 19.7% for GPT-4. This finding underscores the significant impact that semantic shortcuts have on the accuracy of responses, and it also indicates that GPT-4 is considerably more capable of identifying and taking advantage of these shortcuts.

It can also be inferred from the results that the demonstrated human-written examples do help improve the performance. For both ChatGPT and GPT-4, using the _icl_ prompt consistently leads to better performance than the _direct_ prompt. This suggests that these models can benefit from examples that provide explicit and well-crafted human-written reasoning processes. These few-show examples encourage the models to resolve the questions with the correct reason instead of guessing with semantic cues. Nonetheless, even with clear reasoning processes provided, models still tend to

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{hard} & \multicolumn{6}{c}{easy} \\ \cline{2-13} depth & \multicolumn{2}{c}{d=1} & \multicolumn{2}{c}{d=2} & \multicolumn{2}{c}{d=3} & \multicolumn{2}{c}{d=4} & \multicolumn{2}{c}{d=5} & \multicolumn{2}{c}{d=5} \\ \cline{2-13}  & _direct_ & _ic_ & _direct_ & _icl_ & _direct_ & _icl_ & _direct_ & _icl_ & _direct_ & _icl_ & _direct_ & _icl_ \\ \hline ChatGPT & 22.3 & 53.3 & 7.0 & 40.0 & 5.0 & 39.2 & 3.7 & 39.3 & 7.2 & 39.0 & 13.1 & 47.0 \\ Gemini-Pro & 45.0 & 49.3 & 29.5 & 23.5 & 27.3 & 28.6 & 25.7 & 24.3 & 17.2 & 21.5 & 30.6 & 38.9 \\ GPT-4 & 60.3 & 76.0 & 50.0 & 63.7 & 51.3 & 61.7 & 52.7 & 63.7 & 46.9 & 61.9 & 66.4 & 81.8 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Accuracy of ChatGPT, Gemini-Pro and GPT-4 across different depths d of reasoning (number of layers in the questions) as well as the difficulty of the questions. We evaluate two prompt strategies: _direct_ zero-shot prompt and _icl_ with two examples.



fail for incorrect shortcuts.

Another observation is that LLMs do not always perform worse on questions with more depth of reasoning. For instance, ChatGPT with _direct_ prompt is 3.5% more accurate when the depth of reasoning d increases from four to five. We believe this also indicates that LLMs do not always follow the reasoning path and may instead answer the question based on surface-level semantic shortcuts. Further analyses can be found at SS5.2

To identify the errors in these LLM responses, we randomly sampled 20 data points where GPT-4 with _icl_, the best performing method in this benchmark, has generated incorrect answers. The most frequent error pattern lies in hallucination during the reasoning process, encompassing 80% of the cases. In such cases, an intermediate reasoning stage erroneously conceived an inaccurate, albeit plausible answer, culminating in accumulated errors in the final answer. For instance, the "founder of a rocket company" was misconstrued as "_Elon Musk_, the founder of SpaceX", rather than the correct answer of "_Jeff Bezos_, the founder of Blue Origin." This phenomenon proves our claim that LLM relies on semantic cues for problem-solving instead of cognitive reasoning.

## 5 Analysis and Discussions

In this section, we perform further analyses and provide a more in-depth discussion of our findings.

### Can human solve EureQA?

Towards a more comprehensive insight into human proficiency on EureQA, a human analysis was carried out focusing on the hard set characterized by a reasoning depth of five. This evaluation involves two computer science PhD students as the annotators to ensure their expertise. A selection of 50 questions, randomly extracted from the set, was subjected to annotation. Each question received annotations from both annotators independently, ensuring a dual perspective on every query. During the annotation process, annotators are presented with an input question to which they are required to provide an answer. To facilitate this, they are granted access to conduct information searches through the _dbpedia-snapshot-2021-09_ database. This could be achieved either through visiting the specific entity page or by querying via a SPARQL portal. The averaged accuracy of annotators achieves 95% with an Inter-annotator Agreement of Cohen \(\kappa=0.79\).

### Do LLMs take Shortcuts?

One of our main motivations is to test if language models can follow a simple yet effective reasoning

Figure 6: The distribution of entity similarity scores.

Figure 7: GPT-4 performances with different entity similarity scores between the easy and hard sets.

 

chain instead of taking semantic shortcuts based on entity associations. We design an analytical experiment based on entity similarity to investigate whether LLMs take such semantic shortcuts. Our intuition is to model the correlation between the performances and the averaged semantic similarities between the gold answer and other entities mentioned in the question. In an ideal situation where a model takes an optimal reasoning path for each instance, we will see a uniform distribution of performances: the accuracy for various degrees of entity similarities will be relatively the same. If the model relies much on entity biases and takes semantic shortcuts, we will see an increasing performance when the mentioned entities are more closely related to the target answer.

To infer entity similarities, we employ an off-the-shelf sentence Transformer model7 and encode the entity strings into embeddings. With such embeddings, we calculate the dot-product similarity between the target answer and all other Wikipedia entities mentioned in the question and compute an averaged similarity for each instance. We then draw the correlation curve of model performances on instances with certain similarities.8 This method derives a relatively continuous curve based on our limited evaluation data.

Footnote 7: [https://huggingface.co/sentence-transformers/nsnarco-distilbert-cos-v5](https://huggingface.co/sentence-transformers/nsnarco-distilbert-cos-v5).

**Observation 1.** Fig. 5 shows that GPT-4's performance positively correlates with the entity similarities in the instances. This shows that the model relies on semantic shortcuts instead of following the correct reasoning path.

**Observation 2.** GPT-4 performs similarly on different depths if the entity similarity scores are the same, especially on instances with lower similarities (\(<0.25\)). This again suggests that the model relies on spurious entity biases to solve the question, because it does not do better on instances with shorter reasoning paths. That being said, the performance differences between lower and higher depths mostly come from the different data distributions, as shown in Fig. 6.

**Observation 3.**Fig. 7 shows another interesting finding, where GPT-4 does better on easy instances with lower similarity scores (\(<0.25\)) and performs worse on those with higher similarity scores. Our explanation is that GPT-4 tends to take more obvious shortcuts by relying on only one or two entities in the easy instance, and "early exits" without even considering the entities that are closely relevant to the gold answer.

### Do open source LLMs perform better?

To expand the scope of our findings, we experimented with open-sourced LLAMA-2 models across different sizes on hard questions of EUREQA and their results can be found at Tab. 3. Similar to our observations on GPT-series models, there's a notable decline in the accuracy of Llama models as the reasoning depth increases from one to five on the hard set. We've also conducted the entity similarity analysis which led to the same observation as Observation 1 in SS5.2: LLMs' performance positively correlates with the entity similarities in the instances. These conclusions strengthened our claim that models rely on semantic shortcuts instead of following the correct reasoning path.

### Will prompting solve EureQA?

Although the effectiveness of prompting techniques is out of the scope of this paper, we still additionally tested the Tree of Thought(TOT)Yao et al. (2023) method on ChatGPT with a "propose" strategy, which tries to decompose the questions layer by layer and solve them sequentially. Our results show that such a prompting method fails completely even if we provide human-written examples for question decomposition. In no experiment did the TOT method generate a valid answer in the final response.

That being said, it is unlikely that there exists a prompting technique that will significantly improve models' performance on EUREQA over CoT, as long as we do not enforce the correct reasoning chain (which defeats the purpose of testing generalizability) during inference.

### Will optimal retrieval solve EureQA?

Although our knowledge filtering process has already removed the knowledge barrier, it can still be pointed out that whether Retrieval Augmented Generation(RAG) method can solve this task. To address such concerns, we tested GPT-4 on 300 randomly sampled 5-layer hard questions. To minimize the impact of the retrieval method, we investigate a "performance upper bound" setting, which 

directly injects the retrieval result of the visible entities and relations in the input question from DBpedia. To be specific, statements like "Storm Warning (1951 film) stars Actor D" will have "<Sstorm Warning (1951 film), starring, ?>: Ginger Rogers, Steve Cochran, Ronald Reagan, Doris Day" prepended to the input question. The retrieved knowledge will obtain at most 20 candidate entities and the correct entity is guaranteed to be included. A complete example of a question and retrieved knowledge can be found in Fig. 2. We believe this represents an upper bound of the retrieval result. Additionally, we provide each input with a 1-shot demonstration with the retrieved knowledge, input questions, and human-written reasoning thoughts.

The accuracy of GPT-4 through the above process is 62.0%, which is close to our reported performance of 61.9% in the original setting. We can then hypothesize that simply injecting knowledge into the model can hardly solve the problem and the bottleneck remains at the reasoning/question decomposition ability of LLMs. Moreover, it can be concluded that our knowledge-filtering process can effectively estimate parametric knowledge, and it is also reliable after self-consistency.

## 6 Related Work

We discuss two topics of works that are highly relevant to this study.

* misaligns with established world knowledge
- are most widely studied since they can happen in various LLMs to affect multiple tasks' performance Min et al. (2023); Li et al. (2023). Knowledge conflicts also lead to hallucination when in-context information contradicts what LLMs memorize from pretraining Longpre et al. (2021); Zhou et al. (2023); Wang et al. (2023). However, none of these existing works look into hallucination on reasoning-required questions and test whether the models are following the correct reasoning path versus the semantic token bias.

**Reasoning capability of LLMs.** The success of in-context learning Brown et al. (2020) and instruction tuning Wei et al. (2021) have inspired various works to solve reasoning tasks by prompting LLMs. More advanced prompting strategies have been proposed to enhance the reasoning capabilities of LLMs. These investigations take advantage of the heuristic nature of human problem-solving procedures and incorporate them into textual prompts as guidance for the models. An example of this can be seen in chain-of-thought prompting Wei et al. (2022), a method in which intermediate steps are generated leading up to a final solution. Yao et al. (2023) and Zhou et al. (2022b) proposed to decompose the questions into simpler, manageable sub-problems as a method to facilitate complex reasoning. Researchers also attempted to investigate LLMs' reasoning ability over structured knowledge Ding et al. (2023) which necessitates reasoning with uncertainty as well as back-tracking. Studies prior to LLMs also propose to generate rationales to improve with more faithful reasoning Rajani et al. (2019); Wang et al. (2022a). Nonetheless, this work questions the exhibited "reasoning capabilities" of LLMs by distinguishing them from the attempt to take greedy semantic shortcuts during the reasoning process.

## 7 Conclusion

This paper proposes a novel QA benchmark for probing LLM hallucinations induced by semantic shortcuts on reasoning chains. We introduced a systematic method for generating question-answering data with extended reasoning chains. This dataset enables us to examine the reasoning capabilities of LLMs on extended reasoning paths and our experimental results indicate that LLMs predominantly depend on semantic shortcuts for reasoning and such behavior contributes to its failure. Our analyses questioned the validity of current LLMs and also inspired future studies. The problem-solving or reasoning trajectories of humans could be incorporated into the training or inference phase of current Language Modeling systems for more effective computational models.

### Ethical Considerations

Innovations in technology often encounter the moral challenge of dual-use: the same develop 

ment can bring both benefits and risks. With the probing method and benchmark presented in this paper, the line between beneficial and harmful usage largely depends on data. Proper utilization of the technology necessitates the legal and ethical acquisition of input text corpora and other modalities of inputs. Legal frameworks and standards are crucial for ensuring proper data use and for granting individuals the right to remove their data. In the absence of such regulation, the ethical use of data depends on the responsibility of technology users. Additionally, the generated and analysis data may exhibit biases that systematically affect accuracy for less represented groups or in new areas, potentially resulting in performance disparities among sub-populations based on ethnicity, race, gender, and other factors. Moreover, the effectiveness of trained systems diminishes when applied to new data that deviates from their training set. Therefore, issues of generalizability and fairness must be thoroughly examined when implementing the methodologies discussed in this paper. It is crucial to embed ethical considerations as fundamental principles at each stage of system development, ensure high levels of transparency and clarity in data, algorithms, models, and functions within the system, release software under open-source licenses to facilitate public scrutiny and investigate strategies to safeguard at-risk groups.

## Limitations

Our work proposes an analytical framework and dataset to evaluate how well language models can take the right reasoning paths instead of deceptive semantic shortcuts. To this end, we identify the following limitations.

**Limited Baselines.** We only consider two variants of language models as our baselines. With more efforts in the future, we can extend to more families of large language models with different prompting techniques. Although we project that all current systems will be far behind human performances on EureQA, we may identify specific models and methods that are more resistant to deceptive semantic shortcuts.

**Limited Entities.** We only consider popular entities in Wikipedia as our target answer. Future works may benefit from considering a wider range of entities and a more general setting of our data extraction processes.

## Acknowledgement

We appreciate the reviewers for their insightful comments and suggestions. Bangzheng Li is supported by the Faculty Startup Fund of UC Davis, and the Provost's Fellowship. Fei Wang is supported by the Annenberg Fellowship and the Amazon ML Fellowship. Ben Zhou was funded by ONR Contract N00014-23-1-2365. Xingyu Fu was funded by NSF grant IIS-2212433. Muhao Chen is supported by the NSF Grant ITS 2105329, the NSF Grant ITE 2333736, the Faculty Startup Fund of UC Davis, a Cisco Research Award and two Amazon Research Awards.

## References

* 31. Cited by: SS1.
* S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives (2007)Dbpedia: a nucleus for a web of open data. In Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, ISWC'07/ASWC'07, Berlin, Heidelberg, pp. 722-735. External Links: ISBN 978-3-319-3 

use a laptop? a question answering benchmark with implicit reasoning strategies. _Transactions of the Association for Computational Linguistics_, 9:346-361.
* Kuznia et al. (2022) Kirby Kuznia, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Less is more: Summary of long instructions is better for program synthesis. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 4532-4552, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Li et al. (2023) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. _arXiv e-prints_, pages arXiv-2305.
* Liu et al. (2023) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. _arXiv preprint arXiv:2307.03172_.
* Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7052-7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. _arXiv preprint arXiv:2005.00661_.
* Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. _arXiv preprint arXiv:2305.14251_.
* Mishra et al. (2022) Swaroop Mishra, Matthew Finlaysson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. ILLA: A unified benchmark for mathematical reasoning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5807-5832, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report.
* Osowski and Hu (2023) Timothy Osowski and Junjie Hu. 2023. Retrieving multimodal prompts for generative visual question answering. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2518-2535, Toronto, Canada. Association for Computational Linguistics.
* Rajani et al. (2019) Nazreen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself leveraging language models for commonsense reasoning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4932-4942, Florence, Italy. Association for Computational Linguistics.
* Team (2023) Gemini Team. 2023. Gemini: A family of highly capable multimodal models.
* Wang et al. (2023) Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, and Muhao Chen. 2023. A causal view of entity bias in (large) language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 15173-15184, Singapore. Association for Computational Linguistics.
* Wang et al. (2022a) PeiFeng Wang, Aaron Chan, Filip Ilievski, M uhao Chen, and Xiang Ren. 2022a. Pinto: Faithful language reasoning using prompt-generated rationales. In _The Eleventh International Conference on Learning Representations_.
* Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Akanksha Chowdhery, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models. _arXiv preprint arXiv:2203.11171_.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. _arXiv preprint arXiv:2109.01652_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Yao et al. (2023) Shunyau Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate problem solving with large language models.
* Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the ai ocean: A survey on hallucination in large language models. _arXiv preprint arXiv:2309.01219_.
* Zhou et al. (2022a) Ben Zhou, Kyle Richardson, Xiaodong Yu, and Dan Roth. 2022a. Learning to decompose: Hypothetical question decomposition based on comparable texts. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2223-2235, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zhou et al. (2022b) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022b. Least-to-most prompting enables complex reasoning in large language models. _arXiv preprint arXiv:2205.10625_ 

Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompting for large language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 14544-14556, Singapore. Association for Computational Linguistics.

 



## 8 Appendices

We present the prompts used in data generation or model evaluation on EureQA. During the question generation process (SS2.2), The Verbalization Prompt in Fig. 8 transforms a triplet, (\(e_{i}\), \(r_{i}\)\(e_{i+1}\)) or (\(e_{i}\), \(r^{j}_{i}\), \(e^{f}_{i}\)), into a statement by querying ChatGPT with it. Typing Prompt in Fig. 9 is later used when we try to substitute an entity with its hypernyms in a statement but its hypernym is unavailable from DBpedia. Fig. 10 is one of the two examples we used in the _icl_ prompt answering questions with a depth of five. Questions with fewer depths of reasoning adopt two similar examples with corresponding depths. Fig. 11 is an example input of the RAG experiment, as elaborated in SS5.5 and illustrated in Fig. 2.

Tab. 3 is the experimental results of the open-sourced Llama-2 models on EUREQA, as discussed in SS5.3.

Figure 11: Sample knowledge and question in the “performance upper bound” setting of RAG study

Figure 8: Verbalization Prompt: verbalize (entity, relation, entity) into a natural sentence.

Figure 10: One of the two examples used in _icl_ prompt, with depth of five reasoning process.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline depth & \multicolumn{2}{c}{d=1} & \multicolumn{2}{c}{d=2} & \multicolumn{3}{c}{d=3} & \multicolumn{2}{c}{d=4} & \multicolumn{2}{c}{d=5} \\ \cline{2-9}  & _direct_ & _icl_ & _direct_ & _icl_ & _direct_ & _icl_ & _direct_ & _icl_ & _direct_ & _icl_ \\ \hline Llama-2-7b & 29.5 & 22.0 & 21.4 & 11.9 & 15.8 & 13.0 & 10.6 & 11.3 & 10.9 & 14.5 \\ Llama-2-13b & 40.0 & 28.5 & 27.0 & 16.4 & 20.5 & 15.8 & 11.3 & 15.9 & 13.9 & 19.4 \\ Llama-2-70b & 45.0 & 57.0 & 24.5 & 35.2 & 17.8 & 39.0 & 18.5 & 32.5 & 12.1 & 34.5 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of Llama-2 models across different depths d of reasoning (number of layers in the questions). We evaluate two prompt strategies: _direct_ zero-shot prompt and _icl_ with two examples. The 7b and 13b versions are chat variants and the 70b version is instruct variant.