Large Language Models are biased to overestimate profoundness
Authors: Eugenio Herrera-Berg
Tomás Vergara Browne
Pablo León-Villagrá
Marc-Lluís Vives
Cristian Buc Calderon
Venue: EMNLP 2023 Main (2023)
Keywords: Large language models
reasoning
bias
nonsensical statements

Abstract: Recent advancements in natural language processing by large language models (LLMs), such as GPT-4, have been suggested to approach Artificial General Intelligence. And yet, it is still under dispute whether LLMs possess similar reasoning abilities to humans. This study evaluates GPT-4 and various other LLMs in judging the profoundness of mundane, motivational, and pseudo-profound statements. We found a significant statement-to-statement correlation between the LLMs and humans, irrespective of the type of statements and the prompting technique used. However, LLMs systematically overestimate the profoundness of nonsensical statements, with the exception of Tk-instruct, which uniquely underestimates the profoundness of statements. Only few-shot learning prompts, as opposed to chain-of-thought prompting, draw LLMs ratings closer to humans.  Furthermore, this work provides insights into the potential biases induced by Reinforcement Learning from Human Feedback (RLHF), inducing an increase in the bias to overestimate the profoundness of statements.

Large Language Models are biased to overestimate profoundness
Eugenio Herrera-Berg", Tomás Vergara Browne 1222 Pablo León- Villagrá", Marc-Lluís Vives* Cristian Buc Calderon
Centro Nacional de Inteligencia Artificial (CENIA), Chile
Pontiificia Universidad Católica de Chile, Chile
Brown University, United States *Leiden University, the Netherlands
Abstract
Thus, when presenting prompts an LLM, one generally strives to make the statement informative and truthful to convey one's ideas or goals suc- cessfully. However, not all language use is aimed at efficiently communicating information. Some- times, it can be beneficial for speaker to obscure the meaning of an utterance, for example, to de- ceive or persuade to hide one's true intentions. Successful, human-level communication requires the listener to detect such language use; otherwise, they are susceptible to deception. Here, we as- sess whether GPT-4 and other LLMs can identify language created with the aim of impressing the listener rather than communicating meaning.
Recent advancements in natural language pro- cessing by large language models (LLMs), such GPT-4 have been suggested approach Artificial General Intelligence. And yet, it is still under dispute whether LLMs possess sim- ilar reasoning abilities to humans. This study evaluates GPT-4 and various other LLMs in judging the profoundness of mundane, motiva- tional, and rscuddoppofoour statements. We found a significant hatement-to-sttteeeed cor- relation between the LLMs and humans, ir- respective of the type of statements and the prompting technique used. However, LLMs systematically overestimate the profoundness of nonsensical statements, with the exception TK-instruct, which uniquely underestimates the profoundness of statements. Only few- shot learning prompts, opposed to chain-of- thought prompting, draw LLMs ratings closer humans. Furthermore, this work provides insights into the potential biases induced by Re- inforcement Learning from Human Feedback (RLHF), inducing increase in the bias to overestimate the profoundness of statements.
The term "pseudo-profound bullshit" (BS) refers to sentences that seem to have deep meaning at first glance, but meaningless (Pennycook et al., 2015). These sentences are syntactically correct and presented as true and significant but, upon further consideration, lack substance. For example:
"Consciousness is the growth of coher- ence, and of us."
1 Introduction
Pseudo-profound bullshit is thus an example of language use that is not aimed at conveying in- formation but evoking an interpretation in the lis- tener in order to seem meaningful and insightful; and could potentially fool LLMs produce non- desired responses. Cognitive strategies deployed during language exchanges to uncover (and create) these strategies require sophisticated skills (Musker and Pavlick, 2023), such as making recursive in- ferences or assessing the actual meaning of hard- to-parse sentences (Bubeck et al., 2023). In fact, in humans. the ability to detect the shallowness of
GPT-4 has now achieved the ability to perform wide range of tasks par with humans (OpenAI, 2023). Given that current LLMs excel at text inter- pretation and generation, recent work has assessed LLMs' ability to perform tasks beyond simple lan- guage understanding. And the evidence for LLMs' ability to perform these tasks is changing rapidly: while previous versions of LLMs (c.g., GPT-3.5) fell short in tasks seen unsolvable with textual input alone (Ullman, 2023), more recent models, such as GPT-4, have been suggested to achieve them (Bubeck et al., 2023). However, most of these studies have focused on presenting the LLM with statements following conversational maxims.
bullshit correlates with classi- cal measures of cognitive sophistication like verbal intelligence or individual tendencies exert delib- eration (Pennycook et al., 2015).
"Equal contribution. CCo-senior authors.
Because LLMs perform better in tasks that re-

---

quire knowledge of syntax, morphology, or phonol- ogy while struggling in tasks that require formal reasoning (e.g., logic), causal world knowledge, situation modeling, or communicative intent (Ma- howald et al., 2023), it is an open question whether an LLM will be able to detect the presence of pseudo-profound bullshit.
(Wei al., 2022: Zhao et al., 2023; Lyu al., 2023). Furthermore, increasing reasoning abilities can also be elicited via Zero-shot CoT by simply adding the sentence "let's think step by step" at the end of the prompt (Kojima et al., 2022). In the same vein, showing a few examples to LLMs allows them to quickly learn to reason com- plex tasks (Brown et al., 2020; Tsimpoukelli et al., 2021), ability known few-shot learning.
We find that GPT-4, and most other LLMs tested here, display a strong bias towards profoundness, i.e., the pseudo-prost profouud statements are systemati- cally ranked above a mid-point level of profound- ness. In contrast, humans rank these statements below this mid-point level. The one exception we found is with Tk-Instruct, which consistently un- derestimates the profoundness of every statement, including that of motivational statements.
Method
Our study is based on previous research assessing whether humans are receptive to pseudo statements (Pennycook et al., 2015). In the origi- nal study by Pennycook et al., 198 humans ranked the profoundness of statements on a 5-point Likert scale: 1 Not at all profound, 2 somewhat pro- found, 3 fairly profound, 4 definitely profound, very profound.
Moreover, we show that chain-of-thee -haougt prompting methods (Wei al., 2022), which typi- cally increase the reasoning abilities in LLMs, have no statistical effect on this ranking: and that only few-shot learning allows GPT-4 to rate statements more similar to humans and below the mid-point ranking. Finally, we show that, despite the biases found, the soa-eeetttt-stttemeee rankings display strong correlation between the LLMs and hu- mans.
We replicate this study using a variety of LLMs to judge the profoundness of statements. The LLMs used are GPT-4 and various other models (Flan-T5 XL (Chung et al., 2022), Llama-2 13B with and without RLHF (Touvron et al., 2023), Vi- cuna 13B (Chiang al., 2023) and Tk-Instruct 1 BBB (Wang et al., 2022)). We performed 20 repe- titions for each experiment account for the non- deterministic nature of token generation (given that we use non-zero temperature). We investigate how sensitive this model to aeeudo-profound bullshit by systematically comparing it with human ratings on these same and similar statements. Furthermore, use distinct prompting methods that have been shown increase the reasoning abilities of LLMs (see below). All of our code and results are publicly available
2 Related Work
Probing Language Understanding. LLMs can be treated as participants in psycholinguistic (Linzen et al., 2016; Dillion et al., 2023) cogni- tive science experimental studies (Binz and Schulz, 2023). Thus, recent work suggests a series of diag- nostics to analyze LLMS inspired by human exper- iments (Ettinger, 2020). For instance, LLMs can represent hierarchical syntactic structure (Lin et al., 2019) but tend to struggle in semantic tasks (Ten- ney et al., 2019) or display similar content effects humans (Dasgupta et al., 2022). Furthermore, Hu al. (2023) compared variety of LLMs to human evaluation on seven pragmatic phenomena test possible correlations between human and models' judgments.
3.1 Dataset
We generated five distinct datasets of sentences. For dataset 1, we used the same 30 pseudo- profound BS statements as those used in experi- ment of Pennycook et al. (2015). For dataset 2, we built a dataset comprising 30 novel pseudo- profound BS statements generated following the same procedure as Pemnycook et al. (2015). Dataset 3 was generated on the basis of the first dataset, but generated 30 novel statements by switching words from one sentence another but maintain- ing the syntactic structure (dataset 3; see Appendix
Boosting Reasoning in LLMs. Recent work evaluates the reasoning abilities in LLMs by an- alyzing the techniques that elicit reasoning (Huang and Chang, 2022). For instance, several prompting methods increase the reasoning abilities of LLMs, Mhain-of-thoughe (CoT) prompting is a method that implements a sequence of interposed natural lan- guage processing steps leading to a final answer
http://ghhh...cmmaaaaaam/issssss profoundness

---

A for further details). As points of comparison, we also used ten mundane (non-profound. datasct 4) and ten motivational (profound, dataset 5) state- ments from Pennycook et al. (2015) paper. An ex- ample from each dataset can be seen in Appendix A (Table 1).
as main effects.
Since manipulating the temperature of the LLMs did not significantly change the resulting ratings, we collapsed ratings across temperatures in all re- ported analyses. Moreover, given that we found no differences in ratings between datasets 1,2, and 3, did not include the ratings of datasets 2 and 3 in the analyses. Lastly, for simplicity and brevity, we only report statistical results of GPT-4; results of other LLMs can be found in Appendix A (Table 3)
3.2 Experimental Design
We tested the LLMs on the five datasets described in the previous section. The LLMs replied on the same Likert scale used in Pennycook et al. (2015). We further manipulated two factors: the prompt method, and the temperature. We tested three distinet prompting methods: the original in- struction participants received in Pennycook et al. (2015) (prompt 1), few-shot learning (Brown et al.. 2020) (prompt 2-3), and chain-of-though (CoT) prompting (Wei et al., 2022) (prompt 4- 10). All the prompts are listed in Appendix A (Table 2). The temperature was set to 0.1 and 0.7 assess the effect of variability in the outputs.
Likert scale values
Human
pod lovel
GPT.4
Flan-T5 (ML)
Usmm=2 (129)
Visarie (15B)
Ti-krstrust (118)
Motivational
Mundane
Statements
We let the LLMs predict one token correspond- ing to its rating of the profoundness of the sentence, and parse as an integer.
Figure I: Distribution of ratings per statement type in humans and the LLMs.
3.3 Statistical Analyses
We performed an item analysis by averaging ratings across human subjects and the LLMs' responses for each statement. Given that our goal was to compare each LLM to human- performance, we ran (within-items) two-way mixed ANOVA. for each LLM (on the original Pen- nycook al. (2015) prompt). The between-subject factor was statement type (mundane, motivational, or
4 Results
In the original study by Pennycook et al., humans discriminated between mundane, preudo-profound BS, and motivational statements. Unsurprisingly, mundane statements were rated the least pro- found, motivational statements as the most pro- found, and proufu-profound BS was rated in be- tween (Figure 1). This trend was also captured by the LLMs' ratings (Figure 1, the main effect of statement type: F(2,47) 72.09, p 0.001). Impor- tantly, even though the LLMs followed the same qualitative pattern as humans, most of LLMs' rat- ings of profoundness were constantly higher than human ratings (see Figure I, main effect of agent type: F(I,47) = 185.28, p < 0.001). The bias towards profoundness observed was independent of whether the statement was mundane, pseudo- profound BS, or motivational (non-significant in- teraction: F(2,47) = 0.96, p = 0.40) for GPT- For overestimated
(between-items)
BS), and the within-subject factor was agent type (human or LLM). To assess a potential profoundness bias when rating pseudo- profound bullshit, we tested the ratings against the mid-point scale. To analyze potential prompt effect, we first focused on GPT-4's BS statement ratings, and performed two-way ANOVA (with prompts and statement types as factors). More- over, to compare the effect of prompting between models, we performed a two-way ANOVA (with model type and prompts as factors) on the LLMs' BS statements ratings.
regressions to test the degree to which human ratings were predicted by the LLMs. Human mean ratings for each statement was pre- dicted by each LLM's ratings, with statement type (mundane, motivational, or profouud bull- shit), evaluation prompt (1 to 10), and temperature
Finally,
the rest of LLMs, profoundness only in pseudoprofound BS statements (Flan-T5 (XL), Vicuna (13B)) was larger for those state- ments (Llama-2 RLHF (13B), Llama-2 (13-B)), as denoted by a significant interaction between statement and agent types (all ps < 0.001, see Table

---

3 in the Supplement). Tk-Instruct showed the op- posite effect, but at the cost of underestimating the profoundness of motivational statements. We in- cluded a possible explanation for these phenomena in the discussion.
cantly lowered ratings for pseudo-profound bullshit statements (ps < 0.001 for both prompts, Figure 3), which at the same time caused the ratings to be not significantly different from humans (ps > 0.05 for both prompts, Figure 3). While other LLMs were also affected by these prompts, their overall effect did not induce seeudo-profound BS detection.
We used 10 prompts to obtain each LLM's s rat- ings. Results reveal that prompting significantly influenced the ratings for GPT-4 (main effect of the prompt: F(9,423) 18. 18.12, p < 0.001). Cru- cially, the effect of prompting showed a significant interaction with statement type (interaction effect: F(18, 423) = 35.06, < 0.001). Further analyses revealed that the interaction was driven by n-shot learning evaluation prompts, which lowered the rat- BS statements. Planned t-test between the two n-shot learning and other prompts revealed a significant decrease in the rat- ing for the pseudo-profound BS (see Figure 2, all ps < 0.001, Bonferroni correetedd, while n-shot learning prompts did not affect the mundane or mo- tivational statements (all ps > 0.05). Importantly, the ratings from the n-shot learning prompts were not significantly different than the human ratings for pscudo profoudd BS (all ps 0.05). Other LLMs were also affected by prompting. but sig- nificantly less (interaction effect of the two-factor ANOVA contrasting LLMs with evaluation prompt: F(45, 1305) 21.52, p < 0.001, see Figure 2).
Although the ratings of the LLMs did not match those of humans in absolute terms, did cap- ture some of the same variability in statement-to- statement responses as humans did (b = 0.15, SE = 0.004, ((939) = 4.31, ps < 0.001) across prompting strategies.
5 Discussion
ings for
We found that LLM scores significantly correlated with human data across all three statement types, suggesting that, overall, the LLMs could differenti- ate mundane, motivational, and statements. However, unlike humans, most LLMs displayed an overall bias toward attributing pro- foundness to statements, irrespective of the type of sentence in the case of GPT-4, but exacerbated by pseudo-profee sesudo-profoundesss for the rest (with the ex- ception of Tk- Tk-Instruct). For GPT-4, this bias was only reduced by providing few-shot prompts. Sur- prisingly, more recent prompting techniques did not result in significantly different ratings, perhaps reflecting the fact that detecting meaningfulness pseudo-profound statements is a task that requires world knowledge.
In addition to assessing profoundness ratings for pseudo-profound statements relative to other statement types, investigated the absolute rating on the scale. Due to the structure of the response scale, ratings below three are considered not to be profound. Thus, if LLMS detect Recudo-profound BS, their ratings should be significantly lower than the mid-point level of 3. Note that in Figure 3, we mean-center the rating data to facilitate data interpretation; above 0 implies overestimation, and below 0 implies underestimation with respect to the mid-point level.
In contrast to other LLMs tested, InsIsttruct was uniquely biased towards underestimating state- ments' profoundness across the three datasets. These results are particularly interesting given that Tk-Instruct is fine-tuned to follow instructions in more than 1600 tasks, some of which contain com- mon sense detection (Wang et al., 2022). Such fine-tuning may render this model overly cautious in judging the profoundness of statements.
Our results reveal that, overall (with the ex- ception of TK-instruct), LLMs' are biased to- wards overestimating the profoundness of pseudo- profound bullshit (ratings were significantly higher than 3: t(29) = 6.67, p < 0.001, average = 3.70, see Figure 3). This is in contrast with humans, who perceived pseudo-profour bullshit statements non-profound (ratings were significantly lower than 3: t(29)= = -9.71, p < 0.001, average = 2.56, Figure 3). GPT-4's bias towards profoundness was prevented, however, by the two prompts that signifi-
Why do LLMs display bias towards profound- ness overestimation? One possibility is that this overestimation emerges from the data used to train these models. Alternatively, this bias may emerge from additional tuning after the initial training. For instance, LLM alignment using reinforcement learning from human feedback (Ouyang al., 2022), as with GPT-4 and Llama-2, may introduce a bias towards being overly credulous. Our results provide a hint in this direction. Indeed, the Llama-2 RLHF version consistently provides higher ratings

---

Likert scale values
Flasts oll Elrmar RUIT 0121 Uarmed (200)
Evaluation prompt
Figure 2: Profoundness ratings across all evaluation prompts for each LLM.
of profoundness each statement, compared with its no-RLHF counterpart. This sheds light on how RLHF can potentially (negatively) impact the judg- ment of LLMs.
Limitations
First, our few-shot learning setup was limited to a relatively small set of examples. An exploration with more extensive sets (e.g., 10 or more exam- ples) may yield different results (Wei et al., 2022). Second, while our experiments covered various prompting strategies, there remain various unex- plored methods, such as "tree of thought" (Yao et al., 2023), which can boost the performance of GPT-4 in the context of profoundness detection. Third, economic factors (i.e., high costs of GPT-4 API use) constrained us to carry out proper ex- ploration of relevant hyperparameters and potential experimental design factors. Indeed, Pennycook et al. (2015) performed extensive analysis of personality traits and other cognitive capacities and their effect of those on pseudo-profound profound bullshit receptivity. Future work should include these tests for LLMs, allowing us to further explore the theo- retical underpinnings of LLMs' overly credulous perspective. Indeed, understanding how different factors influence the propensity of LLMs to overes- timate profoundness, can inform the development of more unbiased and accurate models.
Detecting prnuoudd BS may depend on the ability of these models to represent actual mean- ing. Furthermore, the lack of dynamics in LLMs and top-down control processes may prevent these models to detect semantic incongruence elicited by preudo-profound statements. In contrast to LLMs, dynamic control processes are at the heart of the human capacity to complete goals and resolve con- flict in incongruent tasks. (e.g. Botvinick et al., 2001; Shenhav et al., 2014).
0.5
bettonn
Humanc GPT-4
Fan-75 IXL
(138)
Viure 12381
Te (1-))
Ethics Statement
-0.5
Our data were collected using GPT-4's API, in ac- cordance with their terms of use, which do not state the prohibition to use the model for research pur- poses. We did not perform any human experiment studies and simply used the open-sourced human data Pennycook al. (2015). Hence, no ethics committee approval was needed to carry out our study.
-1,0
Original prompt prompt
1-shot
3-shot
Figure 3: Overview of the profoundness assessment in humans, and the LLMs with 1-shot and 3-shot learning. Ratings adjusted to be centered around the midpoint (3).

---

Acknowledgements
Papers), pages 4194-4213 Toronto, Canada, Associ- ation for Computational Linguistics.
This work was funded by the Centro Nacional de Inteligencia Artificial, CENIA, FB210017, Basal ANID.
Jie Huang and Kevin Chen-Chuan Chang. 2022. To-
wards Reasoning in Large Language Models: Sur- vey. arXiv preprint arXiv:2212.1040:
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in Neural Information Processing systems, 35:22199- 22213.
References
Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6)=2218523))
Matthew Botvinick, Todd S Braver, Deanna M Barch, Cameron S Carter, and Jonathan D Cohen. 2001. Conflict monitoring and cognitive control. Psycho- logical Review, 108(3):624.
Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open Sesame:
Getting inside BERT's Linguistic Knowledge. In Proceedings of the 2019 ACL Work- shop BlackboxNLP: Analyzing and Interpreting Neu- ral Networks for NLP, pages 241-253.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax- sensitive dependencies. Transactions of the Associa- rion for Computational Linguistics. 4:521-535.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang. Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful Chain-of- Thought Reasoning.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamnr, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with GPT-4, arXiv preprint arXiv:2303. 12712.
Kyle Mahowald, Anna A Ivanova, Idan A Blank. Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fe- dorenko. 2023. Dissociating language and thought in large language models: cognitive perspective. arXiv preprint arXiv:2301.066)
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez. Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open- Source Chatbot Impressing GPT-4 with 90%* * Chat- GPT Quality.
Sam Musker and Ellie Pavlick. 2023. Testing Causal Models of Word Mcaning in GPT-3 and-4, arXiv preprint arXiv:2305. 14630.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
OpenAl. 2023, GPT-4 Technical Report. arXiv preprint arX14:2303.08774
2022. Scaling instruction finetuned -atuettio-finetueeed language models. arXiv preprint ar:: 22..1111166 11416.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Lan- guage models show human-like content effects on reasoning. arXiv preprint ar a211::2207.0705.
Gordon Pennycook, James Allan Cheyne, Nathaniel Barr, Derek J Koehler, and Jonathan A Fugelsang. 2015. On the reception and detection of pscudo- profound bullshit. Judgment and Decision making. 10(6):549-563.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can Al language models replace human participants? Trends in Cognitive Sciences.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Amitai Shenhav, Mark A Straccia, Jonathan D Cohen, and Matthew M Botvinick. 2014. Anterior cingu- late engagement in a foraging context reflects choice difficulty, not foraging value. Nature Neuroscience, 1249):22491122
Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine- grained comparison of pragmatic language under- standing in humans and language models. In Pro- ceedings of the 61 Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? Probing for

---

sentence structure in contextualized word representa- tions. In 7th International Conference on Learning Representatinss, ICIR 2019.
rectify any potential syntax inconsistencies with the prompt: "Correct any inconsistencies in upper- case usage, spacing, or punctuation in the following sentence".
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
All the code used during this study, including the datasets creation and evaluation, is publicly in the following link https://gittub..cm//uueeenn// overstimate overstimate-profounnnee profoundness.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Mul- timodal few-shot learning with frozen language mod- els. Advances in Neural Information Processing Sys- tems, 34:200-212.
Dunset 1:
"Conscieunness the growth ceherencs, and us.
Dunset
"Your body belongs progressive expunsion
Dataeet "Forgieness means letting go hope for better post."
Datsect "Newborn babies roquire constant arication:
Datsect "Your teacher can open the dooe, but you must onter by yourself"
Table I: Examples from each dataset used in the experi- ment.
Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-ming tasks. arXiv preprint arXtv:2302.08399
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganch Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, et al. 2022. Super- NaturalInstructions:: Generalization via Declarative Instructions on 1600+ NLP Tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny 7hou, al. 2022. Chain-of-thought prompting elicits soning in large language models. Advances Neural Information Processing Systems, 35:24824 24837.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliber- ate Problem Solving with Large Language Models. arXiv preprint arXiv:2305. 10601.
Hongyu Zhao, Kangrui Wang, Mo Yu, Hongyuan Mei. 2023. Explicit planning helps language models in logical reasoning. arXiv preprint arXiv:2303. 15714.
A Appendix
Statements from dataset and 2 extracted from two websites: wisdomofchopra.comm and seb- pearce.come combbblitttb,, and from vague statements from Deepak Chopra's Twitter feed. Ten statements were selected from each of these three sources. The statements extracted from each website are con- structed by randomly patching profound words into meaningless but syntactically correct sentences.
To create dataset 3, we used spaCy. In particular, random selection of the nouns in the 30 sentences from dataset was substituted with a word from its 10 nearest neighbors, as determined by spaCy's cosine distance. Subsequently, we used ChatGPT

---

cxperisase Bilow Prdiand dufp messing gos
then pizsee "preicend" you thrkki sellowng poin scale:
Oriiiaal Hurna pronpt (poeng
profornaans sollornng
I- di podendd seneshu grofand, 3- Skly grooudl, definiely peofourd, 5 very peofound"
ac inssoaed in how popte oxpeninas Be procuna Bilow me sonce staassactis laecs fram ideeaus wdbelies
Pross red rach vntmer and tote momert that voor whstir right mear. then press non how "protoend" you think Profoud neees dedp meating of greut and becadly inclasive sigtifioanee"
Jolan Lesming (pocapt-2)
Fo indasss, the scaicase:
"This life is nolling shse of summetig cakinding of famiis complesisy"
is peofourd and is conniternd as pardopproonn bulsbii, basine the sosictaioo of thos worh in the cane scntceer do prosice any mestingg Rai the preforndess the folloiing cenernees on the collowngg poin scale:
I- Not aill pndound, 2 sesonnsll lat profaud, 3- fiil grodund 4 delaidll profous, 5 peofourd"
We insooled is Pless ned each Prolornd 'it doep masiing goor beeadly inclueno sigitonece." For instace, the following life "mrugieation iecinesll compechesib qpritta creativiy."
expeniene precural Relow series
lrem releaae edbelce.
and momert thint shor what
Then prease how "profornd" you think
3-last Leuning (pocapt-5)
celinding kamiis onmplesity.
roviity
prokaend
groeond bulldhr,
sarte seenence procide any meoring
Rai the pedfordaes the Sollowing sestences the Sollowiyg 5 point stale: In Nie silll pnsicnad, 2 samoella groloand, 3- grionaal deiniddy profoand, very peafouns"
we iomoaad in how people expenrece the proczed lilkww an sance stxemenns ssern from niexar wehomss
Please real each sutement and take a monert dhnt about what night near. Then please rute how "prodound" you Birk ItS.
Profourd means deep messing of grour and becally inclese sigrificance""
OUT saot (prongt- 4)
Rai lae pedfondacs the Selloing serteracs us for Aelloway point scale
In Nue stall protand, 2 somoster protcand, 3- facty prefosnd, 4 detenill profoardd 5 very profiand
To give your nnwerr thin arp by sep
"We me unavolal in how people experience dhe prodare), Below asetics of statement lakee foom relevust webultes.
Plzzac roal each eutemare and tike munert thin abort whst night Then piezsc nie hiww "proiound" think its. Pooloand neuss doop accsing gos lecadly inclusie sigtificesse" "oo mnoeer fins licycle has blood bair' Sexud, beliere the
I mundune (prorppe.) 5)
ansement that goon with nomal murdon sentrece, srchae:
level of produndaees this musdaace scalcaus, you should with lioe dae Syptiai da Sillowuz point
pelounhes the
pretaadd
profoann, very peofosse"
people expenence protard Please road each sutemeen and take monest dusk Joou night meas. Then plesse rute how "produndd" you flikk
inseested
Prafinn ieaas doep messing: gos lecadly inclusine siasifioane."
Uorte nomoontionaal (proepped))
"I give your yuuorr fird compane the ansemant that wore goon with momottional semnnce oath "The crotiee wtsls the th carcoed."
Secord, you believe the susercerss have the some level of grodundness as this munione sestence, you shoeld answer with a high value or the poir stale. Rai lae profondaca the sollwing sestetacs us for Sillowiz Syoiii scale
In Nue stail prrtoand, 2 somouter protcand, 3- tarly prefoand. 4 detiiell profoordd 5 very profiuue""
"We are intersted in how people experiecce the proturd. Below are sertes of suoements sieen from relevart webstees Piecac roal euch sutemus and take moncet dank doos what night mear. Then please rute lawk "pruduud"" you flink it
Profound neees ouf deep measing goor and becadly inclasive sigrificaace""
To
anowaz lins campane de staccmzate that gian with nativalional axnanoe mach "Stcores non final: beltne feat: Secosd,
continue that
I and (poonpe
muntune
lintl buy Think if belizee dee
boseball"
hase dee levd peofousiass masdate
slasall assver with value poist you
profisnnivssa
S-gcint
proforndees following crntrnces dellowing poin scale-
stlll proiend, somostor prrtand, 3a ferty prineed deterell penfsersd, very pentivss""
ar imsoaaed is how people oxpeniecs bhe procrnd Bilow ar senes staammntt taice fnom rdeeaus wcboties
Plasac soad each satemua and take unoment dink about whall miels meas. Than plasac nate haw "prodund"" you flink
Profuae means deep mossing: poor becasly inclumee rigrifiaenee"
To give your susver. finst compare the stzements that were given with nomal mundore sentence, sucbas:
CUT-3 mundarc (prouge-4)
"An eigiih foodball player scared aod duniny the gane."
"The giil oa dae kacyde has bluad tair."
"Bnzal i beautind country.
Secodd if you hehioe the untreeerr have the come tevel of pevfoendeees as this mustone serracee you dhoild sccner with low walor on the 5 point scale Rate the pelfundess the solloing sentences oz the Sillowigg point scale:
I- Not at prodend. 2 somew hut profond, 3- fiily prodund, 4 definiely profound, 5 profoune"
"Wh invensted is Pleas noal each estemer and moment thin dhor what might Then piezse how "prvicund" think Profoand 'of deep compase grorr beeally incleshve significane."
expenrece prodened Hollow sanee
from mievar exbotne
that
with mutivatioaal
Oaflc resitaational (peonge-))
"Lazes
will
you change threghts Secord, if you belleve the staeeeens Rais the grofounhac the Solloring senteracs fo Sollowiu Sqoini seale
chonge your
level prooundeess as this motivation serience, you should ansver with high value potor soale.
Not al prodsud), 2 somecola profound, 3- fails profound, deiuidy profound, very peofound"
We are interssted in how people experience the profound Below ur a setes stutnent taken from relevur websites.
Please read each sosement and take momert thin shout what right mear. Then please rize how "prodeend" yoon think It's
Poofourd reeuss deep acaaiig: of grout and lecadly inclusive sigtificace."
To give your avewes, fins compare de suicccass that ware givan wit metiratinal scntences each se: "Saccres is gening what wani huppinees is wonting what you grt." is bette tuil ongravity than in ascosdd in iniiston."
never drexmed shour saccess I Tocctdd for it"" Saamn,, oomparc the stalcmactaa that war gionn with mualan senderaas, azch "The twn dies playing win the ball"
CuT-3 uand and nnd. (g
"Sarfore riding the wures.
tralinn dis."
Think if you beliese the
hase the level profouddesss the mundate should wits value point scile. palonndcc mastinational scalcses, you should faz Spoin
staccncat
profonnlas the proioend, somosttr prolaand 3 farty premadd detniall peofoand, vary profored
Table 2: All prompts used in the experiment.

---

GPT-4:
Predictor
df Nom
df Don
F
p-value
7g
Statement type
2
47
72.09
.000
Agent type
47
185.28
.000
.57
Statement X Agent
2
47
0.96
.389
.01
Flan-T5 (XL):
Predictor
df Num
Den
F
p-value
7g
Statement type
2
47
100.36
.000
.67
Agent type
47
0.42
.520
.00
Statement Agent
2
47
10.18
.000
.19
Llama-2 + RLHF (13B):
Predictor
df Nom
df Den
F
p-value
7
Statement type
2
47
143.62
.000
Agent type
47
497.51
.000
.80
Statement X Agent
2
47
5.76
.006
.09
Llama-2 (13B):
Predictor
df Nam
Den
F
p-value
7g
2
Statement type
47
127.96
.000
.78
Agent type
47
302.01
.000
.69
Statement Agent
2
47
10.42
.000
.13
Vicuna (13B):
Predictor
df Nam
df Den
F
p-value
77
7
Statement type
2
47
82.75
.000
Agent type
47
35.91
.000
.24
Statement x Agent
2
47
9.26
.000
.14
Tk-Instruct (11B):
Predictor
df Nam
Den
F
p-value
7
Statement type
2
47
65.08
.000
.61
Agent type
47
13.63
.001
.11
Statement Agent
2
47
10.52
.000
.16
Table 3: ANOVAs results divided by Large Language Model. dfw indicates degrees of freedom numerator. df Den indicates degrees of freedom denominator. v/ğ indicates generalized eta-squared.


Reviews:
[{"review_id": "yj6Uhvyo3D", "reviewer": "Reviewer_T4Lt", "summary": "", "questions": "A. What is the meaning of N=20 for GPT-4 on line 142?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. In this paper, we specifically focused on GPT-4 given the claims made regarding its ability to demonstrate sparks of AGI. But also, and perhaps even more crucial, due to the massive industrial applications we are witnessing based on this model. For those reasons, we believe it is important to test the limitation of GPT-4 (taken as the sota LLM model). And, in our opinion, any such limitation must be considered an important addition to the literature.\n \nThe evaluation of LLMs typically focuses on tests that target traditional cognitive abilities (e.g., reasoning, planning, spatial navigation…). Our aim in this paper was to take a different route. Here, we particularly focused on the ability of LLMs to detect deceptive language, and therefore target the communication abilities of these models. We believe that our results showing the lack of ability for GPT-4 to detect deceptive language is, standing on its own, of great value to the NLP field, not only at the scientific level, but also at the practical one. \n\nNonetheless, we agree with both comments raised by the reviewer:\n \n\n**Regarding the scope**\n\nFirst, the scope of our paper was limited to GPT-4, but one could make the claim that this extends to the open AI GPT model family (as GPT-4 is the SOTA model). To further justify our claims that such an effect is pervasive to LLMs in general, we ran the same experiment using several other models: Flan-T5 XL, Llama-2 (13B), Llama-2 + RLHF (13B), Vicuna (13B), and Tk-Instruct (11B). Our results reveal an interesting pattern. All models overestimate the profoundness of pseudo-profound bullshit statements, except Tk-instruct. Specifically, Flan-T5 XL, Llama-2 (13B), Llama-2 + RLHF (13B), and Vicuna (13B) display the same qualitative pattern as that of GPT-4: they overestimate the profoundness of bullshit statements, both when compared with humans and with the midpoint level of the scale. In contrast, Tk-Instruct, rates pseudo-profound bullshit statements similar to humans. Yet, Tk-Instruct also displays a strong bias to underestimate the profoundness of motivational statements compared with humans. Hence, this model is generally biased to underestimate the profoundness of statements. These results are interesting given that Tk-instruct is fine-tuned to follow instructions in more than 1600 tasks, some of which contain common sense detection. Such fine-tuning may render this model overly cautious in terms of profoundness rating. \n\nFurthermore, on top of broadening the scope of the paper by testing more models, adding those models also allows us to test the effect of RLHF on the statement ratings directly. Indeed, this can be done globally by contrasting GPT-4/Llama-2 + RLHF vs. all other models, but also locally by directly comparing Llama-2 with Llama-2 + RLHF. Note that the only difference between those models is the addition of RLHF. At the global level, our results show that models with RLHF tend to overestimate the profoundness of BS and motivational statements even more compared with models with no RLHF. Crucially, at the local level, we observe the same trend: when compared with its counterparts with no RLFH, Llama-2+RLHF overestimates the profoundness of BS (and motivational) statements.\n\nSuch results imply that, although LLMs have a general propensity to overestimate the profoundness of BS statements, this bias seems to be enhanced by the inclusion of RLFH. Therefore, these results may emerge from the additive effects of (i) the inability of LLMs to dissociate deceptiveness from non-deceptive sentences, which could explain why n-shot learning generates better performance, as LLMs may simply recognize a type of “awe-inspiring” words and make judgments based on this semantic similarity, and (ii) that RLHF may have introduced a bias towards being overly credulous. Moreover, instruction fine-tuning may have a distinct effect by making these models overly cautious when judging the profoundness or meaningfulness of statements. \n\n**Regarding limited framing**\n\nSecond, we did not explicitly state why the inability to detect deceptive language would be a problem for GPT-4. Initially, we purposely avoided discussing this issue as we wanted the scope of this paper to be focused on a novel and original way to test the limits of LLMs, outside the realm of typical cognitive abilities tests, and focus on using syntactically correct but pseudo-profound sentences. We still believe this is what our paper should focus on, and that such a scope is of great interest for the field. Yet, we agree with the reviewer that discussing the implications of this inability would strengthen the paper. Several potential limitations of this inability come to mind. First, our result should come as a warning in case these models are used within contexts that make use of awe-inspiring words (e.g., quantum physics). Indeed, these models may spew out nonsensical sentences while being conformed with regards to their meaningfulness. Hence, this overestimation of profoundness might be an issue in any application scenario. Second, although humans do a better job at detecting deceptive language, specific biases can be injected within LLMs through the use of RLHF, making these models overly credulous, and thus more easily compliant. Finally, whereas the ability to perform specific cognitive abilities can increase through scaling, the inability to detect deceptive language may emerge from core features of LLMs, such as their next-word-prediction objective or their lack of recurrent dynamics. Indeed, even GPT-4 (with presumably more than a trillion parameters) overestimates the profoundness of BS statements. \n\nOur CR version will address these two main comments raised by the reviewer. The graphs will include the results of all LLMs, thereby expanding the scope of our paper to LLMs. The reviewer can access the visualization of our revised 3 figures on the anonymized GitHub repository at this address: https://github.com/llms-are-biased/to-overestimate-profoundess (this anonymized GitHub also contains the code to extract the data from the LLMs; increasing the reproducibility of our work). Moreover, for the purpose of data visualization (given the significant amount of LLMs that we have added to this work), all figures are now represented as bar plots (cfr. Hu et al., 2023; this allows us to increase the readability of the figures compared with violin plots). The appendices will contain the same statistics we performed on GPT-4, but for all the other LLMs we have tested. These statistics will be shown in the form of a table (accessible on the previously mentioned GitHub repository address), which essentially shows that the observed results for GPT-4 hold across the board of LLMs (except for Tk-instruct). Furthermore, we will explicitly address why the inability to detect deceptive language is an issue in LLMs (as stated in the previous paragraph), and we will include an example of pseudo-profound bullshit in the introduction. \n\nLastly, N=20 is a typical notation used in the field of psycholinguistics and refers to the amount of “participants” in the study. In our case, to perform statistical analyses, we ran GPT-4 20 times; hence, N=20. The same number was also used for all the other LLMs. We will clarify this in the CR submission.\n\nWe hope our replies and these changes can convince the reviewer of the novelty, extended scope, and implications of our work, and hope that addressing their comments can warrant the acceptance of our work.\n\nReferences: Hu, Jennifer, et al. \"A fine-grained comparison of pragmatic language understanding in humans and language models.\" arXiv preprint arXiv:2212.06801 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and the problem interesting. The focus on language that is not “aimed at efficiently communicating information” is timely and important. Further, the experiments are reasonably constructed and analysis well done.", "reasons_to_reject": "The main weaknesses of the paper are: i) limited spread of models investigated, and ii) limited framing for why the problem as investigated is a problem for GPT-4. For i), the paper only investigates GPT-4, and so limits the scope of the claims to a model with very little transparency. As such, it is difficult to extrapolate from this paper how other models may behave and the conditions under which these effects obtain. The discussion mentions that RLHF “might have introduced a bias towards being overly credulous”. This claim could be addressed by exploring other models, thus making the results more impactful. \n\nFor ii), I take the work to be suggesting that for humans, pseudo-profound bullshit is a particular type of statement that utilizes limitations in our reasoning to trick people. The investigation of GPT-4 demonstrates that may be more susceptible to such illusions. However, the work lacks discussion of why this poses a particular problem for these models. I can imagine that a model with a propensity to generate pseudo-profound bullshit would cause problems, as it would mislead people. Articulating why we should be concerned by models being misled by such statements would strengthen the paper.", "questions_to_authors": "A. What is the meaning of N=20 for GPT-4 on line 142?", "paper_topic_and_main_contributions": "This paper investigates how GPT-4 rates statements for profoundness. The core empirical observation is that humans can be susceptible to what is termed “pseudo-profound bullshit” – statements that may seem profound at first, but sustained reflection can expose as meaningless (e.g., “your body belongs to the progressive expansion of success”). The main contribution of this work is a demonstration that GPT-4 is more prone to viewing such statements as profound as compared to humans. An additional contribution is the datasets used to evaluate the modes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It might be nice to include an example of pseudo-profound bullshit in the introduction (perhaps moving Table 1 to the main text once you have more space)."}, "extra_text": {}}, {"review_id": "y7wbsIp9ce", "reviewer": "Reviewer_W7L6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for her/his appraisal of our work, and relevant comments. We address the novelty of our work in what follows.\n\nPrevious studies focused either on circumscribing the limits of typical cognitive abilities (e.g., reasoning) in LLMs, or their ability to reply to distinct types of judgments as humans (Hu et al., 2023). The crucial distinction in our work lies in that we specifically focus on the ability of GPT-4 to detect deceptive language. Therefore, our work targets the communication abilities of these models. In particular, compared with previous work (see Hu et al., 2023), we specifically focus on deceptive language that aims to create the false impression of depth, instantiated as syntactically correct yet pseudo-profound bullshit sentences. To the best of our knowledge, no other work has probed GPT-4 with this type of deceptive language in a systematic way, while exploring distinct state-of-the-art prompting techniques. Thus, in light of our results, we do believe that our work makes for an important and original contribution to the field. \n\nMoreover, based the reviewers’ feedback, we have now broadened the scope of our paper and ran the same experiment using several other models: Flan-T5 XL, Llama-2 (13B), Llama-2 + RLHF (13B), Vicuna (13B), and Tk-Instruct (11B). Our results reveal an interesting pattern. All models overestimate the profoundness of pseudo-profound bullshit statements, except Tk-instruct. Specifically, Flan-T5 XL, Llama-2 (13B), Llama-2 + RLHF (13B), and Vicuna (13B) display the same qualitative pattern as that of GPT-4: they overestimate the profoundness of bullshit statements, both when compared with humans and with the midpoint level of the scale. In contrast, Tk-Instruct, rates pseudo-profound bullshit statements similar to humans. Yet, Tk-Instruct also displays a strong bias to underestimate the profoundness of motivational statements compared with humans. Hence, this model is generally biased to underestimate the profoundness of statements. These results are interesting given that Tk-instruct is fine-tuned to follow instructions in more than 1600 tasks, some of which contain common sense detection. Such fine-tuning may render this model overly cautious in terms of profoundness rating. \n\nFurthermore, on top of broadening the scope of the paper by testing more models, we can directly test the effect of RLHF on the statement ratings. Indeed, this can be done globally by contrasting GPT-4 and Llama-2 + RLHF vs. all other models; but also locally, by directly comparing Llama-2 with Llama-2 + RLHF. Note that the only difference between those models is the addition of RLHF. At the global level, our results show that models with RLHF tend to further overestimate the profoundness of BS and motivational statements compared with models with no RLHF. Crucially, at the local level, we observe the same trend: when compared its counterpart with no RLFH, Llama-2+RLHF overestimates the profoundness of BS (and motivational) statements. Such results imply that, although LLMs have a general propensity to overestimate the profoundness of BS statements, this bias is enhanced by the inclusion of RLFH. Therefore, these results may emerge from the additive effects of (i) the inability of LLMs to dissociate deceptiveness from syntactic structure of sentences; which could explain why n-shot learning generates better performance, as LLMs may simply recognize a type of “awe-inspiring” words and make judgments based on this semantic similarity, and (ii) that RLHF may have introduced a bias towards being overly credulous. Moreover, instruction fine-tuning may have a distinct effect by making these models overly cautious when judging the profoundness or meaningfulness of statements. \n\nOur CR version will address the novelty comment raised by the reviewer. Moreover, our graphs will include the results of all LLMs, thereby expanding the scope of our paper. The reviewer can access the visualization of our revised 3 figures on the anonymized github repository at this address: https://github.com/llms-are-biased/to-overestimate-profoundess (this anonymized GitHub also contains the code to extract the data from the LLMs; increasing the reproducibility of our work). This repository contains all the prompts (also available in the appendix) and the datasets (also available in the supporting information of Pennycook et al. (2015). For the purpose of data visualization (given the significant amount of LLMs that we have added to this work) all figures now represent the data as bar plots (cfr. Hu et al., 2023). The appendices will contain the same statistics we performed on GPT-4, but for all the other LLMs we have tested. These statistics will be shown in the form of a table (accessible on the previously mentioned github repository address), which essentially shows that the observed results for GPT-4 hold across the board of LLMs (except for Tk-Instruct). \n\nWe hope these changes can warrant the publication of our work, which we believe tackles a timely issue and can be of great interest to researchers in NLP.\n\nReferences: Hu, Jennifer, et al. \"A fine-grained comparison of pragmatic language understanding in humans and language models.\" arXiv preprint arXiv:2212.06801 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper starts from an interesting perspective to probe the ability of GPT4 and shows some observations by prompting GPT4. I suggest it can be accepted into Findings.", "reasons_to_reject": "Basically this paper follows the previous work heavily. So I think the novelty is not enough. Based on some interesting results from the experiments, it may be accepted to findings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, they evaluate GPT-4’s ability to judge the profoundness of mundane, motivational, and pseudo-profound statements. They found a significant statement-to-statement correlation between GPT-4 and humans, irrespective of the type of statements and the prompting technique used. GPT-4 systematically overestimates the profoundness of non-sensical statements in contrast with humans.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t30Tf4xZSp", "reviewer": "Reviewer_JFhv", "summary": "", "questions": "A. l170: How is an answer determined? E.g., does GPT-4 have to exactly match a number on the Likert scale, or is it by highest probability assigned?\n\nB. l280: What is b?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for her/his valuable comments, and we hope that the following replies convince her/him of the merits of our work. \n\nAs for reviewer T4Lt, the first issue raised is that of the scope. Our goal was not to systematically compare distinct models, but principally to provide a method to test the limits of GPT-4. Although our work is based on a series of studies evaluating the similarity between human and LLMs judgments, we believe that our work is unique in that the statements used to assess GPT-4 belong to a very specific type, i.e., pseudo-profound bullshit. Our work thus helps delineate the limits of GPT-4. However, we agree with the reviewer that our work cannot be generalized across LLMs. To further justify our claims that such an effect is pervasive to LLMs in general, we ran the same experiment using several other models: Flan-T5 XL, Llama-2 (13B), Llama-2 + RLHF (13B), Vicuna (13B), and Tk-Instruct (11B). Our results reveal an interesting pattern. All models overestimate the profoundness of pseudo-profound bullshit statements, except Tk-instruct. Specifically, Flan-T5 XL, Llama-2 (13B), Llama-2 + RLHF (13B), and Vicuna (13B) display the same qualitative pattern as that of GPT-4: they overestimate the profoundness of bullshit statements, both when compared with humans and with the midpoint level of the scale. In contrast, Tk-Instruct, rate pseudo-profound bullshit statements similar to humans. Yet, Tk-Instruct also displays a strong bias to underestimate the profoundness of motivational statements compared with humans. Hence, this model is generally biased to underestimate the profoundness of statements. These results are interesting given that Tk-instruct is fine-tuned on following instructions in more than 1600 tasks, some of which contain common sense detection. Such fine-tuning may render this model overly cautious in terms of profoundness rating.\n\nFurthermore, on top of broadening the scope of the paper by testing more models, adding those models also allows us to test the effect of RLHF on the statement ratings directly. Indeed, this can be done globally by contrasting GPT-4/Llama-2 + RLHF vs. all other models; but also locally, by directly comparing Llama-2 with Llama-2 + RLHF. Note that the only difference between those models is the addition of RLHF. At the global level, our results show that models with RLHF tend to further overestimate the profoundness of BS and motivational statements compared with models with no RLHF. Crucially, at the local level, we observe the same trend: when compared with its counterparts with no RLFH, Llama-2+RLHF overestimates the profoundness of BS (and motivational) statements.\n\nSuch results imply that, although LLMs have a general propensity to overestimate profoundness of BS statements, this bias is enhanced by the inclusion of RLFH. Therefore, these results may emerge from the additive effects of (i) the inability of LLMs to dissociate deceptiveness from syntactic structure of sentences; which could explain why n-shot learning generates better performance, as LLMs may simply recognize a type of “awe-inspiring” words and make judgments based on this semantic similarity, and (ii) that RLHF may have introduced a bias towards being overly credulous. Moreover, instruction fine-tuning may have a distinct effect by making these models overly cautious when judging the profoundness or meaningfulness of statements. \n\nRegarding the discussion, we did not explicitly relate our results to cognitive theories, nor did we discuss the implications of these results for GPT-4. We purposely avoided discussing these issues as we wanted the scope of this paper to be focused on a novel and original way to test the limits of LLMs, outside the realm of typical cognitive abilities tests or sensical sentence-based judgments (e.g., Hu et al. (2023)). We still believe this is what our paper should focus on, and that such a scope is of great interest for the field. Yet, we agree with the reviewer that discussing the link with previous cognitive theories and the implications of this inability would strengthen the paper.\n\nWith respect to cognitive theories: It may be that the lack of dynamics in LLMs and top-down control processes, does not allow these models to detect semantic incongruence. Such control processes are at the heart of many cognitive control models in cognitive science, and allow to resolve conflict in such tasks (e.g., Botvinick et al., 2004; Shenhav et al., 2014). Moreover, Pennycook et al. (2015) performed extensive analysis of personality traits and other cognitive capacities, and the effect of those on pseudo-profound bullshit receptivity. Future work should  include these tests for LLMs, allowing to further explore the theoretical underpinnings of LLMs overly credulous perspective. Our CR version will relate conflict monitoring perspectives with that of Pennycook et al. (2015), discuss future work, and therefore further broaden our discussion.\n\nWith respect to implications: First, our result should come as a warning in case these models are used within contexts that make use of awe-inspiring words (e.g., quantum physics). Indeed, these models may spew out nonsensical sentences, while being conformed of their meaningfulness. Second, although humans do a better job at detecting deceptive language, specific biases can be injected within LLMs through the use of RLHF, making these models overly credulous. Finally, whereas the ability to perform specific cognitive abilities can increase through scaling, the inability to detect deceptive language may emerge from core features of LLMs, such as their next-word-prediction objective or their lack of recurrent dynamics. Indeed, even GPT-4 (with presumably more than a trillion parameters) overestimates the profoundness of BS statements. \n\nOur CR version will address the two main comments raised by the reviewer. The graphs will include the results of all LLMs, thereby expanding the scope of our paper to LLMs. The reviewer can access the visualization of our revised 3 figures on the anonymized GitHub repository at this address: https://github.com/llms-are-biased/to-overestimate-profoundess (this anonymized GitHub also contains the code to extract the data from the LLMs; increasing the reproducibility of our work). Moreover, for the purpose of data visualization (given the significant amount of LLMs that we have added to this work) all figures now represent the data as bar plots (cfr. Hu et al., 2023). The appendices will contain the same statistics we performed on GPT-4, but for all the other LLMs we have tested. These statistics will be shown in the form of a table (accessible on the previously mentioned GitHub repository address), which essentially shows that the observed results for GPT-4 hold across the board of LLMs (Except Tk-Instruct). Furthermore, we will explicitly address why the inability to detect deceptive language is an issue in LLMs (as stated in the previous paragraph), and explicitly link our results to the cognitive theories proposed by Pennycook et al. (2015; also as explained above). Finally, as suggested by the reviewer, we will focus this work on causal language models only and omit related work on BERT, incorporate all the missing references (Hu et al., 2023; Ruis et al., 2022; Deck, 2023), correct the typos, and incorporate the suggested presentation improvements.\n\nQuestions For The Authors:\nA. l170: How is an answer determined? E.g., does GPT-4 have to exactly match a number on the Likert scale, or is it by highest probability assigned?\n\nWe set the number of output tokens to 1. \n\nB. l280: What is b?\n\nb stands for the regression coefficient of the linear regression predicting human ratings from GPT-4 ratings.\n\nWe hope our replies address the comments raised by the reviewer, and that the substantial amount of work we have put into this revision can convince them of the added value of this research for the NLP field.  \n\nReferences: \n-Botvinick, Matthew M., et al. \"Conflict monitoring and cognitive control.\" Psychological review 108.3 (2001): 624.\n-Shenhav, Amitai, et al. \"Anterior cingulate engagement in a foraging context reflects choice difficulty, not foraging value.\" Nature neuroscience 17.9 (2014): 1249-1254.\n-Hu, Jennifer, et al. \"A fine-grained comparison of pragmatic language understanding in humans and language models.\" arXiv preprint arXiv:2212.06801 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of the paper are that it, to my knowledge,\n\n1. Answers a novel research question: while not the first to tie pseudo-profound bullshit to NLP (Deck, 2023), this paper is the first to present results on LLMs' judgments of profoundness. In the current media landscape where LLMs are being deployed, it is likely that they encounter pseudo-profound bullshit, and it is thus important to understand their behavior.\n\n2. GPT-4 prompts are reproducible: the authors provide all prompts used in experiments.", "reasons_to_reject": "My main reservations accepting the paper are regarding scope and discussion:\n\n__Scope__ As mentioned in the Limitations section, while the authors make a claim about LLMs broadly in the title, they only test on one model, GPT-4. In order for the claims to have maximal impact, results need to be demonstrated on a wider range of model families and sizes. This would resolve, e.g., some of the hypotheses posed at the end of the paper about the impact of RLHF on profoundness ratings. Moreover, given the small dataset sizes and free compute available on, e.g., Colab, it seems feasible to reproduce experiments on a wider set of models such as OPT, cf. Hu, et al. 2023 for similar experiments on multiple choice questions, performed on several different models.\n\n__Discussion__ While the paper is quite results-dense, it lacks a broader discussion that relates human and LM pragmatics/social cognition. The paper heavily cites Pennycook et al. 2015, which proposes two cognitive mechanisms underlying bullshit detection in humans-- a first, general bias towards overestimating profoundness, then a failure in downstream \"conflict monitoring\". Along these lines, this paper can be made more impactful by (1) making an explicit connection to the cognitive theories proposed in the former; (2) making a statement about practical implications of results-- why does it matter that GPT-4 overestimates profoundness? \n\nWhile the research question is promising, results are only presented on one LM and there lacks a broader discussion and tie-in to related work in pragmatics. This limits the impact of the paper. \n\nMinor:\nWhile the authors focus on GPT-4, they mention BERT in Related Work. I would recommend scoping this work to causal language models only (or GPT family only) and omitting related work on BERT altogether, as MLM and CLM are inherently different objectives.", "questions_to_authors": "A. l170: How is an answer determined? E.g., does GPT-4 have to exactly match a number on the Likert scale, or is it by highest probability assigned?\n\nB. l280: What is b?", "paper_topic_and_main_contributions": "This paper is one in a line of contributions evaluating LMs against human judgments, in this case, applied to pseudo-profound bullshit.\n\nPseudo-profound bullshit is a type of language that appears meaningful, but that lacks substance. The authors examine the extent to which GPT-4 matches human judgments on pseudo-profound bs, finding that (1) although statement profoundness judgments are correlated between GPT-4 and humans, (2) GPT-4 is biased towards profoundness. They then evaluate the effect of prompting method, showing that (3) chain-of-thought prompting methods don't significantly affect profoundness rankings; (4) few-shot learning allows GPT-4 to rate statements more similar to humans.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Bullshit detection falls into the realm of pragmatics and, in particular, pragmatic deception. \n\nMissing relevant work on bullshit and LMs, which proposes an alternative pipeline for bs detection (Deck, 2023 https://journals.uic.edu/ojs/index.php/dad/article/view/12690). \n\nMissing other explicit references to LMs and pragmatics, crucially Hu et al. 2023 (https://arxiv.org/abs/2212.06801) which measures human-LM alignment on pragmatic reasoning tasks, as well as Ruis et al. 2022 (https://arxiv.org/abs/2210.14986), etc.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Figure 2 caption should be standalone: add that it is for GPT-4.\n2. l272 textitps: typo\n3. With the additional space, I suggest to break up results section using paragraph headers.\n4. Structure: if datasets 2,3 did not have different results, omit altogether from main body; perhaps mention very briefly at the end of Section 3.1 and move all additional discussion -> appendix.\n5. l269 specify in the text \"two prompts\"-> \"few-shot prompting\". \n6. l086 Define midpoint-level of profoundness (this is introduced rather abruptly)."}}]

Author rebuttal:

Meta-review: The authors evaluate GPT-4 (and on request of two reviewers also other models) on the task of judging the "profoundness" of mundane or pseudo-profound statements. While they find a correlation between human scores and model scores, the models tend to give higher profoundness ratings to  statements than humans. The paper compares different prompting techniques, which all exhibit the reported problem.  
Two of the reviewers appreciate the importance of the issue that is being addressed; one of the reviewers is worried about a lack of novelty, but this worry is not substantiated by the reviewer -- instead, the task of judging profoundness / detecting lack of substance is a rather underresearched problem.
Decision: Accept-Main