# From human artefact to machine output: automating the “art” of psychological measurement

Fernando Marmolejo-Ramos, Okan Bulut, Luis Anunciaçáo, Louise Marques, Abhinava Barthakur, Josef Kundrat, Karel Rečka, Özge Karakale, Juan C Correa, Luis Alberto Pinos Ullauri, et al.

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Fernando Marmolejo-Ramos, Okan Bulut, Luis Anunciaçáo, Louise Marques, Abhinava Barthakur, et al. From human artefact to machine output: automating the “art” of psychological measurement. Journal of Psychology and AI, 2025, 1 (1), pp.2561692. 10.1080/29974100.2025.2561692. hal-05308555

HAL Id: hal-05308555

https://hal.science/hal-05308555v1

Submitted on 10 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

# From human artefact to machine output: automating the “art” of psychological measurement

Fernando Marmolejo-Ramos, Okan Bulut, Luis Anunciaçáo, Louise Marques, Abhinava Barthakur, Josef Kundrat, Karel Rečka, Özge Karakale, Juan C. Correa, Luis Alberto Pinos-Ullauri, Raydonal Ospina & Julian Tejada

**To cite this article:** Fernando Marmolejo-Ramos, Okan Bulut, Luis Anunciaçáo, Louise Marques, Abhinava Barthakur, Josef Kundrat, Karel Rečka, Özge Karakale, Juan C. Correa, Luis Alberto Pinos-Ullauri, Raydonal Ospina & Julian Tejada (2025) From human artefact to machine output: automating the “art” of psychological measurement, Journal of Psychology and AI, 1:1, 2561692, DOI: 10.1080/29974100.2025.2561692

**To link to this article:** https://doi.org/10.1080/29974100.2025.2561692

<figure><img src="image_4.png" /></figure>

© 2025 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

<figure><img src="image_5.png" /></figure>

Published online: 09 Oct 2025.

<figure><img src="image_6.png" /></figure>

Submit your article to this journal

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

View related articles

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

CrossMark

View Crossmark data

<figure><img src="image_11.png" /></figure>
<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

# From human artefact to machine output: automating the "art" of psychological measurement

Fernando Marmolejo-Ramos<sup>D,ª</sup>, Okan Bulut<sup>b</sup>, Luis Anunciação<sup>c</sup>, Louise Marques<sup>c</sup>, Abhinava Barthakur<sup>d</sup>, Josef Kundrat<sup>e</sup>, Karel Rečka<sup>e</sup>, Özge Karakale<sup>f</sup>, Juan C. Correa<sup>g</sup>, Luis Alberto Pinos-Ullauri<sup>h,i,j</sup>, Raydonal Ospina<sup>k</sup> and Julian Tejada<sup>l</sup>

<sup>a</sup>College of Education, Psychology, and Social Work, Flinders University, Adelaide, Australia; <sup>b</sup>Centre for Research in Applied Measurement and Evaluation, University of Alberta, Edmonton, Canada; <sup>c</sup>Department of Psychology, Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil; <sup>d</sup>Education Futures, University of South Australia, Adelaide, Australia; <sup>e</sup>Department of Psychology, University of Ostrava, Ostrava, Czech Republic; <sup>f</sup>School of Psychology, University of Wollongong, Wollongong, Australia; <sup>g</sup>Research & Development Unit, Critical Centrality Institute, Monterrey, Mexico; <sup>h</sup>IMEC research group ITEC, KU Leuven, Kortrijk, Belgium; <sup>i</sup>Faculty of Psychology and Educational Sciences, KU Leuven, Kortrijk, Belgium; <sup>j</sup>Centre for Digital Systems, IMT Nord Europe, Douai, France; <sup>k</sup>Departmento de Estatística, LInCA, Universidade Federal da Bahia, Cidade Universitária, Bahia, Brazil; <sup>l</sup>Department of Psychology, Federal University of Sergipe, São Cristóvão, Brazil

**ABSTRACT**

Creating psychological assessment tools is crucial for research but traditionally expensive and time-consuming. While Large Language Models (LLMs) show promise for automating this process, existing approaches lack systematic, user-friendly methodologies grounded in psychometric principles. This study presents an enhanced Psychometric Item Generator (PIG) method using conversational LLMs with Problem-Solving Plans (PSP) and Chain-of-Thought (CoT) prompting. Three demonstrations validated the approach: Gemini 1.5 Flash generated 20 "propensity to trust AI" items with strong semantic coherence; Claude 3 Opus created 20 "AI anxiety" items that outperformed human-generated versions linguistically; and a 6-item "AI adoption in online learning" scale was developed and validated with 1,233 participants using multiverse analysis. Results demonstrate that LLMs can produce psychometrically sound items. The AI-generated anxiety scale showed superior linguistic properties compared to human alternatives, while the learning scale exhibited good internal consistency, item homogeneity, and clear two-factor structure across multiple analytical teams. The study establishes a PSP-CoT framework that improves LLM output quality, offering researchers a cost-effective, accessible scale development methodology. However, findings emphasize that human oversight, rigorous validation, and ethical considerations remain essential components of the process.

**ARTICLE HISTORY**

Received 21 April 2025
Accepted 15 August 2025

**KEYWORDS**

Large language models;
psychometrics; psychometric
item generator; chain-of-
thought

# 1. Introduction

The use of surveys and/or self-report scales in psychology has been increasing in recent decades (Clark & Watson, 2019). These instruments, after having enough sources of evidence of validity (Association, 2014), are valuable tools for measuring constructs that are not directly observable. Their construction is not an easy process involving a series of steps, from conceptualisation up to validation, including the item generation (Jebb et al., 2021). The latter is a special step in which, usually from a larger set of items, those that best fit the survey objectives are selected, with a special focus on item wording. Clark and Watson (2019) describe this step as one of the most critical in the development of a scale, because by means of psychometric techniques, it is possible to eliminate problematic items, but not to create missing items that should have been included.

Recently, Götz et al. (2023) introduced the Psychometric Item Generator (PIG), a pioneering method that uses the GPT-2 neural network within a Python-based Google Colab environment to automate item generation. Their contribution relies on simplifying and accelerating the process of creating items for psychological assessments by producing content that aligns with the intended constructs, such as personality traits or attitudes. While groundbreaking, their original method has two primary limitations in the

<figure><img src="image_15.png" /></figure>
<figure><img src="image_16.png" /></figure>

current technological landscape: it requires users to interact with and modify Python code, creating a barrier for non-programmers, and it is based on the now-outdated GPT-2 architecture (Götz et al., 2023). Our study proposes a substantial evolution of this concept, creating a more accessible, flexible, and powerful Psychometric Item Generation variant with three key improvements. First, our approach is entirely code-free and platform-agnostic, relying exclusively on conversational prompt engineering. This removes technical barriers, empowering any researcher to leverage automated item generation using readily available web interfaces. Second, we utilise state-of-the-art LLMs (e.g. Google's Gemini 1.5 and Anthropic's Claude 3 Opus), which offer significantly more advanced reasoning, contextual understanding, and text generation capabilities than GPT-2. Third, we introduce a novel Problem-Solving Plan (PSP) and Chain-of-Thought (CoT) prompting framework. This structured approach is designed to improve the coherence and psychometric relevance of the generated items, moving beyond simple one-shot prompts to a guided, iterative dialogue with the LLM. This combination of accessibility, advanced model usage, and a sophisticated prompting strategy represents a significant and necessary advancement over existing automated item generation methods.

LLMs are machine learning models able to generate text in human languages for countless types of contexts, in which the generated text resembles human responses (Giray, 2023; Henrickson & Meroño-Peñuela, 2025). Interaction with these models is conducted through a chat-like command line, where commands are typed as human conversations, and this is the reason why the concept of prompt engineering was born, to assist in the “conversation” with these systems. Prompt engineering is an emerging field focused on the methodical creation and refinement of prompts – the instructions used to interact with LLMs – through iterative cycles of modification (see Figure 1).

There are different types of prompting: Zero-Shot Prompting, in which the instructions are given without examples; Chain-of-thought (CoT), which incorporates intermediate steps to guide the reasoning, breaking down the task into ordering sub-goals and offering examples of step-by-step reasoning (Chu et al.,

<figure><img src="image_17.png" /><figcaption>**Figure 1.** Scheme of how an LLM works. It all starts with a prompt or input text, a human conversation-style instruction that is segregated (Tokenization) into a group of characters based on punctuation marks, spaces, or special characters. These groups can represent words or subwords and are called tokens. Once the tokenizer algorithm has segregated the text, each token is identified with an ID, which in turn, is associated with a dense vector of high-dimensional space (i.e. GPT-3 uses vectors of 12,288 dimensions), forming an embedding matrix. These vector representations are suitable for neural network processing and contain all LLM training data and will allow the algorithm to capture semantic relationships and context. The embedded text sequence is processed via an encoder network (a recurrent neural network [RNN] or transformer-based architecture) which analyses word relationships and builds a contextual representation, capturing knowledge about the overall meaning and flow of the input. In the context with attention phase, the most significant parts of the context generated by the encoder are highlighted (i.e. 'scales' in this context do not refer to devices used for measuring a person's body mass or the small rigid plates that grow out of the skin of a fish). Then, the decoder network, another RNN or transformer-based architecture, uses the encoder context and its own internal state to generate the output text word by word by predicting the next word in the sequence based on the learned information. For more details visit: https://cutt.Ly/lw8PKU7Y.</figcaption></figure>
<figure><img src="image_18.png" /></figure>

2023); Few-Shot Prompting, similar to Zero-Shot, but with some examples; and self-consistency (S-C) in
which similar instructions are given to obtain a pool of responses from which the most frequent is selected
(Ahmed & Devanbu, 2023) (see also Wei et al., 2022). In our case, we propose to use a combination of CoT
with Few-Shot to configure our adaptation of the PIG approach, which can be applied to any LLM available.

The PSP component, inspired by plan-and-solve prompting strategies (L. Wang et al., 2023), specifically advances beyond prior prompt engineering methods by establishing an upfront, top-down plan that defines the scope, goals, and boundaries of the interaction before the CoT begins. While a standard CoT prompt breaks a task into steps (Y. Wang et al., 2024), a PSP ensures those steps are logically coherent and directed towards a pre-defined psychometric objective. For instance, the PSP would first establish the goal: “Develop a 5-item scale for the ‘sociotechnical blindness’ dimension of AI anxiety, ensuring two items are reverse-coded and all items are appropriate for a lay audience.” The subsequent CoT then executes this plan step-by-step. This “plan-and-solve” approach reduces the risk of the LLM deviating into irrelevant or incoherent paths, a common issue with less-structured prompting, thereby making the item generation process more efficient, targeted, and aligned with psychometric goals from the outset.

Our proposed approach proposed herein requires no technical expertise and is available for free use. It overcomes prior barriers to automatic item generation like inflexibility, inaccessibility, and computational demands. Additionally, our approach enables researchers to break free from reliance on manual item writing, allowing algorithms to finally speak the language of psychometrics. Overall, our study aims to provide researchers with a cost-effective, versatile artificial intelligence solution for psychological measurement.

In the next section, we illustrate through three demonstrations, our non-technical proposed approach on
generating psychometric items. First, we employed Gemini 1.5 Flash with zero-shot prompting to generate
20 initial questions measuring “propensity to trust AI”, which were then analysed using natural language
processing (NLP) methods. Second, using Claude 3 Opus, we generated and linguistically evaluated 20 items
targeting AI anxiety, inspired by Y.-Y. Wang & Wang (2022). Third, by intentionally leveraging Gemini 1.5
Flash’s tendency to generate imaginative or fabricated content from minimal prompts, we produced over 20
diverse items assessing AI trust, with a focus on adoption and perception. Expert validation refined this set
to six items, whose psychometric properties were subsequently confirmed using participant responses. The
experts involved in this process were academics specialising in educational technology, artificial intelligence,
online education, data analytics, and psychometrics. The LLM models used in the demonstrations were
released in January 2024. We then discuss how embedding prompts within a problem-solving plan can lead
to fruitful interactions with LLMs and, consequently, informative outputs. Finally, in the discussion section,
we elaborate on psychometrics as a science and the role of LLMs within it.

## 2. Demonstrations

### 2.1. Producing 20 usable items to measure "propensity to trust in AI"

We resorted to a CoT conversational style to query Gemini. We started with a zero-shot prompting
style: “Provide 10 up-to-date psychometric scales to measure people’s propensity to trust AI. Provide
real academic references for each.” Purposefully, we aimed to use Gemini’s hallucinations as a way to
obtain results. That is, instead of the usual approach of avoiding and discarding AI hallucinations, we
used its potential insights as a proving ground. As expected, Gemini provided some real and some
hallucinated references (anecdotally, however, typing these illusory references into Google provided
useful real references). We then continued with the question, “Based on the scales above, what’s the
overall definition of ‘propensity to trust AI’?” Gemini then provided an educated response and broke
down the concept of propensity to trust in AI in four dimensions: i) trust in AI competence, ii) trust in
AI benevolence, iii) trust in AI transparency, and iv) trust in AI fairness. Gemini also provided
definitions for each dimension and examples of how the propensity to trust AI might manifest itself
in real-world behaviours (e.g. Gemini said, “A person with a high propensity to trust AI might be more
likely to use a self-driving car”). Then we asked Gemini: “Provide psychometric items that will measure
‘propensity to trust in AI’ based on your definition above. Provide five items for Trust in AI
competence, 5 items for Trust in AI benevolence, five items for Trust in AI transparency, and 5
<figure><img src="image_19.png" /></figure>

items for Trust in AI fairness". After Gemini's output, we asked to "rewrite the above items so that they are to be answered on a continuous likelihood scale between "very unlikely" and "very likely" and after Gemini's output we finally asked to "rewrite the above items so that they are worded in the first person". Four of the items were:

* I am likely to trust that AI systems can be used to make my life better (item generated for the "trust in AI competence" dimension)

* I am likely to believe that AI systems are designed to act in the best interest of humans (item generated for the "trust in AI benevolence" dimension)

* I am likely to trust that AI systems will be used in a transparent and accountable manner (item generated for the "trust in AI transparency" dimension)

* I am likely to believe that AI systems treat all people fairly and equitably (item generated for the "trust in AI fairness" dimension)

Upon initial examination, the generated items appear to be relevant and align well with the four dimensions of the "propensity to trust AI" as defined by Gemini. More specifically, for the initial human evaluation, we selected several criteria: 1) the generated text must be grammatically coherent, 2) the generated text must be relevant to the given topic, and 3) a researcher unfamiliar with the study's design must be unable to discern whether the items originate from a human-designed test or are AI-generated content.

However, when the items are considered as a corpus, we can assess whether this corpus exhibits textual dimensionality using an NLP metric, such as cosine similarity (Singh & Singh, 2021). Cosine similarity evaluates the similarity between two non-zero vectors by computing the cosine of the angle between them. The values range from -1 to 1: 1 signifies that the vectors are identical (perfectly aligned), 0 indicates that the vectors are orthogonal (completely dissimilar), and -1 means the vectors are diametrically opposed. In the resulting matrix, most values exceed 0.75, showing that the four items are highly related and suggesting that the corpus cannot be divided into distinct dimensions.

However, it is important to note that the results may vary slightly from those reported here if the prompts are used again in the future. This is because LLMs have various hyper-parameters, such as temperature, Top P, frequency penalty, and presence penalty, that influence the randomness of their outputs. The first parameter is temperature, which controls the randomness of the model's output. A higher temperature setting yields more varied and unpredictable responses, whereas a lower temperature setting generates more deterministic and conservative outputs. The second parameter is Top P, also known as nucleus sampling, which determines the probability threshold for token selection. A higher Top P value allows for a wider range of tokens to be considered, resulting in more random output. In contrast, a lower Top P value restricts the selection to more likely tokens, reducing randomness. The third parameter is the frequency penalty, which decreases the likelihood of the model repeating the same words or phrases. A higher penalty setting discourages repetition and promotes the use of unique word choices. Finally, the presence penalty parameter encourages the introduction of new concepts or topics in the output. A higher presence penalty setting makes the model less likely to repeat the same topics, fostering more diverse content generation (see Ding et al., 2023; Zhang et al., 2024). The same argument applies to the results of the next demonstration.

To contextualise these findings, it is instructive to compare the generated items to existing validated scales. The field has several established measures, such as the 12-item Trust in Automation Scale (TIAS) (McGrath et al., 2025) and the 9-item Human-Computer Trust Scale (HCTS) (Gulati et al., 2019). The TIAS, for example, is a robust measure with high internal consistency (Cronbach's α = 0.94) that assesses trust along dimensions of performance and integrity. A key feature of the TIAS is its inclusion of five reverse-scored items (e.g. "The system is deceptive") to mitigate acquiescence bias. Our LLM-generated items for "competence" and "benevolence" align conceptually with the TIAS's "ability" and "integrity" dimensions (McGrath et al., 2025). However, a notable difference is that our initial zero-shot prompting did not produce any reverse-scored items. This comparison highlights a critical point: while LLMs can readily generate topically relevant items, achieving the psychometric sophistication of established scales like the TIAS requires more advanced prompting strategies, such as explicitly instructing the model to generate negatively-worded items to ensure a balanced scale.
<figure><img src="image_20.png" /></figure>

## 2.2. Producing 20 usable items to measure AI anxiety

Y.-Y. Wang and Wang (2022) proposed a 21-item scale to measure AI anxiety (AIA). These items represented four AIA dimensions: eight items related to learning, six related to job replacement, four related to sociotechnical blindness, and three related to AI configuration. In their study, they defined AIA as “as an overall, affective response of anxiety or fear that inhibits an individual from interacting with AI. Thus, AIA may be operationally considered as a general perception or belief with multiple dimensions. Moreover, AIA in the present research context focuses on the variable itself, rather than the process of response or model evaluation, which promotes the operation of AIA as a single variable, independent of numerous antecedents or consequences” (p. 621). Those items were generated in the standard way; researchers created a list of 59 questions to measure AIA and then reviewed the questions with experts to ensure they were clear and covered all aspects of AIA.

We tested our proposed approach by chatting with Claude to generate 20 items suitable for assessing the “learning” dimension of AIA. Table 1 shows the prompt used. Claude generated 20 usable items, and one last prompt required Claude to indicate how to rate the items, as such information was not explicit in its first output (“How are these items to be responded to?”). We list below three of the items generated:

* Trying to understand the advanced algorithms behind AI systems makes me anxious

* Adapting to rapid changes in AI technologies makes me anxious

* Learning new terms and concepts related to AI makes me anxious

It is clear that the generated items followed a linguistic structure similar to the eight items created by Y.-Y. Wang and Wang (2022), but most importantly, the Claude-generated items were semantically related to the keywords AI, anxiety, and learning. When comparing items generated by humans with those produced by Anthrop's LLM, Claude, our application of information theory reveals that the coefficient of mutual

<table><caption>**Table 1.** Prompt submitted to Claude in order to generate 20 items in the “learning dimension” of an AI anxiety scale.</caption><thead><tr><th>New items for the learning dimension of an AI anxiety scale</th></tr></thead><tbody><tr><td>Artificial Intelligence Anxiety (AIA) can be defined as a general, affective reaction of fear or anxiety that prevents an individual from interacting with artificial intelligence. The following eight psychometric items represent the learning dimension of AIA:</td></tr><tr><td>1. Learning to understand all of the special functions associated with AI technique/product makes me anxious.</td></tr><tr><td>2. Learning to use AI techniques/products makes me anxious.</td></tr><tr><td>3. Learning to use specific functions of an AI technique/product makes me anxious.</td></tr><tr><td>4. Learning how an AI technique/product works makes me anxious.</td></tr><tr><td>5. Learning to interact with an AI technique/product makes me anxious.</td></tr><tr><td>6. Taking a class about the development of AI techniques/products makes me anxious.</td></tr><tr><td>7. Reading an AI technique/product manual makes me anxious.</td></tr><tr><td>8. Being unable to keep up with the advances associated with AI techniques/products makes me anxious.</td></tr><tr><td><strong>Generate 20 items similar to the examples above, with the aim of assessing the "learning" dimension of AIA.</strong></td></tr><tr><td><strong>Items generated:</strong></td></tr><tr><td>1. Trying to understand the advanced algorithms behind AI systems makes me anxious.</td></tr><tr><td>2. Learning how to tweak the settings of an AI system to improve its performance makes me anxious.</td></tr><tr><td>3. Reading research papers about new AI techniques makes me anxious.</td></tr><tr><td>4. Attending seminars on AI development methods makes me anxious.</td></tr><tr><td>5. Learning about different AI programming languages and frameworks makes me anxious.</td></tr><tr><td>6. Figuring out how to integrate AI into existing systems and workflows makes me anxious.</td></tr><tr><td>7. Troubleshooting errors in AI systems makes me anxious.</td></tr><tr><td>8. Being asked to explain how an AI system works makes me anxious.</td></tr><tr><td>9. Enrolling in university courses on AI and machine learning makes me anxious.</td></tr><tr><td>10. Attempting to build my own basic AI prototype makes me anxious.</td></tr><tr><td>11. Reading books that explore the social implications of AI makes me anxious.</td></tr><tr><td>12. Learning about advances in AI that could displace human jobs makes me anxious.</td></tr><tr><td>13. Trying to understand an AI system's training data and models makes me anxious.</td></tr><tr><td>14. Learning new terms and concepts related to AI makes me anxious.</td></tr><tr><td>15. Adapting to rapid changes in AI technology makes me anxious.</td></tr><tr><td>16. Taking a professional development course on working alongside AI makes me anxious.</td></tr><tr><td>17. Learning how to audit AI systems for bias makes me anxious.</td></tr><tr><td>18. Figuring out how to integrate my workflow with an AI assistant makes me anxious.</td></tr><tr><td>19. Learning how to customise an AI system's behaviours and outputs makes me anxious.</td></tr><tr><td>20. Being unable to understand how an AI system makes its decisions makes me anxious.</td></tr></tbody></table>
<figure><img src="image_21.png" /></figure>

information (mi) for items generated by Claude was higher (mi = 1.339) than that of the items generated by humans (mi = 0.643). Claude tends to use fewer words than humans when generating items (visit https://cutt.ly/redm74ho for details).

The items generated by Claude 3 Opus can be directly contextualised by comparing them to the original 21-item Artificial Intelligence Anxiety Scale (AIAS) by Wang and Wang (2022), from which our prompt was derived. The original AIAS is a well-validated instrument with a confirmed four-factor structure (Learning, Job Replacement, Sociotechnical Blindness, and AI Configuration) and excellent reliability for its subscales (for example, Cronbach's α was found to be 0.964 for the whole scale and ranged from 0.917 to 0.974 for the four dimensions) (Y.-Y. Wang & Wang, 2022). Subsequent validation studies in different languages and populations have further confirmed its robust psychometric properties (Terzi, 2020). Our AI-generated items are linguistically coherent and semantically related to the “learning” dimension. However, this comparison underscores that such items represent the starting point, not the end point, of rigorous scale development. A full validation study would be required to determine if these new items replicate the established factor structure and high reliability of the original AIAS. This reinforces the principle that while our proposed approach demonstrates a powerful capacity for rapid item pool generation, it must be followed by empirical validation against established benchmarks.

## 2.3. Six items produced to measure adoption and perception of AI in online learning

The study by Marmolejo-Ramos et al. (in press) demonstrates the application of the PIG procedure, leading to the development of a scale that evaluates the inclination to adopt and perceive artificial intelligence. Employing the PIG model, 60 items were created to assess the adoption and perception of AI in online learning. The items were generated using a series of prompts. The first prompt asked for 10 current psychometric scales to measure people's perceptions and adoption of (generative) AI, along with real academic references for each. The items generated were then classified as relating to “adoption of AI” or “perception of AI”. Next, a 60-question survey was created based on the items, with 30 questions for “perceptions of AI” and 30 questions for “adoption of AI”. The questions were phrased using “I” and “me” and were designed to be answered on a rating scale from “strongly disagree” to “strongly agree”, with the scale going from 0 to 1. All questions were rephrased so that any large language model could understand them. These prompts were submitted to ChatGPT 3.5, Claude, and Gemini, and the options “regenerate responses”, “retry”, and “modify response” were used to refine the outputs and generate additional items. Finally, with the goal of creating a concise scale (Rammstedt & Beierlein, 2014) that can be easily administered online, three items were selected for “adoption of AI” and three for “perception of AI” through expert knowledge elicitation (see Table 2 for the selected items and Marmolejo-Ramos et al. (in press) for details).

Moreover, Table 3 presents the specific models and parameters used for each demonstration. It can be seen that the hyper-parameters are shown in default. This is because in traditional web-based chat LLMs, there is no way to adapt or modify the hyper-parameters that affect the generated output. These modifications are normally done via scripts and APIs (e.g. the PIG method). However, this also requires the interested parties to be programming-savvy and modify source code, which limits the accessibility of the PIG approach. Therefore, in our approach, the hyper-parameters are set and adapted by the LLM itself. Moreover, for reproducibility the exact prompts used to generate the items are provided in supplementary materials.

<table><caption>**Table 2.** Items kept for each scale after refining the outputs.</caption><thead><tr><td>Subscale</td><td>Item</td></tr></thead><tbody><tr><td>Perception of IA</td><td>"I am not concerned that AI could replace human teachers in online education."<br>"I think AI can improve the efficiency of online learning."<br>"I believe AI can improve the quality of online learning".</td></tr><tr><td>Adoption of IA</td><td>"I am open to using AI-powered chatbots for online course support."<br>"I am open to using AI-powered writing feedback tools in online courses."<br>"I am willing to allow AI to collect and analyse data about my online learning behaviour."</td></tr></tbody></table>
<figure><img src="image_22.png" /><figcaption>**Table 3.** LLM models and parameters used in demonstrations.</figcaption></figure>

<table><thead><tr><td>Parameter</td><td>Demonstration 1 (Trust in AI)</td><td>Demonstration 2 (AI Anxiety)</td><td>Demonstration 3 (Adoption/Perception)</td></tr></thead><tbody><tr><td>LLM Used</td><td>Google Gemini 1.5 Flash</td><td>Anthropic Claude 3 Opus</td><td>ChatGPT 3.5, Claude 2.1, Gemini Pro</td></tr><tr><td>Model Version</td><td>gemini-1.5-flash-001</td><td>claude-3-opus -20,240,229</td><td>gpt-3.5-turbo, claude-2.1, gemini-pro</td></tr><tr><td>Access Date</td><td>January 2024</td><td>January 2024</td><td>January 2024</td></tr><tr><td>Temperature</td><td>Default</td><td>Default</td><td>Default</td></tr><tr><td>Top P (Nucleus Sampling)</td><td>Default</td><td>Default</td><td>Default</td></tr><tr><td>Frequency Penalty</td><td>Default</td><td>Default</td><td>Default</td></tr><tr><td>Presence Penalty</td><td>Default</td><td>Default</td><td>Default</td></tr><tr><td>Interaction Method</td><td>Web UI</td><td>Web UI</td><td>Web UI with 'regenerate' option</td></tr></tbody></table>

### 2.3.1. Analysis of the generated items

Employing a multiverse analysis approach (Steegen et al., 2016) and a crowd-sourcing data analysis method (Silberzahn et al., 2018), six research groups, who are also co-authors of this paper, were tasked with using their preferred psychometric techniques to assess the quality of the items. This evaluation was based on responses from 1,233 participants who completed both scales in an online data collection conducted via Qualtrics software. The data collection was approved by the University of South Australia (approval number: 205473), and informed consent was obtained from all participants in accordance with the university's ethical standards. The findings are summarised in Table 4. In general, the results suggest that the two scales (perception and adoption of AI) demonstrated good internal consistency (Cronbach's $\alpha$ = 0.63 and 0.78, respectively) and distinctness from each other. Items were mostly similar in difficulty as estimated by the Rasch model of rating scales (Perception items 1, 2, and 3, item difficulty parameters = -0.078, -0.291, and -0.282, respectively; Acceptance items 1, 2, and 3, item difficulty parameters = -0.597, -0.592, and -0.274, respectively). Network analysis revealed strong connections between items, except for one in the perception sub-scale (see Figure 2). Additionally, by estimating the Hopkins statistic, which captures the clustering tendency of a dataset by measuring the probability that the data is generated by a uniform distribution, we found a value of 0.932 (between 0.7 and 1), indicating clustered data. This suggests that the data is likely to contain meaningful clusters.

While the initial findings demonstrate the potential of large language models like ChatGPT 3.5, Claude, and Gemini in generating reliable psychometric items, the process also highlighted the challenges of ensuring clarity and coherence in the outputs. This experience suggests that a more structured approach to interacting with LLMs could further enhance their utility in research settings. By implementing a problem-solving oriented plan and employing tailored prompts, researchers may guide these models more effectively, leading to more consistent and meaningful results. In the following section, we explore

<table><caption>**Table 4.** Qualitative summary of multiverse and crowdsourcing psychometric analyses for six items measuring AI adoption and perception in online learning. The dataset analysed in this study includes responses from 1,233 participants to the items of interest (364 participants from Nigeria, 361 from Finland, 258 from Iran, and 250 from Poland), as well as additional covariates of interest. Analyst groups and authors: g1: A. B.; g2: K. R. and J. K.; g3: L. M. and L. A.; g4: O. B.; g5: F. M-R. And R. O.; and g6: L. A. P-U. GAMLSS: generalised additive models for location, scale, and shape (see (Stasinopoulos et al., 2018)). The dataset and the quantitative analyses that support these qualitative summaries can be accessed at https://cutt.ly/pedEKzwt in R markdown and quarto markdown formats.</caption><thead><tr><td>Methods</td><td>Qualitative overall Results</td></tr></thead><tbody><tr><td>Rasch model as a GAMLSS | g5</td><td>Of the three items in each dimension, only one was estimated to be more difficult. However, the overall range of item difficulty was relatively homogeneous.</td></tr><tr><td>Rating scale model and Many facet Rasch model | g6</td><td>The analysis showed varying item difficulty within each dimension, with the most challenging items relating to concerns about AI replacing human teachers and allowing AI to collect data. Additionally, individuals more involved in online learning had a more open attitude towards AI in both perception and adoption.</td></tr><tr><td>Reliability analysis | g4</td><td>The results indicated that the Perception of AI and Adoption of AI were unidimensional constructs, measured by their respective scale items. There were also strong item-construct associations, with high factor loadings and explained variance.</td></tr><tr><td>Exploratory Factor Analysis | g1</td><td>The reliability of the AI adoption sub-scale was good, but the AI perception sub-scale had poor reliability unless one item was excluded.</td></tr><tr><td>Confirmatory Factor analysis | g3</td><td>There were strong correlations among perception items and among adoption items. However, the correlations between perception and adoption items were weaker, indicating that they were separate factors despite some relationship.</td></tr><tr><td>Network Analysis | g2</td><td>The network was densely connected, except for one item in the AI perception sub-scale. The strongest edges were observed between items 2 and 3 in the AI perception subscale, and between items 1 and 2 in the adoption sub-scale.</td></tr></tbody></table>
<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /><figcaption>**Figure 2.** Estimated Gaussian graphical model of the AI perception and adoption scale. Doughnut charts represent predictability, with a fully filled ring indicating that 100% of an item's variance is explained by its connections with other items. P1 = "I am not concerned that AI could replace human teachers in online education." P2 = "I think AI can improve the efficiency of online learning." P3 = "I believe AI can improve the quality of online learning". A1 = "I am open to using AI-powered chatbots for online course support." A2 = "I am open to using AI-powered writing feedback tools in online courses." A3 = "I am willing to allow AI to collect and analyse data about my online learning behaviour."</figcaption></figure>

how such strategies can optimise dialogues with LLMs, ultimately improving the quality of insights in the context of AI adoption and perception studies.

**3. Using problem-solving plans and tailored prompts to guide chains of thought**

We believe that structuring a logical Chain-of-Thought (CoT) and using tailored prompts (Ps) can greatly improve the clarity and coherence of the dialogue for psychometric item generation when aiming for productive interaction with an LLM. However, a first step is to create an organised, problem-solving-oriented plan (PSP) to provide top-down direction and scope of the psychometric scale before the start of a focused CoT (L. Wang et al., 2023). These prompt engineering strategies have evolved with the goal of better guiding the model in its search for the best possible response. While CoT initially suggested preparing more detailed prompts including examples of step-by-step reasoning (Wei et al., 2022), the Zero-shot strategy improved upon CoT by adding "*Let's think step by step*" (Kojima et al., 2023) before each answer, thus eliminating the need for examples. Finally, the PSP further enhanced the Zero-shot approach by including more detailed instructions on how the system should think step-by-step, adding: "*Let's first understand the problem and devise a plan to solve the problem. Then, let's carry out the plan and solve the problem step by step*" (L. Wang et al., 2023). This upfront framework guides the subsequent staged development of insights in a more linear, interconnected flow. In a psycholinguistic context, problem-solving involves three components: identifying goals, formulating strategies, and monitoring progress (Marmolejo-Ramos & Cevasco, 2014). In the case of a PS-oriented plan, we believe that these three components can be expanded to include components such as *goal setting* (i.e. clearly defining the purpose,
<figure><img src="image_25.png" /></figure>

goals and desired outcomes of the interaction. This provides direction towards a solution), *context and background* (i.e. providing any key context, related information or background that is relevant to achieving the objective. Serves as a common foundation), *scoping boundaries* (i.e. determining which aspects/parameters are in scope and which are out of scope to limit the flow of dialogue. Keeps responses on track), *topic flow* (i.e. outlines the logical sequence of sub-topics, arguments or prompts to be discussed in order to effectively reach the end goal. Defines coherent progression) and *success criteria* (i.e. defining what criteria will be used to determine whether the PSP leads to successful achievement of the original objectives with the LLM. Allows for iterative refinement). It is important to note that while a CoT represents an adaptive, evolving flow of logic, a PSP provides an established guideline for focused problem-solving. Also, CoTs link conceptual insights as they unfold, while PSPs provide a substantive foundation for tackling challenges. This logical precedence of PSP ahead of CoT processes can be denoted as: *PSP ⇒ CoT*. It can also be argued that every CoT is a subset of a broader PSP (i.e. *CoT* ⊂ *PSP*).

Furthermore, anchoring Ps to specifically target the salient aspects of a given CoT trajectory channels attention directly onto unpacking concepts most relevant to the pre-planned reasoning chain. Symbolically, Ps can be represented as a function dependent on the particular CoT (i.e. $P = f(CoT)$). Indeed, it can also be argued that each P is designed for a specific CoT; i.e. $P_1 \subset CoT_1$. Even as new Ps split from the main thread to explore complementary sub-topics, it can be argued that parallel prompts ($P1 \parallel P2$) still fall under the CoT's functional scope; i.e. If $P1, P2 \in P$ and $P1 \parallel P2$, then $P1, P2 \subset f(CoT)$. It can also be argued that parallel prompts share some overlap but remain distinct within the CoT; i.e. ($P_1 \cap P_2) \subset CoT$.

This illustrates how tailored prompting grounds and expands a cohesive chain-of-reasoning through a flexible prompting framework intrinsically tied to overarching objectives. Moving forward, further analysis of relations between Ps and CoTs promises additional techniques for purposeful thinking. The core takeaway is that strategic PSPs inform prompt-driven CoTs to simultaneously enable structured, staged reasoning with latitude for depth. We believe that deftly integrating top-level planning with locally optimised questioning opens new possibilities for optimal interaction with LLMs in the PIG context. Figure 3 illustrates this idea.

We also believe that LLMs can be subjected to the same set of Ps and their outputs compared vertically and horizontally, with assessment performed through unsupervised or supervised verification. Different LLMs can be given the same Ps, and their outputs can be compared within the same prompt (vertical comparison) or across different prompts (horizontal comparison). This approach

<figure><img src="image_26.png" /><figcaption>**Figure 3.** Relationship between prompts (*P*), chains-of-thought (*CoT*), and a problem-solving oriented plan (*PSP*). A *CoT* generates *Ps* until a satisfactory outcome is achieved. These *Ps* trigger parallel *Ps* (e.g. *P*'n*), ensuring coherence (such as generating semantically related questions. This iterative process can persist indefinitely). *Ps* may be different for another *CoT*, even if the new *CoT* (e.g. *CoT'*) is related to the original *CoT*. Preceding any *CoT*, a *PSP* guides its usage effectively. ≈ = some degree of similarity/equivalence between parallel/derivate prompts. *SD* = semantic distance (however, other metrics such as entropy can co-exist).</figcaption></figure>
<figure><img src="image_27.png" /></figure>

allows for a comprehensive evaluation of the LLMs' performance and helps identify the strengths and weaknesses of each model in generating coherent and relevant outputs. Providing a few CoT demonstrations as exemplars in prompting can significantly improve the performance of LLMs on various reasoning tasks (see Chu et al., 2023; Wei et al., 2022). This finding supports the idea that well-crafted prompts, as part of a CoT embedded in a PSP, can guide LLMs to generate higher-quality PIG outputs.

Unsupervised verification involves the LLMs themselves evaluating their outputs through mechanisms such as self-calibration. This approach aligns with the concept of self-consistency, as discussed in Wei et al. (2022), where LLMs generate multiple reasoning paths and select the most frequently occurring final answer. On the other hand, supervised verification involves human evaluators assessing the quality of the outputs generated by the LLMs. This approach is crucial for ensuring the factual accuracy, coherence, and overall quality of the generated outputs. Human evaluation can involve various tasks, such as reading and understanding the text, identifying errors, and judging the output's quality based on predefined criteria (e.g. those described in the first paragraph of section 3). The importance of human evaluation in assessing LLM outputs is further supported by arguments in relation to the role of the reader's neurocognitive systems and structures (NCSS) in the comprehension process Marmolejo-Ramos and Cevasco (2014). The authors argue that text comprehension involves the interaction between the reader's cognitive processes, background knowledge, and the text features. By analogy, human evaluators bring their own NCSS to the assessment of LLM outputs, allowing for a more comprehensive and nuanced evaluation of the generated text. Figure 4 represents the ideas above.

To proactively mitigate response biases, particularly acquiescence bias, the tendency for respondents to agree with items regardless of their content (Davis et al., 2020), the prompting strategy must be refined. LLMs, by default, may generate items with a uniform valence. Therefore, when generating the initial item pool, the PSP should include a specific goal to create a balanced scale. A tailored

<figure><img src="image_28.png" /><figcaption>**Figure 4.** Prompting in large language models (LLMs) embedded in chains-of-thought (CoT) and problem-solving oriented plans (PSPs). Different LLMs can be subjected to the same prompts (*P*). The outputs generated by these LLMs can be compared vertically (i.e. between LLMs within the same prompt) and/or horizontally (i.e. between LLMs across different prompts). The assessment of the outputs can be left to the LLMs themselves (unsupervised verification) or via human supervision (supervised verification).</figcaption></figure>
<figure><img src="image_29.png" /></figure>

prompt (P) within the CoT should explicitly instruct the LLM to generate items in both positive and negative formulations. For example, a follow-up prompt could be: "Now, please rewrite half of the items above so they are reverse-coded. For example, instead of 'I feel confident using AI,' the item could be 'I feel anxious when I have to use AI.'" This ensures the scale will contain items that tap both ends of the construct's continuum. The inclusion of reverse-coded items is a well-established technique for disrupting unreflective "yea-saying" response patterns and reducing method bias (Irfan & Tang, 2025).

A critical step in bridging the gap between AI-generated output and a psychometrically sound scale is the rigorous assessment of content validity. As noted, a set of items can be internally consistent yet fail to capture the full breadth of the intended construct if they are merely semantic paraphrases of one another. To prevent this and ensure the generated item pool adequately represents the construct domain, we propose integrating a formal content validation phase into the PIG workflow, immediately following item generation. This phase should follow the two-stage process described by Lynn (1986), Bertea and Zait (2013); Boateng et al. 2018):

(1) **Development Stage (Expert Panel Review):** A panel of Subject Matter Experts (SMEs) should be convened (Davis, 1992). These experts, who possess in-depth knowledge of the construct domain, are tasked with evaluating the initial pool of AI-generated items against a clear, pre-defined definition of the target construct and its facets (Dumas et al., 2025). Their role is to identify and discard items that are irrelevant, ambiguous, or redundant, ensuring the remaining items comprehensively cover the theoretical domain (Dumas et al., 2025). This qualitative review is the first line of defence against a narrow or repetitive item pool. An optimal way to handle this is through expert knowledge elicitation; a process that transforms expert insights into probability distributions for uncertain quantities. This approach effectively captures and formalises expert understanding in a way that explicitly represents uncertainty (O'Hagan, 2019).

(2) **Judgement-Quantification Stage (CVR Analysis):** To quantify expert agreement and provide an objective criterion for item retention, we recommend using Lawshe's Content Validity Ratio (CVR) (Lawshe, 1975). Each SME rates the necessity of each item for measuring the construct, typically on a 3-point scale: "essential," "useful, but not essential," or "not necessary". The CVR is then calculated for each item using the formula: $CVR = \frac{n_e - \frac{N}{2}}{\frac{N}{2}}$, where $n_e$ is the number of experts rating the item as "essential" and $N$ is the total number of experts. Items that fail to meet a statistically determined minimum CVR value based on the number of experts (e.g. a CVR of 0.78 is a suggested threshold for 5-7 experts) are eliminated (Romero Jeldres et al., 2023). This systematic, evidence-based process ensures that the final item pool is not only AI-generated but also expert-validated for content, a crucial step towards meaningful assessment.

# 4. Discussion

This study introduced a modified approach to automated psychometric item generation using LLMs and demonstrated its effectiveness through three key contributions. First, through demonstrations with Gemini and Claude, we showed that LLMs can generate psychometrically sound items for measuring constructs like "propensity to trust in AI" and "AI anxiety," with outputs that maintained semantic coherence and dimensional alignment. Our analyses revealed that these AI-generated items exhibited comparable or superior linguistic properties to human-generated ones, as evidenced by higher coefficients of mutual information and strong associations between semantic features. Second, our multiverse analysis of AI-generated items measuring adoption and perception of AI in online learning yielded promising psychometric properties, with results showing good internal consistency, item homogeneity, and clear factorial structure across multiple analytical approaches. Network analysis further revealed strong interconnections between items, though with some variation in the perception subscale. Third, we proposed that embedding prompts within a problem-solving-oriented plan provides a more structured and effective framework for interacting with LLMs, potentially leading to more consistent and reliable item generation. This systematic approach, which integrates chains of thought with carefully crafted prompts, offers a promising direction
<figure><img src="image_30.png" /></figure>

for advancing psychometric scale development. In what follows, we elaborate on selected aspects of this work and discuss its implications for the future of psychological measurement.

Together with other recent research (Götz et al., 2023), our findings indicate that LLMs can generate psychometric items with levels of validity and reliability comparable to those created by human experts. While validity and reliability represent the goal-state measures of scale effectiveness, the linguistic attributes of item generation processes represent a separate but equally vital dimension of research. We propose that, within a structured, problem-solving-oriented framework, LLMs can generate items with strong linguistic properties. This observation opens new avenues for research to explore how different CoTs and Ps impact the linguistic quality, validity, and reliability of generated outputs. Thus, an important direction for future work is to quantitatively assess the quality of these linguistic outputs by examining the relationships within and across CoTs and Ps. For example, to ensure the high quality and integrity of items, Ps within a single CoT should exhibit strong consistency in linguistic attributes, such as semantic coherence, while also displaying some semantic divergence from other sets of Ps. Similarly, variations across different CoTs should display close alignment, which can be measured by metrics like entropy. Quantitatively evaluating these linguistic relationships will provide insights into the coherence and stability of language used in scale items, enhancing the psychometric development process and ensuring high-quality measurement tools.

Another crucial aspect is how psychometrics could be affected in the upcoming years. The integration of LLMs with the Internet-of-Things (IoT) allows us to envision the emergence of interconnected systems with the ability to interact with humans, while acquiring a rich source of real-world modalities including motion, thermal, geolocation, imaging, depth, sensors, and audio to recognise the states of humans and the physical objects they interact with (Mo et al., 2024). From a technological viewpoint, the IoT-LLM integration will pave the way for psychometricians to realise and leverage other data collection techniques that do not require surveys, questionnaires, or standardised tests that are limited to provide cross-sectional descriptions of human behaviour. As IoT devices can get data in real time, our understanding of measuring human behaviour will be amplified to dynamical descriptions. A whole new universe unfolds when psychometricians adopt “digital twins” technologies. Roughly speaking, a digital twin is a virtual representation of a real-world entity, system, process, or other abstraction, which can be instanced by a computer program or encapsulated software model that interacts and synchronises with its physical counterpart (Wang et al., 2023). The adoption of these technologies calls for an interdisciplinary cooperation between psychologists, engineers, and software developers. One of the results of such cooperation will be a significant reduction of costs associated with behavioural data collection, though it opens new issues related to privacy, cybersecurity, and ethics, which will need to be addressed by policymakers and regulatory bodies.

A crucial question for the field is whether AI-generated items possess psychometric properties that are on par with, or even superior to, those developed by human experts. Our finding that Claude-generated items exhibited a higher mutual information coefficient than human-generated items suggests that AI may be capable of producing more linguistically efficient and less redundant items. This aligns with an emerging body of research that directly compares the two-generation methods. For example, recent studies have conducted head-to-head comparisons of scales for personality constructs, finding that AI-generated scales can achieve similar reliability (e.g. Cronbach's α) and criterion validity as expert-developed scales for constructs like resilience and conscientiousness (Oeljeklaus et al., 2025). Similarly, in the context of educational assessment, AI-generated multiple-choice questions have been found to have validity, reliability, difficulty levels, and discrimination indices that are nearly equal to those of teacher-made tests (Zeeshan et al., 2024). These findings suggest that, at a surface level, LLMs can produce psychometrically viable items. However, these studies also highlight important and subtle differences. Oeljeklaus et al. found that while their AI-generated scale demonstrated similar reliability and validity to the expert-developed version, it showed poorer fit to a latent measurement model and exhibited a lack of measurement invariance across genders (Oeljeklaus et al., 2025). This critical finding suggests that while LLMs can excel at generating face-valid content that correlates well with other measures, ensuring deep structural validity and fairness across demographic groups may require more sophisticated prompting and more rigorous post-hoc analysis than is currently standard practice. Therefore, while the evidence to date is promising and supports the utility of our PIG approach for generating an initial item pool, it also reinforces the conclusion that AI-generated items must be subjected to the same, if not more stringent, psychometric scrutiny as human-authored items.
<figure><img src="image_31.png" /></figure>

Integrating reasoning, ethics, and contextual awareness into LLM-generated psychometric items presents significant challenges (Etzioni & Etzioni, 2017). These LLMs, acting as agents, may encounter ethical dilemmas similar to the trolley problem (Jin et al., 2024; Takemoto, 2024) when designing items. Ensuring these items are free from ethical biases and promote “the common good” becomes complex, especially given the potential for biased or incomplete training data. This item generation process can be conceptualised as a multi-agent system where multiple LLMs, or internal models within a single LLM, interact. These agents aim to optimise a cost function that balances coherence, ethical considerations, and bias avoidance (Conitzer et al., 2017; Floyd et al., 2017; Sen et al., 2024). However, operating with incomplete and potentially inaccurate contextual information, these LLMs may struggle to reach a stable equilibrium, like a Nash equilibrium, even with ethical guidelines in place (Barron, 2024; Lorè & Heydari, 2024; Silva, 2024). Imagine an environment where various prompts and CoTs guide LLMs to generate technically sound, semantically coherent, and ethically robust items. A PSP in this context would encompass not only clarity and psychometric relevance but also robust ethical guidelines (Jobin et al., 2019; Pellert et al., 2024; Stahl, 2021). Yet, the LLM, in its pursuit of coherence and ethical compliance, may produce outputs that oscillate between different solutions without converging on a stable point. This dynamic interaction with ethical dilemmas can result in items highly sensitive to minor variations in prompts or training data. The lack of a stable equilibrium, such as a Nash equilibrium, highlights that achieving the ideal solution – a set of consistently appropriate, well-calibrated, and ethically sound items – may not be deterministic. Instead, the process likely requires ongoing human oversight and dynamic parameter adjustments (Bostrom & Yudkowsky, 2018). Furthermore, the prioritisation of specific ethical considerations, like promoting inclusion versus emphasising efficiency, will depend on the values embedded in the system and the scientific community (Geisslinger et al., 2021; Raper, 2024). Ultimately, viewing LLMs as agents generating psychometric items within an ethical framework necessitates careful consideration of contextual choice and solution stability. Robust prompt engineering, well-structured problem-solving plans, and vigilant human verification become crucial to ensure the final psychometric items are both technically valid and ethically sustainable.

In this article, we presented a modification of a methodology, the PIG model, for developing psycholo-
gical scales that can be readily aligned with the *Standardisation Of Behaviour Research* (SOBER) guidelines,
which address issues related to proliferation of psychological measures and non-standardised practices
(Elson et al., 2023). The proposed methodology offers a systematic approach to the construction of brief yet
robust scales, thereby reducing the probability of their becoming just another annual addition to the
plethora of existing scales, a concern highlighted in the guidelines (Elson et al., 2023). Efforts in this area
are essential, as the growing integration of AI across various domains indicates that AI-generated psycho-
metric items are set to shape the future of survey and self-report research in social and health sciences. These
tools offer significant benefits to researchers by streamlining and enhancing the scale development process.
As demonstrated in our findings, AI can rapidly generate items with strong linguistic properties, such as
semantic coherence and dimensional alignment, and robust psychometric qualities, including internal
consistency, item homogeneity, and factorial structure. With the ability to produce valid and reliable
items comparable to human-generated items, AI-based approaches reduce the time, effort, and specialised
expertise traditionally required for scale development. For the future of psychological measurement, this
advancement means greater accessibility to high-quality measurement tools, flexibility in adapting to new
constructs, and the potential for iterative improvements with minimal resource investment. By expanding
researchers' capabilities, AI-generated scales could accelerate progress in understanding human behaviour
and mental health, enhancing both the scope and precision of psychological insights.

**4.1. Limitations and ethical imperatives in AI-Driven psychometrics**

While our findings highlight the considerable potential of LLMs in psychometric item generation, their
application is not without significant limitations and ethical challenges that demand vigilant consideration.
One of the most prominent risks is LLM hallucination, where models generate plausible-sounding but
factually incorrect or nonsensical information (Freyer et al., 2024). In our own demonstrations, we
purposefully leveraged hallucinations to spark creative item generation, but we acknowledge this is a high-
risk strategy that requires expert verification. In practice, unverified, hallucinated content could lead to the
<figure><img src="image_32.png" /></figure>

creation of psychometrically invalid items that measure unintended constructs or are based on non-credible sources (Sallam, 2023).

A related challenge is prompt sensitivity, where minor variations in prompt phrasing can lead to significant differences in output, undermining the reliability and reproducibility of the item generation process (Razavi et al., 2025). This is compounded by the “black box” nature of many proprietary LLMs, which limits the transparency and explainability of their reasoning processes (Ajwani et al., 2024). Our proposed PSP-CoT framework aims to mitigate this by structuring the interaction, but it does not eliminate the underlying issue of stochasticity. Researchers must therefore avoid treating LLMs as deterministic “item factories” and recognise the need for critical human verification of all outputs.

The risk of acquiescence bias is another issue that warrants special attention throughout the scale development process. While including reverse-coded items is a crucial design-stage mitigation strategy (Primi et al., 2020; Weijters et al., 2013), it is also imperative to statistically evaluate for such response styles during the validation phase. Method effects can be modelled using techniques like confirmatory factor analysis (CFA) by specifying a latent “acquiescence” factor on which all items, regardless of their direction, have a positive loading (Oeljeklaus et al., 2025). This approach allows researchers to partial out the variance attributable to a general tendency to agree from the variance attributable to the substantive trait, leading to a more accurate and less biased assessment of the scale’s construct validity. Future applications of the PIG methodology should incorporate this statistical check as a standard part of the validation protocol.

Furthermore, the ethical landscape is complex and requires careful navigation. Algorithmic bias is a primary concern, as LLMs trained on vast internet datasets can inherit and amplify societal stereotypes related to gender, race, and culture (Kumar et al., 2025). An AI-generated scale could inadvertently contain biased language or scenarios, leading to unfair or discriminatory assessment outcomes. This necessitates a “human-in-the-loop” approach, where human experts rigorously audit all generated content for potential bias before it is used (Tan et al., 2025). Finally, issues of data privacy and accountability are paramount. Using public LLMs for generating items on sensitive psychological topics requires careful consideration of data handling policies to prevent breaches of confidentiality, as user inputs may be stored and used for model training (Farmer et al., 2025). Ultimately, accountability for the final psychometric product must remain with the human researchers. They are responsible for ensuring the instrument is valid, fair, and aligned with professional and ethical guidelines, such as those proposed by UNESCO and the APA (UNESCO, 2022; Hutnyan & Gottlieb, 2025). The use of AI should assist, not abdicate, this fundamental professional responsibility.

To move from acknowledging these ethical challenges to systematically managing them in practice, we propose that the following multi-stage ethical governance framework be integrated into the PIG workflow:

(1) **Data Provenance and Model Selection:** While researchers using this method do not train the models themselves, they do select them. Whenever possible, researchers should prioritise using LLMs from developers who are transparent about their data curation practices and bias mitigation efforts (Kale et al., 2023).

(2) **Prompt-Level Bias Mitigation:** The PSP should include an explicit ethical goal. Prompts must be carefully engineered to request neutral, inclusive, and stereotype-free language. For example, a prompt should include constraints such as: “*Generate items that are culturally sensitive and avoid stereotypes related to gender, race, age, or disability.*” This proactive step aims to reduce the generation of biased content from the outset.

(3) **Human-in-the-Loop Ethical Auditing:** This is the most critical step for accountability. After generation, a diverse panel of SMEs must audit the entire item pool not just for content validity, but specifically for subtle biases, stereotypes, and potentially harmful or exclusionary language (Amirianiani et al., 2024). This human oversight is the primary mechanism to counter the “black box” nature of LLMs and ensure ethical accountability rests with the researchers.

(4) **Post-Deployment Fairness Assessment:** For scales that are put into practice, researchers have an ethical obligation to monitor their performance. This includes conducting statistical analyses, such as differential item functioning (DIF) tests (Kamata & Vaughn, 2004), to ensure that items function
<figure><img src="image_33.png" /></figure>

equivalently across different demographic groups. This step is crucial for identifying and rectifying
any latent biases that were not caught during the initial review.

This systematic approach ensures that ethical considerations are not an afterthought but are woven into the fabric of the AI-assisted scale development process, aligning with international guidelines that call for fairness, transparency, and accountability in AI systems.

**5. Conclusion**

In this study, we introduced and demonstrated a novel, code-free version of the PIG method, which leverages the conversational power of modern LLMs to automate the development of psychological scales. Our central contribution is the presentation of a systematic framework, grounded in a PSP and CoT prompting, that enhances the accessibility, efficiency, and psychometric soundness of AI-assisted item generation. Our findings across three demonstrations show that this approach can produce items with strong linguistic and psychometric properties, comparable or even superior to human-generated counterparts, for constructs such as AI anxiety and trust in AI. The multiverse analysis of our 6-item scale for AI adoption and perception further validated the method, revealing good internal consistency and a clear factorial structure that was robust to different analytical approaches. However, we urge that this potential be viewed with cautious optimism. Our work also highlights significant limitations and ethical imperatives that must be addressed. The challenges of LLM hallucinations, prompt sensitivity, and inherent algorithmic bias necessitate that this technology be used as a powerful assistant to, not a replacement for, human expertise. Rigorous content validation by subject matter experts, proactive mitigation of response biases through careful item design, and vigilant ethical oversight are non-negotiable components of this process. Future research should focus on several key areas. First, a systematic investigation is needed to determine optimal prompting strategies for generating diverse item types, including those that are reverse-coded or that assess different levels of cognitive complexity. Second, the development of automated, AI-driven pipelines for validating items, such as using LLMs to flag potential biases or assess content validity against a defined semantic space, represents a promising frontier. Finally, longitudinal studies are required to assess the long-term stability and predictive validity of scales developed using these methods. By integrating the power of LLMs with the rigour of traditional psychometrics, we can significantly advance the science of psychological measurement, making it more efficient, innovative, and accessible to the broader research community.

**Disclosure statement**

No potential conflict of interest was reported by the author(s).

**Authors' contributions**

Conceptualisation: FM-R; Methodology: FM-R and JT; Formal analysis: FM-R, OB, LA, LM, AB, KR JC, LAPU, RO, and JT; Investigation: all authors; Resources: all authors; Data curation: FM-R, OB, LA, LM, AB, KR JC, LAPU, RO, and JT; Writing – Original Draft: FM-R and JT; Writing – Review Editing: all authors; Visualisation: FM-R, KR and JT; Supervision: FM-R; Project administration: FM-R.

**Funding**

R.O. gratefully acknowledges the partial financial support received from the Brazilian agencies Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) [Grants 303192/2022-4 and 402519/2023-0] and Fundação de Amparo á Ciência e Tecnologia do Estado da Bahia (FAPESB) [Grant APP0021/2023] for this research. J. K. and K. R. work was supported by Operational Program Jan Ámos Komenský project 'Biography of Fake News with a Touch of AI: Dangerous Phenomenon through the Prism of Modern Human Sciences' (reg. CZ.02.01.01/00/23_025/0008724).

**ORCID**

Fernando Marmolejo-Ramos http://orcid.org/0000-0003-4680-1287
<figure><img src="image_34.png" /></figure>

# Availability of data and materials

Available at https://cutt.ly/pedEKzwt and https://cutt.ly/redm74ho

# Code availability

Available at https://cutt.ly/pedEKzwt and https://cutt.ly/redm74ho

# References

Ahmed, T., & Devanbu, P. (2023). Better patching using LLM prompting, via self-consistency. In *2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)* (pp. 1742–1746). IEEE.

Ajwani, R., Javaji, S. R., Rudzicz, F., & Zhu, Z. (2024). Llm-generated black-box explanations can be adversarially helpful. *arXiv arXiv:2405.06800 [CS]* (https://doi.org/10.48550/arXiv.2405.06800)

Amirianiani, S., Zhou, Y., Suresh, H., Schwettmann, S., & Ghassemi, M. (2024). Llmauditor: A framework for auditing large language models. *arXiv Preprint arXiv*, 2402.09346.

Association, A. E. R. (2014). *Standards for educational and psychological testing*. American Educational Research Association.

Barron, E. N. (2024). *Game theory: An introduction*. John Wiley & Sons.

Bertea, E., & Zait, A. (2013). Scale validity in exploratory stages of research. *Management & Marketing Journal*, (1), 38–46.

Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. *Frontiers in Public Health*, 6, 149. https://doi.org/10.3389/fpubh.2018.00149.

Bostrom, N., & Yudkowsky, E. (2018). The ethics of artificial intelligence. In Roman V. Yampolskiy (Ed.), *Artificial intelligence safety and security* (pp. 57–69). Chapman and Hall/CRC.

Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., & Liu, T. (2023). A survey of chain of thought reasoning: Advances, frontiers and future. https://doi.org/10.48550/arXiv.2309.15402

Clark, L. A., & Watson, D. (2019). Constructing validity: New developments in creating objective measuring instruments. *Psychological Assessment*, 31(12), 1412–1427. https://doi.org/10.1037/pas0000626

Conitzer, V., Sinnott-Armstrong, W., Schaich Borg, J., Deng, Y., & Kramer, M. (2017). Moral decision making frameworks for artificial intelligence. *Proceedings of the AAAI Conference on Artificial Intelligence* (Vol. 31(1)). https://doi.org/10.1609/aaai.v31i1.11140

Davis, L. L. (1992). Instrument review: Getting the most from a panel of experts. *Applied Nursing Research*, 5(4), 194–197. https://doi.org/10.1016/S0897-1897(05)80008-4.

Davis, R. E., Lee, S., Johnson, T. P., Conrad, F., Resnicow, K., Thrasher, J. F., Mesa, A., & Peterson, K. E. (2020). The influence of item characteristics on acquiescence among Latino survey respondents. *Field Methods*, 32(1), 3–22. https://doi.org/10.1177/1525822x19873272.

Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., Yi, J., Zhao, W., Wang, X., Liu, Z., Zheng, H.-T., Chen, J., Liu, Y., Tang, J., Li, J., & Sun, M. (2023). Parameter-efficient fine-tuning of large-scale pre-trained language models. *Nature Machine Intelligence*, 5(3), 220–235. https://doi.org/10.1038/s42256-023-00626-4

Dumas, D., Greiff, S., & Wetzel, E. (2025). Ten guidelines for scoring psychological assessments using artificial intelligence. *European Journal of Psychological Assessment*, 41(3), 169–173. https://doi.org/10.1027/1015-5759/a000904

Elson, M., Hussey, I., Alsalti, T., & Arslan, R. (2023). Psychological measures aren’t toothbrushes. *Communications Psychology, 1*(25). https://doi.org/10.1038/s44271-023-00026-9

Etzioni, A., & Etzioni, O. (2017). Incorporating ethics into artificial intelligence. *The Journal of Ethics*, 21(4), 403–418. https://doi.org/10.1007/s10892-017-9252-2

Farmer, R. L., Lockwood, A. B., Goforth, A., & Thomas, C. (2025). Artificial intelligence in practice: Opportunities, challenges, and ethical considerations. *Professional Psychology, Research and Practice*, 56(1), 19–32. https://doi.org/10.1037/pro0000595

Floyd, M. W., Karneeb, J., & Aha, D. W. (2017, June 26–28). Case-based team recognition using learned opponent models. In David W. Aha, & Jean Lieber (Eds.), *Case-Based Reasoning Research and Development: 25th International Conference, ICCBR 2017* (Vol. 25., pp. 123–138). Springer.

Freyer, O., Wiest, I. C., Kather, J. N., & Gilbert, S. (2024). A future role for health applications of large language models depends on regulators enforcing safety standards. *Lancet Digital Health*, 6(9), 662–672. https://doi.org/10.1016/S2589-7500(24)00124-9

Geisslinger, M., Poszler, F., Betz, J., Lütge, C., & Lienkamp, M. (2021). Autonomous driving ethics: From trolley problem to ethics of risk. *Philosophy & Technology*, 34(4), 1033–1055. https://doi.org/10.1007/s13347-021-00449-4
<figure><img src="image_35.png" /></figure>

Giray, L. (2023). Prompt engineering with ChatGPT: A guide for academic writers. *Annals of Biomedical Engineering*, 51(12), 2629-2633. https://doi.org/10.1007/s10439-023-03272-4

Götz, F. M., Maertens, R., Loomba, S., & Linden, S. (2023). Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development. *Psychological Methods*, 29(3), 494-518. https://doi.org/10.1037/met0000540

Gulati, S., Sousa, S., & Lamas, D. (2019). Design, development and evaluation of a human-computer trust scale. *Behaviour & Information Technology*, 38(10), 1004-1015. https://doi.org/10.1080/0144929X.2019.1656779

Henrickson, L., & Meroño-Peñuela, A. (2025). Prompting meaning: A hermeneutic approach to optimising prompt engineering with ChatGPT. *AI & Society*, 40(2), 903-918. https://doi.org/10.1007/s00146-023-01752-8

Hutnyan, M., & Gottlieb, M. C. (2025). Artificial intelligence in psychological practice: Applications, ethical considerations, and recommendations. In *Professional psychology: Research and practice*. Advance online publication. https://doi.org/10.1037/pro0000631

Irfan, D., & Tang, X. (2025). Evaluating China's electric vehicle adoption with PESTLE: Stakeholder perspectives on sustainability and adoption barriers. *Sustainability*, 17(14), 6258. https://doi.org/10.3390/su17146258

Jebb, A. T., Ng, V., & Tay, L. (2021). A review of key Likert scale development advances: 1995-2019. *Frontiers in Psychology*, 12. https://doi.org/10.3389/fpsyg.2021.637547

Jin, Z., Kleiman-Weiner, M., Piatti, G., Levine, S., Liu, J., Gonzalez, F., Ortu, F., Strausz, A., Sachan, M., Mihalcea, R., Choi, Y., & SchÁłkopf, B. (2024). Language model alignment in multilingual trolley problems. https://arxiv.org/abs/2407.02273

Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. *Nature Machine Intelligence*, 1 (9), 389-399. https://doi.org/10.1038/s42256-019-0088-2

Kale, A., Nguyen, T., Harris, F. C., Li, C., Zhang, J., & Ma, X. (2023). Provenance documentation to enable explainable and trustworthy AI: A literature review. *Data Intelligence*, 5(1), 139-162. https://doi.org/10.1162/dint_a_00119

Kamata, A., & Vaughn, B. K. (2004). An introduction to differential item functioning analysis. *Learning Disabilities a Contemporary Journal*, 2(2), 49-69.

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2023). Large language models are zero-shot reasoners. ArXiv. arXiv:2205.11916 [CS] (https://doi.org/10.48550/arXiv.2205.11916)

Kumar, C. V., Urlana, A., Kanumolu, G., Garlapati, B. M., & Mishra, P. (2025). No LLM is free from bias: A comprehensive study of bias evaluation in large language models. *arXiv. arXiv:2503.11985 [cs]* (https://doi.org/10.48550/arXiv.2503.11985)

Lawshe, C. H. (1975). A quantitative approach to content validity. 1. *Personnel Psychology*, 28(4), 563-575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

Lorè, N., & Heydari, B. (2024). Strategic behavior of large language models and the role of game structure versus contextual framing. *Scientific Reports*, 14(1), 18490. https://doi.org/10.1038/s41598-024-69032-z

Lynn, M. R. (1986). Accessed 2025-07-15 Determination and quantification of content validity. *Nursing Research*, 35 (6), 382-385. https://doi.org/10.1097/00006199-198611000-00017

Marmolejo-Ramos, F., Abadia, R., Karakale, O., Barrera-Causil, C., Männikkö, N., Ghorbani, B., Strzelecki, A., Nwizu, S., Tavares, C., Castillo, M., Som, B., Ngo, G., Özdoğru, A., Ariyabuddhiphongs, K., & Tejada J. (in press). Perceptions and adoptions of artificial intelligence in online learning. *Discover Artificial Intelligence*.

Marmolejo-Ramos, F., & Cevasco, J. (2014). Text comprehension as a problem solving situation. *Universitas Psychologica*, 13(2), 725-743. https://doi.org/10.11144/Javeriana UPSY13-2.tcps

McGrath, M. J., Lack, O., Tisch, J., & Duenser, A. (2025). Measuring trust in artificial intelligence: Validation of an established scale and its short form. *Frontiers in Artificial Intelligence*, 8. https://doi.org/10.3389/frai.2025.1582880

Mo, S., Salakhutdinov, R., Morency, L.-P., & Liang, P. P. IoT-Lm: Large multisensory language models for the Internet of Things. *ArXiv Preprint ArXiv:2407.09801* (2024.) https://doi.org/10.48550/arXiv.2407.09801

Oeljeklaus, L., Höft, S., & Danner, D. (2025). Comparing psychometric properties of expert-developed and AI-generated personality scales. *Psychological Test Adaptation and Development*, 6, 29-43. https://doi.org/10.1027/2698-1866/a000095

O'Hagan, A. (2019). Expert knowledge elicitation: Subjective but scientific. The American Statistician. *American Statistician*, 73(sup1), 69-81. https://doi.org/10.1080/00031305.2018.1518265

Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024). AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. *Perspectives on Psychological Science*, 19(5), 808-826. https://doi.org/10.1177/17456916231214460

Primi, R., Hauck-Filho, N., Valentini, F., & Santos, D. (2020). Classical perspectives of controlling acquiescence with balanced scales. In M. Wiberg, D. Molenaar, J. GonzÁlez, U. Båckenholt & J.-S. Kim (Eds.), *Quantitative psychology* (pp. 333-345). Springer. https://doi.org/10.1007/978-3-030-43469-4_25

Rammstedt, B., & Beierlein, C. (2014). Can't we make it any shorter? *Journal of Individual Differences*, 35(4), 212-220. https://doi.org/10.1027/1614-0001/a000141

Raper, R. (2024). A survey of machine ethics. In *Raising robots to be good: A practical foray into the art and science of machine ethics* (pp. 19-33). Springer. https://link.springer.com/book/10.1007/978-3-031-75036-6

Razavi, A., Soltangheis, M., Arabzadeh, N., Salamat, S., Zihayat, M., & Bagheri, E. (2025). Benchmarking prompt sensitivity in large language models. In C. Hauff, C. Macdonald, D. Jannach, G. Kazai, F. M. Nardini, F. Pinelli,
<figure><img src="image_36.png" /></figure>

F. Silvestri & N. Tonellotto (Eds.), *Advances in information retrieval* (pp. 303–313). Springer. https://doi.org/10.1007/978-3-031-88714-7_29

Romero Jeldres, M., Díaz Costa, E., & Faouzi Nadim, T. (2023). A review of Lawshe's method for calculating content validity in the social sciences. *Frontiers in Education, 8*. https://doi.org/10.3389/feduc.2023.1271335.

Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. *Healthcare, 11*(6), 887. https://doi.org/10.3390/healthcare11060887

Sen, A., Mainali, M., Rauch, C. B., Addison, U., Floyd, M. W., Goel, P., Karneeb, J., Kulhanek, R., Larue, O., Ménager, D., Molineaux, M., Turner, J. T., & Weber, R. O. (2024). Counterfactual-based synthetic case generation. In *International Conference on Case-BasedReasoning* (pp. 388–403). Springer.

Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, S., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Rosa, A. D., Dam, L., Evans, M. H. Cervantes, I. F., . . . Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. *Advances in Methods and Practices in Psychological Science, 1*(3), 337–356. https://doi.org/10.1177/2515245917747646

Silva, A. (2024). Large language models playing mixed strategy Nash equilibrium games. arXiv preprint arXiv:2406.10574.

Singh, R., & Singh, S. (2021). Text similarity measures in news articles by vector space model using NLP. *Journal of the Institution of Engineers (India): Series B, 102*(2), 329–338. https://doi.org/10.1007/s40031-020-00501-5. Accessed 2025-04-03.

Stahl, B. C. (2021). Ethical issues of AI. In Springer (Ed.), *Artificial intelligence for a better future: An ecosystem perspective on the ethics of AI and emerging digital technologies* (pp. 35–53). https://link.springer.com/book/10.1007/978-3-030-69978-9

Stasinopoulos, M. D., Rigby, R. A., & Bastiani, F. D. (2018). GAMLSS: A distributional regression approach. *Statistical Modelling, 18*(3–4), 248–273. https://doi.org/10.1177/1471082X18759144

Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. *Perspectives on Psychological Science, 11*(5), 702–712. https://doi.org/10.1177/1745691616658637. PMID: 27694465.

Takemoto, K. (2024). The moral machine experiment on large language models. *Royal Society Open Science, 11*(2), 231393. https://doi.org/10.1098/rsos.231393

Tan, B., Armoush, N., Mazzullo, E., Bulut, O., & Gierl, M. (2025). A review of automatic item generation techniques leveraging large language models. *International Journal of Assessment Tools in Education, 12*(2), 317–340. https://doi.org/10.21449/ijate.1602294.

Terzi, R. (2020). An adaptation of artificial intelligence anxiety scale into Turkish: Reliability and validity study. *International Online Journal of Education and Teaching, 7*(4), 1501–1515. http://iojet.org/index.php/IOJET/article/view/103.

UNESCO. (2022). Unesco: Recommendation on the ethics of artificial intelligence. UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000380455. Accessed 2025-07-15

Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023). Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. *arXiv. arXiv:2305.04091 [CS]* (https://doi.org/10.48550/arXiv.2305.04091).

Wang, Y., Su, Z., Guo, S., Dai, M., Luan, T. H., & Liu, Y. (2023). A survey on digital twins: Architecture, enabling technologies, security and privacy, and future prospects. *IEEE Internet of Things Journal, 10*(17), 14965–14987. https://doi.org/10.1109/JIOT.2023.3263909

Wang, Y.-Y., & Wang, Y.-S. (2022). Development and validation of an artificial intelligence anxiety scale: An initial application in predicting motivated learning behavior. *Interactive Learning Environments, 30*(4), 619–634. https://doi.org/10.1080/10494820.2019.1674887

Wang, Y., Zhao, S., Wang, Z., Huang, H., Fan, M., Zhang, Y., Wang, Z., Wang, H., & Liu, T. (2024). Strategic chain-of-thought: Guiding accurate reasoning in LLMs through strategy elicitation. *arXiv. arXiv:2409.03271 [cs]* (https://doi.org/10.48550/arXiv.2409.03271)

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. https://arxiv.org/abs/2201.11903

Weijters, B., Baumgartner, H., & Schillewaert, N. (2013). Reversed item bias: An integrative model. Psychological Methods. *Psychological Methods, 18*(3), 320–334. https://doi.org/10.1037/a0032121.

Zeeshan, M., Iqbal, M., Shamim-Ur-Rasul, S., Sami, F., Malik, G. M., Imdad, D., & Sherazi, G. Z. (2024). A comparative analysis of psychometric properties in AI-generated and teacher-made MCQs test. *Kurdish Studies, 12*(4), 1808–1820. https://doi.org/10.53555/ks.v12i4.3653

Zhang, S., Bao, Y., & Huang, S. (2024). Improving large language models’ generation by entropy-based dynamic temperature sampling. https://arxiv.org/abs/2403.14541