# Conversation Progress Guide: UI System for Enhancing Self-Efficacy in Conversational AI

Daeun Jeong, Sungbok Shin, Jongwook Jeong

► To cite this version:

Daeun Jeong, Sungbok Shin, Jongwook Jeong. Conversation Progress Guide: UI System for Enhancing Self-Efficacy in Conversational AI. CHI 2025 - Conference on Human Factors in Computing Systems, ACM, Apr 2025, Yokohama, Japan. Article No.: 180, 11p., 10.1145/3706598.3714222. hal-04977135v2

HAL Id: hal-04977135

https://hal.science/hal-04977135v2

Submitted on 5 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI

Daeun Jeong

Jeonbuk National University

Jeonju, Republic of Korea

daeun3736@jbnu.ac.kr

Sungbok Shin*

Inria, Université Paris-Saclay

Saclay, France

sungbok.shin@inria.fr

Jongwook Jeong†

Jeonbuk National University

Jeonju, Republic of Korea

jwjeong55@jbnu.ac.kr

## ABSTRACT

In this study, we introduce the Conversation Progress Guide (CPG), a system designed for text-based conversational AI interactions that provides a visual interface to represent progress. Users often encounter failures when interacting with conversational AI, which can negatively affect their self-efficacy—an individual’s belief in their capabilities, reducing their willingness to engage with these services. The CPG offers visual feedback on task progress, providing users with mastery experiences, a key source of self-efficacy. To evaluate the system’s effectiveness, we conducted a user study assessing how the integration of the CPG influences user engagement and self-efficacy. Results demonstrate that users interacting with a conversational AI enhanced by the CPG showed significant improvements in self-efficacy measures compared to those using a conventional conversational AI.

## CCS CONCEPTS

* Human-centered computing → Natural language interfaces; Graphical user interfaces; Interaction design theory, concepts and paradigms.

## KEYWORDS

Self-efficacy, Progress Bar, Conversational AI, Conservation Interface, Human-AI Interaction

**ACM Reference Format:**

Daeun Jeong, Sungbok Shin, and Jongwook Jeong. 2025. Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI. In *CHI Conference on Human Factors in Computing Systems (CHI '25)*, April 26-May 1, 2025, Yokohama, Japan. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3706598.3714222

## 1 INTRODUCTION

Self-efficacy, or belief in one's abilities, is crucial for learning and work efficiency. If generative conversational AI lowers self-efficacy,

users may develop negative views and avoid these services. Systems like ChatGPT and Copilot have gained widespread use, but their inherent limitations lead to issues such as misunderstanding user intent, abrupt conversation endings, repetitive responses, and hallucinations, resulting in user dissatisfaction.

These issues are particularly important to address because in tasks that require multi-turn conversations, they can have negative impacts on users' self-efficacy beyond mere dissatisfaction. Doubts about information accuracy and transparency, as well as hallucinations, can impose excessive cognitive load on users performing tasks, potentially leading to a decrease in self-efficacy [8, 13]. Furthermore, when repetitive responses and failures to understand user intent occur, users experience failures that become major factors in lowering self-efficacy.

It is well known that self-efficacy greatly affects task performance and learning efficiency [6, 27], and interfaces designed to enhance self-efficacy already exist [1, 12, 30]. However, these are mostly limited to interfaces that improve specific learning situations using digital devices. While applications utilizing conversational agents to increase self-efficacy have been proposed, research aiming to promote self-efficacy in the context of using generative conversational AI has not yet been conducted. As the importance and application domains of generative conversational AI continue to expand, improving self-efficacy in this environment is a significant challenge that needs to be addressed.

In this paper, we present the Conversation Progress Guide (CPG), a visual interface that illustrates the progress of a conversation, contributing to the enhancement of users' self-efficacy. Our research goal is not to improve the quality or performance of existing conversational services but to achieve positive effects on self-efficacy through an interface designed to enhance self-efficacy without altering the underlying technology. Motivated by the effect of progress bars [22], we designed a progress interface for conversational AI as a method to help strengthen users' self-efficacy.

(1) **The CPG indicates the completion of the subtasks that comprise the user's task.** The problem-solving process using conversational AI often includes many interactions, and this process may prevent users from gaining successful experiences, potentially causing frustration. By presenting the completion of subtasks to the user, the system offers mastery experiences, the most powerful source of self-efficacy.

(2) **The CPG provides users with feedback on task progress by displaying a visual progress bar.** While conversational AI can present feedback or encouragement through dialogue, it does not provide fixed performance feedback visually. Performance feedback on current progress aids self-efficacy and encourages users to continue the task [2, 10]. Additionally, the progress bar can positively

<footnote>* The work was done while the author was affiliated with the University of Maryland, College Park.</footnote>

<footnote>† corresponding author.</footnote>

<footnote>Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.</footnote>

<footnote>CHI '25, April 26-May 1, 2025, Yokohama, Japan</footnote>

<footnote>© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.</footnote>

<footnote>ACM ISBN 979-8-4007-1394-1/25/04</footnote>

<footnote>https://doi.org/10.1145/3706598.3714222</footnote>
affect users' physiological and emotional states by presenting the task's progress status to reduce anxiety.

To verify the effectiveness of the CPG, we developed a prototype and conducted a user study. Participants were divided into a control group using the default conversational AI interface and an experimental group using the AI with the CPG applied. Both groups performed the same task. To evaluate the impact of the CPG on users, we measured self-efficacy before and after task performance, cognitive load, satisfaction, task time, and interaction count with the conversational AI.

Analysis revealed that both groups improved in self-efficacy, with significantly greater gains in the CPG group. This shows that CPG fosters self-efficacy, encouraging more proactive use of conversational AI in problem-solving. Furthermore, no significant differences in other measures were found, confirming the system does not increase cognitive load.

In summary, our work makes the following contributions. We have:

(1) introduced the CPG, a novel interface concept that illustrates task progress in conversational AI to enhance user self-efficacy.

(2) developed a structured design approach for the CPG, which evaluates the completion of subtasks and provides users with structured feedback to maintain engagement during interactions.

(3) conducted a user study demonstrating that the CPG enhances self-efficacy while maintaining cognitive load, task performance, and user satisfaction.

## 2 BACKGROUND AND RELATED WORK

We present the background about self-efficacy, interfaces that enhance self-efficacy, and failures on generative conversational AI.

**Self-Efficacy.** Self-efficacy refers to an individual's belief in their ability to accomplish specific goals [3, 4, 7]. It reflects confidence in one's ability to control motivation, behavior, and environmental factors. The level of self-efficacy influences the goals individuals set, the energy they invest in achieving those goals, and their behavioral performance. Low self-efficacy diminishes motivation and leads individuals to avoid challenges [7]. It also increases anxiety, reduces persistence, and ultimately results in decreased learning ability and performance [26, 33]. In his theory of self-efficacy, Bandura identifies four sources that build and enhance self-efficacy: mastery experiences, vicarious experiences, social persuasion, and physiological and emotional states. Mastery Experiences are direct personal achievements that most effectively enhance self-efficacy. Vicarious Experiences are the experiences gained by observing others' success, which help individuals believe they can succeed as well. Social Persuasion is the positive encouragement or feedback from others that promotes self-efficacy. Physiological and Emotional States are the physical and emotional conditions that influence confidence, with positive states enhancing self-efficacy.

**Interfaces that Enhance Self-Efficacy.** Park et al. developed a conversational agent-based application designed to enhance the self-efficacy of children with Attention Deficit Hyperactivity Disorder (ADHD) [29]. Low cognitive ability children with ADHD often have

fewer successful experiences and consequently lower self-efficacy compared to their peers. Through interactions with the conversational agent, children are able to plan, execute, and review their task performance. The application encourages consistent task completion, which was found to result in an improvement of self-efficacy among the participants.

Huang et al. proposed strategies to build the four sources of self-efficacy within an online learning environment [16]. Each of the four strategies demonstrated a positive effect on improving self-efficacy, and applying all four strategies together proved more effective. Among their strategies, Attributional Feedback entailed displaying effort-based feedback after each practice problem, with messages tailored to whether participants solved the problem correctly or incorrectly, reducing anxiety and improving self-efficacy.

Our approach aligns with the mechanisms used in these previous studies to strengthen self-efficacy. However, limited research has explored self-efficacy in the context of conversational AI. Given that conversational AI may negatively impact self-efficacy, further investigation is needed.

*Failures on Generative Conversational AI.* See and Manning [34] conducted an earlier case study on the neural generative model deployed in the social chatbot, Chirpy [28]. They investigated conversations between users and the bot, analyzing the relationship between errors and user dissatisfaction. Based on their findings, they defined taxonomies to systematically categorize these issues. Kim et al. [18] conducted a study to understand the dissatisfaction users experience during interactions with generative conversational AI. They reviewed 59 papers to collect failure cases and performed open coding on these instances, ultimately classifying user dissatisfaction factors arising from generative conversational AI responses and interactions into seven categories. Similarly, Lin et al. [20] extracted signals of satisfaction from user inputs when using generative conversational AI to identify patterns of satisfied and dissatisfied conversations, summarizing the related dissatisfaction patterns in the process. By synthesizing the categorized dissatisfaction factors from these three studies, we present them into ten categories (see Table 1).

These studies indicate that users experience a wide range of dissatisfaction when interacting with generative conversational AI. However, users rarely adopt strategies to address these issues, and even when they do, 72% of the dissatisfactions remain unresolved [18]. These dissatisfaction and failure experiences may result in a decrease in users' self-efficacy when using generative conversational AI. To address this issue, our approach focuses on utilizing the partial mastery experiences that users acquire. We aim to enhance users' self-efficacy by visually displaying the achievement of subtasks and associated progress during their interactions with the AI.

## 3 SOLUTION APPROACH

Below we describe our solution approach.

### 3.1 Design Requirements

We designed CPG based on three key design requirements. First, we prioritized on the efficient use of screen space. Accordingly, we
<table><caption>Table 1: Categories of Dissatisfaction in Conversational AI Responses</caption><thead><tr><td>Category</td><td>Description</td><td>References</td></tr></thead><tbody><tr><td>Intent Understanding</td><td>Failing to accurately reflect user intent or context</td><td>[18, 20]</td></tr><tr><td>Content Depth and Originality</td><td>Responses are overly generic or lack originality</td><td>[18, 20]</td></tr><tr><td>Information Accuracy</td><td>Providing incorrect or inconsistent information</td><td>[18, 20, 34]</td></tr><tr><td>Transparency</td><td>Lack of clarity on the basis or rationale for responses</td><td>[18]</td></tr><tr><td>Refusal to Answer</td><td>Avoiding or refusing to answer specific questions</td><td>[18, 34]</td></tr><tr><td>Content Ethics and Integrity</td><td>Including unethical or harmful content</td><td>[18, 34]</td></tr><tr><td>Response Format and Attitude</td><td>Inappropriate format, tone, or length of responses</td><td>[18, 20, 34]</td></tr><tr><td>Bot Repetitive</td><td>Repeating the same information excessively</td><td>[34]</td></tr><tr><td>Bot Hallucination</td><td>Generating non-existent or incorrect information</td><td>[34]</td></tr><tr><td>Sudden End</td><td>Abruptly ending the conversation without resolution</td><td>[20]</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>Figure 1: The system architecture of the CPG, highlighting the interaction between the Task Agent and the Progress Feedback Agent in providing feedback to users.</figcaption></figure>

allocated a significant portion of the interface showing the conversation history between the user and conversational AI, as it is the main part of our work. Second, we wanted users to easily understand the progress. We focused on the design's affordances to ensure users could easily understand the tool's utility. Third, we wanted the tool to support continuous progress monitoring and explicitly communicate achievements to reinforce users' self-efficacy. Providing clear feedback on subtask completion helps users build a sense of accomplishment, which is vital for maintaining motivation. Based on these considerations, we decided that a progress bar, in combination with markers is the desiderata to support our goal. Its compact format efficiently conveys information, while markers for subtask completion provide clear feedback [21, 23, 24].

## 3.2 Conversation Progress Guide Architecture

CPG requires two AI agents to operate, as shown in Figure 1. The first agent is the Task Agent, which functions identically to a conventional conversational AI; users interact with this agent to solve problems. The second agent is the Progress Feedback Agent, which monitors the task's progress and evaluates the user's advancement.

This agent analyzes the user's inputs and the Task Agent's responses during their interaction to evaluate whether the conversation aligns with the set goal and to determine if the subtasks required to achieve that goal have been completed.

When the agent evaluates that a subtask has been completed, it provides visual feedback by activating an additional step marker on the progress bar. This allows users to visually confirm the completion status of subtasks, their overall progress, and whether the task has been fully accomplished during the conversation.
## 3.3 User Interface

The user interface of the CPG comprises two core components: the Progress Bar and the Subtask Marker (see Figure 2). These elements collaboratively offer a visual representation of task progress and feedback on subtask completion.

<figure><img src="image_3.png" /><figcaption>Figure 2: User Interface of Conversation Progress Guide</figcaption></figure>

**Progress Bar.** The CPG offers two main visual elements: the Progress Bar and the Subtask Marker. The progress bar is an indicator that visually supports users in recognizing the progress of conversations in problem-solving. Initially, an empty progress bar is presented with all markers deactivated; only the marker symbolizing the final goal is displayed in a deactivated state at the end of the progress bar. This design helps users clearly recognize the goal they need to achieve.

Through interactions with the Task Agent, users incrementally accomplish predefined subtasks. When a subtask is completed, an activated marker is added to the progress bar in the predefined order, displaying only the markers for subtasks the user has achieved to highlight their efforts in the problem-solving process.

This design is based on the concepts of goal setting and motivation presented in social cognitive theory. Visual feedback indicating progress toward goal achievement can act as a factor in promoting self-efficacy and help users maintain motivation toward their goals [31]. Therefore, the progress bar in this study serves as an important tool that helps users clearly recognize their goals and reinforces the belief that they are making progress toward achieving them.

**Subtask Marker.** The Subtask Marker is an indicator that visually supports users in recognizing the accomplishment of subtasks. Tasks requiring multi-turn conversations have a high likelihood of involving multiple failures in the process of achieving the goal.

Mastery experience, one of the main sources of self-efficacy, has the most powerful influence on self-efficacy; negative experiences such as failures or mistakes can significantly diminish it [14]. We focused on enhancing self-efficacy by emphasizing the sense of accomplishment when even small tasks are successfully completed, rather than dwelling on the negative impacts of failures.

To this end, we subdivided the user's goal into subtasks and designed the progress bar into multiple stages. Additionally, instead of exposing all subtasks, we adopted a method of adding markers only for successful subtasks, allowing users to actively experience mastery. This design aims to encourage users to focus on successful experiences rather than on failed tasks.

Bandura [4] presents three elements that explain active mastery: first, the event must actually occur; second, the individual must directly perform the action and experience a sense of success; third, the event must contribute to achieving important immediate or long-term goals. Based on these elements, we designed the system to enhance self-efficacy by allowing users to incrementally experience how their small achievements contribute to accomplishing larger goals [32].

# 4 USER STUDY AND RESULTS

This section describes the user study conducted to evaluate the effectiveness of the CPG. We present the user study, and results, including a comparative analysis of the control and experimental groups.

## 4.1 User study

To evaluate the effectiveness of the proposed CPG, we developed a prototype and conducted a user study. Participants were recruited and randomly assigned into two groups: a control group using the default conversational AI interface, and an experimental group using the AI with the CPG applied. Both groups were instructed to perform the same task using the conversational AI, and we compared their processes and outcomes.

In total, 22 participants took part in the experiment, comprising 14 females and 8 males. Their average age was 22.05 years, with the youngest being 20 and the oldest 26.

We recruited participants who were undergraduate students at a university majoring in Computer Science, and were knowledgeable about information encryption tasks. These criteria aimed to recruit motivated individuals with an interest in solving the assigned tasks, thereby reflecting a real-world scenario in the experimental results. Additionally, this approach minimized biases stemming from variations in participants' AI familiarity and task expertise. Each participant had completed at least three semesters, ensuring a consistent baseline of academic experience. Notably, none of the participants had a clear understanding of the steps involved in RSA encryption. Each participant received approximately $9 as a reward.

Prior to the experiment, we provided a brief introduction to the system and gave simple instructions on how to use it. The task was printed on paper and handed to the users. All participants had prior experience using conversational AI; therefore, we did not provide a training session on its usage, ensuring basic familiarity with the conversational interface. Users were informed that they could discontinue the task at any time if they wished and were instructed
<table><caption>Table 2: RSA Encryption Task Self-Efficacy Survey Questions</caption><thead><tr><th>No.</th><th>Survey Question</th></tr></thead><tbody><tr><td>1</td><td>Do you think you fully understand the basic concepts of RSA encryption?</td></tr><tr><td>2</td><td>Are you confident that you can obtain the RSA private key using prime numbers p and q?</td></tr><tr><td>3</td><td>Do you think you can correctly calculate the public key using the private keys p and q?</td></tr><tr><td>4</td><td>Are you confident that you can accurately derive the private key d based on the calculated public key?</td></tr><tr><td>5</td><td>Do you think you can successfully encrypt the string "JBNU_CSAI" using RSA encryption?</td></tr><tr><td>6</td><td>Are you confident that you can clearly explain the entire RSA encryption process?</td></tr></tbody></table>

<table><caption>Table 3: Satisfaction Survey Questions.</caption><thead><tr><th>No.</th><th>Survey Question</th></tr></thead><tbody><tr><td>1</td><td>Were you overall satisfied while using this system?</td></tr><tr><td>2</td><td>How helpful do you think this system was in performing the task?</td></tr><tr><td>3</td><td>Would you like to use this system again?</td></tr><tr><td>4</td><td>Were you satisfied with the design of this system?</td></tr></tbody></table>

to explicitly indicate when they believed they had completed the task.

To assess the impact of the proposed system, we measured participants' self-efficacy before and after the task using a task-specific self-efficacy survey. The survey consisted of six questions (see Table 2). The first and last questions assessed the participants' belief in their understanding of the concepts necessary to solve the task and their ability to explain the problem-solving process. Questions 2 to 5 measured their confidence in completing the processes required to achieve the subtasks necessary for solving the problem. All items were designed to be answered on a 5-point Likert scale (1:very unconfident to 5:very confident).

This task-specific survey provides a precise evaluation of participants' confidence levels related to the RSA encryption task. By focusing on concrete steps within the task, it ensures content validity by directly relating survey items to the essential skills required. By administering this survey before and after the experiment, we observed changes in self-efficacy and analyze differences between the two groups.

To further examine the system's impact, we measured cognitive load after the task was completed using the NASA Task Load Index (TLX) questionnaire [15], a validated tool for assessing mental workload. Additionally, we surveyed user satisfaction for both groups after the task. The satisfaction survey included the following questions (see Table 3).

Furthermore, to compare the efficiency of the task completion process, we measured participants' task time and interaction count, defined as the number of times users interacted with the conversational AI by asking questions.

By collecting data on self-efficacy, cognitive load, user satisfaction, task time, and interaction count, we aimed to comprehensively

evaluate the impact of the CPG on participants' performances and experience.

## 4.2 Task and System Implementation

### <RSA Encryption Exercise>

Your task is to encrypt the given text using RSA encryption. Follow the steps below to obtain the ciphertext using RSA:

1. Find the private keys, *p* and *q*.

2. Using the private keys *p* and *q*, calculate the smallest possible public key.

3. Use the calculated public key to derive the private key *d*.

4. Using the derived private and public keys, encrypt the string "JBNU_CSAI" using the RSA encryption method.

<caption>Figure 3: The task instruction for user study.</caption>

**Task.** We designed the task as shown in Figure 3. Measuring self-efficacy through math problems is a common approach [16, 26, 27], and encryption tasks, being inherently mathematical and involving clearly defined linear subtasks, are an appropriate choice for our experimental task. The ultimate goal of this task is to generate an encrypted message using RSA encryption. Participants were provided with this final goal along with key steps associated with the subtasks required to achieve it: "Understanding RSA Encryption Method," "Multiplication of Primes," "Euler's Totient Function," "Public Key Generation," "Private Key Generation," and "Encryption Using RSA." For each subtasks, the Progress Feedback Agent evaluates the conversation and returns a completion value based on predefined evaluation rules. For example, the "Multiplication of Primes" step's rules are shown in Figure 4. Similar detailed
<figure><img src="image_4.png" /><figcaption>**Figure 4: Evaluation rules for the "Multiplication of Primes" subtask, as used by the Progress Feedback Agent to assess completion.**</figcaption></figure>

rules were established for each subtask to guide the Progress Feed-
back Agent in its evaluations, and the comprehensive rules for the
other subtasks are shown in Figure 9 (Appendix). Determining an
appropriate number of subtasks is crucial for effective progress
evaluation. After empirical testing, we found that 3 to 7 subtasks
generally provide a manageable range. The exact number varies by
display environment.

**System Implementation.** We implemented an web application based on React.js and Node.js, utilizing two GPT agents. For the Task Agent, we used GPT-3.5-turbo, as it has a higher likelihood of encountering errors during task execution compared to the latest LLM models. This choice was intentional to simulate realistic scenarios where users might face challenges, thereby allowing us to assess the impact of our system on self-efficacy under such conditions. For the Progress Feedback Agent, employed the GPT-4 model to more precisely evaluate the conversation topics.

When the user inputs a question into the prompt, the Task Agent generates a response. Additionally, the previous 10 exchanges (5 pairs of questions and answers) are stored in an array and provided to the Task Agent as part of the conversation history. This enables the agent to understand the context of the conversation and generate more accurate responses.

The generated question-answer pairs are then evaluated by the
Progress Feedback Agent. The system divides the conversation
goals into subtasks and stores them as objects, where each object
includes attributes such as step, label, and active status, which are
used to track the current progress of the conversation. The Progress
Feedback Agent evaluates whether the conversation matches the
label of each subtask, and returns the assigned value for the cor-
responding step if there is a match. The Progress Bar component
observes this step value and inserts the appropriate step object into
an array to visually display the user's progress through the tasks.

The Subtask Marker is implemented to appear on the Progress
Bar when its active status is set to “TRUE.” Markers are displayed in
a predefined order on the Progress Bar, regardless of the sequence
in which the user completes the subtasks. Uncompleted subtasks
are not displayed on the Progress Bar, and no additional hints
are provided. For example, if a user completes the first and third
subtasks in sequence, their corresponding markers will appear
consecutively on the Progress Bar. When the second subtask is
later completed, its marker is added between the first and third
markers. This approach is designed to clearly convey the predefined
problem-solving process to users, enhancing learning outcomes.

When the Progress Feedback Agent determines that the final
goal has been achieved, a modal window is presented to the user to
confirm whether the task is complete or if they wish to continue the
conversation. This approach provides users with more autonomy by
allowing additional opportunities to interact further if the Progress
Feedback Agent incorrectly judges the task as complete or if the
user does not feel that the task has been sufficiently completed.
When users proceed with the conversation, the system continuously
tracks progress and updates the status of completed subtasks. If the
user completes the task again, a modal window is displayed. This
allows users to refine their work and complete the task.

## 4.3 Results

**Self-Efficacy.** First, we performed a t-test to compare the pre-
task self-efficacy between the experimental group and the control
group. The results showed that for all six items in the Self-Efficacy
Survey, the p-values were above 0.05, indicating no statistically
significant differences between the two groups. This suggests that
the two randomly assigned participants groups had similar levels
of self-efficacy prior to performing the task.

<figure><img src="image_5.png" /><figcaption>**Figure 5: Average scores of Pre- and Post-Self-Efficacy Survey for the control group. Error bars represent standard deviations.**</figcaption></figure>

We then compared the pre- and post-task self-efficacy within
the control group. Figure 5 visualizes the scores of each item in the
Self-Efficacy Survey before and after the task. Initially, we observed that the mean scores increased across all items. Moreover, *t*-test results revealed significant differences in all items except Q3 at the 0.05 significance level, with *p*-values ranging from 0.00016 to 0.02465, excluding Q3. These results suggest that performing the task using the default conversational AI Interface enhances task-specific self-efficacy.

<figure><img src="image_6.png" /><figcaption>**Figure 6:** Average scores of Pre- and Post-Self-Efficacy Survey for the experimental group. Error bars represent standard deviations.</figcaption></figure>

Next, we compared the pre- and post-task self-efficacy within the experimental group. Figure 6 visualizes the scores of each item in the Self-Efficacy Survey before and after the task. Similar to the control group, we observed increases in the mean scores across all items. The *t*-test results revealed significant differences at the 0.05 significance level for all questions, with *p*-values ranging from 0.00202 to 0.02542. These results suggest that performing the task using the CPG also enhances task-specific self-efficacy.

Q3 showed a significant improvement only in the experimental group, while all other items demonstrated significant self-efficacy improvements in both groups. To further compare the degree of improvements of these improvements between the two groups, we calculated the effect size using Cohen's *d* for pre- and post-task self-efficacy scores in each group. The results are presented in Table 4. According to these results, Q1 (experimental: 1.128 vs. control: 1.977) and Q5 (experimental: 1.03 vs. control: 1.29) exhibited greater improvements in the control group, suggesting a larger magnitude of change among its participants. However, for Q2 to Q4, the experimental group demonstrated larger improvements in self-efficacy (Q2: experimental 1.512 vs. control 1.036; Q3: experimental 1.217 vs. control 0.672; Q4: experimental 1.351 vs. control 1.07). For Q6, the extent of improvements was similar between the two groups (experimental: 1.256, control: 1.303). These findings suggest that using the default conversational AI Interface also contributed to increases in self-efficacy. Participants in the control group also experienced improvements in self-efficacy through successful task completion and learning from problem-solving experiences. However, the use

of the CPG facilitated greater improvements in self-efficacy across more items, further demonstrating its effectiveness in supporting task-specific self-efficacy.

<figure><img src="image_7.png" /><figcaption>**Figure 7:** Average Cognitive Load scores for the experimental and control group. Error bars represent standard deviations.</figcaption></figure>

**Cognitive Load.** Participants measured their cognitive load after task completion using the NASA-TLX questionnaire, and the results are illustrated in Figure 7. For Q1-5, the average cognitive load of the experimental group was lower than that of the control group. However, the *t*-test results showed that the *p*-values for Q1-5 were all above 0.05, indicating no statistically significant difference in the cognitive load perceived by users between the two groups. Additionally, Q6 also showed no significant difference, with a *p*-value above 0.05. These findings suggest that performing the task using the CPG does not impact on user's cognitive load.

<figure><img src="image_8.png" /><figcaption>**Figure 8:** Average satisfaction scores for the experimental and control group. Error bars represent standard deviations.</figcaption></figure>
<table><caption>Table 4: Effect sizes (Cohen's d) for self-efficacy improvements in the experimental and control group</caption><thead><tr><th></th><th>Q1</th><th>Q2</th><th>Q3</th><th>Q4</th><th>Q5</th><th>Q6</th></tr></thead><tbody><tr><td>Experimental group</td><td>1.128</td><td>1.512</td><td>1.217</td><td>1.351</td><td>1.03</td><td>1.256</td></tr><tr><td>Control group</td><td>1.977</td><td>1.036</td><td>0.672</td><td>1.07</td><td>1.29</td><td>1.303</td></tr></tbody></table>

**Satisfaction.** Participants conducted the satisfaction survey, and the results are illustrated in Figure 8. For all questions, the difference in averages between the two groups was small, and the *t*-test results showed that the *p*-values were all above the significance level of 0.05, indicating no statistically significant differences between the groups. These findings suggest that performing the task using the CPG does not impact on user's satisfaction and experience.

**Task Time & Interaction Count.** The average task time observed in the experiment was 613 seconds for the control group and 649 seconds for the experimental group. The *t*-test results showed a *p*-value of 0.897, indicating no statistically significant difference between the two groups. In terms of interaction count, the control group and experimental group had averages of 11.5 and 8.82, respectively. A *t*-test resulted in a *p*-value of 0.365, also showing no statistically significant difference between the groups.

The analysis revealed no statistically significant differences between the two groups in terms of average task time or the number of interactions. This suggests that the CPG did not lead to measurable improvements in task performance. Regarding task time, it can be influenced by the response time of the GPT model used in the experiment, and due to the high variability of this response time, task time may be limited as an accurate measurement metric.

Although the difference in average task time was not statistically significant, the longest task time in the experimental group (2441 seconds) exceeded that in the control group (1458 seconds). This difference may explain the slightly longer average task time in the experimental group. Additionally, the reduced variance in task times suggests a more consistent user experience in the experimental group compared to the control group. In terms of interaction count, since the steps required to perform the task were roughly outlined and users followed them similarly, the number of utterances did not differ significantly between interfaces. Given the current experimental settings, measuring and comparing efficiency using task time and interaction count appears challenging, indicating a need for alternative metrics or measurement methods.

**User Feedback.** Below we delineate notable observations from our participants (See Table 5). Feedback about system responsiveness was reported by participants in both experimental and control groups (P2, P4), specifically noting delays in the Chat GPT API's response time, unrelated to our interface. P14 successfully obtained the encrypted value but failed to recognize it as the correct answer, repeatedly performing the encryption process. Unable to realize task completion, P14 abandoned the experiment. P7 completed a task in a single prompt by addressing all subtasks at once, triggering the CPG's task completion modal. This caused confusion, as reflected in his feedback.

# 5 DISCUSSION AND LIMITATION

Below we discuss the implication of results, discuss important points in detail, as well as limitations behind our work.

## 5.1 Implication of Results

In our study, we evaluated the impact of the CPG on self-efficacy and task performance. Participants in both the control and experimental group showed improvements in self-efficacy. However, the self-efficacy of participants in the experimental group, who used the CPG, improved significantly compared to the control group. This suggests that the proposed system is effective in enhancing users' self-efficacy. Furthermore, we observed promising indications of the benefits of CPG, such as an effective guidance tool.

Additionally, no significant differences were observed in task time, interaction count, cognitive load, or user satisfaction between the two groups. This indicates that the CPG does not impose additional cognitive burden or negatively affect task performance and user experience. Below we further discuss in detail, about the implications of our work.

## 5.2 Discussion

We present additional discussions about our work.

**Applicability.** While the CPG framework provides meaningful progress feedback for goal-oriented tasks that can be decomposed into linear subtasks, its applicability is limited in other situations. For instance, non-task-oriented dialogues that lack a specific goal are not well-suited to progress evaluations [17]. Similarly, when tasks cannot be broken down into well-defined subtasks (e.g., creative writing), the CPG can only offer final outcome feedback rather than incremental guidance. Moreover, although some flexibility is allowed in how subtasks are approached, the CPG remains unsuitable for open-ended or unstructured conversations.

Even with well-defined subtasks, the CPG is limited in supporting users who do not follow a linear, step-by-step approach, as observed in our user study (e.g., P7), where tasks were not completed in a structured manner as expected. Potential solutions include integrating automated prompt suggestions to better guide users through the subtasks [35] or providing a post-task summary of all completed steps to reinforce learning and comprehension [19]. We leave these issues as future work.

**Cognitive load.** Cognitive load impacts not only user experience and task performance but also self-efficacy [9]. Excessive cognitive load caused by additional interface elements can overwhelm users' cognitive resources. This can potentially lead to decreased self-efficacy, frustration, and reduced confidence in completing tasks [25, 36]. Conversely, insufficient cognitive load does not support improvements in self-efficacy. An optimal level of cognitive
<table><caption>Table 5: Participants' Feedback</caption><thead><tr><td>Group</td><td>Feedback</td></tr></thead><tbody><tr><td>Control</td><td>"Performance improvement is necessary." (P4)<br>"I repeatedly attempted without realizing that the correct answer had already been derived." (P10)</td></tr><tr><td>Experimental</td><td>"The response time seems to be slow." (P2)<br>"I noticed the window indicating task completion a few times, but I am not sure what triggered it." (P7)</td></tr></tbody></table>

load is crucial for maintaining user engagement and facilitating effective performance [37]. We plan to explore this issue.

*Negative feedback in progress evaluation.* Currently, the CPG evaluates progress based on subtask completion. Incorporating failures as part of progress could provide users with a more comprehensive view of their performance. Nonetheless, incorporating negative feedback requires careful consideration, as it may conflict with the provision of mastery experiences intended to enhance self-efficacy. This is especially crucial in cases of repeated failures, where negative feedback could diminish motivation. Conveying negative feedback positively, such as through humor, could mitigate these issues. For example, Dybala et al. [11] suggest that humor can effectively deliver negative feedback without reducing user engagement, which could be a valuable consideration for future enhancements of the CPG.

*Self-efficacy measurement.* Self-efficacy is inherently linked to beliefs in specific domains, and its accurate assessment requires the refinement of measurement scales. In this study, self-efficacy was evaluated for both the control and experimental groups using a consistent measurement procedure and survey questions. However, the evaluation employed was specifically developed for the tasks used in our experiments; as Bandura [5] suggests, task-specific self-efficacy scales should be refined to capture the nuances of different domains. For different tasks, the development of domain-specific measurement scales (or assessment instruments) may be necessary, and a thorough review and subsequent refinement of the current scale could further enhance its accuracy. Moreover, cultural and environmental factors may influence self-efficacy outcomes, and these should be carefully considered in relation to the specific tasks and characteristics of the participants [5].

## 5.3 Limitations and Future Work

There are limitations related to the generalizability of our findings due to the small and homogeneous sample used in this study. Future research should involve replicating the study with a larger and more diverse participant pool to increase the reliability of the results and explore the long-term effects on self-efficacy. Future work could involve generating evaluation rules or integrating machine learning techniques to enhance accuracy, thereby increasing the system's robustness and effectiveness across diverse contexts.

In addition, integrating automated subtask decomposition and evaluation can enhance progress tracking, thereby enabling the system to manage complex tasks more effectively. Furthermore, future work also may explore extending the CPG to accommodate

less structured scenarios, such as entertainment-driven chatbot, where task goals are harder to define.

## 6 CONCLUSION

In this study, we introduce the CPG, a system designed to provide a visual interface for representing progress in conversations with conversational AI. The proposed interface tracks the user's goal and related subtasks, offering visual feedback on progress as each subtask is completed. This enables users to receive feedback on their progress and offers mastery experiences by highlighting the completion of each subtask.

Results from the user study demonstrate that the CPG significantly improved self-efficacy compared to the default conversational AI interface. While additional measures such as task time, interaction count, cognitive load, and satisfaction are also evaluated, no statistically significant differences are observed. This suggests that the proposed interface did not negatively impact task efficiency or user satisfaction. In conclusion, the CPG is more effective at improving self-efficacy than the default conversational AI interface. This system is particularly beneficial for tasks involving multi-turn conversations, where repeated failures are likely, as it provides users with a better experience by enhancing self-efficacy.

## REFERENCES

[1] Iman Al Hashmi, Hilal Alsabti, Omar Al Omari, Yusra Al Nasseri, and Atika Khafaf. 2022. Development, feasibility and acceptability of a self-efficacy-enhancing smartphone application among pregnant women with gestational diabetes mellitus: single-arm pilot clinical trial. BMC Pregnancy and Childbirth 22, 1 (2022), 358.

[2] Roger Azevedo and Jennifer G Cromley. 2004. Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of educational psychology 96, 3 (2004), 523.

[3] Albert Bandura. 1978. Self-efficacy: Toward a unifying theory of behavioral change. Advances in behaviour research and therapy 1, 4 (1978), 139-161.

[4] Albert Bandura et al. 1986. Social foundations of thought and action. Englewood Cliffs, NJ 1986, 23-28 (1986), 2.

[5] Albert Bandura et al. 2006. Guide for constructing self-efficacy scales. Self-efficacy beliefs of adolescents 5, 1 (2006), 307-337.

[6] Albert Bandura, Linda Reese, and Nancy E Adams. 1982. Microanalysis of action and fear arousal as a function of differential levels of perceived self-efficacy. Journal of personality and social psychology 43, 1 (1982), 5.

[7] Albert Bandura and Sebastian Wessels. 1997. Self-efficacy. Cambridge University Press Cambridge.

[8] Roy F Baumeister, Ellen Bratslavsky, Mark Muraven, and Dianne M Tice. 2018. Ego depletion: Is the active self a limited resource? In Self-regulation and self-control. Routledge, 16-44.

[9] John M Carroll. 1997. Human-computer interaction: Psychology as a science of design. International journal of human-computer studies 46, 4 (1997), 501-522.

[10] Edward L Deci and Richard M Ryan. 2000. The "what" and "why" of goal pursuits: Human needs and the self-determination of behavior. Psychological inquiry 11, 4 (2000), 227-268.
[11] Pawel Dybala, Michal Ptaszynski, Rafal Rzepka, and Kenji Araki. 2009. Activating Humans with Humor–A Dialogue System That Users Want to Interact with. *IEICE TRANSACTIONS on Information and Systems* 92, 12 (2009), 2394–2401.

[12] Morgan C Evans, Adela Kapuscinska, Maya Greenholt, Junchao Lin, Xuanyuan Liu, Tianyi Zhang, Jessica Hammer, and Geoff Kaufman. 2021. Designing a self-efficacy game for health literacy in marginalized communities. In *Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems*. 1–6.

[13] David F Feldon, Rebecca Brockbank, and Kaylee Litson. 2023. Direct effects of cognitive load on self-efficacy during instruction. *Journal of Educational Psychology* (2023).

[14] Jessica Gale, Meltem Alemdar, Christopher Cappelli, and David Morris. 2021. A mixed methods study of self-efficacy, the sources of self-efficacy, and teaching experience. In *Frontiers in Education*, Vol. 6. Frontiers Media SA, 750599.

[15] Sandra G Hart. 1986. NASA task load index (TLX). (1986).

[16] Xiaoxia Huang, Richard E Mayer, and Ellen L Usher. 2020. Better together: Effects of four self-efficacy-building strategies on online statistical learning. *Contemporary Educational Psychology* 63 (2020), 101924.

[17] Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents/chatbots classification and design techniques. In *Web, Artificial Intelligence and Network Applications: Proceedings of the Workshops of the 33rd International Conference on Advanced Information Networking and Applications (WAINA-2019)*. 33. Springer, 946–956.

[18] Yoonsu Kim, Jueon Lee, Seoyoung Kim, Jaehyuk Park, and Juho Kim. 2024. Understanding users’ dissatisfaction with chatgpt responses: Types, resolving tactics, and the effect of knowledge level. In *Proceedings of the 29th International Conference on Intelligent User Interfaces*. 385–404.

[19] Claudia Leopold, Elke Sumfleth, and Detlev Leutner. 2013. Learning with summaries: Effects of representation mode and type of learning activity on comprehension and transfer. *Learning and Instruction* 27 (2013), 40–49.

[20] Ying-Chun Lin, Jennifer Neville, Jack W Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, et al. 2024. Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models. *arXiv preprint arXiv:2403.12388* (2024).

[21] Samiha Marwan, Preya Shabrina, Alex Milliken, Ian Menezes, Veronica Catete, Thomas W Price, and Tiffany Barnes. 2021. Promoting students’ progress-monitoring behavior during block-based programming. In *Proceedings of the 21st Koli Calling International Conference on Computing Education Research*. 1–10.

[22] Richard E Mayer. 2002. Multimedia learning. In *Psychology of learning and motivation*. Vol. 41. Elsevier, 85–139.

[23] Brad A Myers. 1983. Incense: A system for displaying data structures. *ACM SIGGRAPH Computer Graphics* 17, 3 (1983), 115–125.

[24] Brad A Myers. 1985. The importance of percent-done progress indicators for computer-human interfaces. *ACM SIGCHI Bulletin* 16, 4 (1985), 11–17.

[25] Fred Paas, Alexander Renkl, and John Sweller. 2003. Cognitive load theory and instructional design: Recent developments. *Educational psychologist* 38, 1 (2003), 1–4.

[26] Frank Pajares. 1996. Self-efficacy beliefs in academic settings. *Review of educational research* 66, 4 (1996), 543–578.

[27] Frank Pajares and M David Miller. 1994. Role of self-efficacy and self-concept beliefs in mathematical problem solving: A path analysis. *Journal of educational psychology* 86, 2 (1994), 193.

[28] Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyen Minh Phu, Dilara Soylu, and Christopher D Manning. 2020. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. *arXiv preprint arXiv:2008.12348* (2020).

[29] Doehn Park, Myounglee Choo, Bohyun Jin, Un Sun Chung, Jinwoo Kim, Junghan Lee, and Yee-Jin Shin. 2023. Utilizing a Conversational Agent to Promote Self-efficacy in Children: A Pilot Study on Low Cognitive Ability Children with Attention Deficit Hyperactivity Disorder. In *Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems*. 1–7.

[30] André Rodrigues, André RB Santos, Kyle Montague, and Tiago Guerreiro. 2021. Promoting Self-Efficacy Through an Effective Human-Powered Nonvisual Smartphone Task Assistant. *Proceedings of the ACM on Human-Computer Interaction* 5, CSCW1 (2021), 1–19.

[31] Dale H Schunk and Maria K DiBenedetto. 2021. Self-efficacy and human motivation. In *Advances in motivation science*. Vol. 8. Elsevier, 153–179.

[32] Ralf Schwarzer. 2014. *Self-efficacy: Thought control of action*. Taylor & Francis.

[33] Ralf Schwarzer, Matthias Jerusalem, J Weinman, Stephen Wright, M Johnston, et al. 1995. Measures in health psychology: A user's portfolio. Causal and control beliefs. *Causal and control beliefs* 1, 011 (1995), 35–37.

[34] Abigail See and Christopher D Manning. 2021. Understanding and predicting user dissatisfaction in a neural generative chatbot. In *Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue*. 1–12.

[35] Guinan Su, Yanwu Yang, and Jie Guo. 2023. Prompt Your Mind: Refine Personalized Text Prompts within Your Mind. *arXiv preprint arXiv:2311.05114* (2023).

[36] John Sweller. 1988. Cognitive load during problem solving: Effects on learning. *Cognitive science* 12, 2 (1988), 257–285.

[37] Jeroen JG Van Merrienboer and John Sweller. 2005. Cognitive load theory and complex learning: Recent developments and future directions. *Educational psychology review* 17 (2005), 147–177.
A APPENDIX

a { role: "system", content: `1. If the question and answer are inappropriate or unrelated to RSA encryption, return only the word 'false'.`,

b { role: "system", content: `2. Return 1 if the following conditions are met:
1) If the user asks a question related to the steps of RSA encryption.
2) If GPT's response includes content related to the steps of RSA encryption.
If these conditions are not met, return only the word 'false'.`,

c { role: "system", content: `3. Return 2 if any of the following conditions are met:
1) If the user requests arbitrary selection of p and q, and the GPT provides two prime numbers p, q, and their product.
2) If the user requests the multiplication of primes, and the GPT provides an appropriate answer by calculating that value.
3) If the user requests the value of N, and the GPT provides an appropriate answer by calculating the product of the two primes.
If none of these conditions are met, return only the word 'false'.`,

d { role: "system", content: `4. Return 3 if the following conditions are met:
1) If the user requests a calculation related to the Euler's totient function, and GPT's response is deemed appropriate.
2) If the user requests the calculation of φ(N), and GPT's response is deemed appropriate.
3) If the user requests a calculation related to the Euler function, and GPT's response is deemed appropriate.
4) If GPT provides the result of the φ(N) calculation.
If none of these conditions are met, return only the word 'false'.`,

e { role: "system", content: `5. Return 4 if the following conditions are met:
1) The user's question must include content related to the public key.
2) GPT's response must include at least one of the following: greatest common divisor, GCD, coprime, or e.
If these conditions are not met, return only the word 'false'.`,

f { role: "system", content: `6. Return 5 if the following conditions are met:
1) The user's question must include content related to the private key.
2) The content must include at least one of the following: the Euclidean algorithm or mod.
If these conditions are not met, return only the word 'false'.`,

g { role: "system", content: `7. Return 6 if the following conditions are met:
1) GPT's response must include the encrypted result of the text "JBNU_CSAI".
2) GPT's response must include the actual numeric result of the encrypted text.
3) The user's question must include content related to the public key and private key.
4) The user's question must include a request to encrypt a specific text using RSA.
If these conditions are not met, return only the word 'false'.`,

<caption>Figure 9: Evaluation rules used by the Progress Feedback Agent to assess task completion and trigger progress markers. These rules were specifically developed for this experiment and include (a) a fundamental rule applicable to all subtasks for relevance checking, as well as subtask-specific rules for: (b) Understanding RSA Encryption Method, (c) Multiplication of Primes, (d) Euler's Totient Function, (e) Public Key Generation, (f) Private Key Generation, and (g) Encryption Using RSA.</caption>