Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents

## Abstract

Abstract This paper presents a novel application of large language models (LLMs) to enhance user comprehension of privacy policies through an interactive dialogue agent. We demonstrate that LLMs significantly outperform traditional models in tasks like Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering, setting new benchmarks in privacy policy analysis. Building on these findings, we introduce an innovative LLM-based agent that functions as an expert system for processing website privacy policies, guiding users through complex legal language without requiring them to pose specific questions. A user study with 100 participants showed that users assisted by the agent had higher comprehension levels (mean score of 2.6 out of 3 vs. 1.8 in the control group), reduced cognitive load (task difficulty ratings of 3.2 out of 10 vs. 7.8), increased confidence in managing privacy, and completed tasks in less time (5.5 minutes vs. 15.8 minutes). This work highlights the potential of LLM-based agents to transform user interaction with privacy policies, leading to more informed consent and empowering users in the digital services landscape.

## 1 Introduction

The pervasive collection and processing of personal data by online services have elevated privacy concerns in digital interactions (Vicario et al., ). To address these issues, websites and applications are legally mandated to publish privacy policies detailing practices related to data collection, usage, sharing, and protection. However, these policies are *notoriously difficult* for the average user to comprehend due to their *legal complexity* (McDonald & Cranor, ), *dense language, and considerable length*. Consequently, they are frequently ignored or misunderstood, undermining the principle of informed consent and exposing users to potential privacy risks. This disconnect poses significant challenges (Harkous et al., ): it impedes users’ ability to make informed decisions about their data and complicates efforts by regulators and organizations to ensure transparency, enforce compliance, and build trust. As privacy regulations become more stringent and policies more intricate, innovative approaches are urgently needed to bridge this comprehension gap.

Recent advances in natural language processing (NLP), particularly through the development of large language models (LLMs), offer promising solutions to these challenges. LLMs, such as those based on GPT architectures, have demonstrated remarkable capabilities in understanding and generating human-like text across various domains, including legal documents. These models have been successfully applied to tasks such as information extraction, content summarization, and question answering, showing potential in automating the analysis and interpretation of complex legal texts like privacy policies.

In this paper, we present a novel application of LLMs to improve user interaction with website privacy policies by developing an AI agent based on GPT-4o-mini. Our research aims to *empower users by improving their ability to navigate and comprehend privacy agreements, thus fostering better control over their personal data*. The study is structured in two parts.

Figure: Figure 1: Overview of the workflow for benchmarking large language models on usable privacy policy tasks and designing a guided agent to assist consumers.
Refer to caption: extracted/5927103/paper_overview2.jpg

In the first part, we evaluate the performance of LLMs across several core tasks relevant to privacy policies, including Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering (Wilson et al., ; Mysore Sathyendra et al., ; Ahmad et al., ; Keymanesh et al., ). By systematically replacing traditional models with LLMs, we establish new benchmarks and demonstrate that *LLMs achieve state-of-the-art performance* across these tasks, significantly improving the *accuracy and efficiency* of natural language understanding, particularly in the analysis and interpretation of privacy policies.

Building on these empirical results, the second part introduces an innovative LLM-based agent built upon GPT-4o-mini to assist users in comprehending website privacy policies. The agent employs a heuristic interaction model that simplifies complex legal language without requiring users to know what specific questions to ask. As an expert system, it processes privacy agreements swiftly, executes multiple tasks such as identifying key information, and adapts to user needs by highlighting critical policy components autonomously (Ravichander et al., ).

To evaluate the effectiveness of our LLM-based agent, we conducted a user study involving 100 participants, divided into two groups: one interacted directly with privacy policies, and the other used the LLM agent for assistance (Kelley et al., ). We measured comprehension, efficiency, cognitive load, and user confidence. The results indicate that users who utilized the agent had a significantly better understanding of the privacy policy, reported reduced cognitive load, and completed tasks in less time compared to the control group. These findings demonstrate that our AI agent not only improves user comprehension but also enhances efficiency and boosts user confidence (Kelley et al., ).

The key contributions of this work are threefold:

- •
We provide empirical evidence of the *superiority* of LLMs over traditional models in interpreting privacy policies, setting new benchmarks.
- •
We develop an AI agent that *reduces cognitive load*, *increases user confidence*, and *enhances comprehension* of privacy policies. This agent simplifies complex legal language, assists users without requiring *prior knowledge* or *specific queries*, and autonomously highlights critical policy components to improve decision-making.
- •
We demonstrate that our LLM-based agent significantly improves users’ ability to understand privacy policies, reducing task completion time and cognitive effort. This work not only *empowers users to better assert their privacy rights* and make more informed decisions about their data, but also provides valuable insights and a foundation for future research in developing AI-driven tools for legal document comprehension, encouraging further advancements in user-centered privacy solutions.

## 2 Related Work

As privacy becomes a key concern in modern society, regulations like GDPR (General Data Protection Regulation) have shaped global privacy policies by enforcing stricter transparency requirements (Wang et al., ). Despite these advancements, privacy policies are still often long and complex, making them difficult for users to fully understand. McDonald and Cranor (McDonald & Cranor, ) estimated that reading all privacy policies a user encounters would take 201 hours per year, highlighting the need for more user-friendly formats.

##### History of Privacy Policies Development

The development of privacy rights began with Warren and Brandeis’s concept of "the right to be let alone" (Warren & Brandeis, ), which laid the foundation for privacy as a legal right. The Universal Declaration of Human Rights (UN General Assembly, ) recognized privacy as a fundamental human right, while Westin (Westin, ) further expanded the concept, emphasizing individuals’ control over personal information. As the digital age progressed, Introna (Introna, ) highlighted the importance of privacy in networked societies, and the Federal Trade Commission (Federal Trade Commission, ) introduced the "Notice and Choice" framework. Cate (Cate, ) criticized its effectiveness, arguing that it fails to adequately protect user privacy. More recent work has focused on the gap between the language of privacy policies and users’ understanding (Reidenberg et al., ), and how big data and social inequalities affect privacy (Jain et al., ; Madden et al., ).

##### Privacy Policies with Natural Language Models

Natural Language Processing (NLP) has been instrumental in automating the analysis of privacy policies, addressing the increasing complexity of these documents. Early efforts by Costante et al. (Costante et al., ) and Ammar et al. (Ammar et al., ) focused on identifying data practices, while Liu et al. (Liu et al., ) and Ramanath et al. (Ramanath et al., ) worked on aligning privacy statements. The introduction of the OPP-115 dataset by Wilson et al. (Wilson et al., ) and the development of tools like Polisis (Harkous et al., ) advanced privacy policy classification and summarization. Machine learning techniques, such as CNNs for text classification and risk assessment tools (Zaeem et al., ), have improved the accessibility of privacy policies. Question-answering systems have also been applied to enhance user interaction with privacy policies, as shown by Ravichander et al. (Ravichander et al., ) and the creation of the PolicyQA dataset by Ahmad et al. (Ahmad et al., ).

##### LLM Agent

LLM Agents, powered by large language models like GPT and BERT, have shown significant promise in various fields, including social sciences and engineering (Bubeck et al., ). These agents excel in natural language understanding and interaction, making them ideal for automating tasks like policy summarization and compliance analysis. The application of LLM Agents to privacy policy analysis can enhance user accessibility, allowing for more efficient and understandable interpretations of complex documents.

## 3 Benchmarking Large Language Model ability in usable privacy policy tasks

In this section, we present a comprehensive evaluation of OpenAI’s state-of-the-art closed-source large language models (LLMs) on key tasks within the domain of usable privacy policies. The models assessed include GPT-4o to GPT-3.5. To accurately reflect the models’ inherent capabilities and generalization potential, we employed both zero-shot and few-shot prompt engineering approaches, testing the models directly via their respective APIs. This methodology aligns with our objective of utilizing LLMs to develop agents capable of addressing a wide range of issues in privacy policy comprehension.

We selected the following four tasks for evaluation, as they represent critical aspects of user interaction with privacy policies: Data Practice Identification, Choice Identification, Policy Summarizing, Privacy Question Answering.

### 3.1 Data Practice Identification

In our experiments, we evaluated several models, including GPT-4o, GPT-4o-mini, GPT-4-turbo, and GPT-3.5, on the task of Data Practice Identification. The testing procedure involved direct API calls, employing the same prompt across all models to ensure consistency. These experiments were conducted on the entire OPP-115 dataset (Wilson et al., ), a comprehensive collection of annotated privacy policies.

Given that the task required the models to perform classification, we set the temperature parameter to zero to ensure deterministic outputs and eliminate randomness in predictions. This decision was made to maintain consistency and reliability in the classification process.

After considering factors such as response times, performance metrics, and computational cost, we selected GPT-4o-mini for further evaluation. Table 1, Table 2 present the performance of GPT-4o-mini on the Data Practice Identification task compared to the baseline(Wilson et al., ). Notably, GPT-4o-mini, under zero-shot learning conditions without additional context, outperformed the baseline model on average.

It is important to note that (Wilson et al., ) divided the Other category into three smaller subcategories, which are displayed in the final chart. Consequently, we do not have baseline data for the classification of the whole Other category. The results demonstrate the capability of GPT-4o-mini in policy classification tasks and its potential for application in constructing real-world agents.

**Table 1: Performance metrics for GPT-4o-mini and LR across categories.**
|  | GPT-4o-mini | LR |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Category | Precision | Recall | F1-score | Precision | Recall | F1-score |
| First Party Collection/Use | 0.95 | 0.64 | 0.77 | 0.73 | 0.67 | 0.70 |
| Third Party Sharing/Collection | 0.84 | 0.69 | 0.75 | 0.64 | 0.63 | 0.63 |
| User Choice/Control | 0.88 | 0.43 | 0.58 | 0.45 | 0.62 | 0.52 |
| User Access, Edit, & Deletion | 0.90 | 0.59 | 0.71 | 0.47 | 0.71 | 0.57 |
| Data Retention | 0.96 | 0.16 | 0.27 | 0.10 | 0.35 | 0.16 |
| Data Security | 0.97 | 0.44 | 0.61 | 0.48 | 0.75 | 0.59 |
| Policy Change | 0.86 | 0.59 | 0.70 | 0.59 | 0.83 | 0.69 |
| Do Not Track | 0.64 | 0.88 | 0.74 | 0.45 | 1.00 | 0.62 |
| International & Specific Audiences | 0.95 | 0.77 | 0.88 | 0.49 | 0.69 | 0.57 |
| Other | 0.91 | 0.35 | 0.51 | NaN | NaN | NaN |
| Micro-Average | 0.90 | 0.53 | 0.67 | 0.53 | 0.65 | 0.58 |

**Table 2: Performance metrics for SVM and HMM across categories.**
|  | SVM | HMM |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Category | Precision | Recall | F1-score | Precision | Recall | F1-score |
| First Party Collection/Use | 0.76 | 0.73 | 0.75 | 0.69 | 0.76 | 0.72 |
| Third Party Sharing/Collection | 0.67 | 0.73 | 0.07 | 0.63 | 0.61 | 0.62 |
| User Choice/Control | 0.65 | 0.58 | 0.61 | 0.47 | 0.33 | 0.39 |
| User Access, Edit, & Deletion | 0.67 | 0.56 | 0.61 | 0.48 | 0.42 | 0.45 |
| Data Retention | 0.12 | 0.12 | 0.12 | 0.08 | 0.12 | 0.09 |
| Data Security | 0.66 | 0.67 | 0.67 | 0.67 | 0.53 | 0.59 |
| Policy Change | 0.66 | 0.88 | 0.75 | 0.52 | 0.68 | 0.59 |
| Do Not Track | 1.00 | 1.00 | 1.00 | 0.45 | 0.40 | 0.41 |
| International & Specific Audiences | 0.70 | 0.70 | 0.70 | 0.67 | 0.66 | 0.66 |
| Other | NaN | NaN | NaN | NaN | NaN | NaN |
| Micro-Average | 0.66 | 0.66 | 0.66 | 0.60 | 0.59 | 0.60 |

### 3.2 Choice identification

We evaluated the performance of GPT-4o, GPT-4o-mini, GPT-4-turbo, and GPT-3.5 under the zero-shot learning setting without additional context on the Choice Identification task. The models exhibited performance slightly below the baseline but remained noteworthy. Specifically, we observed that precision was lower, while recall was relatively high. This suggests that the models were able to identify the majority of texts containing opt-out options, with minimal instances of missed true positives.

This trend indicates that the models rarely failed to detect texts that genuinely contained opt-out provisions. To further enhance performance, we employed a few-shot learning approach, deliberately increasing the number of examples with true opt-out options in the prompt. After this adjustment, the models achieved performance on par with, or slightly better than, the baseline.

The comparative results are illustrated in Table 3.

**Table 3: Performance metrics comparison across different models.**
| Metric | GPT-4o-mini (zero shot) | GPT-4o-mini (few shots) | LR | BERT | fastText |
| --- | --- | --- | --- | --- | --- |
| Precision | 0.74 | 0.88 | 0.90 | 0.83 | 0.90 |
| Recall | 0.94 | 0.95 | 0.86 | 0.98 | 0.76 |
| F1-score | 0.83 | 0.91 | 0.88 | 0.90 | 0.82 |
| Accuracy | 0.94 | 0.93 | NaN | NaN | NaN |

### 3.3 Policy Question Answer

In our experiments on the Privacy Question Answering task, we tested the performance of GPT-3.5 and GPT-4o-mini on the PolicyQA test dataset, designed to assess their ability to answer questions within the context of privacy policies. Additionally, we compared the performance of these models against the BERT-base model to further evaluate their effectiveness.

Our results indicate that the GPT-4o-mini model, utilizing a top-10 selection strategy, outperformed the BERT-base model in answering questions within the privacy policy context. The zero-shot performance of GPT-4o-mini demonstrates its ability to generalize across unseen data, effectively extracting relevant information and generating accurate answers from the privacy policies. However, we also observed that autoregressive large language models like GPT-4o-mini tend to hallucinate responses, even when strict output controls are in place. Despite our efforts to minimize such occurrences, hallucination remains a persistent challenge.

To address cases where the model failed to generate a meaningful response, we applied a post-processing step. Specifically, we filtered out samples where the model provided no answer at all, ensuring that the final evaluation included only valid and relevant outputs. This step helped refine the model’s overall performance assessment by removing uninformative outputs.

The comparative results are illustrated in Table 4.

**Table 4: Performance metrics comparison for Policy Question Answer.**
| Metric | GPT-3.5 | GPT-3.5(top-10) | GPT-4o-mini | GPT-4o-mini(top-10) | BERT-base |
| --- | --- | --- | --- | --- | --- |
| Rouge-1 | 0.35 | 0.48 | 0.44 | 0.57 | 0.56 |
| Rouge-2 | 0.23 | 0.37 | 0.37 | 0.50 | 0.44 |
| Rouge-L | 0.32 | 0.45 | 0.43 | 0.56 | 0.56 |

### 3.4 Policy Summarization

In this experiment, following the methodology of Keymanesh et al. (2020), we used their dataset to evaluate GPT-4o and GPT-4o-mini on ten publicly available user agreements from platforms like Google, Amazon, and CNN. These agreements were processed to extract the ’riskiest’ sentences, focusing on privacy and data handling at content ratios of 1/16 and 1/64. While GPT-4o slightly underperformed compared to domain-specific supervised models, it showed strong generalization in a zero-shot setting. The dataset’s age and lack of task-specific fine-tuning likely contributed to the performance difference, but GPT-4o’s ability to achieve comparable results demonstrates its robust generalization.

**Table 5: Performance metrics comparison for different compression ratios.**
|  | Compression Ratio 1/64 | Compression Ratio 1/16 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | GPT-4o | GPT-4o-mini | CNN+RF | CNN+CF | GPT-4o | GPT-4o-mini | CNN+RF | CNN+CF |
| ROUGE-1 | 0.338 | 0.335 | 0.340 | 0.379 | 0.429 | 0.427 | 0.431 | 0.404 |
| ROUGE-2 | 0.248 | 0.246 | 0.250 | 0.288 | 0.310 | 0.308 | 0.312 | 0.287 |
| ROUGE-L | 0.246 | 0.244 | 0.248 | 0.292 | 0.366 | 0.364 | 0.368 | 0.340 |
| METEOR | 0.398 | 0.396 | 0.400 | 0.439 | 0.418 | 0.416 | 0.420 | 0.416 |

## 4 LLM Agent for Usable Privacy Policy

### 4.1 Overview of the Agent

We designed the AI agent as a guided and heuristic system to assist users in comprehending complex privacy policies without requiring expertise in privacy law. It proactively identifies critical points that warrant user attention and employs guided, heuristic dialogues to effectively communicate this information(Ouyang et al., ; OpenAI, ; Wei et al., ). Subsequently, it facilitates an open-ended question-answering session to help users address any remaining uncertainties.

As illustrated in Figure 2, the agent operates through a multi-stage process, beginning with the retrieval of the privacy policy from a specified URL, followed by document preprocessing, segmentation, summarization, data practice identification, opt-out choice extraction, and question answering. The system autonomously recognizes essential sections—such as data-sharing practices, user rights, and opt-out mechanisms—that are often obscured within extensive legal texts. Additionally, it features an interactive interface that enables users to engage with the policy by posing specific inquiries regarding their privacy rights and options. By simplifying the process of navigating and understanding privacy policies, the AI agent enables users to make informed decisions about their privacy with ease and confidence. It provides clarity and support, allowing users to focus on what matters most without the need for extensive legal knowledge.

Figure: Figure 2: The figure illustrates the development workflow of this agent. The system integrates multiple tools to automate the extraction and interpretation of privacy practices, providing users with summarized insights and question-answering capabilities.
Refer to caption: extracted/5927103/Agent.jpg

### 4.2 System Architecture

The AI agent is built upon the LangChain framework (Følstad & Skjuve, ), integrating large language models with specialized tools for privacy policy analysis. The architecture comprises the following key components:

Document Retrieval and Preprocessing:
The agent accepts a user-provided URL to retrieve the corresponding privacy policy. Utilizing LangChain’s request_url, it extracts relevant content, filtering out non-essential elements like advertisements to focus on the policy text.

Document Segmentation:
We employ ASDUS (Athreya, ) to segment the HTML document into titles and prose, organizing content into <h2> and <p> tags. This process aids in identifying critical sections such as user rights, data-sharing practices, and opt-out options, enhancing the agent’s ability to analyze and interpret the policy effectively.

Policy Analysis Modules:
The agent incorporates two primary modules:
- Policy Practice Identification: Using LangChain’s custom tools, the agent classifies policy segments into ten predefined categories (e.g., data collection, third-party sharing) based on established taxonomies (Wilson et al., ).
- Content Summarization: The summarize_tool condenses complex policy content into concise summaries, allowing users to quickly grasp essential privacy practices without sifting through dense legal text.

Opt-Out Choice Detection:
To detect opt-out options, the agent scans the policy text for relevant keywords and hyperlinks leading to opt-out mechanisms. It then uses the underlying LLM to analyze the context and confirm valid opt-out choices, which are stored for easy user access.

Interactive Dialogue Mechanism:
The agent’s interactive capabilities are facilitated by LangChain’s RunnableWithMessageHistory, enabling dynamic question-and-answer sessions. Users can inquire about specific aspects of the policy, and the agent responds by leveraging appropriate tools and the LLM for contextually relevant answers. Conversation history is maintained to provide a coherent and personalized interaction experience.

Backend Integration:
Managed by LangChain’s AgentExecutor, the system orchestrates the execution of multiple tools efficiently. Integration with the underlying LLM (e.g., GPT-4o-mini) allows for deep contextual understanding and accurate response generation. The agent’s architecture is designed for scalability, enabling parallel processing of multiple queries while ensuring robustness through memory databases and external modules for advanced tasks.

## 5 Questionnaire Design

We designed a questionnaire to evaluate participants across five dimensions—comprehension, user experience, time efficiency, cognitive load, and trust intention—based on established evaluation methods (Brooke, ; Hart & Staveland, ; McKnight & Chervany, ). This allowed us to assess differences between participants who read privacy policies directly and those assisted by our AI agent. The complete questionnaire content is attached in the appendix.

Comprehension Assessment:
Comprehension was assessed using three multiple-choice questions on key aspects of privacy policies: data collection types, data sharing practices, and user rights (Reidenberg et al., ). Participants scored 1 point for each correct answer, with a total possible score ranging from 0 to 3.

User Experience Evaluation:
We evaluated user experience using a 5-point Likert scale to measure ease of use, satisfaction, and information quality (Brooke, ). Participants interacting with the AI agent answered additional AI-specific questions.

Time Efficiency:
We recorded the time each participant took to complete the task—either reading the privacy policy or using the AI agent—to compare time efficiency between the two groups.

Cognitive Load Evaluation:
Cognitive load was assessed using an adapted NASA-TLX questionnaire (Hart & Staveland, ). Participants rated mental demand, task difficulty, and frustration on a scale from 0 ("Very Low") to 10 ("Very High").

Trust and Intention Assessment:
Participants rated their trust in the information received and their intention to use similar tools in the future using a 5-point Likert scale (McKnight & Chervany, ).

Open-ended Feedback:
We collected open-ended feedback on any difficulties encountered and suggestions for improving the reading experience or the AI agent, providing qualitative insights for future enhancements.

## 6 Result Analysis

This section presents the analysis of data collected from participants who directly read the privacy policies (Control Group) and those who used the AI agent (Experimental Group). We evaluated the results across five dimensions: comprehension, user experience, time efficiency, cognitive load, and trust/intention.

### 6.1 Participants and Data Analysis

A total of 100 participants were recruited for the study, comprising 52 males and 48 females, ranging in age from 18 to 65 years old (mean age = 35.2, SD = 10.4). Participants were randomly assigned to the Control Group or the Experimental Group, with 50 participants in each. The sample included individuals with diverse educational backgrounds: 20% had a high school diploma, 50% held a bachelor’s degree, and 30% had a graduate degree (master’s or doctoral). Regarding digital literacy, participants self-reported their proficiency levels, with 45% identifying as beginners, 45% as intermediate users, and 10% as advanced users. This diversity in demographics ensures that the findings are generalizable across different user profiles.

Prior to analysis, we verified that the data met the assumptions for parametric tests, including independence of observations, normality, and homogeneity of variances using Shapiro-Wilk and Levene’s tests. All assumptions were satisfied, allowing for the use of independent samples t-tests.

**Table 6: Summary of Results Across Measured Dimensions**
| Dimension | Measure | Control Group | Experimental Group |
| --- | --- | --- | --- |
| Comprehension | Mean Score (0–3) | 1.8 (0.7) | 2.6 (0.6) |
|  | t-test ($p$-value) | $t(98)=6.23$, $p<0.001$, $d=1.24$ |  |
| User Experience (1–5) | Ease of Use | 2.8 (0.9) | 4.1 (0.7) |
|  | Info Accessibility | 2.5 (1.0) | 4.2 (0.6) |
|  | Info Clarity | 2.9 (0.8) | 4.3 (0.6) |
|  | t-tests ($p$-value) | $p<0.001$, $d$ = 1.69–2.13 |  |
| Time Efficiency | Time (minutes) | 15.8 (2.5) | 5.5 (1.8) |
|  | t-test ($p$-value) | $t(98)=11.62$, $p<0.001$, $d=2.32$ |  |
| Cognitive Load (0–10) | Mental Demand | 7.5 (1.2) | 3.5 (1.1) |
|  | Task Difficulty | 7.8 (1.1) | 3.2 (1.0) |
|  | Frustration Level | 7.2 (1.3) | 3.0 (1.2) |
|  | t-tests ($p$-value) | $p<0.001$, $d$ = 2.23–2.81 |  |
| Trust and Intention (1–5) | Trust in Info | 2.8 (0.7) | 4.5 (0.5) |
|  | Confidence in Privacy Mgmt | 2.6 (0.8) | 4.4 (0.6) |
|  | t-tests ($p$-value) | $p<0.001$, $d$ = 1.57–1.62 |  |

The detailed analysis of each measured dimension is summarized below, in conjunction with the results presented in Table 6.

- •
Comprehension: Participants in the Experimental Group demonstrated significantly better comprehension of the privacy policies compared to the Control Group. An independent samples t-test confirmed this difference ($t(98)=6.23$, $p<0.001$, Cohen’s $d=1.24$).
- •
User Experience: The Experimental Group reported a significantly better user experience across all measured statements. Differences in ease of use, information accessibility, and information clarity were all statistically significant with large effect sizes (all $p<0.001$, $d$ ranging from 1.69 to 2.13).
- •
Time Efficiency: Using the AI agent significantly reduced the time required to complete the task. The difference was statistically significant ($t(98)=11.62$, $p<0.001$, Cohen’s $d=2.32$), demonstrating the AI agent’s efficiency.
- •
Cognitive Load: Participants in the Experimental Group experienced significantly lower cognitive load across all dimensions. Reductions in mental demand, task difficulty, and frustration level were statistically significant (all $p<0.001$, $d$ ranging from 2.23 to 2.81), indicating the AI agent effectively reduced cognitive effort.
- •
Trust and Intention: The Experimental Group reported higher levels of trust in the information received and greater confidence in managing their privacy. These differences were statistically significant with large effect sizes (all $p<0.001$, $d$ ranging from 1.57 to 1.62). Participants also expressed a strong intention to use the AI agent for understanding future privacy policies.

## 7 Discussion & Limitations

While our study demonstrates the potential of LLM-based agents in enhancing user comprehension of privacy policies, it is important to acknowledge the limitations and consider areas for future work. One significant concern is the possibility of biases and hallucinations inherent in LLMs (Bender et al., ), which could affect the accuracy and reliability of the information provided to users. These issues may inadvertently mislead users or reinforce existing biases, potentially impacting their understanding and decisions regarding privacy. Addressing these concerns requires implementing robust validation mechanisms and incorporating human oversight to ensure the agent’s outputs are trustworthy.

Additionally, our study focused on short-term interactions with the agent. To fully understand its impact on users’ privacy management behaviors, longitudinal studies are necessary. Evaluating the agent’s effectiveness over extended periods will provide insights into its sustainability and long-term benefits, including whether users continue to engage with the agent and how it influences their privacy decisions over time. Future research should also explore user retention rates, the agent’s impact on long-term privacy awareness, and its integration into daily digital practices.

## 8 Conclusion

In this work, we have pioneered the application of large language models to enhance user comprehension of privacy policies through an interactive LLM-based agent. We investigated the challenges users face when interpreting complex legal language and demonstrated how our agent significantly improves understanding, reduces cognitive load, and increases confidence in managing personal data. Simultaneously, we demonstrated new benchmarks in privacy policy analysis tasks, showcasing the superior performance of LLMs over traditional models. We strongly encourage further research in this area, including the development of more advanced agents and evaluation methods. By fostering a collaborative and iterative approach to user empowerment and privacy management, we eagerly anticipate continued advancements in promoting informed consent and user autonomy in the digital landscape.

## Appendix A Appendix

### A.1 Prompt used in bench-marking and agent

#### A.1.1 Practice Identification

#### A.1.2 Policy Summarization

#### A.1.3 opt-out choice Identification

#### A.1.4 Policy QA

### A.2 Agent Usage Examples

To illustrate the practical application of our LLM-based agent, we provide screenshots demonstrating its interaction with users.

#### A.2.1 User Interface

Figure: Figure 3: The initial screen of the Privacy Policy Analysis Agent, prompting users to enter a URL for detailed policy analysis.
Refer to caption: extracted/5927103/images/agent_fig0.jpg

#### A.2.2 Data Practices Identification

Figure: Figure 4: Comprehensive breakdown of the IMDb Privacy Notice, showing various sections and the extent of information provided.
Refer to caption: extracted/5927103/images/agent_fig1.jpg

#### A.2.3 Third-party Sharing/Collection

Figure: Figure 5: Detailed explanation of the third-party sharing and collection practices outlined in the IMDb Privacy Notice.
Refer to caption: extracted/5927103/images/agent_fig2.jpg

#### A.2.4 Opt-Out Detection

Figure: Figure 6: An overview of the privacy policy analysis agent interface showing the opt-out options available to users.
Refer to caption: extracted/5927103/images/agent_fig3.jpg

#### A.2.5 Policy Summarization

Figure: Figure 7: A concise summary of the IMDb Privacy Notice, highlighting key aspects of data collection, sharing, and user control.
Refer to caption: extracted/5927103/images/agnet_fig4.jpg

### A.3 User Privacy Policy Reading Experience Questionnaire

### Basic Information

- 1.
Have you ever read a privacy policy? [   ] Yes [   ] No
- 2.
Have you ever used an AI agent?      [   ] Yes [   ] No

### Experimental Group

Please select the group you participated in:

- [   ] Group A: Directly read the privacy policy
- [   ] Group B: Used an AI agent to assist in reading the privacy policy

## I. Comprehension Assessment

Based on your understanding after reading the privacy policy, please answer the following questions:

- 1.
Types of Data Collected: What types of personal data are collected in this privacy policy?
(A)
Contact information (e.g., email, phone)
(B)
Location information
(C)
Browsing history
(D)
All of the above
- 2.
Data Sharing: Does the company share your personal data with third parties?
(A)
Yes
(B)
No
(C)
Only with user consent
(D)
Uncertain
- 3.
User Rights: According to the privacy policy, what rights do you have to manage your personal data?
(A)
Access and modify your data
(B)
Request deletion of your data
(C)
Opt-out of data processing
(D)
All of the above

## II. User Experience Assessment

Based on your experience, please rate the following statements:

1 = Strongly Disagree     2 = Disagree     3 = Neutral     4 = Agree     5 = Strongly Agree

- 1.
Usability
a)
I found reading/using the privacy policy/AI agent very easy. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
b)
I could easily find the information I needed. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
c)
The navigation and operation were intuitive to me. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
- 2.
Satisfaction
a)
I am satisfied with my understanding of the privacy policy. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
b)
I am satisfied with the overall reading/usage experience. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
- 3.
Information Quality
a)
I believe the information provided was clear and useful. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
b)
The content helped me better understand my privacy rights. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
- 4.
AI Agent Specific (Group B only)
a)
The AI agent’s responses were accurate and relevant. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
b)
I found interacting with the AI agent pleasant. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
c)
I am willing to continue using AI agents to understand privacy policies in the future. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]

## III. Time Efficiency

Please indicate the total time you spent completing the task (in minutes):                            minutes

## IV. Cognitive Load Assessment

Based on your experience, please rate the following statements:

0 = Very Low     …     10 = Very High

- 1.
Mental Demand: How much mental and cognitive effort was required to complete the task?
0 [   ] 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ] 6 [   ] 7 [   ] 8 [   ] 9 [   ] 10 [   ]
- 2.
Task Difficulty: How difficult did you find the task?
0 [   ] 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ] 6 [   ] 7 [   ] 8 [   ] 9 [   ] 10 [   ]
- 3.
Stress Level: How much stress or frustration did you feel while completing the task?
0 [   ] 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ] 6 [   ] 7 [   ] 8 [   ] 9 [   ] 10 [   ]

## V. Trust and Willingness

Based on your experience, please rate the following statements:

1 = Strongly Disagree     2 = Disagree     3 = Neutral     4 = Agree     5 = Strongly Agree

- 1.
I believe the information I read/received is accurate. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
- 2.
I am confident in my ability to understand and manage personal privacy. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]

(Group B only)

- 3.c.
I trust the answers provided by the AI agent. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]
- 4.d.
I am willing to use AI agents in the future to help me understand other privacy policies. 1 [   ] 2 [   ] 3 [   ] 4 [   ] 5 [   ]

## VI. Open-ended Feedback

- 1.
What difficulties did you encounter during the reading/usage process?
- 2.
How do you think the experience of reading privacy policies could be improved?

(Group B only)

- 3.
Do you have any suggestions or comments regarding the AI agent?

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: Why was the user comprehension assessment only three questions? Do you think such a short assessment meaningfully measures user comprehension? How were the three questions chosen?

Did you track instances of hallucination in experimental group user sessions? How frequent and severe were they? How correlated was user trust with the accuracy of the information provided by the agent?
soundness: 3
strengths: Originality: moderate. As the authors themselves note, there is substantial prior work on the problems with user comprehension of privacy policies and terms, and this paper is largely a straightforward application of a new model to an existing task. However, the agent the authors developed to assist users is a novel contribution, especially providing the ability to automatically surface opt-out mechanisms for users.

Quality: moderate. It might not be earth-shattering, but the execution nonetheless seems thorough.

Clarity: high. The presentation of the experiments and analyses performed is very clear.

Significance: moderate. The effects of the agent on user comprehension are notable, though practical impact feels limited given that it still takes nearly 6 minutes to read a privacy policy.
summary: The authors assess the performance of OpenAI's GPT suite of LLMs on a set of text classification tasks using an existing privacy policy dataset. They compare the models' performance to baseline, non-LLM models from the dataset's creators. Additionally, they develop an LLM-powered agent to assist with reading and interpreting privacy policies, measuring its effect on comprehension and cognitive effort in a population of 100 users.
weaknesses: The authors state that "GPT-4o-mini, under zero-shot learning conditions without additional context, outperformed the baseline model on average" on the Data Practice Identification task. However, the model suffered from consistently poor recall, which the authors do not meaningfully address.

Statistical tests in section 6 not corrected for multiple comparisons.

As noted above, given that it takes nearly 6 minutes to read a privacy policy even with assistance, I feel skeptical that this approach would make a meaningful difference in the number of users that actually read privacy policies. Coupled with the models' poor recall and tendency to hallucinate, it seems likely that users would still miss the most important information in the privacy policy or even be presented with false information. It might be informative to conduct time-limited trials, where user comprehension is measured after e.g. a 30 second time limit. Another idea might be to measure the time it takes the user to be able to achieve an 80% score on a comprehension test (allowing multiple attempts).

The assessment of user comprehension is extremely coarse (three questions). A more fine-grained assessment might provide interesting insights for where further improvements (either to the agent or the UX) are most needed.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1.	What are the traditional models in page 2?
2.	There is a missing citation in page 3, CNNs for text classification(?)
3.	Figure 1 was never described.
4.	In page 6, what is the process of ensuring valid and relevant outputs? 
5.	Why different metrics were used to evaluate different tasks? The explanation along with a short description of the metrics will benefit the clarity. Same comment for t-test.
soundness: 3
strengths: 1.	The entire paper is well-written and presented the ideas in very clear way. 
2.	The authors explored a very specific and less explored use case of LLM agents in recent times.
3.	The empirical analysis is comprehensive and make sense of the idea the authors proposed.
4.	Multiple open-sourced and close-sourced LLMs were used and compared their performance. 
5.	The built agents archive comparable performance like benchmarks and sometimes outperforms the baselines.
summary: In the paper, the authors address a very specific issue of understanding the privacy policies of the users of various websites in a comprehensive manner from different aspects by LLM agents. It was built with an aim to help the general users of the websites about the privacy concerns and the policies of the personal or other data they share. The performance of the build LLM agent was evaluated on 100 people where half of them studied the policies by themselves and the rest used the LLM agent. The empirical results infer the users who used the LLM agent they got a better understanding of the websites user policies than the manually readers.
weaknesses: 1.	As a whole, this paper is more like a building a new tool for the various website users, than a theoretical or technical presentations of ideas and experimental analysis. However, before making it available for the public usage, it needs several things to be considered, e.g., misinformation, hallucination, privacy leakage of company policies. 
2.	It doesn’t include any novel technical or theoretical contributions in terms of the finding the research gaps of LLMs agents to be utilized for specific use cases. 
3.	Usually, LLMs agents for particular task are more likely to hallucinates its users. The risks of LLMs hallucinations were not explored in this paper in details. The built LLM agents might not work well under such vulnerabilities. At least a few results with analysis should have been discussed. Apart from this, LLM agents might face several potential security and privacy issues as described in https://arxiv.org/pdf/2407.19354; this paper does not explore or discuss such vulnerabilities. 
4.	Building the agent only on one privacy policy dataset (though it is large) may not be sufficient to use the LLMs agent in practice.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Tables 1–3: The rationale for not including fine-tuned models is not sufficiently explained. Fine-tuning could potentially yield stronger baselines or comparative insights in this setting.
soundness: 3
strengths: The paper is clearly written and easy to follow.
summary: This study presents a competent investigation into the use of GPT family models for privacy policy comprehension support. However, it may not fully align with ICLR’s intended contribution areas, as it reads more like a system-oriented paper that might be more suited for an HCI venue.

The authors first assess the performance of GPT models, both in zero-shot and few-shot settings, and compare these results against traditional approaches. They conclude that GPT models exhibit reasonable performance levels in this context. Following this evaluation, the study introduces an LLM-driven agent designed to assist users in understanding privacy policies and completing related tasks. Through questionnaires, the study demonstrates that the agent helps reduce cognitive load and enhances both comprehension and user confidence.

While this research is well-executed, I question whether its contributions are significant enough to justify a full paper. The study does not introduce new models or corpora, nor does it directly address gaps in current models related to privacy management, although limitations are acknowledged.

Moreover, it is unclear how the proposed system substantially differs from other reading comprehension and summarization systems. A deeper comparison in this area could provide useful context for assessing the novelty of the approach.

Specific Comments:

Tables 1–3: The rationale for not including fine-tuned models is not sufficiently explained. Fine-tuning could potentially yield stronger baselines or comparative insights in this setting.
weaknesses: While this research is well-executed, I question whether its contributions are significant enough to justify a full paper. The study does not introduce new models or corpora, nor does it directly address gaps in current models related to privacy management, although limitations are acknowledged.

Moreover, it is unclear how the proposed system substantially differs from other reading comprehension and summarization systems. A deeper comparison in this area could provide useful context for assessing the novelty of the approach.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 2
strengths: 1. Applying the LLM in the digital privacy management is an interesting topic.
summary: This paper applies the  large language models (LLMs) to enhance user comprehension of privacy policies through an interactive dialogue agent.  The authors first demonstrate that LLMs significantly outperform traditional models in tasks like Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering. Building on these findings, they then introduce an LLM-based agent that functions as an expert system for processing website privacy policies, guiding users through complex legal language without requiring them to pose specific questions. A user study with 100 participants showed that users assisted by the agent had higher comprehension levels, reduced cognitive load, increased confidence in managing privacy, and completed tasks in less time .
weaknesses: 1. The main technical contribution of this paper appears limited given its current scope and the expectations of ICLR. It may be better suited for HCI venues such as CHI or IUI, which align more closely with the type of work presented.

2. The current study appears to lack IRB approval, and details of the user study are insufficiently reported. Key information such as where did you recruit participants and what is the compensation for participants are missing. Without this information, it is challenging to ensure that the study’s conclusions are reasonable and generalizable to other populations.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Did the authors explore different kinds of privacy policies in the user study — for example, are the gains from using the LLM tool greater when the privacy policy is longer / more complex?
soundness: 2
strengths: **Originality**

- Includes a systematic user study of ML-assisted privacy policy interpretation — appears to be novel relative to related work, which relies mostly on benchmark datasets (though I am not familiar with this literature).
- Constructs a system for applying state-of-the-artx LLM models to help users interpret privacy policy interpretation — including a broad range of features (interactive QA, classification, summarization) not unified in prior work.

**Quality**

- Appears to analyze the results of a user study competently and with appropriate statistics and measures of error, though some methodological details are missing.
- Appears to correctly apply benchmarks from prior work to evaluate LLM agent performance.
- From the details provided, the LLM tool seems to be well constructed and appropriate for the task.

**Clarity**

- Well written, and for the most part easy to follow.
- Does an excellent job making clear the goals & contributions of the research.

**Significance**

- Provides a technological solution to a clear privacy & transparency issue for internet users.
- Seems to present a clear, significant finding that an LLM agent could help lay users more easily interpret complex privacy policies——this could be a potentially useful workaround, barring systematic improvements to transparency requirements.
summary: The authors construct an LLM-based tool for assisting users in interpreting and summarizing privacy policies. They evaluate the tool using several benchmarks for privacy policy comprehension/summarization from prior work. In a study of 100 users, they find that users report greater comprehension and greater ease of interpretation when assisted by the LLM tool.
weaknesses: **Originality**

**[Minor]** The idea to use ML tools to assist users in interpreting privacy policies is not new—in this sense the contribution of this study is marginal. Still, there is certainly value in evaluating this idea using the most recent large language models, and there is certainly value in conducting a study with actual users to see whether the tool really makes interpretation easier. I am not sure whether there are many user studies in prior work on this idea — perhaps the authors could clarify whether this is the first user study of its kind and, if not, whether it tells us anything new.

**Quality**

Missing methodological details make it hard to tell whether the empirical findings support the broad claims in the abstract and introduction, and I have lingering questions about some of the results.

- **[Minor]** Section 3: Need a clear description of all the benchmark tasks to understand exactly what’s being evaluated — Section 3 seems to assume the tasks have already been defined. Examples:
    - Section 3.1: What does it really mean to identify “User Choice/Control” or “Data Retention”, e.g., as a practice? Does this simply mean the privacy policy describes their user choice allowances or data retention practices, which could range from quite benign to quite egregious? How is this useful to a user?
    - Section 3.2: What is the Choice Identification task? Is this the task described in A.1.3? Was this task defined in Wilson et al. (2016) too?
    - Section 3.3: What’s “Privacy Question Answering”? (Or is it “Policy Question Answering”? Both terms are used.)
    - Section 3.4: What’s in this dataset, as compared to the dataset used in the previous tasks? Who defined the “risky” sentences (what were the human-generated references for the ROUGE score)? Any examples?
    - Section 4 provides a bit more detail, and the examples in the Appendix are somewhat helpful. Perhaps this Section could come before Section 3; or alternatively, move parts of Section 3 to the appendix, and just summarize the most important findings (GPT models perform better on X benchmarks) in a paragraph, using that space instead to better explain the tasks at hand.
- **[Major]** Section 4: These results are striking — users seem to comprehend the privacy policies much more easily with LLM assistance! But there are some key methodological details missing that could determine how rigorous the results are:
    - Did the Experimental Group also have a copy of the privacy policy that they could read directly during the task (not through QA), or did they rely solely on information from the LLM agent? From the Appendix, I infer they did have access to the raw text — do the gains decrease/increase if the user cannot cross-check the LLM agent responses with the raw legal text?
    - Section 6.1: Where/how were users recruited? How many privacy policies did each participant review? How were the privacy policies selected — from one of the previous datasets? Did every participant review the same privacy policy? (How likely is it that these policies appeared in the training data — i.e. leakage?) Where/how was questionnaire administered? This information is key for determining how internally and externally valid these results might be.
    - Was the study IRB approved?
    - L393: What about racial, economic diversity in the sample? How well might these results generalize to other groups, especially marginalized groups?
    - I’m surprised by the finding that the Experimental Group had *higher* trust in info scores than the control group — and I wonder if there’s an issue with construct validity for this question. The relevant question is (L978): “I believe the information I read/received is accurate (1-5).” Given that the control group had direct access to the privacy policies, why would they respond with a 2.6, on average, compared to 4.5 in the experimental group, since the underlying information (the privacy policy) is the same for both groups? My best guess is that the Control Group suspected the company was misrepresenting its privacy practices in its privacy policy, and answered based on their distrust in the company; I suspect the Experimental Group, on the other hand, responded based on their level of trust in the accuracy of the LLM agent’s responses. So the scores may not be directly comparable. The alternative is that using the LLM agent somehow increased people’s confidence in the accuracy of the privacy policy itself, which seems less likely but still possible.
- **[Major]** Generally, it’s not clear how well the benchmarks measure the “correctness” of the agent’s responses — what is the ground truth for each of these tasks? The comprehension questions seem good, but they’re short, and not very granular — whereas the examples in the Appendix show LLM responses with much, much more detailed information about data practices. As the authors point out in the discussion, LLMs often produce incorrect and misleading text, especially when prompted for specific details that are less likely to be represented in training data. Can the authors say anything about the factuality of those more specific responses? How likely are those responses to contain falsehoods about the privacy policy that could mislead users? Can users easily identify false responses by cross-checking with the raw text or the QA feature?

**Clarity**

Generally the paper is easy to follow, with the exception of the omitted methodological details listed above. Some **minor** points of clarity that would be worth addressing: 

- L132: Have ML techniques actually improved privacy policy accessibility in practice? Or is this just a summary of research, not practice?
- L130: What is the OPP-115 dataset? Readers may not know.
- L131: Broken cite here.
- L136: What’s the difference between an LLM and an LLM agent? Is there a definition the authors can give? What makes this application an LLM agent, rather than just an LLM (the fact that the program scrapes hyperlinks, maybe)?
- Fig. 2: Text is too small to read, and often cropped, so it’s not clear what the different elements are. Simple labels might be better.
- Table 1-2: Suggest combining numbers side-by-side, so it’s easy to compare.
- Table 2, L192: SVM F1-score has a misplaced decimal.

**Significance**

- **[Minor]** This is a neat idea, and it seems like it could certainly help users in particular cases. But to frame the significance more precisely, it would be helpful to comment on the scope of a technological solution like this (e.g. in the discussion) — there is a structural issue here with privacy regulations, and with GDPR in particular, that require companies to disclose information about their privacy policies but do not require companies to make that information, and users’ options with respect to their data, truly accessible. In a perfect world, this tool may not be necessary — companies could be required to produce interpretable “privacy labels” similar to Apple’s Privacy Nutrition labels. How does the performance of this LLM-based solution compare to other policy alternatives? (These questions probably cannot be answered in this study, but it is worth mentioning that a technological solution is not necessarily the best solution.)
- **[Major]** Section 3: On a similar note, can the authors report any non-ML baselines here? How does a person do on this task, on their own? It seems less important to know how GPT models compare to BERT or other ML models, and more important to know how this method compares to what users would otherwise be doing in practice. (Unless those traditional models are actually being used by lay users in practice — that would be worth mentioning.)
    - L094: “We provide empirical evidence of the superiority of LLMs over traditional models”:  I’m assuming these sentence refers specifically to *ML* models (would be worth clarifying).  But is this approach superior to the practical alternatives available to users/policymakers? Superior to things like Apple’s “Privacy Nutrition” labels? Superior to writing a simpler privacy policy? Superior to hiring a lawyer? It would help to be more precise with this and similar claims of LLM “superiority”—superior to what?
    - Section 3: It seems like the GPT models perform better than traditional ML models, but stepping back, are these scores good enough to be relied on? For example, the recall scores seem really low here — as far as I can tell, the GPT models miss as many as 30% of instances of third party sharing, and as many as 84% of instances of “data retention”? Can this tool be used to balance precision and recall? Is this the right balance for this kind of task? Recall might well be more important to users in this kind of task.

Meta-review: This paper investigates the use of the GPT model suite for enhancing user comprehension of privacy policies. The authors develop LLM-powered agent to assist users in comprehending website privacy policies and they evaluate its effectiveness by conducting a user study involving 100 participants. Measured by comprehension, efficiency, cognitive load, and user confidence, their results indicate that users who utilized the agent had a significantly better understanding of the privacy policy.  

This is a well-executed study on an underexplored use case of LLM agents with comprehensive empirical analysis showing that GPT models exhibit reasonable performance levels compared to traditional approaches.

Decision: Accept (Poster)