Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents
Authors: BOLUN SUN
Yifan Zhou
Haiyun Jiang
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
Agent
Usable Privacy Policies
Benchmarking
HCI
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: This paper presents a novel application of large language models (LLMs) to enhance user comprehension of privacy policies through an interactive dialogue agent. We demonstrate that LLMs significantly outperform traditional models in tasks like Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering, setting new benchmarks in privacy policy analysis. Building on these findings, we introduce an innovative LLM-based agent that functions as an expert system for processing website privacy policies, guiding users through complex legal language without requiring them to pose specific questions. A user study with 100 participants showed that users assisted by the agent had higher comprehension levels (mean score of 2.6 out of 3 vs. 1.8 in the control group), reduced cognitive load (task difficulty ratings of 3.2 out of 10 vs. 7.8), increased confidence in managing privacy, and completed tasks in less time (5.5 minutes vs. 15.8 minutes). This work highlights the potential of LLM-based agents to transform user interaction with privacy policies, leading to more informed consent and empowering users in the digital services landscape.

Published as a conference paper at ICLR 2025
EMPOWERING USERS IN DIGITAL PRIVACY MANAGE- MENT THROUGH INTERACTIVE LLM-BASED AGENTS
Bolun Sun
Yifan Zhou
SNF Agora Institute Johns Hopkins University Baltimore, MD 21218, USA bsun2 60 edu
Institute for Artificial Intelligence University of Georgia
Athens, GA 30602, USA
yi rifan- houl euga. edu
Haiyun Jiang
School of Computer Science Fudan University
Shanghai, China ha melyynnjiangnnp@mmall com
ABSTRACT
This paper presents a novel application of large language models (LLMs) to en- hance user comprehension of privacy policies through an interactive dialogue agent. We demonstrate that LLMs significantly outperform traditional models in tasks like Data Practice Identification, Choice Identification, Policy Summariza- tion, and Privacy Question Answering, setting new benchmarks in privacy policy analysis. Building on these findings, we introduce an innovative LLM-based agent that functions as an expert system for processing website privacy policies, guid- ing users through complex legal language without requiring them to pose specific questions. A user study with 100 participants showed that users assisted by the agent had higher comprehension levels (mean score of 2.6 out of 3 vs. 1.8 the control group), reduced cognitive load (task difficulty ratings of 3.2 out of 10 vs. 7.8), increased confidence in managing privacy, and completed tasks in less time (5.5 minutes vs. 15.8 minutes). This work highlights the potential of LLM- based agents to transform user interaction with privacy policies, leading to more informed consent and empowering users in the digital services landscape.
INTRODUCTION
The pervasive collection and processing of personal data by online services have elevated privacy concerns in digital interactions (Vicario et al. 2019). To address these issues, websites and applica- tions are legally mandated to publish privacy policies detailing practices related to data collection, usage, sharing, and protection. However, these policies notoriously difficult for the average to comprehend due to their legal complexity (McDonald & Cranor) 2008), dense language, and considerable length. Consequently, they frequently ignored misunderstood, undermining the principle of informed consent and exposing users to potential privacy risks. This disconnect poses significant challenges (Harkous al. 2018): it impedes users' ability make informed decisions about their data and complicates efforts by regulators and organizations to ensure transparency, en- force compliance, and build trust. As privacy regulations become more stringent and policies more intricate, innovative approaches are urgently needed to bridge this comprehension gap.
Recent advances in natural language processing (NLP), particularly through the development of large language models (LLMs), offer promising solutions to these challenges. LLMs, such those based on GPT architectures, have demonstrated remarkable capabilities in understanding and gener- ating human-like text across various domains, including legal documents. These models have been successfully applied to tasks such as information extraction, content summarization, and question
Haiyun Jiang corresponding author.

---

Published as a conference paper at ICLR 2025
answering, showing potential in automating the analysis and interpretation of complex legal texts like privacy policies.
In this paper. present novel application of LLMs to improve interaction with website pri- vacy policies by developing an Al agent based on GPT-4o-mini. Our research aims to empower users by improving their ability navigate and comprehend privacy agreements, thus fostering bet- ter control over their personal data. The study structured in two parts and the Figure fillustrates the overall workflow of the proposed approach in this paper.
Frequently Ignored Difficult to understand Ambiguous Length...
language models have abilicies salve these peoblome?
OpenAi
LangChain
python
Streamlit
Usable Privacy Policy
Chase lodo cnavanmr-hdtte
Kenchmurking Large Language Model Ability In Usable Privacy Policy Tasks
Deigg apeat
A guided agent to help consumers understand usuble privacy policies
How Can NLP Benefit it?
Demonsiratial better pesformance gencraliaation abiliry compared traditiocal NLP
Mentiliattonn
Evaluation with Questionnaire
Ope-Oal Mentiicctionn
Anilysi
Figure 1: Overview of the workflow for benchmarking large language models on usable privacy policy tasks and designing a guided agent to assist consumers.
In the first part, we evaluate the performance of LLMSs across several core tasks relevant to privacy policies, including Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering (Wilson et al. 2016 Mysore Sathyendra et al.) 2017] Ahmad et al, 2020 Keymanesh al., 2020). By systematically replacing traditional models such as SVM, HMM, and CNN, as well as pre-trained models like BERT, with LLMs, we establish new benchmarks and demonstrate that LLMs achieve state-of-the-ar performance across these tasks, significantly improving the accuracy and efficiency of natural language understanding, particularly in the analysis and interpretation of privacy policies.
Building on these empirical results, the second part introduces innovative LLM-based agent built upon GPT-4o-mini to assist users comprehending website privacy policies. The agent employs heuristic interaction model that simplifies complex legal language without requiring users to know what specific questions to ask. As an expert system, processes privacy agreements swiftly, executes multiple tasks such as identifying key information, and adapts to user needs by highlighting critical policy components autonomously (Ravichander et al. (019)).
To evaluate the effectiveness of our LLM-based agent, we conducted a user study involving 100 participants, divided into two groups: one interacted directly with privacy policies, and the other used the LLM agent for assistance (Kelley al.) (2099). We measured comprehension, efficiency, cognitive load, and user confidence. The results indicate that users who utilized the agent had a sig- nificantly better understanding of the privacy policy, reported reduced cognitive load, and completed tasks in less time compared to the control group. These findings demonstrate that our Al agent not comprehension but also enhances efficiency and boosts user confidence (Kelley
only improves et al. 2013).
The key contributions of this work threefold:
2

---

Published as a conference paper at ICLR 2025
We provide empirical evidence of the superiority of LLMs over traditional models in inter- preting privacy policies, setting new benchmarks.
We develop an Al agent that reduces cognitive load, increases user confidence, and en- hances comprehension of privacy policies. This agent simplifies complex legal language, assists users without requiring prior knowledge specific queries, and autonomously high- lights critical policy components to improve decision decision-making making.
We demonstrate that our LLM-based agent significantly improves users' ability to under- stand privacy policies, reducing task completion time and cognitive effort. This work not only empowers users better assert their privacy rights and make more informed de- cisions about their data, but also provides valuable insights and a foundation for future research in developing Al-driven tools for legal document comprehension, encouraging further advancements in user-centered privacy solutions.
RELATED WORK
As privacy becomes a key concern in modern society, regulations like GDPR (General Data Protec- tion Regulation) have shaped global privacy policies by enforcing stricter transparency requirements (Wang et al. 2018). Despite these advancements, privacy policies are still often long and complex, making them difficult for users to fully understand. McDonald and Cranor (McDonald & Cranor 2008) estimated that reading all privacy policies a user encounters would take 201 hours per year, highlighting the need for more user-friendly formats.
History of Privacy Policies Development The development of privacy rights began with Warren and Brandeis's 's concept of "the right to be let alone" (Warren & Brandeis 1890). which laid the foundation for privacy as a legal right. The Universal Declaration of Human Rights (UN General Assembly 1948) recognized privacy as a fundamental human right, while Westin (Westin) 1968) further expanded the concept, emphasizing individuals' control over personal information. As the digital age progressed, Introna (Introna (997) highlighted the importance of privacy networked societies, and the Federal Trade Commission (Federal Trade Commission 1998) introduced the "Notice and Choice" framework. Cate (Cate) 2010) criticized its effectiveness, arguing that it fails to adequately protect user privacy. More recent work has focused on the gap between the language of privacy policies and users' understanding (Reidenberg et al. 2015), and how big data and social inequalities affect privacy (Jain et al. 2016
Madden et al. 2017)
Privacy Policies with Natural Language Models Natural Language Processing (NLP) has been instrumental in automating the analysis of privacy policies, addressing the increasing complexity of these documents. Early efforts by Costante et al. (Costante et al., (012) and Ammar et al. (Ammar ett al. 2012) focused identifying data practices, while Liu et al. (Liu al. 2014) and Ramanath al. (Ramanath et al.) 2014) worked on aligning privacy statements. The introduction of the OPP- dataset by Wilson et al. (Wilson al. 2016 and the development of tools like Polisis (Harkous al.,
2018) advanced privacy policy classification and summarization. Machine learning techniques, such as CNNs for text classification and risk assessment tools (Zeem al.) 2018), have improved the accessibility of privacy policies. Question-answering -n---aiiwweeege systems have also been applied to enhance user interaction with privacy policies, as shown by Ravichander et al. (Ravichander et al. 2019) and the creation of the PolicyQA dataset by Ahmad et al. (Ahmad et al., 2020).
LLM Agent LLM Agents, powered by large language models like GPT and BERT, have shown significant promise in various fields, including social sciences and engineering (Bubeck et al.) 2023). These agents excel in natural language understanding and interaction, making them ideal for au- tomating tasks like policy summarization and compliance analysis. The application of LLM Agents to privacy policy analysis can enhance user accessibility, allowing for more efficient and understand- able interpretations of complex documents.
3

---

Published as a conference paper at ICLR 2025
3 BENCHMARKING LARGE LANGUAGE MODEL ABILITY IN USABLE PRIVACY POLICY TASKS
In this section, present comprchensive evaluation of OpenAl's ofte-of-theear closed-source large language models (LLMs) on key tasks within the domain of usable privacy policies. The mod- els assessed include GPT-4o to GPT-3.5. To accurately reflect the models' inherent capabilities and generalization potential, we employed both zero-shot and few-shot prompt engineering approaches, testing the models directly via their respective APIs. This methodology aligns with our objective of utilizing LLMs to develop agents capable of addressing a wide range of issues in privacy policy comprehension.
We selected the following four tasks for evaluation. as they represent critical aspects of user interac- tion with privacy policies: Data Practice Identification, Choice Identification, Policy Summarizing, Privacy Question Answering.
3.1 DATA PRACTICE IDENTIFICAION
In our experiments, we evaluated several models, including GPT-4o, GPT-4o-mini, GPT-4-turbo, and GPT-3.5, on the task of Data Practice Identification. The testing procedure involved direct API calls, employing the same prompt across all models to ensure consistency. These experiments conducted the entire OPP-115 dataset (Wilson al. 2016), comprehensive collection of annotated privacy policies.
Given that the task required the models perform classification, set the temperature parameter to zero to ensure deterministic outputs and eliminate randomness in predictions. This decision was made maintain consistency and reliability in the classification process.
After considering factors such as response times, performance metrics, and computational cost, we selected GPT-44-mini for further evaluation. Table Table present the performance of GPT-4o- mini on the Data Practice Identification task compared to the baseline (Wilson et al. 2016). No- tably, GPT-4o-mini, under zero-shot learning conditions without additional context, outperformed the baseline model on average.
is important note that (Wilson egories, which are displayed in the final chart. Consequently, we do not have baseline data for the classification of the whole Other category. The results demonstrate the capability of GPT-4o-mini in policy classification tasks and its potential for application in constructing real-world agents.
2016) divided the Other category into three smaller subcat-
GPT-4o-mini Recall
LR
Category
Precision
Fi-score
Precision
Recall
FI F1-score
First Party Collection/Use
0.95
0.64
0.77
0.73
0.67
0.70
Third Party Sharing/Collection
0.84
0.69
0.75
0.64
0.63
0.63
User Choice/Control
0.88
0.43
0.58
0.45
0.62
0.52
User Access, Edit, & Deletion
0.90
0.59
0.71
0.47
0.71
0.57
Data Retention
0.96
0.16
0.27
0.10
0.35
0.16
Data Sccurity
0.97
0.44
0.61
0.48
0.75
0.59
Policy Change
0.86
0.59
0.70
0.59
0.83
0.69
Do Not Track
0.64
0.88
0.74
0.45
1.00
0.62
International & Specific Audiences
0.95
0.77
0.88
0.49
0.69
0.57
Other
0.91
0.35
0.51
NaN
NaN
NaN
Micro-Average
0.90
0.53
0.67
0.53
0.65
0.58
Table Performance metrics for GPT-4o-mini and LR across categories.
3.2 CHOICE IDENTIFICATION
We evaluated the performance GPT-4o, GPT-4o-mini, GPT-4-turbo, and GPT-3.5 under the zero- shot learning setting without additional context the Choice Identification task. The models ex- hibited performance slightly below the baseline but remained noteworthy. Specifically, we observed that precision lower, while recall relatively high. This suggests that the models were able
4

---

Published as a conference paper at ICLR 2025
SVM Recall
HMM Recall
Category
Precision
FI- -ccor
Precision
FI Fl-score
First Party Collection/Usse
0.76
0.73
0.75
0.69
0.76
0.72
Third Party Sharing/Collection
0.67
0.73
0.07
0.63
0.61
0.62
User Choice/Control
0.65
0.58
0.61
0.47
0.33
0.39
User Access, Edit, & Delction
0.67
0.56
0.61
0.48
0.42
0,45
Data Retention
0.12
0.12
0.12
0.08
0.12
0.09
Data Security
0,66
0.67
0.67
0.67
0.53
0.59
Policy Change
0.66
0.88
0.75
0.52
0.68
0.59
Do Not Track
1.00
1.00
1.00
0.45
0.40
0.41
International & Specific Audiences
0.70
0.70
0.70
0.67
0.66
0.66
Other
NaN
NaN
NaN
NaN
NaN
NaN
Micro-Average
0.66
0.66
0.66
0.60
0.59
0.60
Table 2: Performance metrics for SVM and HMM across categories.
identify the majority of texts containing opt-out options, with minimal instances of missed true positives.
This trend indicates that the models rarely failed to detect texts that genuinely contained opt-out pro- visions. To further enhance performance, we employed few-shot learning approach, deliberately increasing number of examples with true opt-out options the prompt. After this adjustment, the models achieved performance on par with, or slightly better than, the baseline.
The comparative results are illustrated in Table
Metric
GPT-4o-mini (zero shot)
GPT-4o-mini (few shots)
LR
BERT
fastText
Precision
0.74
0.88
0.90
0.83
0.90
Recall
0.94
0.95
0.86
0.98
0.76
Fl-score
0.83
0.91
0.88
0.90
0.82
Accuracy
0.94
0.93
NaN
NaN
NaN
Table 3: Performance metrics comparison across different models.
3.3 POLICY QUESTION ANSWER
In our experiments the Privacy Question Answering task, tested the performance of GPT-3.5 and GPT-4o-mini on the PolicyQA test dataset, designed to assess their ability to answer questions within the context of privacy policies. Additionally, we compared the performance these models against the BERT-base model to further evaluate their effectiveness.
Our results indicate that the GPT-4o-mini model, utilizing top-10 selection strategy, outperformed the BERT-base model in answering questions within the privacy policy context. The zero-shot per- formance of GPT-4o-mini demonstrates its ability to generalize across unseen data, effectively ex- tracting relevant information and generating accurate answers from the privacy policies. However, we also observed that autoregressive large language models like GPT-4o-mini tend to hallucinate responses, when strict output controls are in place. Despite efforts to minimize such occur- rences, hallucination remains persistent challenge.
To address cases where the model failed to generate meaningful response, applied post- processing step. Specifically, we filtered out samples where the model provided no answer at all, ensuring that the final evaluation included only valid and relevant outputs. This step helped refine the model's overall performance assessment by removing uninformative outputs.
The comparative results are illustrated in Table
5

---

Published as a conference paper at ICLR 2025
Metric
GPT-3.5
GPT-3.5(top-10)
GPT-4o-mini
GPT-4o-min(top-10)
BERT-base
Rouge-1
0.35
0.48
0.44
0.57
0.56
Rouge-2
0.23
0.37
0.37
0.50
0.44
Rouge-L.
0.32
0.45
0.43
0.56
0.56
Table 4: Performance metrics comparison for Policy Question Answer.
3.4 POLICY SUMMARIZATION
In this experiment, following the methodology of Keymanesh et al. (2020), we used their dataset evaluate GPT-4o and GPT-4o-mini on publicly available agreements from platforms like Google, Amazon, and CNN. These agreements were processed to extract the 'riskiest' sentences, focusing on privacy and data handling at content ratios of 1/16 and 1/64. While GPT-4o slightly underperformed compared to domain-specific supervised models, it showed strong generalization in zero-shot setting. The dataset's age and lack of task -asc-spec fine-tuning likely contributed the performance difference, but GPT-4o's ability to achieve comparable results demonstrates its robust generalization.
Compression
Compression
Metric
GPT-4o
GPT-4o-mini
CNN+RF
CNN+CF
GPT-4o
GPT--o-mini
CNN-RF
CNN+CF
ROUGE-1
0.338
0.335
0,340
0.379
0.429
0.427
0.431
0.404
ROUGE
0.246
0.250
0.308
ROUGE-L
0.246
0.244
0,248
0.292
0.366
0,364
0,368
0.340
METEOR
0.396
0.400
0,439
0.418
0.416
0.420
0,416
Table Performance metrics comparison for different compression ratios.
4
LLM AGENT FOR USABLE PRIVACY POLICY
4.1 OVERVIEW OF THE AGENT
We designed the Al agent as guided and heuristic system to assist users in comprehending complex privacy policies without requiring expertise privacy law. proactively identifies critical points that warrant user attention and employs guided, heuristic dialogues to effectively communicate this CoormationOOuyan et al.) 2022; OpenAl 2023) Wei et al., 2022). Subsequently, it facilitates an open-ended question-answerinn answering session to help users address any remaining uncertainties.
As illustrated in Figure 2, the agent operates through a multi-stage process, beginning with the re- trieval of the privacy policy from specified URL, followed by document preprocessing, segmenta- tion, summarization, data practice identification, opt-out choice extraction, and question answering. The system autonomously recognizes essential sections-such data-sharing practices, user rights, and opt-out mechanisms- are often obscured within extensive legal texts. Additionally. fea- tures an interactive interface that enables regarding their privacy rights and options. By simplifying the process of navigating and understand- ing privacy policies, the Al agent enables users to make informed decisions about their privacy with ease and confidence. It provides clarity and support, allowing users to focus on what matters most without the need for extensive legal knowledge.
engage with the policy by posing specific inquiries
4.2 SYSTEM ARCHITECTURE
The AI agent is built upon the LangChain framework (Følstad Skjuve 2019), integrating large language models with specialized tools for privacy policy analysis. The architecture comprises the following key components:
Document Retrieval and Preprocessing: The agent accepts user-provided URL retrieve the corresponding privacy policy. Utilizing LangChain's 's request ur1, it extracts relevant content, filtering non-essential elements like advertisements focus the policy text.

---

Published as a conference paper at ICLR 2025
Tools Build up
Langchan
OpenAl
URL
TASKS
CSe LUM FOOLS
CTRWAY
Sep
Segmentation And Cleaning
Streamii
Sheash
Doo
Monafication
Agent
Data Presiic
Mentilfcation
Optist Optine
Choice MacnVcaace
Policy Summarizing
QuaetIon
The figure illustrates the development workflow of this agent. The system integrates multiple tools to automate the extraction and interpretation of privacy practices, providing users with summarized insights and question-answering capabilities.
Figure 2:
Document Segmentation: We employ ASDUS (Athreya 2018) segment the HTML document into titles and prose, organizing content into <h2> and <p> tags. This process aids in identify- ing critical sections such as user rights, data-sharing practices, and opt-out options, enhancing the agent's 's ability to analyze and interpret the policy effectively.
primary modules: Policy Practice Iden- rification: Using LangChain's 's custom tools, the agent classifies policy segments into ten prede- fined categories (e.g.. data collection, third-party sharing) based on established taxonomies (Wilson) al. 2016). Content Summarization: The surmari ze_tool condenses complex policy content into concise summaries, allowing users to quickly grasp essential privacy practices without sifting through dense legal text.
Policy Analysis Modules:
The agent incorporates
Opt-Out Choice Detection:
To detect opt-out options, the agent scans the policy text for relevant keywords and hyperlinks leading opt-out mechanisms. It then uses the underlying LLM to analyze the context and confirm valid opt-out choices, which are stored for easy user access.
Interactive Dialogue Mechanism: The agent's interactive capabilities
facilitated by enabling dynamic question-and- iesstion-and-nsseer ses- sions. Users inquire about specific aspects of the policy, and the agent responds by leveraging appropriate tools and the L.M for contextually relevant answers. Conversation history is maintained provide coherent and personalized interaction experience.
LangChain's RunnableWi haaebeeeWttMessseeeeeese hMes waMessaaeeistor......
Backend Integration: Managed by LangChain's AgentExecutor the system orchestrates the execution of multiple tools efficiently. Integration with the underlying LLM (e.g., GPT-4o-mini) allows for deep contextual understanding and accurate response generation. The agent's architecture designed for scalability, enabling parallel processing of multiple queries while ensuring robustness through memory databases and external modules for advanced tasks.
4.3 DATA SECURITY IN AGENT
Ensuring robust data security is paramount in privacy-oriented systems. Drawing on established best practices in the literature (He et al.) 2018), our design adopts the principle data minimization: user interactions with agent neither require store personally identifiable information (PII).

---

Published as a conference paper at ICLR 2025
Specifically, the system only processes publicly available privacy policies supplied by users and returns structured, human-readable explanations without retaining input or output logs.
Our system relies the OpenAl API for natural language processing tasks, but does not cache any user queries or responses a persistent database. By integrating these design choices -ccuuee communications, ephemeral processing, and strict non-retention of user provided information align with privacy by design guidelines and maintain the confidentiality of user interactions. This approach is well-suited to privacy research settings, it ensures that users' sensitive data are neither collected nor exposed during the policy comprehension process.
5 QUESTIONNAIRE DESIGN
We designed questionnaire to evaluate participants across five dimensions-ccoppreheeeee comprehension, user experience, time efficiency, cognitive load, and trust bntention-based on established evaluation methods (Brooke 1996 Hart & Staveland, 1988 McKnight & Chervany (002). This allowed us to assess differences between participants who read privacy policies directly and those assisted by our Al agent. The complete questionnaire content is attached in the appendix.
Comprehension Assessment: Comprehension was assessed using three multiple-choice questions on key aspects of privacy policies: data collection types, data sharing practices, and user rights (Reidenberg et al.) (0T6). Participants scored 1 point for each correct answer, with total possible score ranging from to
User Experience Evaluation: We evaluated user experience using 5-point Likert scale to measure ease of use, satisfaction, and information quality (Brooke. 1996). Participants interacting with the Al agent answered additional Al-specific questions.
Time Efficiency: We recorded the time each participant took to complete the task either reading the privacy policy or using the AI agent-to compare time efficiency between the two groups.
Cognitive Load Evaluation: Cognitive load was assessed using adapted NASA-TLX question- naire (Hart Staveland 1988). Participants rated mental demand, task difficulty, and frustration on scale from 0 ("Very Low")) to 10 ("Very High").
Trust and Intention Assessment: Participants rated their trust in the information received and their intention use similar tools in the future using 5-point Likert scale (McKnight & Chervany 2002).
We collected open-ended feedback any difficulties encountered and suggestions for improving the reading experience or the Al agent, providing qualitative insights for future enhancements.
Open-ended Feedback:
RESULT ANALYSIS
This section presents the analysis of data collected from participants who directly read the privacy policies (Control Group) and those who used the AI agent (Experimental Group). We evaluated the results across five dimensions: comprehension, user experience, time efficiency, cognitive load, and trust/intention.
6.1 PARTICIPANTS AND DATA ANALYSIS
total of 100 participants were recruited for the study, comprising males and 48 females, rang- ing in age from 18 to 65 years old (mean age 35.2, SD 10.4). Participants were randomly assigned to the Control Group or the Experimental Group, with 50 participants in each. The sample included individuals with diverse educational backgrounds: 20% had high school diploma, 50% held a bachelor's 's degree, and 30% had graduate degree (master's 's or doctoral). Regarding digital literacy, participants self-reported their proficiency levels, with 45% identifying as beginners. 45% as intermediate users, and 10% as advanced users. This diversity in demographics ensures that the findings generalizable across different user profiles.
8

---

Published as a conference paper at ICLR 2025
Prior to analysis, verified that the data met the assumptions for parametric tests, including in- dependence of observations, normality, and homogeneity of variances using Shapiro-Wir Wilk and Lev- ene's tests. All assumptions were satisfied, allowing for the use of independent samples r-tests.
Dimension
Measure
Control Group
Experimental Group
Mcan Score (0-3)
1.8 (0.7) ((88) = 6.23, p < 0..01,, 1.24
(.66)
Comprehension
[-test (p-value)
Easc Use
2.8 (0.9)
4.1 (0.7)
Info Accessibility
2,5 (1.0)
(0.6)
User Experience (1-5)
Info Clarity
2.9 (0.8)
4.3 (0.6)
(p-value)
p 0.001, d 1.69-2.13
Time (minutes) (p-value)
15.8 (2.5)
5..
Time Efficiency
f(98) 11.62, p 0.001, d 2.32
Mental Demand
7.5 (1.2)
(1.1)
Task Difficulty
7.8 (1.1)
3.2 (1.0)
Cognitive Load (0-10)
Frustration Level
7.2 (1.3)
3.0
1-tests (p-valuc)
< 0.001, d= 2.23-2.81
Trust in Info
2.8 (0.7)
4.5 (0.5)
Trust and Intention (1-5)
Confidence in Privacy Mgmt
2.6 (0.8)
(0.6)
1-tests (p-value)
p 0.001, d 1,57-1.62
Table 6: Summary of Results Across Measured Dimensions
The detailed analysis of each measured dimension is summarized below, in conjunction with the results presented in T.ble
Comprehension: Participants in the Experimental Group demonstrated significantly better comprehension of the privacy policies compared to the Control Group. An independent samples f-test confirmed this difference (t(98) 6. 6.23, p < 0.001, Cohen's 's d = 1.24).
User Experience: The Experimental Group reported significantly better user experience across all measured statements. Differences in ease of use, information accessibility, and information clarity were all statistically significant with large effect sizes (all p 0.001, d ranging from 1.69 to 2.13).
Time Efficiency:
Using the Al agent significantly reduced the time required to complete the task. The difference was statistically significant (I(98) 11.62, p 0.001, Cohen's d = 2.32), demonstrating the Al agent's efficiency.
Participants in the Experimental Group experienced significantly lower cognitive load across all dimensions. Reductions mental demand, task difficulty, and frustration level were statistically significant (all p 0.001, ranging from 2.23 to 2.81), indicating the AI agent effectively reduced cognitive effort.
Cognitive Load:
Trust and Intention:
The Experimental Group reported higher levels of trust in the infor- mation received and greater confidence in managing their privacy. These differences were statistically significant with large effect sizes (all < 0.001, d ranging from 1.57 to 1.62). Participants also expressed strong intention use the agent for understanding future privacy policies.
DISCUSSION & LIMITATIONS
While our study demonstrates the potential of LLM-based agents enhancing user comprehension of privacy policies, it is important to acknowledge the limitations and consider areas for future work.
One significant concern is the possibility of biases and hallucinations inherent LLMs (Bender et al. 2021), which could affect the accuracy and reliability of the information provided to users. These issues may inadvertently mislead users or reinforce existing biases, potentially impacting their understanding and decisions regarding privacy. Addressing these concerns requires implementing

---

Published as a conference paper at ICLR 2025
robust validation mechanisms and incorporating human oversight ensure the agent's outputs trustworthy.
Additionally, study focused short-term interactions with the agent. To fully understand its impact on users' privacy management behaviors, longitudinal studies are necessary. Evaluating the agent's effectiveness term benefits, including whether users continue to engage with the agent and how it influences their privacy decisions over time. Future research should also explore user retention rates, the agent's impact on long-term privacy awareness, and its integration into daily digital practices.
extended periods will provide insights into its sustainability and long-
8 CONCLUSION
In this work, we have pioneered the application of large language models to enhance user compre- hension of privacy policies through an interactive LLM-based agent. We investigated the challenges users face when interpreting complex legal language and demonstrated how agent significantly improves understanding, reduces cognitive load, and increases confidence in managing personal data. Simultaneously,
demonstrated new benchmarks in privacy policy analysis tasks, show- casing the superior performance of LLMs over traditional models. We strongly encourage further research in this area, including the development of more advanced agents and evaluation methods. By fostering a collaborative and iterative approach to user empowerment and privacy management, we eagerly anticipate continued advancements in promoting informed consent and user autonomy in the digital landscape.
ACKNOWLEDGMENTS
We would like to clarify that this research did not require formal IRB approval, as it involved no collection of personally identifiable information (PII) and presented minimal risk to participants. The study focused exclusively on comprehension tasks related to publicly available privacy policies, without requesting sensitive information or posing risks beyond those of everyday activities.
Since this was a personal research endeavor without organizational sponsorship. we followed ethical guidelines for miinimal-risk research. Informed consent was obtained from all participants, who were fully informed about the study's purpose, the nature of the tasks, and their right withdraw at any time. No personal or sensitive data were gathered or stored, ensuring that the project remained strictly centered on analyzing privacy policy content.
Participants were recruited on voluntary basis through social platforms and personal networks. They received appropriate compensation for their time, ensuring fairness and ethical engagement. By adhering to these measures, maintained a high standard of ethical conduct and respected the confidentiality of all individuals involved.
REFERENCES
W. Ahmad, J. Chi, Y. Tian, and K.-W. Chang. PolicyQA: A reading comprehension dataset for privacy policies. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 743-749, 2020.
W. Ammar, S. Wilson, N. Sadeh, and N. A. Smith. Automatic categorization of privacy policies: A pilot study. Technical Report CMU-LTI-12-019, Carnegie Mellon University, 2012.
Abhijith Athreya, Asdus: Automatic segment detection using unsupervised and supervised learn- ing, 2018. GitHub repository, available at: htttps://gttub com/ abhi athreya/
ASDUS
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency (FAccT), pp. 610 623, New York, NY, USA, 2021. ACM. doi: 10.1145/33421888.44455
URL Attps: /ttttp:://ci..rooo /
3442 88 3445922
10

---

Published as a conference paper at ICLR 2025
John Brooke.
SUS-a quick and dirty usability scale. In Patrick W. Jordan, Bruce Thomas, Bernard Weerdmeester, and lan L. McClelland (eds.), Usability Evaluation in Industry, pp. 1894194. Tay- lor & Francis, 1996.
S. Bubeck, V. Chandrasekaran, R. Eldan, G. Elidan, K. LeJeune, and M. Teyssier. Sparks Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303. 2023.
H. Cate. The limits of notice and choice. IEEE Security & Privacy, 8(2), 2010.
E. Costante, J. den Hartog, and M. Petkovic. What websites know about you. In Data Privacy Management and Autonomous Spontaneous Security, pp. 146-159. Springer, 2012.
Federal Trade Commission. Privacy online: report Congress. Technical report, Federal Trade Commission, Washington, DC, June 1998.
Asbjørn Følstad and Marita Skjuve. Chatbots for customer service: user experience and motivation. In Proceedings of the Ist International Conference on Conversational User Interfaces (CUI '19), pp. Article 1, 1-9, New York, NY, USA, 2019. Association for Computing Machinery. doi: 10.1145/3342755334477..
H. Harkous, K. Fawaz, Lebret, F. Schaub, G. Shin, and K. Aberer. Polisis: Automated analysis and presentation of privacy policies using deep learning. In Proceedings of the 27th USENIX Security Symposium, pp. 531-548, 2018.
Sandra G. Hart and Lowell F. Staveland. Development of NASA-TIXX (task load index): Results of empirical and theoretical research. In Peter A. Hancock and Najmedin Meshkati (eds.), Human Mental Workload, volume 52 Advances in Psychology, pp. 139-183, North North-Holland, 1988.
Jiawen He, Wei Zhang, and John Smith. Toward privacy by design in online services. Proceedings of the 25th International Conference on Web Services (ICWS), pp. 123-131. IEEE, 2018.
L. D. Introna. Privacy and the computer: why we need privacy in the information society. Metaphi- losophy, 28(3):259-273 275, 1997.
Jain, M. Gyanchandani, and N. Khare. Big data privacy: A technological perspective and review. Journal of Big Data, 3(1):25, 2016.
G. Kelley, J. Bresee, L. F. Cranor, and R. W. Reeder. nutrition label for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security (SOUPS), pp. 1-12, 2009.
P. G. Kelley, L. F. Cranor, and N. Sadeh. Privacy as part of the app decision-making process. In Proceedings of the ACM CHI Conference Human Factors in Computing Systems, pp. 3393- 3402, 2013.
M. Keymanesh, M. Elsner, and S. Parthasarathy. Toward domain domain-guided controllable summarization of privacy policies. In Natural Legal Language Processing Workshop. KDD, 2020.
Liu, R. Ramanath, N. Sadeh, and N. Smith. step towards usable privacy policy: Auto- matic alignment of privacy statements. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 884-894, 2014.
M. Madden, Gilman, Levy, and A. Marwick. Privacy, poverty, and big data: A matrix of vulnerabilities for poor Americans. Washington University Law Review, 95:53, 2017.
A. M. McDonald and L. F. Cranor. The cost of reading privacy policies. ISJLP, 4:543,
D. Harrison McKnight and Norman L. Chervany. What trust means in E-Commerce customer lationships: An interdisciplinary conceptual typology. International Journal of Electronic Com- merce, 6(2):35-59, 2002.
K. Mysore Sathyendra, S. Wilson, F. Schaub, S. Zimmeck, and N. Sadeh. Identifying the provision of choices in privacy policy text. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2774-2779 2017.
OpenAl. Gpt-4 technical report. CoRR, abs/2303.08774, 2023.

---

Published as a conference paper at ICLR 2025
L. Ouyang, J. Wu, X. Jiang, and et al. Training language models to follow instructions with human feedback. Proceedings of NeurIPS, 2022.
R. Ramanath, F. Liu, N. Sadeh, and N. A. Smith. Unsupervised alignment of privacy policies using hidden Markov models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 605-610, 2014.
A. Ravichander, A. W. Black, S. Wilson, T. Norton, and N. Sadeh. Question answering for privacy policies: Combining computational and legal perspectives. Proceedings of the 2019 Con- ference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4947-4958, 2019.
J. R. Reidenberg, T. D. Breaux, L. F. Cranor, B. French, A. Grannis, T. Graves, and R. Ramanath. Disagreeable privacy policies: Mismatches between meaning and users' understanding. Berkeley Technology Law Journal, 30:39, 2015.
Joel R. Reidenberg, Jaspreet Bhatia, Travis Breaux, and Thomas B. Norton. Ambiguity in privacy policies and the impact of regulation. The Journal of Legal Studies, 45(S2):S163 S190, 2016.
UN General Assembly. Universal declaration of human rights. UN General Assembly, 302(2), 1948.
M. D. Vicario, A. Bessi, Zollo, F. Petroni, A. Scala, G. Caldarelli, and W. Quattrociocchi. The spreading of misinformation online. Proceedings of the National Academy of Sciences, 113(3): 554-559, 2019.
Z. Wang, S. Zimmeck, P. Story, and N. Sadeh. The Impact of the General Data Protection Regu- lation (GDPR) on Mobile App Privacy Practices. Proceedings of the ACM on Human- Human-Compute Computer Interaction, 2:200, 2018.
S. D. Warren and L. D. Brandeis. The right to privacy. Harvard Law Review, 4(5):193-220) 1890.
J. Wei, Y. Tay, R. Bommasani, and et al. Emergent abilities of large language models. Transactions of Machine Learning Research, 2022.
A. F. Westin. Privacy and Freedom. Atheneum, New York, 1968.
S. Wilson, F. Schaub, S. Amini, J. Bhatia, J. Liu, and N. Sadeh. The Creation and Analysis of Website Privacy Policy Corpus. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1330-1340, 2016.
R. N. Zaeem, C. German, and K. S. Barber. Automated risk assessment for privacy policies using machine learning approach. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 1533- 1546, 2018.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROMPT USED IN BENCH-MARKING AND AGENT
A.1.1 PRACTICE IDENTIFICATION
Prompt
You are given an annotation scheme for a website's privacy policy, which consists of the following ten categories:
Instruction:
1. First Party Collection/Use: How and why the service provider collects user information.
2. Third Party SharingCCollection: How user information may be shared with or collected by third parties.
3. User Choice/Control: What choices and control options are available to users.
User Access, Edit, & Deletion: Whether and how users may access, edit, delete their information.
Data Retention: How long user information is stored.
6. Data Security: How user information is protected.
Policy Change: Whether and how users will be informed about changes to the privacy policy.
Whether and how Not Track signals for online tracking and advertising honored.
Do Not Track:
9. International & Specific Audiences: Practices that pertain to specific groups users (e.g., children, Europeans, California residents).
10. Other: Additional labels for introductory or general text, contact information, or practices not covered by other categories.
For the privacy policy text content below, please select the most appropriate category (by number) and return only the number.
Content: (Your Text Here)
Answer:
A.1.2 POLICY SUMMARIZATION
Prompt
Instruction:
Please select the 6 sentences from (TEXT) that you consider to be the most (RISKY) from the provided privacy policy text. You must strictly choose only sentences from the original text without adding, modifying, or including any other content. (RISKY) refers to sections or clauses that could potentially expose users to privacy threats, data misuse, security vulnerabilities.
TEXT);
13

---

Published as a conference paper at ICLR 2025
Prompt
Instruction:
Please select the 29 sentences from (TEXT) that you consider to be the most (RISKY) from the provided privacy policy text. You must strictly choose only sentences from the original text without adding, modifying, or including any other content. (RISKY) refers sections or clauses that could potentially expose users privacy threats, data misuse, or security vulnerabilities.
(EXT)):
A.1.3 OPT-OUT CHOICE IDENTIFICATION
Prompt
Instruction: You are an intelligent assistant trained to identify whether a hyperlink in website privacy policies provides an opt-out option for users. Your task is to determine whether a hyperlink offers users the ability to withdraw consent for data collection or processing ('opt-out'). When making a decision, follow these stricter steps:
1. Review the context of the hyperlink carefully. Check if the link explicitly refers to an option for users to decline, refuse, or stop data collection or usage. The link should offer clear action to withdraw consent change data preferences.
Look for specific keywords or phrases like 'opt-out,' 'unsubscribe," 'do not sell,' 'withdraw consent,' "manage preferences, the hyperlink refers to generic terms such as "privacy policy,' 'learn more,' 'terms of service," 'support,"
'disable tracking."
'about us,' return 'False.'
4. Be very cautious in interpreting implicit meanings. If there is any doubt about whether the link provides an opt-out action, return 'False.'
5. Only return 'True' if the link explicitly offers an opt-out or similar function directly related data privacy or user preferences.
Here is the content to analyze. Please predict whether the hyperlink contains an opt-out choice based the following information. Return only 'True' or 'False.'
Content:
(Your Content Here)
Answer:
14

---

Published as a conference paper at ICLR 2025
A.1.4 POLICY QA
Prompt
You are an expert in privacy policies. I will provide you with (Reading Material) and (Question).
Your task is to extract the most precise and relevant information from the (Reading Material) that answers the (Question).
Guidelines:
1. Only information directly from the (Reading Material) generate your response. the relevant information specific word, phrase, or sentence, use only that. Avoid unnecessary elaboration.
3. If the exact answer is not present in the reading material, return the closest direct match in the RReading Material).
Output Requirements:
1. The Output must consist only of words or sentences that appear in the (Reading Material).
A.2 AGENT USAGE EXAMPLES
To illustrate the practical application of our LLM-based agent, we provide screenshots demonstrat- ing its interaction with users.
A.2.1
USER INTERFACE
Privacy Policy Analysis Agent
Hellc, your usable privacy policy assistant. I'm here help you quickly understand the content of this privacy pol please enter the URL of the Privacy Policy:
Figure 3: The initial screen of the Privacy Policy Analysis Agent, prompting users to enter a URL for detailed policy analysis.
15

---

Published as a conference paper at ICLR 2025
A.2.2 DATA PRACTICES IDENTIFICATION
Privacy Policy Analysis Agent
Hello, I'm your usable privacy policy assistant. I'm here to help you quickly understand the content of this privacy policy, please enter the URL Privacy Policy:
https:/wwww.mmmdb.omm/rrvaaaaa
+ The IMDb Privacy Notice contains the following information:
1. First Party Collection/Use: 16 paragraphs detailing how and why IMDb collects user
information.
2. Third Party Sharing/Collection: 11 paragraphs explaining how user information may be shared with or collected by third parties.
User Choice/Control: 11 paragraphs outlining choices and control options available to users.
User Access, Edit, Deletion: paragraphs if and how may access, edit, delete their informaii..........
5. Data Retention: paragraph on how long user information is stored.
Data Security: paragraphs on how user information protected.
Policy Change: paragraphs and how users will be informed about changes to the
privacy policy.
8. Do Not Track: 0 paragraphs on if and how Do Not Track signals for online tracking and
advertising honored.
9. International & Specific Audiences: 4 paragraphs on practices that pertain only to specific groups of users (e-g,, children, Europeans, or California residents).
10. Other: paragraphs for introductory or general text, contact information, and practices covered by the other categories.
Additionally, there opt-out options available for users.
Ask a question about the privacy policy:
Figure 4: Comprehensive breakdown of the IMDb Privacy Notice, showing various sections and the extent of information provided.
16

---

Published as a conference paper at ICLR 2025
A.2.3
THIRD-PARTY SHARING/COLLECTION
O
Tell me more about the Third Party Sharing/Collecttoon
+ The IMDb Privacy Notice provides detailed information about Third Party Sharing/Collection in the following ways:
1. Transactions Involving Third Parties: Db makes certain services available through third parties,such purchasing movie tickets. In these cases, information related the transactions is shared with the third party involved.
2. Third-Party Service Providers: IMDb employs other companies and individuals perform functions on its behalf, such as processing payments, analyzing data, and providing marketing g isaaneece. These service providers have access to personal information needed perform their functions but are not permitted use for other purposes.
3. Business Transfers: in the event that IMDb or assets are acquired or merged with another company, user info rmation may be transferred as part of the business assets. However, the information remains subject promises made any pre existissting Privacy Notice unless the consents otherwise.
4. Protection of IMDb and Others: IMDb may release account and other personal information comply with the law, enforce agreements, or protect the rights, property, or safety of IMDb, its users, or others. includes exchanging information with other companies organizations for fraud protection and credit risk reduction.
5. Third-Party Advertising and Links: IMDb Services may include third party advertising and links to other websites and apps. Third-party advertising partners may collect information about users when they Interact with their content, advertising, and services.
6. of Third-Party Advertising Services: IMDb provides ad-serving co oopaniee, ad networks, and advertising companies with information that allows them measure their effectiveness. However, IMDb does not share users' names or other information that directly identifies them.
relevant ads and
These practices ensure that while user information may be shared with third partles, is done controlled manner with specific purposes lim ltmatonions.
Ask question about the privacy policy:
Figure 5: Detailed explanation of the hiirt-party sharing and collection practices outlined in the IMDb Privacy Notice.
17

---

Published as a conference paper at ICLR 2025
A.2.4
OPT-OUT DETECTION
3
Show the options available
+ Here the opt-ou: options available the IMDb Privacy Notice, along with their hyperlinks:
Interest- Brreett-Basdddeee Preferences: may set your preferences interest- brarst---addddd ads and other preferences in user account settings. Learn more here.
2. Cookie Management: You manage browser cookies through your browser settings to prevent browser from accepting new cookies, block cookies, or have the browser notify you when you receive new cookie. Learn more here.
Third- Party Information Sharing:
receive notice when personal information about you might he shared with third parties, and you will have apportunity choose not share the information. Conditions of Use.
4. Advertising Preferences: You can adjust your advertising prefe rences by visiting the "Your Ads Privacy Choices" page. Amazon Interest- Based Ads and Your Privacy Choices.
5. Email and Communication Preferences: if u do not want to recelve e-mail other adjusi your preferences in your account setting ngs. What information Can Access?.
communications IMDb, you can log your
5. An-App Notifications: You can adjust your notification settings the device If you do not want to receive in-app notifications, Your Privacy Choices.
7. Browsing History: You can browse IMDb websites without linking the browsing history to your account by logging g out and blacking ng cookies on your browser, Contact Help Center.
8. Consent Withdrawal: may withdraw your consent at any time for the processing of your personal information for specified purpose, Customer Service.
9. Data Access and Deletion: You have the right to request access to, correct, and delete your personal data, and to ask for data portability, You may also object to or restrict the processing your personal data certain
Help Center.
These options provide users with control how their personal information used shared IMDb.
question about the privacy policy:
Figure 6: An overview of the privacy policy analysis agent interface showing the opt-out options available to users.
18

---

Published as a conference paper at ICLR 2025
A.2.5 POLICY SUMMARIZATION
Summarize the policy in one paragraph and keep the risky and most important information for me
The IMDb Privacy Notice outlines how personal Information is collected, used, and shared through its services. It collects information provided by users, automatically through cookies, and from third-party sources to enhance services, personalize experiences, and comply with legal ohligations. User data may be shared with Amazon, third-party service providers, and during business transfers, but IMDb does sell personal information. Users have contral their through account settings, including opting of interest-based and managing cookies. IMDb employs security measures to protect user data advises users to safeguard their account credentials. The policy also addresses data retention, user rights to access and delete data, and specific provisions for European and UK users under GDPR. Notably, services are not intended for children under 13, and users are informed of any significant policy changes.
Ask question about the privacy policy:
Figure 7: concise summary of the IMDb Privacy Notice, highlighting key aspects of data collec- tion, sharing, and user control.
19

---

Published as a conference paper at ICLR 2025
A.3 USER PRIVACY POLICY READING EXPERIENCE QUESTIONNAIRE
BASIC INFORMATION
1. Have you read privacy policy?
[ Yes
[ ] No
2. Have you ever used an Al agent?
[ ] Yes
[ ] No
EXPERIMENTAL GROUP
Please select the group you participated in:
J Group A: Directly read the privacy policy
] Group B: Used an Al agent to assist reading the privacy policy
I. COMPREHENSION ASSESSMENT
Based on your understanding after reading the privacy policy, please answer the following questions:
1. Types of Data Collected: What types of personal data are collected in this privacy policy?
(A) Contact information (e.g.. email, phone)
(B) Location information
(C) Browsing history
(D) All of the above
2. Data Sharing: Does the company share your personal data with third parties?
(A) Yes
(B) No
(C) Only with user consent
(D) Uncertain
3. User Rights:
According to the privacy policy, what rights do you have to manage your personal data?
(A) Access and modify your data
(B) Request deletion of your data
(C) Opt-out of data processing
(D) All of the above
II. USER EXPERIENCE ASSESSMENT
Based on your experience, please rate the following statements:
= Strongly Disagree
2 = Disagree
= Neutral
4= Agree
5 = Strongly Agree
1. Usability
a) I found reading/using the privacy policy/Al agent very easy.
1[ ]2[ ]3[ ]4[
]
I could easily find the information needed.
1[ ]2[
]3[ ]4[
]5[
c) The navigation and operation were intuitive to me. 1[
]2[
]3[
]4[
]5[ ]
2. Satisfaction
I am satisfied with my understanding of the privacy policy. 1
]2[ ]3[ ]4[
5
b) I am satisfied with the overall reading/usage experience.
1[ ]2[ ]3[ ]4[ ]5
[
3. Information Quality
20

---

Published as a conference paper at ICLR 2025
I believe the information provided was clear and useful. 1[ ]2[ ]3[ 14[ 15
[
b) The content helped me better understand my privacy rights. [[ ]2[ ]3[ ]4[
5
4. AI Agent Specific (Group B only)
a) The Al agent's responses were accurate and relevant.
[ ]2[ ]3[
]4[ ]5[
b) I found interacting with the Al agent pleasant. 1[ ]2[ ]3[ ]4[ ]5[
I am willing continue using Al agents understand privacy policies in the future.
[[ ]2[ 13 ]4[ ]5[
III. TIME EFFICIENCY
Please indicate the total time you spent completing the task (in minutes):
minutes
IV. COGNITIVE LOAD ASSESSMENT
Based on your experience, please rate the following statements:
Very Low
10 Very High
1. Mental Demand: How much mental and cognitive effort required to complete the task?
0 [ JIL ]2[ ]3[ ]4[ ]S[ ]61 171 ]8[ 191 ]
2. Task Difficulty: How difficult did you find task?
0 ]1[ ]2[ ]3[ ]4[ ]5[ ]6[ ]7[ ]8[ 19[ ]10
3. Stress Level: How much stress frustration did you feel while completing the task?
00 ] ]2[ ]3[ [ ]4[ ]5[ ]61 17[ ]8[ 191 ]
V. TRUST AND WILLINGNESS
Based your experience, please rate the following statements:
Strongly Disagree
2 = Disagree
= Neutral
= Agree
5 = Strongly Agree
believe the information read/received accurate.
I[ ]2[
]3[ ]4[ ]5[ ]
2. am confident in my ability to understand and manage personal privacy.
1[ ]2[ ]3[
[4[ ]5[
(Group only)
1[ ]2[ ]3[ ]4[ ]5[ 4.d. am willing to use Al agents in the future to help me understand other privacy policies.
3.c. trust the answers provided by the Al agent.
]2[ ]3[ ]4[ ]5[
VI. OPEN-ENDED FEEDBACK
1. What difficulties did you encounter during the reading/usage process?
2. How do you think the experience of reading privacy policies could be improved?
(Group B only)
3. Do you have any suggestions or comments regarding the Al agent?
21


Reviews:
[{"review_id": "lsS6G5BxrM", "reviewer": "Reviewer_rBni", "summary": "The authors assess the performance of OpenAI's GPT suite of LLMs on a set of text classification tasks using an existing privacy policy dataset. They compare the models' performance to baseline, non-LLM models from the dataset's creators. Additionally, they develop an LLM-powered agent to assist with reading and interpreting privacy policies, measuring its effect on comprehension and cognitive effort in a population of 100 users.", "questions": "Why was the user comprehension assessment only three questions? Do you think such a short assessment meaningfully measures user comprehension? How were the three questions chosen?\n\nDid you track instances of hallucination in experimental group user sessions? How frequent and severe were they? How correlated was user trust with the accuracy of the information provided by the agent?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: moderate. As the authors themselves note, there is substantial prior work on the problems with user comprehension of privacy policies and terms, and this paper is largely a straightforward application of a new model to an existing task. However, the agent the authors developed to assist users is a novel contribution, especially providing the ability to automatically surface opt-out mechanisms for users.\n\nQuality: moderate. It might not be earth-shattering, but the execution nonetheless seems thorough.\n\nClarity: high. The presentation of the experiments and analyses performed is very clear.\n\nSignificance: moderate. The effects of the agent on user comprehension are notable, though practical impact feels limited given that it still takes nearly 6 minutes to read a privacy policy.", "weaknesses": "The authors state that \"GPT-4o-mini, under zero-shot learning conditions without additional context, outperformed the baseline model on average\" on the Data Practice Identification task. However, the model suffered from consistently poor recall, which the authors do not meaningfully address.\n\nStatistical tests in section 6 not corrected for multiple comparisons.\n\nAs noted above, given that it takes nearly 6 minutes to read a privacy policy even with assistance, I feel skeptical that this approach would make a meaningful difference in the number of users that actually read privacy policies. Coupled with the models' poor recall and tendency to hallucinate, it seems likely that users would still miss the most important information in the privacy policy or even be presented with false information. It might be informative to conduct time-limited trials, where user comprehension is measured after e.g. a 30 second time limit. Another idea might be to measure the time it takes the user to be able to achieve an 80% score on a comprehension test (allowing multiple attempts).\n\nThe assessment of user comprehension is extremely coarse (three questions). A more fine-grained assessment might provide interesting insights for where further improvements (either to the agent or the UX) are most needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xlb7c3X7hi", "reviewer": "Reviewer_9U5o", "summary": "In the paper, the authors address a very specific issue of understanding the privacy policies of the users of various websites in a comprehensive manner from different aspects by LLM agents. It was built with an aim to help the general users of the websites about the privacy concerns and the policies of the personal or other data they share. The performance of the build LLM agent was evaluated on 100 people where half of them studied the policies by themselves and the rest used the LLM agent. The empirical results infer the users who used the LLM agent they got a better understanding of the websites user policies than the manually readers.", "questions": "1.\tWhat are the traditional models in page 2?\n2.\tThere is a missing citation in page 3, CNNs for text classification(?)\n3.\tFigure 1 was never described.\n4.\tIn page 6, what is the process of ensuring valid and relevant outputs? \n5.\tWhy different metrics were used to evaluate different tasks? The explanation along with a short description of the metrics will benefit the clarity. Same comment for t-test.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1.\tThe entire paper is well-written and presented the ideas in very clear way. \n2.\tThe authors explored a very specific and less explored use case of LLM agents in recent times.\n3.\tThe empirical analysis is comprehensive and make sense of the idea the authors proposed.\n4.\tMultiple open-sourced and close-sourced LLMs were used and compared their performance. \n5.\tThe built agents archive comparable performance like benchmarks and sometimes outperforms the baselines.", "weaknesses": "1.\tAs a whole, this paper is more like a building a new tool for the various website users, than a theoretical or technical presentations of ideas and experimental analysis. However, before making it available for the public usage, it needs several things to be considered, e.g., misinformation, hallucination, privacy leakage of company policies. \n2.\tIt doesn’t include any novel technical or theoretical contributions in terms of the finding the research gaps of LLMs agents to be utilized for specific use cases. \n3.\tUsually, LLMs agents for particular task are more likely to hallucinates its users. The risks of LLMs hallucinations were not explored in this paper in details. The built LLM agents might not work well under such vulnerabilities. At least a few results with analysis should have been discussed. Apart from this, LLM agents might face several potential security and privacy issues as described in https://arxiv.org/pdf/2407.19354; this paper does not explore or discuss such vulnerabilities. \n4.\tBuilding the agent only on one privacy policy dataset (though it is large) may not be sufficient to use the LLMs agent in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTpMVhElMO", "reviewer": "Reviewer_o4tn", "summary": "This study presents a competent investigation into the use of GPT family models for privacy policy comprehension support. However, it may not fully align with ICLR’s intended contribution areas, as it reads more like a system-oriented paper that might be more suited for an HCI venue.\n\nThe authors first assess the performance of GPT models, both in zero-shot and few-shot settings, and compare these results against traditional approaches. They conclude that GPT models exhibit reasonable performance levels in this context. Following this evaluation, the study introduces an LLM-driven agent designed to assist users in understanding privacy policies and completing related tasks. Through questionnaires, the study demonstrates that the agent helps reduce cognitive load and enhances both comprehension and user confidence.\n\nWhile this research is well-executed, I question whether its contributions are significant enough to justify a full paper. The study does not introduce new models or corpora, nor does it directly address gaps in current models related to privacy management, although limitations are acknowledged.\n\nMoreover, it is unclear how the proposed system substantially differs from other reading comprehension and summarization systems. A deeper comparison in this area could provide useful context for assessing the novelty of the approach.\n\nSpecific Comments:\n\nTables 1–3: The rationale for not including fine-tuned models is not sufficiently explained. Fine-tuning could potentially yield stronger baselines or comparative insights in this setting.", "questions": "Tables 1–3: The rationale for not including fine-tuned models is not sufficiently explained. Fine-tuning could potentially yield stronger baselines or comparative insights in this setting.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written and easy to follow.", "weaknesses": "While this research is well-executed, I question whether its contributions are significant enough to justify a full paper. The study does not introduce new models or corpora, nor does it directly address gaps in current models related to privacy management, although limitations are acknowledged.\n\nMoreover, it is unclear how the proposed system substantially differs from other reading comprehension and summarization systems. A deeper comparison in this area could provide useful context for assessing the novelty of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oePL0SkhvH", "reviewer": "Reviewer_hLXw", "summary": "This paper applies the  large language models (LLMs) to enhance user comprehension of privacy policies through an interactive dialogue agent.  The authors first demonstrate that LLMs significantly outperform traditional models in tasks like Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering. Building on these findings, they then introduce an LLM-based agent that functions as an expert system for processing website privacy policies, guiding users through complex legal language without requiring them to pose specific questions. A user study with 100 participants showed that users assisted by the agent had higher comprehension levels, reduced cognitive load, increased confidence in managing privacy, and completed tasks in less time .", "questions": "See above.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Applying the LLM in the digital privacy management is an interesting topic.", "weaknesses": "1. The main technical contribution of this paper appears limited given its current scope and the expectations of ICLR. It may be better suited for HCI venues such as CHI or IUI, which align more closely with the type of work presented.\n\n2. The current study appears to lack IRB approval, and details of the user study are insufficiently reported. Key information such as where did you recruit participants and what is the compensation for participants are missing. Without this information, it is challenging to ensure that the study’s conclusions are reasonable and generalizable to other populations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8hI4jBvbR8", "reviewer": "Reviewer_5oSx", "summary": "The authors construct an LLM-based tool for assisting users in interpreting and summarizing privacy policies. They evaluate the tool using several benchmarks for privacy policy comprehension/summarization from prior work. In a study of 100 users, they find that users report greater comprehension and greater ease of interpretation when assisted by the LLM tool.", "questions": "Did the authors explore different kinds of privacy policies in the user study — for example, are the gains from using the LLM tool greater when the privacy policy is longer / more complex?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality**\n\n- Includes a systematic user study of ML-assisted privacy policy interpretation — appears to be novel relative to related work, which relies mostly on benchmark datasets (though I am not familiar with this literature).\n- Constructs a system for applying state-of-the-artx LLM models to help users interpret privacy policy interpretation — including a broad range of features (interactive QA, classification, summarization) not unified in prior work.\n\n**Quality**\n\n- Appears to analyze the results of a user study competently and with appropriate statistics and measures of error, though some methodological details are missing.\n- Appears to correctly apply benchmarks from prior work to evaluate LLM agent performance.\n- From the details provided, the LLM tool seems to be well constructed and appropriate for the task.\n\n**Clarity**\n\n- Well written, and for the most part easy to follow.\n- Does an excellent job making clear the goals & contributions of the research.\n\n**Significance**\n\n- Provides a technological solution to a clear privacy & transparency issue for internet users.\n- Seems to present a clear, significant finding that an LLM agent could help lay users more easily interpret complex privacy policies——this could be a potentially useful workaround, barring systematic improvements to transparency requirements.", "weaknesses": "**Originality**\n\n**[Minor]** The idea to use ML tools to assist users in interpreting privacy policies is not new—in this sense the contribution of this study is marginal. Still, there is certainly value in evaluating this idea using the most recent large language models, and there is certainly value in conducting a study with actual users to see whether the tool really makes interpretation easier. I am not sure whether there are many user studies in prior work on this idea — perhaps the authors could clarify whether this is the first user study of its kind and, if not, whether it tells us anything new.\n\n**Quality**\n\nMissing methodological details make it hard to tell whether the empirical findings support the broad claims in the abstract and introduction, and I have lingering questions about some of the results.\n\n- **[Minor]** Section 3: Need a clear description of all the benchmark tasks to understand exactly what’s being evaluated — Section 3 seems to assume the tasks have already been defined. Examples:\n    - Section 3.1: What does it really mean to identify “User Choice/Control” or “Data Retention”, e.g., as a practice? Does this simply mean the privacy policy describes their user choice allowances or data retention practices, which could range from quite benign to quite egregious? How is this useful to a user?\n    - Section 3.2: What is the Choice Identification task? Is this the task described in A.1.3? Was this task defined in Wilson et al. (2016) too?\n    - Section 3.3: What’s “Privacy Question Answering”? (Or is it “Policy Question Answering”? Both terms are used.)\n    - Section 3.4: What’s in this dataset, as compared to the dataset used in the previous tasks? Who defined the “risky” sentences (what were the human-generated references for the ROUGE score)? Any examples?\n    - Section 4 provides a bit more detail, and the examples in the Appendix are somewhat helpful. Perhaps this Section could come before Section 3; or alternatively, move parts of Section 3 to the appendix, and just summarize the most important findings (GPT models perform better on X benchmarks) in a paragraph, using that space instead to better explain the tasks at hand.\n- **[Major]** Section 4: These results are striking — users seem to comprehend the privacy policies much more easily with LLM assistance! But there are some key methodological details missing that could determine how rigorous the results are:\n    - Did the Experimental Group also have a copy of the privacy policy that they could read directly during the task (not through QA), or did they rely solely on information from the LLM agent? From the Appendix, I infer they did have access to the raw text — do the gains decrease/increase if the user cannot cross-check the LLM agent responses with the raw legal text?\n    - Section 6.1: Where/how were users recruited? How many privacy policies did each participant review? How were the privacy policies selected — from one of the previous datasets? Did every participant review the same privacy policy? (How likely is it that these policies appeared in the training data — i.e. leakage?) Where/how was questionnaire administered? This information is key for determining how internally and externally valid these results might be.\n    - Was the study IRB approved?\n    - L393: What about racial, economic diversity in the sample? How well might these results generalize to other groups, especially marginalized groups?\n    - I’m surprised by the finding that the Experimental Group had *higher* trust in info scores than the control group — and I wonder if there’s an issue with construct validity for this question. The relevant question is (L978): “I believe the information I read/received is accurate (1-5).” Given that the control group had direct access to the privacy policies, why would they respond with a 2.6, on average, compared to 4.5 in the experimental group, since the underlying information (the privacy policy) is the same for both groups? My best guess is that the Control Group suspected the company was misrepresenting its privacy practices in its privacy policy, and answered based on their distrust in the company; I suspect the Experimental Group, on the other hand, responded based on their level of trust in the accuracy of the LLM agent’s responses. So the scores may not be directly comparable. The alternative is that using the LLM agent somehow increased people’s confidence in the accuracy of the privacy policy itself, which seems less likely but still possible.\n- **[Major]** Generally, it’s not clear how well the benchmarks measure the “correctness” of the agent’s responses — what is the ground truth for each of these tasks? The comprehension questions seem good, but they’re short, and not very granular — whereas the examples in the Appendix show LLM responses with much, much more detailed information about data practices. As the authors point out in the discussion, LLMs often produce incorrect and misleading text, especially when prompted for specific details that are less likely to be represented in training data. Can the authors say anything about the factuality of those more specific responses? How likely are those responses to contain falsehoods about the privacy policy that could mislead users? Can users easily identify false responses by cross-checking with the raw text or the QA feature?\n\n**Clarity**\n\nGenerally the paper is easy to follow, with the exception of the omitted methodological details listed above. Some **minor** points of clarity that would be worth addressing: \n\n- L132: Have ML techniques actually improved privacy policy accessibility in practice? Or is this just a summary of research, not practice?\n- L130: What is the OPP-115 dataset? Readers may not know.\n- L131: Broken cite here.\n- L136: What’s the difference between an LLM and an LLM agent? Is there a definition the authors can give? What makes this application an LLM agent, rather than just an LLM (the fact that the program scrapes hyperlinks, maybe)?\n- Fig. 2: Text is too small to read, and often cropped, so it’s not clear what the different elements are. Simple labels might be better.\n- Table 1-2: Suggest combining numbers side-by-side, so it’s easy to compare.\n- Table 2, L192: SVM F1-score has a misplaced decimal.\n\n**Significance**\n\n- **[Minor]** This is a neat idea, and it seems like it could certainly help users in particular cases. But to frame the significance more precisely, it would be helpful to comment on the scope of a technological solution like this (e.g. in the discussion) — there is a structural issue here with privacy regulations, and with GDPR in particular, that require companies to disclose information about their privacy policies but do not require companies to make that information, and users’ options with respect to their data, truly accessible. In a perfect world, this tool may not be necessary — companies could be required to produce interpretable “privacy labels” similar to Apple’s Privacy Nutrition labels. How does the performance of this LLM-based solution compare to other policy alternatives? (These questions probably cannot be answered in this study, but it is worth mentioning that a technological solution is not necessarily the best solution.)\n- **[Major]** Section 3: On a similar note, can the authors report any non-ML baselines here? How does a person do on this task, on their own? It seems less important to know how GPT models compare to BERT or other ML models, and more important to know how this method compares to what users would otherwise be doing in practice. (Unless those traditional models are actually being used by lay users in practice — that would be worth mentioning.)\n    - L094: “We provide empirical evidence of the superiority of LLMs over traditional models”:  I’m assuming these sentence refers specifically to *ML* models (would be worth clarifying).  But is this approach superior to the practical alternatives available to users/policymakers? Superior to things like Apple’s “Privacy Nutrition” labels? Superior to writing a simpler privacy policy? Superior to hiring a lawyer? It would help to be more precise with this and similar claims of LLM “superiority”—superior to what?\n    - Section 3: It seems like the GPT models perform better than traditional ML models, but stepping back, are these scores good enough to be relied on? For example, the recall scores seem really low here — as far as I can tell, the GPT models miss as many as 30% of instances of third party sharing, and as many as 84% of instances of “data retention”? Can this tool be used to balance precision and recall? Is this the right balance for this kind of task? Recall might well be more important to users in this kind of task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)