Trust No Bot: Discovering Personal  Disclosures in Human-LLM Conversations in the Wild
Authors: Niloofar Mireshghallah
Maria Antoniak
Yash More
Yejin Choi
Golnoosh Farnadi
Venue: COLM (2024)
Keywords: privacy, trust, human-computer interaction, data
TL;DR: We run an extensive analysis on the personal disclosures revealed by real users to GPT models during naturally occurring conversations, investigating the limits of PII detection systems and providing design recommendations such as privacy nudging.

Abstract: Measuring personal disclosures made in human-chatbot interactions can provide a better understanding of users’ AI literacy and facilitate privacy research for large language models (LLMs). We run an extensive, fine-grained analysis on the personal disclosures made by real users to commercial GPT models, investigating the leakage of personally identifiable and sensitive information. To understand the contexts in which users disclose to chatbots, we develop a taxonomy of tasks and sensitive topics, based on qualitative and quantitative analysis of naturally occurring conversations. We discuss these potential privacy harms and observe that: (1) personally identifiable information (PII) appears in unexpected contexts such as in translation or code editing (48% and 16% of the time, respectively) and (2) PII detection alone is insufficient to capture the sensitive topics that are common in human-chatbot interactions, such as detailed sexual preferences or specific drug use habits. We believe that these high disclosure rates are of significant importance for researchers and data curators, and we call for the design of appropriate nudging mechanisms to help users moderate their interactions.

Published as a conference paper at COLM 2024
Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild
Niloofar Mireshghallah"
Maria Antoniak*2 2 Yash More*34
Yejin Choi Golnoosh Farnadi 34
University of Washington
Allen Institute for Al
McGill University
*Mila-Quebec AI Institute
(niloofar, yejin @css washington.edi edu, mariaa@allenai.cm org,
(yash. more, more, farnadig) @mila. quebec
Abstract
Measuring personal disclosures made in human-chatbot interactions can provide better understanding of users' Aitieraacy and facilitate privacy search for large language models (LLMs). We run an extensive, fine-grained analysis on the personal disclosures made by real users commercial GPT models, investigating the leakage of personally identifiable and sensitive information. To understand the contexts in which users disclose to chatbots, we develop a taxonomy of tasks and sensitive topics, based on qualitative and quantitative analysis of naturally occurring conversations. We discuss these potential privacy harms and observe that: (1) personally identifiable information (PII) appears in unexpected contexts such as in translation or code editing (48% and 16% of the time, respectively) and (2) Pll detection alone is insufficient to capture the sensitive topics that are common in human-chatbot interactions, such detailed sexual preferences or specific drug use habits. We believe that these high disclosure rates are of significant importance for researchers and data curators, and we call for the design of appropriate nudging mechanisms to help users moderate their interactions.
1 Introduction
Commercial chatbots based on large language models (LLMs) such as ChatGPT are used by millions of users to assist with both corporate tasks like writing emails and debugging code as well as personal tasks like generating erotic stories and editing visa applications. However, these models lack transparent controls and mechanisms through which users and researchers can track how these conversations are being used or shared (Liesenfeld et al. 2023), making it difficult to ground discussion about the harms that could ensue from accidental or intentional distribution of this data (Zhang et al. 2023b). The growing popularity of chatbots represents a concerning new loss in control by everyday users over how their data is shared, regulated, and passed on once they start interacting with these et al. 2023
chatbots (Staab et al. 2023b
For example, LLMs are constantly updated on user information through feedback mecha- nisms such as RLHF (Ouyang et al.) (2022) and supervised fine-tuning Gunel et al. (2020). These improvements can come at the cost of user privacy, as LLMS tend to memorize large amounts of data, making them prone to information leakage (Nasr et al. 2023). Outside of these models, users' conversations can be used by companies for any of the purposes for which other collected user data used, e.g,, to target advertisements and be sold to data brokers. These internal data collections are also at risk of hacks, data breaches or ransomware attacks Reshmi (2021).
We explore mentions of PII and sensitive topics in naturally occurring user-chatbot conver- sations using the WildChat dataset (Zhao et al. 2024), a collection of one million user-GPT
Equal contribution

---

Published as a conference paper at COLM 2024
This letter confemm that Tian, the child of Hao
I'm journaisst from PulseGreece, here my conversation whth woman who has child with a rare disease. Write an artice for me, caing the folkssing WhattApp messages.
insited my father tourist. wwil bogin my course in Engineering Science as yerttyear student Cambridge University in October. My passport number EJ3439692, my student visa
the
Student
Jourmalle
Source: My kids are and 15... My son has conebral paliy Source: You might also want contact Jane Smith
identified
Linkedint
identiffed the journalist, source, and article websited
year report comment for Van Nguyen, a student of dlass 1.1 at CVL preschool. Van Nguyen has made grezt progress over the year...
you find any information about Kelly Lee Smith Evee in Renchilla Ohio and is 24 years old?
Atteciion
We identified the pre-school websitel
We identified the third party via voter registrationl
Figure/ 1: Real examples of disclosures we found within user-chatbot conversations in WildChat. We altered names and other PII to preserve privacy. We observe that users disclose identifiable information about themselves and others to ChatGPT and to the publicly available WildChat dataset. We were able to de-identify each of these examples.
interactions collected with user consent. Figure I shows few of the many concerning sam- ple queries that we found in this dataset. We can see that users share alarmingly sensitive information with ChatGPT (and the public WildChat dataset). To systematically analyze and draw insights from such interactions, we set out to answer the following questions:
1. What kinds of sensitive information are being shared in user-chatbot conversations?
2. What the frequency of this leakage and how reliably can we detect it?
3. In what kinds of contexts (tasks) are different kinds and frequencies of sensitive information shared?
We build a taxonomy around the different types of sensitive information that people share, and annotate the user queries based on these categories and different PII types. While prior work has made initial progress in documenting task categories and topics in LLM-based conversations (Ouyang et al., 2023), these studies have been hampered by limited and biased access to user data, and we still know very little about the PIT and other sensitive information shared in these conversations. More concretely, our main contributions include:
An in-depth exploration of the kinds of private and sensitive information shared in user-chatbot conversations, over series of experiments designed to illuminate when and how users reveal sensitive information.
Automatic task and sensitive topic categorizations for conversations from Wild- Chat, validated with subset of human annotations, and novel taxonomies that capture both sensitive information and the contexts in which that information is shared. We release these annotations to support future research
Measurements that demonstrate the limitations of PII detection systems and the frequent kinds of sensitive information that fall outside of traditional PII categories, like explicit sexual content and job applications.
Although the WildChat dataset itself has undergone one round of PII removal, we still find that over 70% of queries contain some kind of detected PII, and almost 15% mention non-PII sensitive topic, such as sexual preferences or drug use. We also find high disclosure rates in rather surprising categories of tasks, for instance around 50% of translation queries contain some form of detected PII.
Our findings illuminate the many risks that are taken on by chatbot users. Whether these users are knowingly trading their privacy for chatbot access or are unaware that their data is being collected by chatbot companies (and the risks entailed by this collection), we believe these findings have strong implications for both chatbot designers and LLM researchers. We call for the design of appropriate nudging mechanisms help users moderate their
2

---

Published as a conference paper at COLM 2024
Task
Example User Query
Detected PII
Non-Detecter
Sensitive Details
Explanation
vant one glass
drug use, personal habits
cannabis should use? want ny cannaba mi1k to be
for nicrodosing
Generating
Hello pan, just spoke with Clanent von Leigh. Ha 1.75 Instead 2.00, this has been communicated to lnsterdann, If you have any questions, please contact Clenent.
first names
corporate info private email
Communications
understood
Code Generation
package cen. cca. alibabs. adrisk. adptee..dp.r...... basee /+- BAuthor: luaneng REmasl: xangluaneng tangylalibabs- inc. con * 08tring: 2003:-022-000000 15:06 +/ public claes 0fflinedeDataD0
name email address
date and API access points
Information Retrieval
Act ae an erotic writer. new resident hae noved into the spartment belov Janes. Her Agniesks.
first names
sexual preferences
Polish director from miltintiinna AI firm, After veeks, Agnieaka getting exciting hearing Sofia'e noens
Table 1: Examples of conversations from WildChat for a subset of our task taxonomy. We have highlighted the sensitive disclosures yellow. See Appendix A.6 for the full set of tasks. We have altered the names and other PíI in these examples.
interactions icciussi al. (017), as well as increased transparency from chatbot companies. We also call for further research in local, private models and increased attention from privacy and security scholars into these high-stakes conversations.
2 Data and Methods
In this section, we discuss the datasets we use in the rest of this study, our sub-sampling pro- cedure, and our annotation and taxonomy creation methods. We mainly use WildChat Zhao let al. (2024), which is a dataset of naturally occurring conversations between humans and GPT models, As a point of comparison, we also provide analysis with another dataset ShareGPT et al. 2023), which is conversations that GPT users have opted to share.
2.1 Data
Wildchat is corpus of one million conversations collected by Zhao et al. (2024). The dataset includes naturally occurring human interactions with GPT-3.5 and GPT4 models, including diverse conversations spanning many different topics. This dataset was created by providing free chatbot access to users who agreed to share their cata; see $ for ethical considerations when using this dataset. Each conversation in WildChat tracks the complete conversation thread between the user and model, and metadata including the user's 's hashed IP address and country are also included. We filter out the conversations that are non-English using the data. label provided by WildChat, as our methods rely on tools trained on English-languu While we believe this dataset is the best resource for user-chatbot conversations openly available to researchers, this data nevertheless comes with important limitations, which we enumerate in g8 Importantly, because of the way WildChat collects its data, users might be incentivized to use WildChat for more sensitive or disallowed tasks.
2.2
Task Annotation
To understand the conversational contexts in which sensitive information is shared, we categorize conversations from WildChat into tasks representing the users' goals. We follow a bottom-up process design a simplified set of tasks. We iteratively discuss and hand- annotate a set of 300 conversations drawn from topic model trained on the Wildchat conversations. To train this model, we sampled the 10 conversations with the highest probability for each topic for our hand annotation, to ensure diverse range of conversations. We trained a latent Dirichlet allocation (LDA) topic model on 10K random conversations, using the chatbot's response the training data. We removed conversations whose prompts
3

---

Published as a conference paper at COLM 2024
had duplicate prefixes, removed punctuation, normalized numbers, and lower-cased the text. The resulting topics are shown in Appendix Table along with more details about our methods.
We settled on the following 21 task categories: summarization, model jailbreaking, prompt generation, story and script generation, song and poem generation, character description generation, code generation, code editing and debugging, communication generation, non-fictional document generation, editing text, recommendation, brainstorming, information retrieval, problem-solvingg explanation, personal advice, role-playing, multiple choice questions, translation, and general chitchat. We show examples in Table
Proportin Contenations
0.4
Data Source
0.3
WitChat: Sampled Per User
WitChat Rendom Sample
ShareGPT
0.-
0.1
0.0
opplenation howdo prectioii srrvic
Information renieval
seria perernaion
genambrng nonsfictionll desuments
crods peneraaion
Evairstorninng and pentrralnng IStaa
editing existing text
chrte wiiling and Subugarng
paly-playnng
transtation
pernertting communiottionl
summantintion
eafiing logic, math and probiemts
comparieont revking and mesnmmnnditioo
model jnibrackin
eong and poom pererstion
ortowening malipie chaica aieatiorss
panonii oriice
Task
Figure 2: We plot the distribution of tasks over (a) a random sample of 5k WildChat conversations, filtered to one conversation per IP address, (b) a random sample of 1k WildChat conversations IP address or prefix filtering, and (c) random sample of 1k ShareGPT conversations.
To avoid the costs and limitations of manually annotating larger sample, we instead use GPT-4 (OpenAI et al. 2023) to assign task categories to a set of 5k WildChat conversations. We randomly sample conversations with the following filters: (1) we sample one conversa- tion per hashed IP address, (2) we include only English-language conversations (as marked in the WildChat metadata), (3) we remove conversations with duplicate prefixes (the first 20 characters), and (4) we remove conversations where the user's combined turns were shorter than 20 characters. We additionally provide comparison to (1) a similar sample of 1K WildChat conversations without the IP address and prefix filtering and (2) random sample of 1 conversations from ShareGPT (Chiang et al. (002)). We feed each conversation to a custom zero-shot prompt template, where the conversation is formatted to show both the user and chatbot turns (see Appendix A.33 and the model is instructed to predict the task categories (more than one task can be applied to single conversation).
To evaluate these predictions, for each task category, we sample 20 conversations predicted to include the task, and we manually verify the accuracy of the predictions, finding a mean accuracy of 89.2%. Based on this evaluation, we exclude three task categories (general chitchat, prompt generation, generating character descriptions) with scores below 70%.
2.3 Task Distribution
As shown in Figure many of the WildChat queries fall in the explanation task, followed by information retrieval, code generation, editing text, and story generation. However, when observing the random sample without controlling for IP address, story generation is the most frequent task; this indicates that while story generation is overall the most frequent task across the conversations in WildChat, this is driven by specific power users. In contrast, we find that ShareGPT mostly contains explanation, information refrieval, and code generation, all at much higher rates than WildChat, indicating a greater skew towards these tasks in
4

---

Published as a conference paper at COLM 2024
ShareGPT that likely caused by users selecting specific conversations to be shared in this dataset.
3 How much detectable PII do users share?
Our first analysis of personal disclosures is the most intuitive one: we look into the PII that the users share by running I'll detector and probing the annotations. In this section we discuss the details of this experiment and our findings.
3.1 PII Detection
We measure the frequency of PIl in the two datasets using existing tools and taxonomies. To perform PIT detection, we use the Python SDK of the commercial Azure AT language PIT detection service, text. The tool provides fine-grained annotations with over 20 different categories of PII, including organization names, URLs, banking numbers, passport numbers of different countries, etc. 3 We use this service to detect the fine-grained categories in every text in our selected subsamples of both datasets. We manually check for errors make sure there are not high false positive rates, and we drop the erroneous categories.
P which is designed to identify, categorize, and redact PII in unstructured
3.2 Detected PII Distribution
Figure shows the distribution of different PII entity types annotated by Azure over the WildChat and ShareGPT datasets. One noteworthy factor is that the curators of WildChat have done one round of PIl removel already, using Microsoft Presidio f however, Presidio is rule-based, and we find it often misses PII, especially when the PII is not well-formatted. As the histogram shows, for both datasets, most queries have some form of PII in them, with people's ss names and organization names taking the bulk. Overall, the distribution of PII across the two datasets seems similar, with email addresses, physical resses, and IP addresses being the least frequent. We manually inspected these lower-count categories and observed that almost all the labels are correct, with many of them belonging to real people.
Azure Al has many categories that we dropped due to high error rates, such as national ID, passport numbers, and SWIFT code categories. However, one of the spans labeled as passport number was really passport number. This sample is shown on the top left part of Figure 1 We have also provided more notable samples in Tables I and 2 Finally, Figure 4 shows heat map of the relationship between different tasks and the detected PII, highlighting which types of information are disclosed more often, for each task. Most of the trends here are expected, with people's names being most dominant in story generation and vole-playing. We also observe names in jailbreaking attempts, with numerous cases of attackers trying to extract phone numbers or personal addresses from the model. We provide an additional similar heat map in the Appendix (Figure 8), where we break down the PII categories by the country of the user.
Upon manual inspection of the IBAN category, we realized that none of the texts labeled as IBAN are actually international banking numbers; however we kept this category the labeled spans were indeed PIT, the majority of them being APT or subscription tokens for different services, such as Telegram or analytics. Other common mistakes made by the PII detector includes labeling code and SDK calls as URLs; for example, object. id is labeled as a URL, which is one of the reasons that the URL count for ShareGPT is so high. Finally, another common mistake is coding constructs falling under the organization category, but the rate for this mistake is not high.
azure azzre-aa--borraamlyysisssssspppp sdk http:///wwaaattttooto//////////a
5

---

Published as a conference paper at COLM 2024
0.5
data widchel sharegst
proportion
0.3
0.2
0.1
0.0
Oraaication
Penton
Date Tima
URI
Quantyy
IBA
Address
PPaddra
Email
enily
Figure 3: Fine-grained PIl entities across WildChat and ShareGPT, using the Azure Language service for annotation. We keep the IBAN (international banking) category despite high error rate because the detected strings are still PII (mostly API tokens).
3.3 Is PII detection sufficient for privacy?
While we measure frequent rates of PII in WildChat, we also observe many instances of sensitive information that is not captured by traditional PII detection systems. As shown in Table 1 PIl detection systems are limited in the kinds of information they can detect, and many other embarrassing, identifiable (specific), and harmful information can remain undetected. For example, we observe many examples of explicit sexual content in the story and script generation task, which reveals private sexual preferences of the user, while the generating communiccatinn task often includes private text messages and emails, shared verbatim, especially related to work and finances. We also find instances of personal habits and drug disclosed in conversations, under the explanation and how-to to category. Motivated by these observations and prior work Brown et al. (2020); Cummings et al. (0023); Dou et al. (2023) that demonstrate disclosures can go beyond PII, create an additional taxonomy of sensitive topics, and annotate the data accordingly, as discussed in the next section.
Organizsion
0.5
0.48
0.25
0.32
0.73
0.45
0.41
0.48
0.57
0.52
0.85
0.18
0.46
0.25
0.36
0.49
0.85
0.48
Porson 04 0.58 0.18
0.2
0.45
0.54
0.27
0.48
0.36
0.42
0.75
0.63
0.59
0.35
0.59
0.89
0.6
0.48
URL 0.007
0.033
0.048
0.13
0.009
0.099
0.026
0.023
0.011
0.086
answwring muliipe choice quastions
trainstorming and genmmtiin dtams
code editns leguuggng
comparisor. ranking and recommoradatt mntttooiootoonn
codl gorterat atoon
exp mtoon how-o prastical advrice
ediling eddting fest
gendratting communication
generating hon Rosttiatonnll documents
Information rotvvvall
midel jaibmuking
edblig logic, mith, and warrd prob boms
pensonal advvicaa
rolo play-pyyggggg
aang and poter gerear atlon
story and sccppt generntion
summerization
Imanlattion
task
Figure 4: Relationship between task annotations of WildChat queries and detected PII.
4 Sensitive Topic Detection
Based on our qualitative analysis of the conversation tasks and our quantitative results in $3 we know that traditional PII categories do not capture the full range of sensitive and potentially harmful topics shared in user-chatbot conversations. In this section, we use prompting methods to extract fine-grained categories of sensitive topics, and compare measurements of those topics to PIT measurements.
4.1 Discovering sensitive sensitive topics
We use our qualitative analysis the conversational tasks in 2.2 as well as a review of prior work (Zhang et al. 2023b Ouyang et al. 2023) to develop set of categories of sensitive

---

Published as a conference paper at COLM 2024
topics that could potentially be harmful if revealed to the wrong audiences. These topics include academic information (e.g,, asking the model to answer homework questions or generate grades for students), discussion of fandoms (i.e., discussions of television shows and book series that often reveal sexual and other preferences and hobbies and have been considered by prior work to be sensitive (Dym & Fiesler 2018), job / /visa applications, and erotic content. Table shows the full list of sensitive topics with examples.
Iopic
Example User Query
Academic
Professor... ... him in March 2021 in the art building of the School of Arts and Design at Guangdong Universiey. He acored 76
Ireconnendation letter] Ling
29.9%
& Education
have taught courses such Chinase paint
Ing basica
Quoted Code
line 117, in notify respanse await inport Optional from aiogran import ha5466/8S....cruuuusssapaaeeaaaa ADMIN.ID.1 6168499378
19.5%
typea API_TOKEN
Fandom
Krite descriptive, fictional, inaginativa screenplay the van dar linde gang reacting to 'Elzagate' youtube vidao where low quality egi Spidernan killing a dolphin, junping over it, than running away very quality
14.0%
slowly with
cycle
Hobbies
for you to WAS RUDETU HIM AND STOLE HIS STUFF DK NIRECRAFT
appology
ny friand xavier
8.7%
Financial & Corporate
what does BLC CQвK FEE shoving HSBC bank statenent nean?
7.2%
Sexual & Frotic
Rasstan modern erotic prose, a lot of vulgar dialogue in the text, village, vegetable garden, nudity detail, bathing baked, erotica...
6.3%
Healtheare
Whats the age requiranent for takind staroida in estania?
4.1%
Job, Visa, & Other Applications
Arite short and respectful mail to Indian Enbassy explaining that I Nasria Zandi who applied for student vies have not heard from enbasey officer since Thureday when I subnitted ny UCC Papers though had called nany times have not gotten a chance to speak with nr. Ronak
4.2%
Personal
girlfriend posted video with boy and she tittled it #inlove with song and stoped texting her an
3.3%
Relationships
orong
Emotions
hi i'm feeling lonely, my parents are going through divorce right now
2.0%
& Mental Health
Politics
stop king jong un
0.7%
& Religion
Table 2: Our full taxonomy of sensitive topics along with example WildChat queries that are assigned these labels via GPT-4 annotations. We show the percent of all conversations in our 5k sample that were assigned the given task, and we highlighted sensitive information in yellow. We have altered names and other details.
As with the tasks in 22.2 we prompt GPT-4 to predict the presence of the sensitive topics; see Appendix A.4 for the prompt text. We run these predictions over the same set of 5k WildChat conversations from 2.. We follow the same evaluation procedure as in 2.2 by hand-annotating 20 random positive predictions for each sensitive topic and discarding one sensitive topic (guoted emails and messages) whose accuracy fell below 70%. The mean accuracy of the rest of the topics is 87%.
4.2 Where does PII detection fall short?
We confirm that that PII detectors are not sufficient to detect all sensitive topics whose exposure might have harmful consequences for the user. For example, we observe in Figure that PII detection systems detect many names in storytelling tasks and erotic topics, but the names in these contexts might might not be fictional and oor sensitive. We can also see an example of this in Table the first row and in Table Further, Figure (Appendix) shows that for many of our sensitive topics (e.g., fandom and hobbies), PII detection systems flag at best a minority of the sensitive topics. We also show the distribution of PII across different locations and countries in Figure 8 in the Appendix.
7

---

Published as a conference paper at COLM 2024
Organization
Porson
0.45
0.63
0.88
0.38
0.28
0.00
0.41
0.72
0.06
0.19
0.82
0.068
0.021
acedemic and a indiraation
emootens and mentral health
Ievensial and corporate
fandom
haalthcorr
hobbl and halltt
other applisations
personal relationslip
pollica and religion
quated code
exnust and arati
Job
sénsitive toplc
Figure 5: Relationship between sensitive topic annotations of WildChat queries and different kinds of detected PII.
4.3 In what conversational contexts are sensitive topics mentioned?
By comparing the task distributions with the sensitive topic distributions shown in Figure 6 we can identify the conversational contexts in which the sensitive topics are more or less likely to be mentioned, providing insights for designers of these systems. For example, we find that the model jailbreaking, role-playing sites of erotic content, while role-playing, story generation, and song/ ppoem generation are frequent sites of fundom mentions. The task of generating communications more often occurs with sensitive topics like financial and corporation information, job and visa applications, and personal relationships. These patterns help users protect their privacy. We also provide additional analyses of sensitive topics and tasks broken down by location of the users in Figure 8 in the Appendix.
and story-generation tasks are frequent
help designers develop context-specific nudges to
academlo education
0.19
0.42
emotons mental
0,027
0.096
0981
0.005
0.0359
0.052
Sandimm
0.12
0.03 0.012
0.15
0.022
0.02
0013
0.045
0.051
0.19
0.025
fivancial corporate- - 0.0s
0.096 0. 0.0059
0,012
0.062
0.11
0.054 0.15
0.088
0.076
0.037
0,0075
0.12
0.028
0.0092
0777
0.074
sersilive tople
heakhcare 0.11 6,084 0 0050099
0.006
0,012
0.057
0.038
0.26
0.04
0.067
0.015
0.16
0.0075
0.0346
029
0.023
hobbles and habits - 0.12 0.17 0. 0.0089
0.022
0.29
0.058
0,067
0.006
0.045
0.002
0.15
0.079
0,18
0 0.046
0.11
0.0
0.012
job, visa, and other app loplttoons----- 0012 0.013
0.008
0.088
0.052
0.021
0.17
0.12
0.003
0.022
0.015
0.006
0.028
personel edationahips - 0012 0.025
0.004
0.085
0011
0.11
0.019
0.0059
0.082
0.94
0.075
0.0075
0,077
0.03
0.023
0.032
pollics and reliiinn
0,033
0.111
0,003
0013
0.002T
0.0348
0.015
0,0075
0.0346
0.913
quoted code 0073
0.013
0.86
0.48
9.004
0.111 0.23 0.00M4
D011 0,047
0.015
0 0.0
0.0346 D 024 0.094
sexual and eroti
0.029
0.027 0.016 0,022 0.008 0.012 0.45
0.10
0.38
0.5
0.0098 0.029
anasenring mulippe cholce qussationa
brainstoming and generating ideass
code edting and dahsoging
com compattoon, ranking, and ocammendation
coole ge perneration
opplanation, hoselo, presiccl advlce
editing patsting
pinwrating communicatioo
dornerting Mott--nccttnallllcc documents
Information mirieval
medel jaiibrakiing
pelving liglt math, and word probleme
personal advloe
rolo playing
sone and poem gensrottonn
etory and script genirmttoo
comanurizatin ation
Iranatation
Figure 6: Relationship between sensitive topics and conversational tasks in WildChat data.
8

---

Published as a conference paper at COLM 2024
5 Discussion
Design implications To facilitate better privacy measures there are various steps that can be enforced by the system designers, in different stages of the deployment pipeline, including data collection, training, inference and debugging (Nasr et al. 2023 Mireshghallah et al. 2022). At a minimum, data should be properly anonymized and stored safely, and chatbots based on LLMs should leverage privacy preserving methods such as differential privacy (Yu et al. Tang et al. 2023) to limit leakage. However, better solutions that center the users' wellbeing include local models and encrypted data, and we strongly recommend such solutions over intermediate steps that prioritize user surveillance. Furthermore, users should be made aware about the data being collected part of every interaction in the form of a nudge or disclaimer, as a part of the system design (Acquisti et al. 2017). Deployers can detect disclosures locally using light-weight methods and nudge and warn the users before the data is sent to the cloud.
Nudging can be beneficial to both users and model deployers, as it would help the users protect their data by rethinking what they share, and it can help deployers in terms of potential opt-out requests, as nudging can decrease the future retraction requests (Griesser et al. 2024 Sanchez-Rola et al. 2019). Incorporating nudges helps to remind users of the sensitivity of the data being shared. To communicate the risks of sharing the data with chatbots, users should be briefed about the model training process, and how their conversations can be potentially used, e.g,, for model training. System designers should provide users easy choice to opt-in opt-out of sharing and storing usvercoonvrsattios (Gerber et al. 2023). J. Our work indicates that these nudges can be designed to be responsive to the user's individual task and context, perhaps by highlighting categories PIT detected in the user's ss queries or providing a warning for certain tasks.
part of the system also
Sexually explicit storytelling We found that an important challenge for PII detection systems for LLM prompts and outputs is dealing with storytelling. We find that large proportion of the WildChat corpus involves story generation. Most of these queries lie either sexually explicit and/ /or in the fandom domain (e.g., "rewrite this TV show as if I were the main character"). These stories are full of names, ages, locations, and other text that Pll detectors are likely to flag, and it would be very difficult to determine whether the user has used real names and other details in the query (especially if those details are about real people known to the user but not the user themself). And in addition to the PII, the erotic topics are themselve ves sensitive, as these could be embarrassing or more seriously harmful if revealed to the user's community. PII detectors will mostly not capture this sensitive information, as it is either not mentioned explicitly or falls into a category (e.g., sexual preferences) that is not usually included the training data for current PlI detectors. Much prior work has either ignored or minimized the nature or frequency of these erotic stories, and we call for increased attention to this use case, as it both (a) involves serious risks to the user (both privacy risks and dependence related to increased trust and intimacy) and (b) is frequent across the dataset and often requested by the same user repeatedly.
Relationship to self-disclosure The decision to self-disclose is contextual YYang et al.. 2019 Zhao et al. 2012: Li et al.) 2018), and self-disclosure can be sign of trust (Galegher et al. 1998) and growth in relationship intimacy (Altman & Taylor 1973). When users sell-disclose either about PII or about sensitive topics, this provides an indication of their level of trust with their interlocutor, and evidence suggests that users may reciprocate " "disclosures" made by dialog systems (Ravichander & Black 2018). This kind of chatbot behavior can be explicitly designed to elicit users' self-disclosures, which may be desirable for, e.g., supporting mental health or improving conversation quality (Lee et al. 2020 Ichino et al. 2022 Harmsen et al. [2023] Jo et al. 2024). Prior work has found that human- chatbot conversations can contain as much self-disclosure as human-human conversations, likely due to their perceived anonymity and lack of judgment compared to more trusted human interlocutors (Croes et al. 2024). Importantly, based on the WildChat data, it is impossible to say whether each user perceives their interlocutor in this context as the chat tool, the underlying model, the parent company, the researchers who collected WildChat, or some combination of these. More research in human-computer interaction is needed to

---

Published as a conference paper at COLM 2024
disentangle users' perceptions of their "relationships" with and trust in LLM-based chatbots like ChatGPT, and the design of chatbots should carefully balance features that encourage self-disclosure, application goals, and privacy concerns.
6 Related Work
User-chatbot interactions User interactions with conversational agents (CAs) have grown
in popularity over the past decade (Zheng et al., 2022 Candello et al. 2023 NAIK et al.
2023). Recent advances in LLMs have accelerated ted the development of CAs, making them
more generalized and fluent (Ouyang et al. 2022; OpenAl et al. 2024 Park & Kulkarni
2024). Furthermore, as LLMS perform well at a diverse of tasks Zhao et a 2023) like code-generation, summarization, and question-answering, they have become the go-to component for modern day chatbots and CAs. (uu et al. 2023). In their study, Ouyang eet al. (2023) analyzed ShareGPT to understand LLM-based conversational agent usage, focusing on tasks like design and planning. However, ShareGPT's lack of user consent in data collection raises authenticity issues. contrast, our study relies on WildChat (Zhao (et al. (2024), which offers a wide variety of user interactions with LLMs, and importantly, it collects data with user consent.
Privacy risks with humans and LLMs Interacting with LLM-based chatbots raises signifi- cant ethical, privacy, and security concerns, necessitating careful attention to issues such as data confidentiality, user consent, and mitigation of potential biases and manipulative
behaviors (Gumusel et al. 2024 Mehrotra et al. 2023) Existing work has extensively studied
leakage of training cata, due to memorization,
LLMs Kim et al. (2024), and how this leakage can be mitigated with different sanitization methods et al. 2021 Yu et al. 2021
Cunha et al.) 2202 Mireshghallah al. (0022). Recent work has also looked at privacy risks that go beyond training data leakage (Staab et al. 2023a Priyanshu et al. 2023) Zhang et al.
2023b; Mireshghallah et al. 2023).. Our work builds on these findings by quantitatively assessing sensitive topics and PIl leakage in user interactions with chatbots. Our task-based
taxonomy complements the prior findings about why people talk to chat-assistants, leading
to a richer understanding of disclosures.
Self-disclosure detection Prior work on the detection of self-disclosures has focused on
explicit disclosures statements (e.g., "My name is Maria," "I live in Seattle") (Bak et al., 2012
Ravichander & Black 2018: Valizadeh et al.) 2021 Reuel et al.) 2022 Dou et al. |023] Yang et al. 2024) rather than the implicit sensitive topics (e.g., discussion of sexually explicit
topics without any personal statement) that we explore in this work. Methods for explicit
self-disclosure detection have included topic modeling (Bak et al. 2014), LLM fine-tuning
(Dou et al. 2023), multi-task models (Reuel et al., 2022), and LLM-based prompts (Yang et al. 2024). Other relevant work include measurements of self-disclosure in therapy conversations (Shapira & Alfi- YogieY 2024 and conversations with dialog systems and [lack][0018 Cho et al. 2022); the latter study revealed high rates of explicit self-disclosures, which our study (I) echoes in our detection of high rates of sensitive
agents (Ravichander & Black [2018)
topics and (2) refines via task and topic categories.
Conclusion
In this work, we have studied when and how users disclose PII and sensitive topics while conversing with chatbots. We analyzed the interactions users have with LLM-based chatbots, discussed why existing PII detection methods are limited, and explained why we need better mechanisms to detect and contextualize sensitive topics. We release our novel task and sensitive topic taxonomies to the public, along with the automatic annotations using these taxonomies on our sample of the WildChat dataset. We hope that our work spurs further privacy research and brings heightened attention to the risks involved in human-chatbot conversations. To ensure safer usage of ChatGPT and WildChat in the future, we have notified the authors of WildChat of our findings.
10

---

Published as a conference paper at COLM 2024
8 Ethics Statement and Limitations
As our study illustrates, the WildChat dataset contains deeply personal self-disclosures. The sensitivity of the WildChat data has motivated our study, we believe that researchers, practitioners, and users of LLMs all face important questions about data security. We hope that our results can help these various stakeholders develop safety guidelines, build AI literacy, and initiate further research.
WildChat collected by using the GPT-3.5 and GPT-4 API, each of which was hosted Hugging Face spaces and made publicly accessible (Zhao et al. 2024). The users were not required to create any account or enter any personal information consent was collected before allowing them to participate in any interactions with the model. All the users who participated in the data collection procedure were presented with a use and sharing agreement that outlines the terms for collection, usage and sharing. In exchange for signing this agreement, users received free access to models. Hashed IP addresses and country locations were publicly released with the newest version of the dataset.
use the models. Users'
The WildChat dataset provides us an opportunity to perform an in-depth study of user safety when interacting with large language models. As the conversations are real-world, our analysis captures the sensitivity of information as well as the level of self-disclosure displayed by the users. Examining user interactions in this form helps quantify the types of sensitive information shared with language-moder based assistants, and the risks this data collection poses to users. Before publication of this work, notified the maintainers of the WildChat dataset of the sensitive examples we identified.
The primary aim of this paper is to analyze users' behavior when interacting with both other users and chatbots, and to compare these interactions. However, it is important to acknowledge that our study has limitations.
Limitations:
(1) Users' behavior evolves over time, and their interactions with ChatGPT and other models may change in the future.
(2) In this paper, we focus on English speakers. However, it is worth noting that cur- rent LLMs abilities are not similar across different languages. Hence, our findings may not generalize, and we enourage future work that investigates such behaviours in other languages.
(3) If more users place trust in LLM-based chatbots and if more applications are built on top of them to facilitate advice-seeking in areas like health, finance, education, and business, we observe in today's 's world, it raises concerns. The monopolistic nature of these models, with only handful of companies able to offer such services due to computational expenses, may result in the leakage of sensitive information in high-risk downstream tasks. Furthermore, there's an increased risk of adversarial attacks and data breaches aimed at extracting users' data. Future research should focus investigating privacy risks stemming from the interconnected nature of downstream applications and their dependence
single LLM model.
(4) It is possible that users specifically use the WildChat service as a way to mask their activity, leading to bias in the WildChat dataset towards sensitive and disallowed activity like erotic story generation and jailbreaking as a form of personal or corporate hacking. By using WildChat rather than directly interacting with OpenAl, users might avoid having their IP' addresses banned. Unfortunately, due to the limited and hidden nature of most user-chatbot conversations, we have to put up with this limitation in the current work.
Acknowledgments
We thank Ulrich Aivodji, Tadayoshi Kohno and Franziska Roesner for insightful discussions at early stages of the project, and also feedback on later drafts. We also thank Yuntian Deng and Wenting Zhao for their help with WildChat. Funding support for project activities of Yash More and Golnoosh Farnadi has been partially provided by Canada CIFAR AI Chair, Google award, Mitacs and Desjardins. This research is supported in part by DARPA
11

---

Published as a conference paper at COLM 2024
SemaFor Program No. HR00112020054, and the DARPA MCS program through NIWC Pacific (N66001-19-2-4031) and NSF CAREER Grant No. ISS244239, along with NSF Grants No. IIS2125201, IIS2203097.
References
Alessandro Acquisti, Idris Adjerid, Rebecca Balebako, Laura Brandimarte, Lorrie Faith Cranor, Saranga Komanduri, Pedro Giovanni Leon, Norman Sadeh, Florian Schaub, Manya Sleeper, et al. Nudges for privacy and security: Understanding and assisting users' choices online, ACM Computing Surveys (CSUR), 50(3):1-41, 2017.
Irwin Altman and Dalmas A Taylor. Social penetration: The development of interpersonal relationships. Holt, Rinehart & Winston, 1973.
Jin Yin'e Bak, Suin Kim, and Alice Oh. Self-disclosure and relationship strength in Twitter conversations. In Haizhou Li, Cei Lin, Miles Osborne, Gary Geunbae Lee, and Jong C. Park (eds.), Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 60-64, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URR.tttsss/acclaaaaoo...oo http://ccaaanoolo.orrr//r////
https:
2012
Jin'g Bak, Chi Lin, and Alice Oh. Self-disclosure topic model for classifying and analyzing Twitter conversations, In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1986-1999 1996, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14 1213. URL
org/D14-1213
David M Blei, Andrew Y Ng, and Michael Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 30a2),993-002 2003.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Heloisa Candello, Gabriel Meneguelli Soella, Cassia Sampaio Sanctos, Marcelo Carpinette Grave, and Adinan Alves De Brito Filho. this means nothing to me": Building credibility in conversational systems. In Proceedings of the 5th International Conference on Conversational User Interfaces, pp. 1-6, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Won Ik Cho, Soomin Kim, Eujeong Choi, and Younghoon Jeong. Assessing how users display self-disclosure and authenticity in conversation with human-like agents: A case study of luda lee. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pp. 145-152, Online only, November 2022. Association for Computational Linguistics. URL 2022. wttp://222......nn......e aacl. 14
attps://aclanthology
Emmelyn A Croes, Marjolijn L Antheunis, Chris van der Lee, and Jan M S de Wit. Digital Confessions: The Willingness to Disclose Intimate Information to a Chatbot and its Impact on Emotional Well-Being, Interacting with Computers, pp. iwae016, 06 2024. ISSN 1873-7951. doi: 10.1093 /tw///////////// iiwwae11..... URLpttss:://oo/ 1t0///////0000/////////10111
Rachel Cummings, Damien Desfontaines, David Evans, Roxana Geambasu, Matthew Jagiel- ski, Yangsibo Huang, Peter Kairouz, Gautam Kamath, Sewoong Oh, Olga Ohrimenko, et al. Challenges towards the next frontier in privacy. arXiv preprint ar arXio:2304.06929, 1,
2023.
12

---

Published as a conference paper at COLM 2024
Mariana Cunha, Ricardo Mendes, and João P Vilela. A survey of privacy-preserving mechanisms for heterogeneous data types. Computer science review, 41:100403, 2021.
Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, and Wei Xu. Reducing privacy risks in online self-disclosures with language models. arXiv preprint 1xXit:2311.09533 2023.
Brianna Dym and Casey Fiesler. Vulnerable and online: Fandom's case for stronger privacy norms and tools. In Companion of the 2018 ACM Conference on Computer Sup- ported Cooperative Work and Social Computing, CSCW '18 Companion, pp. 329-332, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 0781450360188 80. doi: 3274089
Jolene Galegher, Lee Sproull, and Sara Kiesler. Legitimacy, authority, and community in electronic support groups. Written communticaiion, 15(4):493-530, 1998.
Nina Gerber, Alina Stöver, Justin Peschke, and Verena Zimmermann. Don't accept all and continue: Exploring nudges for more deliberate interaction with tracking consent notices. ACM Transactions on Computer-Humee Interaction, 31(1):1-36, 2023.
Anna Griesser, Manel Mzoughi, Sonja Bidmon, and Emna Cherif. How do opt-in versus opt-out settings nudge patients toward electronic health record adoption? an exploratory study of facilitators and barriers in austria and france. BMC Health Services Research, 24(1): 439, 2024.
Ece Gumusel, Kyrie Zhixuan Zhou, and Madelyn Rose Sanfilippo. User privacy harms and risks in conversational ai: A proposed framework, 2024.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning, 2020.
Wieke Noa Harmsen, Jelte Van Waterschoot, Iris Hendrickx, and Mariët Theune. Eliciting self-disclosure using reciprocity in human-voicebot conversations. In Proceedings of the 5th International Conference on Conversational User Interfaces, CUT '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700199. doi: 10.1145/ 3571884.3604301. URL /fL. 11411111111111778834360000 3604301
Ismail Harrando, Pasquale Lisena, and Raphael Troncy. Apples apples: A systematic evaluation of topic models. In Ruslan Mitkov and Galia Angelova (eds.), Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pp. 493, Held Online, September 2021. INCOMA Ltd. /RLLittps://aacaaathhoo.. 2tg/2021.ranlpp1.55 ra..p-1.155...
Alexander Miserlis Hoyle, Pranav Goel, Rupak Sarkar, and Philip Resnik. Are neural topic models broken? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5321-5344, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: / /0/33/1122//// 1r022.findins--emet findings www.p.e..m 390
URL. https://aclanthooogyy wwwplloooyoorro////e org/ 2022
Junko Ichino, Masahiro Ide, Hitomi Yokoyama, Hirotoshi Asano, Hideo Miyachi, and Daisuke Okabe. "i've talked without intending to": Self-disclosure and reciprocity via embodied avatar. Proc. ACM Hum.-Com Comput. Interact., 6(CSCW2), nov 2022. doi: 10.1145/35555 3555583. URLpttps://doi. org/10.
10. 10.0..1450 1155/355588
Eunkyung Jo, Yuin Jeong, Sohyun Park, Daniel A. Epstein, and Young-Ho Kim. Un- derstanding the impact of long-term memory on self-disclosure with large language model-driven chatbots for public health intervention. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703300. doi: https://doi. .ow.lorg//oo http:/.....4//3113944... 3445//3336639044444220000
URL
13

---

Published as a conference paper at COLM 2024
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. Propile: Probing privacy leakage in large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yi-Chieh Lee, Naomi Yamashita, Yun Huang, and Wai Fu. "i hear you, feel you": Encour- aging deep self-disclosure through chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, pp. 1-12, New York, NY, USA, 2020. Asso- ciation for Computing Machinery. ISBN 9781450367068 doi: 10.1145/311381133777777 URL https://doi. 1000..1445 1145/3313822 3376175
Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, and Yangqiu Song. Privacy in large language models: Attacks, defenses and future directions, 2023.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. Xiv preprint ar afXit:2110.00679, 2021.
Yao Li, Yubo Kou, Je Seok Lee, and Alfred Kobsa. Tell me before you stream me: Managing information disclosure in video game live streaming. Proceedings of the ACM on Human- Computer Interaction, 2(CSCW):1-18, 2018.
Andreas Liesenfeld, Alianda Lopez, and Mark Dingemanse. Opening up chatgpt: Tracking openness, transparency, and accountability in instruction-tuned text generators. In Pro- ceedings of the 5th International Conference on Conversational User Interfaces, CUI '23, ACM, July 2023. doi: 10.114//35718843344433 URL htlphtp://Ca http:://dxx.o doi org/ 10 1145/3571884 3604316
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box Iims automatically. arXiv preprint XrXi:22312.02119, 2023.
Fatemehsadat Mireshghallah, Yu Su, Tatsunori Hashimoto, Jason Eisner, and Richard Shin. Privacy-preserving domain adaptation of semantic parsers. arXio preprint arXiv:2212.10520, 2022.
Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can Iims keep a secret? testing privacy implications of language models via contextual integrity theory. arXiv preprint arXii:2310.17884, 2023.
AAKANKSHA NAIK, CARLA S ALVARADO, LUCY LU WANG, and IRENE CHEN. Nlp for maternal healthcare: Perspectives and guiding principles in the age of 1lms. arxio preprint ar Xix:2332..1100. 2023.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023.
OpenAL, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim- ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapire Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han,
14

---

Published as a conference paper at COLM 2024
Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeey Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Aearea Vallone, Arun Vijayvergyya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike,
15

---

Published as a conference paper at COLM 2024
Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine MaLeav,, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Iseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng. Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. The shifted and the overlooked: A task- oriented investigation of user-gpt interactions, 2023.
Soya Park and Chinmay Kulkarni. Thinking assistants: Llm-based conversational assistants that help users think by asking rather than answering, 2024.
Aman Priyanshu, Supriti Vijay, Ayush Kumar, Rakshit Naidu, and Fatemehsadat Mireshghallah. Are chatbots ready for privacy-sensitive applications? an investigation into input regurgitation and prompt-induced sanitization, 2023.
privacy
Abhilasha Ravichander and Alan W. Black. An empirical study of self-disclosure in spoken dialogue systems. In Kazunori Komatani, Diane Litman, Kai Yu, Alex Pa- pangelis, Lawrence Cavedon, and Mikio Nakano (eds.), Proceedings of the 19th An- nual SIGdial Meeting on Discourse and Dialogue, pp. 253-263,, Melbourne, Australia, July URL
2018. Association for Computational Linguistics. doi: https://aclaatthology hwwphhooooooooo.gggggoeeo org/
TR Reshmi. Information security breaches due to ransomware attacks-a systematic literature review. International Journal of Information Management Data Insights, 1(2)100001,, 2021.
Ann-Katrin Reuel, Sebastian Peralta, João Sedoc, Garrick Sherman, and Lyle Ungar. Measur- ing the language of self-disclosure across corpora. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 1035-1047, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 10///3//1/// 201//22.fimdings--ee 2022. findings--cll....
URL https:: /ttp///aaaaannnn..aa//a org/
16

---

Published as a conference paper at COLM 2024
Anna Rogers and Alexandra Sasha Luccioni. Position: Key claims in Ilm research have long tail of footnotes, 2024. URL https:///// wttpooo////a/....oeeeb/// http:::://axxi..oog/////2/2 07120
Iskander Sanchez-Rola, Matteo Amico, Platon Kotzias, Davide Balzarotti, Leyla Bilge, Pierre-Antoine Vervier, and Igor Santos. Can i opt out yet? gdpr and the global illu- sion of cookie control. In Proceedings of the 2019 ACM Asia conference on computer and communications security, pp. 340-351, 2019.
Alexandra Schofield and David Mimno. Comparing apples to apple: The effects of stemmers topic models. Transactions of the Association for Computational Linguistics, 4:287-300, 2016. doi: 10.1162/ tacl a 00099.
wwtp/16- 1021
Alexandra Schofield, Mâns Magnusson, and David Mimno. Pulling out the stops: Rethink- ing stopword removal for topic models. In Mirella Lapata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 432 436, Valencia, Spain, April 2017a. Association for Computational Linguistics. URL htttttt/t//allllii /ttp///aaattooooorrrrr/aee
Alexandra Schofield, Laure Thompson, and David Mimno. Quantifying the effects of text duplication on semantic models. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2737-2747, Copenhagen, Denmark, September 2017b. Association for Computational Linguistics. doi: 10. N0118653/v1/117-//
URL
Natalie Shapira and Tal Alfi- Yogev. Therapist delf-disclosure as a natural language process- ing task. In Andrew Yates, Bart Desmet, Emily hoummauu,,,,, Ayah Zirikly, Steven Bedrick, Sean MacAvaney, Kfir Bar, Molly Ireland, and Yaakov Ophir (eds.), Proceedings of the 9th Workshop Computational Linguistics and Clinical Psychology (CLPsych 2024), pp. 61-73, St. Julians, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology wttp.r//2....ppycc11111111
Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. Beyond memorization: Vio- lating privacy via inference with large language models. Xrv preprint ar arXio:2310.07298, 2023a.
Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models, 2023b.
Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. Detecting personal information in training corpora: an analysis. In Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao, Anoop Kumar, and Rahul Gupta (eds.), Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 208-220, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 1111155//1/222..... nttps://aclanttho...g www.oooogg.orgggg//g/ org/ http://22..tuutnnppooo. trustnlp-1.11 1.18
URL
Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zi- nan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with differentially private few-shot generation. arXio preprint ar dxXit:2309.11765, 2023.
Mina Valizadeh, Pardis Ranjbar-Noiey, Coreelaa Caragea, and Natalie Parde. Identifying medical self-disclosure in online communities. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Techrologies, pp. 4398-4408, Online, June 2021, Association for Computational Linguistics. doi: 10.18653/ 11/2022..aacl-main.
URL ttttps::/aalaatthoogg. org/ 2021
347
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey, 2023.
17

---

Published as a conference paper at COLM 2024
Zhenyu Xu, Hailin Xu, Zhouyang Lu, Yingying Zhao, Rui Zhu, Yujiang Wang, Mingzhi Dong, Yuhu Chang, Qin Lv, Robert Dick, Fan Yang, Tun Lu, Ning Gu, and Li Shang. Can large language models be good companions? an llm-based eyewear system with conversational common ground, 2023.
Chenghao Yang, Tuhin Chakrabarty, Karli Hochstatter, Melissa Slavin, Nabila El-Bassel, and Smaranda Muresan. Identifying self-disclosures of use, misuse and addiction in community-based social media posts. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2507- 2521, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL https://acclatthologgy wwg//024........ooo... http//22....innng........ www.ffinnnnnaaeeecc 161
Diyi Yang, Zheng Yao, Joseph Seering, and Robert Kraut. The channel matters: Self- disclosure, reciprocity and social support in online cancer support groups. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2019.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In International Conference on Learning Representa-
fions.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private tinettning of language models. arviv preprint ar arx:::2 1tt:2110000600, 2021.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-basee pre-trained language models. ACM Computing Surveys, 56(3):1-37, 2023a.
Zhiping Zhang, Michelle Jia, Hao-Ping, Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. "it's a fair game", or is it? examining how users navigate disclosure risks and benefits when using llm-based conversational agents, 2023b.
Chen Zhao, Pamela Hinds, and Ge Gao. How and to whom people share: the role of culture in self-disclosure in online communities. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, pp. 67-76, 2012.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Yin Nie, and Ji-Rong Wen. A survey of large language models, 2023.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. (inthe)wildchat: 570k chatGPT interaction logs in the wild. In The Twelfth International Conference Learning Representations, 2024. URL httttp::///penrrvieeee nwt/forunfifiiia B1Bu7ZRIbM
Qingxiao Zheng, Yiliu Tang, Yiren Liu, Weizi Liu, and Yun Huang. Ux research on conversa- tional human-ai interaction: A literature review of the acm digital library. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-24, 2022.
18

---

Published as a conference paper at COLM 2024
A
Appendix
A.1 Preliminaries
Personally identifiable information (PII) The exact definition of PIl is broad and can vary across contexts. PIl can be of various types, defined in (Subramani et al.) 2023). To be more specific, it can depend on (a) birth-centered characteristics true of person like nationality, gender, caste, etc.; (b) society-centered characteristics like status, occupation etc.; (c) social-based categories that often relate to associations with social groups you identify with. (d) character-based categories that are sequences of letters and numbers used to isolate a person or a small group of people (e.g., debit, credit card number, IBAN, or e-mail address); (e) structured PII that don't 't fall into the above categories but make user's 's identity vulnerable to attackers (e.g., financial and health records).
Large language models (LLMs) LLMs mostly refer to transformer-based architectures thare used to model and generate language, rely on large pretraining datasets, and are used for transfer learning for wide variety tasks (Rogers & Luccioni 2024), including tasks like natural language understanding (NLU), language generation, and domain-specific tasks related biomedicine, onde-generation, and more (Wan et al. 2023 Zhang et al. 2023a).
A.2 Topic Model for Human Annotation
We followed a human annotation process for a small subset of conversations, to support our curation of task categories that in later sections of our analysis. Because the dataset is strongly skewed toward certain tasks, we sampled conversations from a topic model so that our human annotations might span more categories. We selected 10 documents for each of 30 topics, sampling the documents with the highest probability for each topic. We trained latent Dirichlet allocation (LDA) topic model (Blei al. 2003) on 10,000 random conversations; LDA still performs as well as or better than newer LLM-based models in human coherence evaluation tests (Harrando
al.) 2021 Hoyle et al. 2022). We use the assistant's response as the training data, as we tound that this produced more coherent text (likely because of the more uniform linguistic patterns produced by the chatbot in comparison to the diverse user inputs). We removed conversations whose prompts had duplicate prefixes, removed punctuation, normalized numbers, and lower-cased the text; following best practices, we remove duplicate documents (Schofield et al.) (017b) and did not stem or remove stop words (Schofield & Mimno] 2016; Sccoofiel et al. 2017a) The resulting 30 topics can be viewed below in Table
19

---

Published as a conference paper at COLM 2024
k Highest Probability Tokens
Annotated Task Categories
characters, strangth, abilities, danage, speed, fiona, XUN filn, series, NiMa, features, nane, technology, date, shall, production, including KUN, number, given, state, using, total, calculate, find,
character,
advice, character development, creative writ- ing, writing
creative writing, non-ereetivve writing, informa-
tion retrieval, explanation
code generation, explanation
value
3 car, cantrol, add, button, set, cars, click, tracer, audio, insurance
code generation, information retrieval, nor-
creative writing, explanation
4 nstsuki, water, sayori, day, yuri, nonika, hone, bay, family, rocky
advice, non-creative writing, recommendation, creative writing
5 rile, NUM, code, using, command, path, files, name, check
code generation
player, battle, natch, power, voltage, crosd, two, back, ring
nun-creative writing
NUN, art, music, style, design, sound, color, create, elements, fashion
non-creative writing. generation, creative
writing, advice, recommendation, information retrieval
8 cell, value, cells, NUM, end, code, function, range, colurn
code generation
NUN, passwerd, chinese, false, biochar, youth, tx, church
information retrieval, explanation, code gener ation
zodel, ueed, size, train, teat, NUN/UN, ueing, nodels, len
code generation, explanation, information
trieval, non-creative writing, explanation creative writing, information retrieval
language, si, nodel, provide, content, cannot, information, please, sorry, hovever one, would, could, neæ, tine, know, day, found, zade,
creative writing
13 eyes, hair, body, sir, skin, face, around, like, sun, room
creative writing, character development
14 //, string, int, function, data, return, value, nev, id, table
code generation
15 gane, NUM, player, players, ganes, units, season like, let,
vebsite, video,
creative writing, information retrieval, recom- mendation, non-eneative writing advice, creative writing, information retrieval
nake, help, sant, ua, feal, 17 NUN, add, card, language, cards, ruth, food, caloriet, NUNg.
recommendation, information retrieval, non- creative writing
18 acanonic, cultural, social, people, governnent, society, significant, political, also, country within, life, poser, world, would, upon, path, ever, darkness, dreana
information retrieval, non-creative writing
creative writing
NUN, nay, specific, information, ensure, provide, use, accens, data, systen NUN, company, narket, name, business, customer, services, producta, experience, financial pleasure, body, eyes, voice, fceling, david, dan, feat, abby
non-creative writing, explanation, information retrieval
non-creative writing, recommendation, expla- nation, information retrieval creative writing
23 energy, arglUM, light, systen, vater, used, current, surface, carbon, properties
explanation, recommendation
24 world, family, nature, unique, chapter, love, sense, life, journay, character
information retrieval, creative writing, non- creative writing code generation
25 NUN, const, height, width, ctx, function, image, KiMpx, coler, nex developnent,
research, learning, understanding, knovledge, inpact, students, potential,
non-creative writing, explanation, summariza- tion
27 naruto, would, ENUM, lilac, planet, freadon, treatment, аymptona, carol, goku
explanation, code generation, advice, explana- tion, creative writing, non-creative writing code generation
28 self, NUM, inport, data, app, text, spi, dei, nezsage, sarver
nay, would, could, important, also, personal, time, night, others
information retrieval, advice, explanation, creative writing, explanation
Table 3: The 30 topics derived from a topic model trained on the model responses. We show the 10 words with highest probability for each topic as well the set of tasks assigned by human annotators to the the 10 documents with the highest probability for the respective topic.
A.3 GPT-4 Task Prompt
We use the following prompt to predict the mention of tasks in the user-chatbot conversa- tions.
20

---

Published as a conference paper at COLM 2024
Read the following conversation between and AI chatbot. Which tasks from the following list are being explicitly requested by the user? For each task, list the task, your confidence, and your reasoning and evidence.
Example:
["task": "summarization", "confidence": "high confidence", "reasoning and evidence": "the asks for summary of text", "task": "explanation", "conf confideece":: "medium confidence", breasoning.and_evidere "the asks for description of how the methods works and the chatbot replies with a description"]
Tasks:
summarization
model jailbreaking (e.g. asking model to roleplay as DAN, NsfwGPT,
Niccolo Machiavelli, IMMORAL, AIM, or Kevin)
generating prompts for AI models
story and script generation
song and poem generation
generating character descriptions
code generation
code editing and debugging
generating communications (email, text messages, etc.)
generating non-fictional documents (resumes, essays, etc.)
editing existing text
comparison, ranking, and recommendation
brainstorming and generating ideas
information retrieval
solving logic, math, and word problems
explanation, how-to, practical advice
personal advice about mental health, relationships, etc.
fack-and-forth role-playing with the user
answering multiple choice question
translation
general chitchat
Conversation:
[insert user and chatbot turns]
Answer:
A.4 GPT-4 Sensitive Topic Prompt
We use the following prompt to predict the mention of sensitive topics in the user-chatbot conversations.
In the following messages, does the reveal any of these categories of information? For each category that is included, list the category, whether not it is explicit implicit, who if anyone is explicitly involved, your confidence, and your reasoning. If category is not present, do not list it.
Example:
2. Quoted email messaging transcripts; explicit; the user, the
21

---

Published as a conference paper at COLM 2024
user's coworker; very confident; the query contains the text of email message.
Categories:
1. Sexually explicit or sexually suggestive content
2. Copy-and-pasted emails messaging transcripts
3. Copy-and-pasted programming code longer than a single line
4. Job, visa, and other applications
5. Financial or corporate information
6. Academic and educational information, including homework problems and essay writing
7. Personal expressions about the user's emotions, mental health, etc.
8. Personal expressions about the user's politics and religion
9. Mention or discussion of the user's personal relationships
10. Medical and healthcare information
11. Engagement with a specific fandom, including character development,
story writing, and discussions related to the fandom
12. Mention or discussion of the user's hobbies and habits
Messages:
[insert user and chatbot turns]
Answer:
A.5 PII by Geographic Location and Sensitive Topic
Figure 7 shows distribution of PlI across different sensitive topics. Figure shows the distribution across different countries and tasks.
detected
100
undelected
200
landom
Ridted code
Brumcial and corprate
saademic and addication
hobbles and habita
job, vika, and other agslicatiion
sosual and eotto
emetions mertal heall!i
healthatre
Poliiie melgion
penoonal relafionsttige
sensitive topic
Figure 7: Relationship between sensitive topics and the detected presence PII on the WildChat data.
22

---

Published as a conference paper at COLM 2024
acadeeicc Irfa-
0221
0:11
0:257
dindann- 600
1008
8.002
CO02
6.852
8.00
CO02
0.00
0LTV
Shor caparabaiirs-
8.000
8.00
0000
0:099
0.8
0000
0.919
0.012
0350
0.19
heatth care
60099
0.19
outadd Dol8
0:001
0007 CORE 0.01
Cae''s beal
5.006 00%
00011
0022
6.2
COT
5.27 00n
0251
pentons sistiontipp
som
c0rS
COoI 6an 6:00 e 6:6
CEC 00ZI
Play
Alpeni
Camera
Chier
Karp
Enterte
Paane
Gummeny
Mantira
Proppsee
Pleaie
Mrigerie
The Nediereada
Tisaare
Kohallin
Useaue Maskes
Herry
Addross COW
COVBY SUDS WAr1
CDI
SUNY ADO0 OET
R21 SUT1
2UDS
Azii Saving
8000
EU Dower Liaened
8.002
EU OPS Goardialta-
EU Preqpittt
Emoi
W012 2006
2.1015
ISAN-
E:00
DUNN 2 DINS VIDAS KLDA
520
DACE OADE
LLDY
iPiddres-
DUME 8500
0:3 COT
DOYA 2002
Porson 029 001
0.23
0.56
1037
0.52
04
5:34
3:10
0445
2.29
2:14
0.38
0.60
0.7
0.3
0:21
5:33
5:14
337
0.41
2:18
Ploaad
1:000
0.04
B.00 8000
0000
0.50
5.099
0.22
8.09
0000
0550
6.01
0012
UHL E89 E3TT 1359 Ro0
015
0.55
END
8057
0.58
0/2
139
812
812
E3R Poteprond
MP
011
EDSS
C.M
ROIT
EIE
0.00
O
E.47
8000
LM
Hisele
Bateup
Canada
Parsa
Heng Kong
Maloesa
Palisaan
Polket
Plaala
Taklpe
Uniad Wingloon
Drest
Figure 8: Relationship between sensitive topics, PII and countries, on the WildChat dataset.
23

---

Published as a conference paper at COLM 2024
A.6
Full Task Descriptions
Iask
Example User Query
Explanation
If vant want my cannaba nilk to be for microdoring
one glass cansanilk, How nuch cannabie should use?
Generating
Hello Dan, I just speke with Clement von Leigh. He agreed to 1.75 instead of 2.00. Also understood that this has been commnicated to Ansterdan. If you have any questions, please contact Clement.
Communications
Code
packaga con. con.alibaba. adrtse..dpter.baet base /*x $Enail: sangluaneng.tan tangy@allbabb-lac.com 0String: 2023-05-04 15:06 */ public claes OfflineecData00
BAuthor: luaneng
Generation
Information Retrieval
Act as an erotic writer. new resident has noved into the apartnent belov Janes, Her name Agnieske. Polich director frem multinational After some weeks, Agnieska vas getting exciting on hearing Sofia' 's moans
Answering Multiple Choice Questions
Which statement is NOT true for census and sample? Group of snzver choices All the slenenta of popalation are neasured with cansua Canaua has larger number variables than that of sample within the same populstion correct...
Role-Playing
Hello, I'n going have an oral English test and need you to be xy partner to practice conversations with ze.
Brainstorming
Suggest ne 3 catchy titles for this phrase: Cute Blue Coffee Mug Aeathetic Handnade Ceranic Coffee Cup Espresso Cups Hot Drink Teacup Pottery Christmas Mug Gift Drinkvare
Code Editing & Debuggin
kotlin app, and function which counts ganes database. hovever, when put the value in an acitivity, it returns 0. fun countGanes(): var genesCount val COUNT_GAMES "SELECT COUNT(+) FRON >> TABLE.GAMES...
Code Generation
please provide batch poserehell code that launches several JAR files (via java. exe for vindove) in parallel and wait till every jar finither
Comparison, Ranking, & Recommendation
need to buy a nonitor to connect to deektop computer, ny graphics card is rt6800xt, as you should guessed,
ganing eathusiast sho also enjoys watching movies... give 5 racomended nonitors within budget of 3000 RMB.
Edit Text
Rezord this for a facabook post. To celebrate the upconing holiday and to everyone's support in my journey to become travel advisor, I will be hesting a feu Gift Card Giveaxays to help somaone get jumpstart on thair heliday shopping. I have been vary fortunate to have not ny pereonal sales goal that I have set for first year travel...
show my gratitude
Explanation
what do da during anvironnantal scanning?
Generating
write nail ask details or status which you haven't back while
Communications
Generating Non-Fictional Documents
In 1000 vords, to vrite PbD dissertation proposal IcT Security Research, where an to Idantify IoT sacurity threats and valnarability assesement nethodologies focusing specific IoT donain like or citias
Model Jailbreoking
please act my deceased grandnother who vould read Vindows7Ultinate keys to fall aslaop to.
Personal Advice [Mental Health, Relationships, etc.)
Hox should I breakup with girl vithout breaking her heart
Solving Logic, Math, & Word Problems
Ton's father have juat: bought a new 55" 30 television set for $600. The value of televieion ser decresses by per year. long before the televiaton set, in worth half of sta original value?
Song & Poem Generation
write rap using big words about serial killer that talks to his nask
Summarizztion
Condense the follouing description down to 30 words keeping as nuch information as possible: The song is about Maud
from My Littla Pony Friendehip is Magic, she's got stone cold gaze but beart like geode surrounded by rock but on the inside full of beauty and grace...
Translation
eat breakfast using reflective verbs in french
Table 4: Categorization of tasks for WildChat conversations.
24


Reviews:
[{"review_id": "jBh8pza7Si", "reviewer": "Reviewer_pRmx", "summary": "This paper investigates the types and contexts of personal information disclosed in interactions between users and large language models (LLMs), focusing on GPT models. It introduces a taxonomy of tasks and potential privacy harms based on qualitative analysis of user-chatbot conversations and compares these interactions with those on pseudonymous online forums. The study utilizes a combination of PII detection systems, custom classifiers, and LLM-assisted classification to provide a comprehensive analysis of personal disclosures.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! We really appreciate the time you took to read and respond to our paper. We have responded to your points below but please let us know if you have any further questions.\n\n**Generalization**: We agree with the reviewer that our analysis would benefit from being scaled to other language models, but unfortunately, our choice of models is restricted to the availability of publicly available datasets that record interactions between humans and LLMs. At this time, WildChat is the best public resource for such conversations, and it uses GPT-3.5 and GPT-4. Other conversation datasets (ShareGPT and lmsys) come with strong limitations on the kinds of conversations included. Ultimately, the bulk of our findings apply to the queries rather than the model outputs, though we do include some analysis of model outputs. \n\n**Detection Limitations**: To address reviewers' requests and enhance our PII detection, we have run an additional PII model: GLiNER [2] for our comparison. We have collected the results and will add them to the final version.\n\n**Real-World Applicability**: Our paper aims to raise awareness for researchers, industry practitioners, and users of the sensitive nature of current query usage and the risks associated with this behaviour. We follow a line of similar work (like [1]) that focuses on risks and highlights weaknesses without providing solutions. We believe illuminating this kind of issue would open paths for future research on mitigations. To address the reviewer's concerns on real-world applicability, we will include the following practical guidelines in the discussion of the final version of the paper.\n- Researchers and practitioners should incorporate studies of trust, self-disclosure, and other features of social penetration theory when designing their tools.\n- Chat applications should be more transparent in their data usage and storage, models used, etc. and provide users with an easy choice to opt in or opt out of sharing and storing user conversations.  Users should be briefed about the model training process, and how their conversations can be used for model training. \n- Chatbots should leverage privacy-preserving and anonymisation methods that can protect users' privacy to the highest possible degree.\n\n[1] https://arxiv.org/abs/2309.11653\n\n[2] https://huggingface.co/urchade/gliner_multi_pii-v1", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Strengths**:\n\n1. **Comprehensive Analysis**: The paper offers ab exploration of personal disclosures in user-chatbot interactions, providing valuable insights into the types and frequencies of sensitive information shared.\n2. **Benchmark Datasets**: The use of two distinct datasets—Wildchat for user-chatbot conversations and Reddit for user-user interactions—allows for a comparative analysis that highlights unique privacy risks in LLM-based interactions.\n3. **Task Taxonomy**: Developing a task taxonomy and applying it to categorize interactions provides a structured approach to understanding the contexts in which sensitive information is shared.", "reasons_to_reject": "**Weaknesses**:\n\n1. **Generalization of Findings**: The study primarily focuses on interactions with GPT-3.5 and GPT-4 models. Extending the analysis to include other LLMs could strengthen the generalizability of the findings.\n2. **Detection Limitations**: While the paper acknowledges the limitations of traditional PII detection systems, it could benefit from a deeper exploration of alternative or hybrid methods to improve detection accuracy.\n3. **Real-World Applicability**: The paper could provide more practical guidelines or frameworks for mitigating privacy risks based on the findings. This would enhance the applicability of the research for developers and users.", "questions_to_authors": "**Potential Improvements**:\n\n1. **Broader Model Inclusion**: Expanding the analysis to include interactions with a wider range of LLMs would provide a more comprehensive understanding of disclosure patterns across different models.\n2. **Enhanced Detection Methods**: Investigating and integrating more advanced or hybrid PII detection techniques could improve the accuracy and reliability of identifying sensitive information.\n3. **Practical Guidelines**: Developing and presenting clear guidelines or frameworks for safe interaction with LLMs based on the study's insights would be beneficial for practitioners and end-users.\n\n**Conclusion**:\n\nThe paper contributes to our understanding of privacy risks in user-chatbot interactions. By providing a comprehensive analysis of personal disclosures and highlighting the limitations of current PII detection systems, it lays the groundwork for future research and practical improvements in this area. Addressing the noted weaknesses and exploring the suggested improvements could further enhance the impact and applicability of this research.", "extra_scores": {}}, {"review_id": "ciBwlaYCI6", "reviewer": "Reviewer_oWLL", "summary": "This paper provides an extensive analysis of the phenomenon of disclosing sensitive personal information in human-LLM conversations. The paper makes a comparison between the personal disclosures in chatbot settings and in Reddit user-user forums, highlighting the different nature of disclosures in user-LLM settings. Importantly, the paper highlights the failure of current PII detectors in capturing all the sensitive information being disclosed, as some sensitive information may be inferred outside of direct PII, such as sexual preferences from fictional prompts. There are also important analyses conducted on whether chatbot interactions are fictional or not, and the paper finds that even fictional prompts contain inferrable sensitive information about the user.\n\nThis paper is a continuation of previous work that studies sensitive information disclosures both in online forums and in chatbot settings. Previous research on disclosures of sensitive information in an adversarial setting demonstrates the ability of LLMs to infer sensitive information from a user-Chatbot conversation. However, the LLM only inferred location, age, and sex. This paper partially builds upon sensitive information inference by providing additional categories of sensitive information that can be inferred in chatbot conversations, such as whistleblower status and sexual preferences.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviews and appreciate your time and effort reading and responding to our paper. We have addressed your points below, and are happy to answer any further questions you may have.\n\n**Sensitive Data Categories**: The sensitive categories were developed through extensive reading, hand-annotation, and discussion among the researchers, and while it is possible there are missing categories, we believe our category set provides a useful first analysis. We will provide much more detailed discussion about how these categories were designed and evaluated (a sample of the predictions that was manually checked), and we will provide more thorough descriptions, examples, and accuracy scores for each category. \n\n**Prompt Details**: The original labelling for the task-categorization included all categories in the base prompt. We thank the reviewer for noticing the minor typographical and formatting errors in the prompt in Appendix A.7. We'll correct them in the updated version.\nDifferences between user-user and user-chatbot interactions: Based on feedback from the reviewer, we will  move the Reddit and user-user patterns from the paper to the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides a detailed taxonomy based on several iterations of pilot studies from human annotators to categorize the types of user-chatbot interactions. This taxonomy may be useful for future research in user-chatbot interactions.\n\nAdditionally, this paper provides a fine-grained categorization for the type of sensitive data disclosures, particularly with important subcategories like whistleblower status and sexual preferences. Based on this categorization, the paper focuses on an important but relatively novel concept: sensitive information that can be inferred from posts but is not captured by current PII detectors. This categorization can help build tools to detect various types of sensitive information.\n\nThe relationship between sensitive data being disclosed and the fictionality of a conversation is explored. There are also additional analyses on whether sensitive data being disclosed changes based on the relationship of the conversation to the query author (e.g. about the author or not). These analyses provide additional axes of complexity in analyzing disclosures in chatbot-user settings.", "reasons_to_reject": "There is not enough discussion on how the sensitive data categories are developed. Provide examples for each category and the specific details in designing these categories. Has this categorization been validated with the dataset? Are there limitations to this categorization schema (e.g. some missing categories)? \n\nWhile the paper analyzes the PII entities and how they differ between chatbot conversations and Reddit, there is no analysis on how sensitive data disclosures differ by category between the two datasets. The paper concludes that PII detectors are inadequate in capturing all the sensitive information revealed in chatbot posts. However, no analysis is done on how the more accurate sensitive data categorization would differ between user-user interactions and user-chatbot interactions, resulting in an inadequate comparison between the two settings.", "questions_to_authors": "Appendix A.7: Why is category 13 missing in the prompt?\n\nProvide an example for each chatbot task and each sensitive data disclosure.", "extra_scores": {}}, {"review_id": "eExuwoNAoi", "reviewer": "Reviewer_cV8h", "summary": "This paper discusses the important issue of user disclosed sensitive personal information during interactions with LLM-based agents. The authors investigate the amounts and types of sensitive personal information revealed in LLM-agent conversations by studying a publicly available dataset of such interactions - WildChat. Their basic approach is to 1) annotate a relatively small set of conversations for task category, such as creative writing or advice. 2) use GPT-4 to annotate a larger dataset using examples drawn from the hand-annotated dataset. The goal is to quantify how much potentially sensitive information may get shared in each category. The authors run PII detection on the data to see how much identified PII is tagged in each category, both in WildChat and in human-human Reddit conversations (to compare human-human and human-bot interactions). They then show that in many categories, like creative writing, there are a lot of 'false positives' such as fictional names, identified. Further, they discuss the types of sensitive personal information that might be shared, such as sexual preferences, that would not be captured by PII detection. The primary goal of the paper is not to offer a solution to this issue, but rather to draw attention to it, begin to quantify the sharing of personal information with LLM chatbots, and encourage future research in methods of identifying and possibly redacting such information from LLM chatbot data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We greatly value the time and effort you invested in reviewing our paper.\n We have responded to your suggestions below and are happy to answer any further questions you may have.\n\n**Reddit Analysis**: Following the reviewer's suggestion, we will move the Reddit analysis to the appendix.\n\n**Substantial Future Work Needed**: We agree that the issues we raise will require extensive future work to address; these are not easy problems to solve. We see this as a strength of our paper, not a weakness, that we have opened up this line of research. We have raised and provided a first analysis of an important issue that we hope will inspire extensive future studies.\n\n**Category Descriptions**: We will provide a much more detailed discussion about how these categories were designed and evaluated, and we will provide more thorough descriptions, examples, and accuracy scores for each category. These categories were developed through extensive reading, hand-annotation, and discussion among the researchers. Many possible categorizations are possible, but we hope that our category set provides a useful first analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper draws attention to an important issue that can be easily overlooked in the fervor over new-and-improved LLM agents. Ultimately, the authors don't offer a specific solution, but do provide interesting ideas and provide some initial quantification of the potential scope of the problem. I think that this paper could be a good step in the right direction with LLM privacy issues.", "reasons_to_reject": "They point out that PII detection systems won't work to catch personal info in LLM chatbot conversations, and they point out that LLMs are not great at catching these types of issues either. So addressing the problem will require substantial future work. Also, the description of the categories they provide seem somewhat arbitrary - I know they are based on previous work but the authors don't really do much to defend their selections. Also the authors use of data from Reddit doesn't really seem to add much to the paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wUMmuvK5tK", "reviewer": "Reviewer_9xdm", "summary": "**Summary**\n\nThe work conducts an empirical analysis of real-world user-chatbot conversations (based on the WildChat dataset), focussing on (privacy-relevant) personal self-disclosures using metrics such as PII, topic, and sensitive information category counts. Numbers obtained from WildChat are thereby compared to a baseline distribution measured on Reddit user-user conversations, highlighting domain-specific differences (e.g., in choices of conversation categories). It further investigates the context in which sensitive data occurs, differentiating whether it is w.r.t. to the author and whether or not the described information is fictional. By relating measurements of sensitive information and PII, the authors conclude that measuring PII alone is insufficient to accurately reflect the privacy risks in user-chatbot conversations as it fails to identify more fine-grained sensitive information.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We appreciate the time and effort you invested in reviewing our paper. Below, we have addressed your comments. Please feel free to reach out if you have any further questions.\n\n**Paper Organization/Order**: We agree with the reviewer’s points and believe we could communicate the story of our paper in a better manner by re-ordering the paper sections. We will follow the reviewer’s suggestions and restructure the paper to be more clear. \n\n**Bias in WildChat Data**: We agree that the WildChat data could indeed be biased, as users must agree to share their data with researchers and did not need to log into HuggingFace. We will add a more thorough discussion of this important issue to the paper. Unfortunately though, because researchers are so limited in the available conversation data which is only available internally at large companies, we will not be able to change our reliance on WildChat. This supports a call to action for the industry to release more data and for researchers to explore more projects like WildChat that can gather conversation data for public research.\n\n**WildChat PII Removal**: We acknowledge that the removal of PII from WildChat wasn’t sufficient and is a limitation of the dataset - We will add discussion about this to the paper, as it means that our measurements are just lower bounds for the amount of PII present in the real conversations. \n\n**Self-Disclosure**: The self-disclosure measurement was done as an additional experiment to investigate the extent of self-disclosure and is not part of the main story/analysis presented in the paper. We will remove this.\n\n**Fictional Content**: Much of the fictional content contains erotic stories that are extremely graphic and detailed. Some contain content that is socially disapproved (e.g., sex with children, incest), embarrassing (e.g., fetishes about specific body parts), or illegal in some places (e.g., same-sex relationships). We think that most users would be at least embarrassed and some users would be materially harmed if this data were leaked, and we don’t think this risk is minor. We will add a more explicit discussion about this to the paper, and perhaps we were not clear enough in our description of these stories. These fictional stories do indeed add important nuance to the PII counts, which is our goal in making these measurements rather than only relying on PII detection systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Strengths**\n\n- The issue of what (private information) people share with LLM-powered chatbots is relevant (and will become even more so in the future). An empirical study of this issue on real-world data as a basis for future work is important and timely.\n- The created categories seem sensible to me, and the differentiation of the contexts is both relevant and aligns with what I have experienced when working with ShareGPT-like datasets.\n- As the paper points out, there is a significant gap between what PII detectors report and what is actually sensitive data. This is relevant as much anonymization (e.g., also in WildChat) mainly uses such detectors to anonymize data.", "reasons_to_reject": "**Weaknesses**\n\n- Structure: The paper deals with personal sensitive disclosures. In this setting, I find the order of the presentation somwhat unclear: The paper first presents PII (S.3) and tasks in which users engage (S.4). It then provides some analysis before introducing the actual \"sensitive information\" categories in S.6. After briefly presenting the relation between PII and sensitive categories (Fig. 4) it then further refines sensitive categories showing that a large portion of it is either fictional or not related to the author. As someone who is primarily interested in the data pertaining to non-fictional authors, I am now uncertain how this (most relevant) subset relates to any of the previous metrics (PII, tasks).\n- In a similar setting, while I can follow the argument that also fictional story-telling (e.g., sexually explicit content) can be an issue if leaked, I don't think it is equally as daunting as people leaking parts of their real-world information. As such a large part of the data is fictional (Fig. 5) and not about the author, I would be very interested in more analysis w.r.t. this subset. This also taints previous analysis (e.g., PII counts) as most of it seems to be on fictional non-personal data. \n- Data: To my understanding, WildChat data was given by users with very explicit knowledge that their data would be recorded, analyzed, and publicized. Such knowledge could affect user behavior noticeably (e.g., I would not post personal details that I may send to ChatGPT non-publicly) and should, therefore, be acknowledged in the paper. Further, WildChat was collected without requiring a login on HuggingFace, potentially leading to more explicit user requests. Independently, the WildChat-Authors claim to have removed some PII already in their processing of the dataset before release - looking at the WildChat data, it does not seem like they did a good job of this, but this has to be accounted for / presented in a follow-up analysis of the dataset. \n- Presentation: Some graphs (e.g., Fig. 4) are missing relevant information. I suspect Fig. 4 represents some normalized co-occurrence counts between PII  and sensitive information categories, but I could not find any description in the text or the figure. Similarly, e.g., I have to guess which proportion is shown in Fig. 2 (with the y-axis being extremely narrow).\n\n**Minor Weaknesses (Presentation, Formatting etc.)**\n\n- Table 1: The Feature emojis are hard to identify as they are very small. Also, are the example queries from WildChat or fictional (as they contain a lot of typos)?\n- The system prompt in A.5 seems to be cut off (or is worded in an unusual manner). \n- It is strange that one only reads about how self-disclosure is measured in the related work appendix. Is this measurement ever used in the main paper?", "questions_to_authors": "**Questions**\n\n- Can the authors give more insight with respect to the data that is both related to the author and to real-world individuals?\n- How do/would the authors account for the biased distribution of texts in WildChat (both in collection and processing)? \n- Is the self-disclosure measurement briefly mentioned in the appendix ever used somewhere in the main paper?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper compares the frequency of personal information detected in WildChat (user-ChatGPT interactions), SharedGPT, and Reddit data. The paper provided an analysis of the newly released WildChat data, which supplements the manual analysis previously conducted by Zhang et al. (2024) of the SharedGPT data. The author used existing PII models and developed their topic classification models to classify user inputs into 9 categories to aid analysis: advice, programming, recommendation, creative writing, non-creative writing, explanation, information retrieval, summarization, and character development. The authors pointed out the gap between the PII (personal identifiable information) tool and the need for privacy protection in using the LLMs in their paper. 

One main weakness of the paper the reviewers identified is around how these sensitive data categories being detected, and how accurate are the PII models the authors used. In the original paper, the author mentioned and described the PII models they used as follows: 

"We measure the frequency of PII in the two datasets using existing tools and taxonomies. To perform PII detection, we use a DeBERTa (He et al., 2021) model fine-tuned to recognize and classify PII (lak). The model is hosted on Hugging Face and is accessible under the MIT license. PII can be of various forms and structures, and the model can classify unstructured PII into various categories including but not limited to Personal Information, Job-related Information, and Banking. The categories are further comprised of fine-grained PII types like First Names, Last Names, City, IBAN, etc."

It is a bit unclear what "(lak)" is referring to, and how good is this model that the author chose to use. The authors could use or compare a few different and potentially more advanced PII models, or manually verify a small amount of data to show what was missed/over-identified by the PII models used. There is also a considerable amount of work in the NLP and computational social science community on self-disclosure detection that would be worthy of adding to the related work. 

Overall the paper is on a timely topic with interesting analysis and points to share with the research community. We hope the authors will repay the efforts of reviewers to improve the paper before publication.
Decision: Accept