# "Lying makes me look suspicious": Users' Perspectives and Analysis of Security Questions

Xin Sun, Kavous Salehzadeh Niksirat, Kévin Huguenin, Mohammad Mannan, Amr M. Youssef

<figure><img src="image_1.png" /></figure>

## To cite this version:

Xin Sun, Kavous Salehzadeh Niksirat, Kévin Huguenin, Mohammad Mannan, Amr M. Youssef. "Lying makes me look suspicious": Users' Perspectives and Analysis of Security Questions. European Symposium on Usable Security (EuroUSEC), Sep 2025, Manchester, United Kingdom. pp.49-62, 10.1109/EuroUSEC69254.2025.00015. hal-05130189

HAL Id: hal-05130189

https://hal.science/hal-05130189v1

Submitted on 16 Sep 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# “Lying makes me look suspicious”: Users’ Perspectives and Analysis of Security Questions

Official version available on IEEEXplore: 10.1109/EuroUSEC69254.2025.00015

Xin Sun†*, Kavous Salehzadeh Niksirat‡§*, Kévin Huguenin‡, Mohammad Mannan†, and Amr Youssef†

†Concordia University

‡University of Lausanne

§Max Planck Institute for Security and Privacy

**Abstract**—Despite recommendations from security authorities, such as those of the US NIST, against the use of security questions for online authentication, these methods are still used for login and account recovery processes. Although research on security questions has a long history, key gaps remain, particularly regarding user perceptions and the requirements used by websites for selecting and answering security questions. In this paper, we address these gaps through a two-part study: (i) a user survey ($N = 292$) capturing insights from a diverse US sample, and (ii) an analysis of an extensive set of 26 security requirements across 73 websites, totaling 1913 security questions. Our findings reveal notable user misconceptions, such as users’ believing that websites already possess correct answers to personal security questions. Additionally, we identify widespread insecure practices, such as accepting single characters and allowing identical answers across multiple security questions. By addressing both user perceptions and website security requirements, we provide a comprehensive understanding of weaknesses in current security question practices and contribute to the ongoing discourse on strengthening authentication methods.

**Index Terms**—security questions; authentication; user survey

## I. INTRODUCTION

Authenticating users online, typically to safeguard access to their accounts, is of paramount importance. There exist multiple ways to do so, the most popular is the use of passwords, despite the repeated attempts to replace them with alternatives (recently with passkeys) [1]. Security questions (SQs), also known as personal knowledge questions (PKQs), is another way to authenticate users, alone or in conjunction with other ways. SQs/PKQs are questions for which the answer depends on the user and that are used to add some level of security to user account management. Typical questions include “What is your mother’s maiden name?”. Generally, during account registration, users are required to select a few questions and set their answers and then, during usage (e.g., login/account recovery), to answer one or multiple of these questions. Real-world incidents such as the compromise of Sarah Palin’s email account back in 2008 [2] illustrate the risks of weak SQ practices: the hacker used OSINT to find the answers to her SQs and used them to reset her e-mail account password. Some major online services (e.g., Google, since 2014) dropped the use of SQs primarily for security

reasons.<fnref n="1" /> In 2017, the US NIST issued a recommendation against the use of SQs for specific use cases, including login and account recovery [3].

The design and evaluation of SQs, from both security and usability perspectives, have been studied in prior work (e.g., see the 1991 paper by Haga and Zviran [4], and several other publications since then [5, 6, 7, 8, 9, 10]). However, as apparent from our literature review, several important research questions remain unaddressed. For example, prior studies often focus on specific user groups, such as university students [6, 8, 9, 10, 11, 12, 13]. There is also a lack of comprehensive collection and systematic analysis of the SQs used by online services (e.g., types of questions), the types of services that use such questions (e.g., banking and transportation), and their usage (e.g., number of questions). Such data collection could make it easier to explore the security (e.g., entropy/guessability) of SQ answers and the usability of SQs. Even more critical for security are the requirements of SQ answers, as imposed by websites, that largely remained under-explored. Existing studies also lack connecting user perceptions of SQs with real-world question-and-answer requirements that could identify important gaps in the security provided by SQs vs. users’ mental models. We believe all these aspects demand a closer look, as many users are still faced with SQs in their day-to-day use of the web. Accordingly, we ask the following research questions (RQs):

* **RQ1.** How do users perceive SQs, and what strategies do they use to select, use, and answer them?

* **RQ2.** What (mis)conceptions do users hold about the purpose and validation of SQs by online services?

* **RQ3.** What types of SQs and answer requirements are currently implemented by websites, and how do these align with recommended best practices?

To explore the above-mentioned RQs, we conducted an online survey with a diverse sample of the US population ($N = 292$) and a manual collection, categorization, and requirements analysis of 212 websites that offer SQs. Our survey (see Section III) not only collects the names of websites that use SQs but also explores users’ perceptions and

<footnote n="1">¹According to an industry survey, 22% of US adults suffered online account takeover, and among the victims, 36% had SQs set on the compromised accounts; https://www.security.org/digital-safety/account-takeover-prevention.</footnote>

<footnote>*Xin Sun and Kavous Salehzadeh Niksirat contributed equally to this work.</footnote>
experiences with SQs, providing insights into their usability and security concerns. More specifically, we collect data about (1) users' experiences and practices (e.g., prevalence and purpose of SQs, strategies for selecting SQs); (2) users' strategies for answering (e.g., providing truthful vs. untruthful answers, attitude towards answering privacy-sensitive questions); (3) answer recall (e.g., mechanisms used for recall, forgetting answers); (4) perceived effectiveness of the protection level provided by SQs and the difficulty of guessing answers by various adversaries; and (5) preference of security mechanisms (e.g., comparing SMS vs. SQs, different types of SQs). Including both closed- and open-ended questions, the survey provides the most comprehensive understanding of users' perceptions and insights on SQs compared to existing work. Our website analysis (see Section V) collects a large set of SQs provided by our survey respondents to understand the different types of SQs users encounter; this set is further augmented with SQs gathered through a manual web search.<fnref n="2" /> As many websites generally do not mention much about their use of SQs, finding such websites via an online search is not straightforward. We base our analysis on the recommendations provided by OWASP [14] for legacy use cases for SQs.<fnref n="3" />

Our survey findings reveal important insights into users' perceptions and behaviors regarding SQs. Notably, users report a declining reliance on SQs for authentication, yet they remain frequently used for account recovery. Respondents commonly select questions based on ease of recall, yet a substantial proportion prioritizes privacy and security by choosing less predictable answers and answers that are not publicly available (e.g., from users' social media profiles). The survey highlights a significant gap in user awareness, with some believing that services already possess the correct answers to personal SQs. Although most users provide truthful answers, some choose untruthful responses for added security. The findings show that though users generally find SQs easy to recall, they express mixed feelings about the security and privacy implications of sharing personal information.

The manual analysis of the websites that use SQs (see Section VI) confirms some of the results of our user survey (e.g., regarding the number of SQs used). It also shows that many websites do not follow OWASP recommendations when building their SQ authentication systems. For example, 31 out of 67 websites (42.5%) do not re-authenticate users before allowing them to set SQs, and 15 out of 36 websites (20.5%) rely solely on SQs during password recovery, without combining them with e-mail verification. Furthermore, 16 out of 54 websites (21.9%) appear vulnerable to brute-force attacks, as they allow users to make more than ten attempts during SQ verification.<fnref n="4" />

Our study makes several key contributions to the understanding and improvement of SQs: (1) We address a gap in prior research by surveying a diverse sample, thus offering new insights into user perceptions, practices, and misconceptions about SQs. (2) We highlight critical weaknesses in the current use of SQs, such as users' tendencies to provide consistent answers across services and misconceptions about service providers possessing the correct answers. (3) We demonstrate the significant security risks caused by website administrators not following established guidelines, such as those from OWASP, and the continued use of easily guessed or overly personal questions (e.g., “mother’s maiden name”). (4) We provide actionable insights for improving both user practices and the design of SQs on websites.

## II. RELATED WORK

Academic research in SQs has a long history (e.g., see the 1991 study by Haga and Zviran [4]). However, compared to passwords, they have been studied far less extensively [15, 16]. Researchers from various communities use different terminologies to refer to, such as “security questions” [6, 17], “secret questions” [7], “personal knowledge questions” [5, 8], “question-and-answer passwords” [4], and “password choice and challenge questions” [9]. Hence, for consistency, we will use the term “security questions (SQs)”, regardless of the term used in the papers we describe.

### A. Security Aspects and Users' Perspectives

We first summarize, in chronological order, the studies that focus on the analysis of SQs, from security and users' perspectives. Rabkin [8] focused on banking/financial websites in the US and categorized 215 SQs from 11 such websites. “Names” of friends and family and “Favorites” were the most common categories. Rabkin identified serious usability and security weaknesses, including, memorability and unsuitability, in the use of SQs. Using a short small-scale survey (N = 46), Rabkin also found that most respondents considered the memorability of SQs more important than security and that they answer these questions truthfully most of the time. Just and Aspinall [9] collected 282 questions from the inputs of respondents (rather than from real-world websites). Analyzing possible answers by using a novel attacker model based on the knowledge level of potential attackers, they concluded that the answers to the majority of the questions would fail to resist attacks from strangers (in contrast to the beliefs of the respondents).

Schechter et al. [7] conducted a user study to measure the reliability and security of SQs used by four major webmail providers (AOL, Google, Microsoft, and Yahoo!). They asked participants (N = 130) to answer these questions and then asked them to guess their partners' answers to these questions. They found that acquaintances (e.g., spouse, relatives, and friends) guessed 17% of the answers; participants also forgot 20% of their own answers within six months.

Analyzing a large real-world dataset from Google, on security and memorability of SQs and 11 million account recovery claims, Bonneau et al. [5] found that SQs have poor security

<footnote n="2">²146 out of 212 were collected from our survey and 86 out of 212 were collected from Google Search. 73 out of 212 were analyzed with security requirements and 194 out of 212 were used to collect SQs for categorization and characteristics evaluation.</footnote>

<footnote n="3">³While NIST advises against using SQs, OWASP acknowledges that some systems still rely on them and provides security best practices in such cases.</footnote>

<footnote n="4">⁴We cannot analyze some security requirements on all websites because some websites do not provide usage page of SQs.</footnote>
and memorability. They also found that several answers that are expected to be unique (e.g., phone numbers) were identical across different users, significantly compromising the security of the questions. A complementary survey (N = 1000) revealed that the primary reasons for providing untruthful answers are that the answers are difficult to guess, lack private information, and are easy to remember.

Golla and Dürmuth [18] analyzed a leaked database of 3.9 million answers to SQs from AshleyMadison.com (Aug. 2015). They found that the question “What is Your Favorite Sports Team?” is the least secure, using statistical entropy to measure the success rate of guessing answers for four different questions as used by the website. Furthermore, they found the security of SQs is slightly higher than the four-digit user-chosen PINs. Golbeck and Li [17] collected screenshots of SQs via crowd-sourcing, and identified 283 unique SQs (total 607) from 52 unique websites. They summarized a set of categories and themes: “Locations”, “Firsts”, “Time and Dates”, “Numbers”, “Dreams and Aspirations”, and “Other”, in addition to “People’s Names” and “Favorites”.

Dhekane [12] conducted a survey (N = 153) to analyze the security and usability of SQs, and found that many respondents prefer customized questions over those offered by the services. To further study the memorability of both types of questions, the author surveyed students with knowledge in security (N = 48). The majority of the students found it difficult to recall the answers. Micallef and Arachchilage [13] studied (N = 30) users’ strategies and perceptions of SQs. They found that respondents who selected their own answers favored questions with personally relevant, fixed responses and memorable answers. Most respondents reused the same questions and answers across multiple services.

More recently, Lassak et al. [10] conducted a user study (N = 97) over a period of 18 months to compare various fallback authentication mechanisms (e-mail, SMS, SQs, and trusted contacts). They found that SQs had the lowest success rate and were perceived as the least usable option for password recovery. In another survey (N = 281), Höltervennhoff et al. [6] reported that respondents indicated that SQs have poor performance, both in terms of security and usability, compared to other account recovery methods (e.g., recovery code).

## B. Improvements of Security Questions

To improve the security and usability of knowledge-based authentication, Senarath et al. [19] designed an interface with mnemonics to help users set secure answers. They measured answer strength via five parameters, similar to passwords: capital letter, digit, special character, letter, and character length. They also implemented a strength checker to provide feedback to users about the strength of their answers. AlHusain and Alkhalifah [11] introduced SQs based on user behavior and preferences. Through an experiment (N = 23), they evaluated the security and usability of answers to their questions. They reported that using user behavioral details in selecting topics for SQs improves recall and resistance to guessing.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Flow chart of the study procedure.</figcaption></figure>

## C. Research Gap

Prior research on SQs has largely focused on memorability, usability, and user strategies, leaving several critical aspects underexplored. First, existing studies have typically examined limited user groups, such as university students or crowd-sourced participants with unclear demographic diversity. We address this gap by conducting a large-scale survey with a diverse US sample, ensuring representation across location, age, sex, and ethnicity. Second, previous research has primarily assessed guessability through empirical attack success rates rather than user perceptions. We address this gap by examining how users perceive SQ guessability across a broader range of adversarial entities. Third, existing literature lacks a comprehensive evaluation of real-world SQ security requirements. We systematically analyze 26 security requirements, following established security guidelines (e.g., OWASP), across a set of 73 websites obtained through a user survey and a systematic manual search. Finally, prior studies have examined SQs, but their scope has been limited—both in the number of questions analyzed and the range of services covered — hindering a comprehensive understanding of SQ design and implementation. Our work overcomes this gap by building the most extensive and structured dataset to date (1913 questions).

# III. METHODOLOGY OF USER SURVEY

To address RQ1 and RQ2, which focus on users’ perceptions, usage strategies, and (mis)conceptions regarding SQs, we conducted an online user survey (N = 292). Figure 1 depicts the study procedure with the key numbers regarding the collected data.

## A. Recruitment

We aimed to recruit approximately 300 participants to ensure sufficient coverage across demographics and behavioral patterns. This sample size aligns with, and often surpasses, those used in prior research on SQs (e.g., N = 46 [8]; N = 130 [7]; N = 153 [12]; N = 281 [6]). We relied on the Prolific platform, considered a reliable source of respondents for scientific research [20], and one of the few platforms ensuring fair pay and ethical treatment of workers [21]. We requested a “representative sample” of the US population from Prolific (w.r.t. location, age, sex, and ethnicity, as per the US Census Bureau). We chose the US as Prolific has a large user base in the country and the English language used in this article is a commonly used language in the US. After two weeks of data collection, we obtained the
<table><caption>TABLE I
DEMOGRAPHICS OF THE SURVEY RESPONDENTS PROVIDED BY PROLIFIC (FINAL DATASET, N = 292).</caption><thead><tr><th></th><th>n</th><th>%</th><th></th><th>n</th><th>%</th></tr></thead><tbody><tr><td><strong>Sex</strong></td><td></td><td></td><td><strong>Age</strong></td><td></td><td></td></tr><tr><td>Male</td><td>146</td><td>50%</td><td>18-24</td><td>36</td><td>12.3%</td></tr><tr><td>Female</td><td>146</td><td>50%</td><td>25-34</td><td>60</td><td>20.5%</td></tr><tr><td><strong>Ethnicity</strong></td><td></td><td></td><td>35-44</td><td>64</td><td>21.9%</td></tr><tr><td>Asian</td><td>16</td><td>5.5%</td><td>45-54</td><td>54</td><td>18.5%</td></tr><tr><td>Black</td><td>30</td><td>10.3%</td><td>55-64</td><td>45</td><td>15.4%</td></tr><tr><td>Mixed</td><td>29</td><td>9.9%</td><td>65-74</td><td>31</td><td>10.6%</td></tr><tr><td>Other</td><td>19</td><td>6.5%</td><td>75-100</td><td>2</td><td>0.7%</td></tr><tr><td>White</td><td>198</td><td>67.8%</td><td><strong>SeBIS [23]</strong></td><td colspan="2">M = 29.8, SD = 5.4</td></tr></tbody></table>

targeted number of respondents in all categories (i.e., for the sample to be representative) except for two respondents in the 55-64 age group, four in the 65-74, and 25 in the 75-100. Following Prolific's recommendation, we re-assigned the unfilled quotas to the other age groups. Our survey was advertised to the respondents as a 15-minute questionnaire about people's perceptions of and responses to SQs on online services. In the advertisement, we specified that (eligible) respondents should have at least one account on an online service that uses SQs. We also asked prospective respondents to choose one such online service in preparation for the survey. Following Prolific's recommendations [22], we compensated the respondents with £2.75 for their participation.

## B. Respondents' Demographics

Table I provides the summary statistics about the respondents in our final sample (i.e., after our data-reliability procedure described next in Procedure Section). As Prolific does not provide information about gender (only sex), education, and tech-savviness, we deployed a follow-up survey to collect this information. The distribution of gender among respondents was nearly balanced between women and men, with 49.6% identifying as women and 48.3% as men. A small percentage identified as non-binary (1.2%) or preferred to self-describe their gender (1.7%), like "agender". A total of 82.2% of the (valid) respondents of the main survey completed the follow-up survey. Most respondents (59.6%) held at least a bachelor's degree or higher, and 84.2% perceived themselves as being tech-savvy.<fnref n="5" /> The respondents were further compensated with £0.7 for their participation.

## C. Questionnaire

Depending on the survey logic, respondents were asked 36 to 38 questions, distributed across eight parts. Almost all questions were closed-ended, but four were open-ended in order to collect deeper responses about the more complex aspects (e.g., describe the respondent's rationale behind a certain behavior). The questionnaire covered various topics, including users' experiences, their strategies for answering SQs, and their perceptions of security mechanisms. Next, we

describe the structure of the survey questionnaire. The survey transcript is provided in Supplementary 1.<fnref n="6" />

**Part A. Introduction and Consent.** The questionnaire started with a consent form (Q1 and Q2) that detailed the purpose of the study, the entities involved in conducting the study, the inclusion/exclusion criteria, as well as the respondents' rights in the research process. Next, respondents were required to provide explicit consent to partake in the survey.

**Part B. Sample Website that use Security Question.** We first asked the respondents if they use SQs (Q3 and Q4) and further asked them to provide the URL and a screenshot of a website that uses SQs (Q5 and Q6). The respondents were instructed that the screenshot should provide evidence that the website indeed uses SQs but that it should not include their answers to the SQs (for obvious security reasons).<fnref n="7" /> The goal of this question was two-fold: (1) to make sure that the respondents know about SQs (and have some experience with them); and (2) to gather names of websites that use SQs for subsequent analysis (see Figure 1 and Section V). We warned the respondents that only accurate URLs and relevant screenshots will be accepted (otherwise, their data would be discarded and they would not be compensated).

**Part C. User Experiences and Practices.** We assessed the respondents' experiences and practices with SQs in online services. We covered topics such as the prevalence of SQs (Q7), their purpose (Q8), the number of SQs offered by the service providers (Q9), the number of SQs required by the service providers to be answered for registration (Q10) and login (Q12), and respondents' strategies for selecting the SQs offered by the service providers (Q11). Lastly, we asked about the proportion of SQs that rely on their usage of a given service, such as the phone number they call most often, in the case of a mobile operator (Q13).

**Part D. User Strategies for Answering.** We investigated respondents' strategies for setting answers to the SQs and recall them. We first asked about the frequency of providing consistent answers to the same SQ (Q14), and later assessed the respondents' attitudes towards providing truthful vs. untruthful answers (Q15) to SQs. Respondents were asked about their reasons for doing so (Q16 and Q17). We also assessed their comfort level about sharing their personal information with the service provider when answering SQs (Q18).

**Part E. Answer Recall.** We assessed respondents' perceptions and practices in recalling their SQ answers. We asked how difficult or easy it is for them to remember their answers (Q19), the mechanisms they use to facilitate the recalling process (Q20), and the extent which these mechanisms are utilized across online services relying on SQs (Q21). We also asked about the frequency of forgetting SQ responses (Q22).

**Part F. Perceived Effectiveness.** We assessed respondents' perceptions of the effectiveness of the protection provided

<footnote n="6">⁶Supplementary materials are available in the OSF repository at https://doi.org/10.17605/osf.io/qa8rx.</footnote>

<footnote n="7">⁷None of them did; if some of them had, we would have securely removed the data.</footnote>

<footnote n="5">⁵Agreed or strongly agreed with "I am considered tech-savvy [...]" (Q42).</footnote>
by SQs. First, we asked the respondents to compare and explain the level of protection provided by SQs compared to passwords (Q23 and Q24). The insights indicated whether there was a prevailing opinion among respondents regarding the relative effectiveness of passwords vs. SQs in providing protection. Next, we asked about the perceived effectiveness of the protection provided by the SQs (Q25). Furthermore, we asked respondents to select the difficulty level of their answers being guessed by different adversaries such as strangers, close relatives, and mobile operators (Q28).

*Part G. Preference of Security Mechanisms.* We asked respondents to take into account both security and usability, and rank various security mechanisms (e.g., SMS, SQs, e-mail [24, 25, 26]) as a second factor of authentication, from the most preferred to the least preferred (Q29). Additionally, we asked the respondents to rank different categories of SQs that they would prefer to be queried about (Q30). We followed the study of Golbeck and Li [17] to determine the following categories: Aspirations, Favourites, Firsts, Locations, Names, and Numbers. We randomized the order of categories.

*Part H. General Security Behavior.* We used the 8-item Security Behavior Intentions Scale (SeBIS) [23] to evaluate individuals’ intentions and behaviors concerning online security practices. Through these questions (from Q31 to Q38), we aimed to gain insights into several key aspects: (i) awareness of security measures; (ii) behaviors in response to security incidents; (iii) strategies for managing security; and (iv) consistency in security practices.

### D. Procedure

Prior to survey deployment, we conducted cognitive pretests and a soft launch to proactively identify and address potential issues in the questionnaire design. For the survey, we asked seven master's students, from Concordia University, with backgrounds in computer security and two doctoral students, from the University of Lausanne, with backgrounds in HCI. None of these students were part of the research team of this work. Using their feedback, we refined the question wording to enhance clarity and optimized the answer formats to improve efficiency and comprehension. For questions that respondents found confusing or difficult to understand, we rephrased them to make them more understandable and highlighted certain key elements in bold to draw respondents' attention. Taking into account their suggestions, we modified some options by changing the adverbs. Subsequently, before deployment, we published a soft launch with a sample size of 20 respondents to assess the survey's comprehensiveness and to identify any necessary modifications for enhancement. During the soft launch, we discovered that one question was never displayed due to an error in the condition; we later changed this for the final survey. We did not use the soft launch data for the final analysis. We deployed the survey through Qualtrics.

### E. Data Reliability

Although Prolific serves as a valuable resource for recruiting respondents in order to facilitate high-quality data for

academic research, it might not entirely prevent distracted, unfocused, or impatient respondents. Consequently, we implemented several strategies to mitigate these issues and ensure data quality. More specifically, we removed the respondents<fnref n="8" /> who (1) disagreed with the consent (Q2); (2) selected “no online service relied on SQs” (Q3) or “none” for the proportion of online services where their accounts rely on SQs (Q4); (3) did not complete the questionnaire; in practice, mostly dropped out the survey during the task requiring submission of example URLs and screenshots (Q5 and Q6)—despite being instructed (in the invitation e-mail) to prepare them in advance;<fnref n="9" /> (4) selected the wrong answer in the attention check question (Q26). We also (5) discarded the individuals whose screenshots (Q6) did not display original SQs;<fnref n="10" /> (6) investigated the responses of speeders who were exceptionally fast. Following the guidelines of Matsuura et al. [27], we kept them as their response looked reliable: their answers to open-ended questions were meaningful. Ultimately, we collected N = 292 responses. The median time to complete the survey was 17:28.

### F. Coding Process

To analyze the responses to the open-ended questions, we used the affinity diagramming method [28] for categorization. The second author performed the initial coding. The codebook for the open-ended questions is in Supplementary 2.

### G. Survey Limitations

Our respondents are all drawn from the US, which, while offering meaningful regional insights, may limit the generalizability of our findings. Furthermore, most respondents have university-level education and technical expertise, which may introduce bias by excluding the perspectives of those with lower educational or technical backgrounds. Also, the collected data is self-reported and can be affected by recall bias; yet, note that by asking the respondents to review their accounts and prepare a URL and screenshot of a website that uses SQs, we limited this bias.

### H. Ethics

The survey was approved by our institutional review board (IRB). All aspects of this study were conducted with the utmost attention to respondent privacy and data protection. We informed the respondents about the nature of the survey and obtained their consent. The respondents were informed of their right to withdraw at any time, with a guarantee that their data would be deleted upon withdrawal. No personally identifiable information was collected. Respondents were also reminded to not upload screenshots of the SQs with their responses, thus further ensuring confidentiality.

<footnote n="8">⁸Note that Prolific’s representativeness quotas are applied only to the final eligible respondents who complete the study.</footnote>

<footnote n="9">⁹It ensured that the dataset comprised only truly eligible respondents, not those misrepresent their eligibility-enhancing the validity of our findings.</footnote>

<footnote n="10">¹⁰We used Google Image to identify the images copied from the Web.</footnote>
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2. Practices of websites regarding SQs (reported by the survey respondents).</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 3. Evolution of the proportion of websites that rely on SQs (perceived and reported by the survey respondents).</figcaption></figure>

# IV. RESULTS OF USER SURVEY

## A. User Practices and Experiences

Regarding the prevalence of SQs (Q4), the majority of the respondents reported that less than half of the online services, where they have accounts, rely on SQs (see Figure 2a). Note, however, that people who do not have any accounts on a service that relies on SQs were excluded from our pool (Q3). As for the evolution of the prevalence of SQs (Q7), overall, respondents had the impression that the proportion of online services that rely on SQs decreased over the last years (see Figure 3).

In terms of use cases (Q8), the majority of the respondents (n = 267, 91.4%) reported that SQs are used by online services for letting users regain access to their account (e.g., password recovery). The other two use-cases were for accessing their account, e.g., login (n = 144, 49.3%), and for performing sensitive operations, e.g., money transfer (n = 104, 35.6%). A few respondents (n = 7, 2.4%) provided other purposes than the three offered in the questionnaire. Interestingly, one respondent mentioned that they do not know the purpose of SQs, as they “set up the security questions and never saw them again”. Two respondents provided purposes that correspond to the previously mentioned option of performing sensitive operations (e.g., “cash out [their] earnings”). Finally, four respondents mentioned improved security, related to the context (hence not systematic, like CAPTCHAs) and not to the operation performed (e.g., “using a different device”).

The vast majority of the respondents reported that the services, where they have accounts, provided less than ten

SQs (Q9) to choose from when registering their accounts; see Figure 2b. Regarding the number of SQs they were required to set up upon registration (Q10), the most frequent response was one of three questions (n = 157, 53.8%), followed by two (n = 85, 29.1%) and one (n = 33, 11.3%). As for the number of SQs they were required to answer on the usage page (Q12), e.g., login and password recovery, the vast majority reported only one (n = 156, 53.4%) or two (n = 82, 28.1%).

Regarding the type of SQs used by online services, the respondents were asked about the prevalence of questions based on their use of that service (Q13), e.g., “What is the approximate amount of the last transaction on your bank account?”. Apparently, such questions are not widely used in practice with the large majority of respondents reporting that none (n = 116, 39.7%) or almost none (n = 87, 29.8%) of the services rely on such questions.

We asked the respondents about their general approaches to selecting SQs when registering their accounts (Q11); see Table II at Appendix A for the results summary. About half of the respondents (n = 153, 52.4%) reported selecting questions for which the answers are easy to remember: “Ones that I know off the top of my head without using any memory whatsoever.” [Woman, 48 y.o., White]. A substantial proportion of the respondents (n = 52, 17.8%) mentioned selecting questions with definitive and unique answers: “Many of the questions are too wishy-washy for me since there are multiple possible answers. Those include questions about ‘my favorite [thing],’ since my favorite changes frequently.” [Agender, 61 y.o., White]. Many respondents (n = 89, 30.5%) mentioned choosing questions whose answers are not publicly available or easily found online or questions that require complex answers: “I choose really offbeat ones: Things that I’d never talk to anyone about and things that they’d have a really hard time figuring out.” [Man, 32 y.o., White]. A few respondents (n = 6, 2.1%) mentioned privacy concerns, emphasizing that the questions should not compromise their privacy by asking personal questions or those about their relatives (i.e., interdependent privacy [29]). We also observed that a small proportion of respondents (n = 15, 5.1%) did not have a deterministic approach: they would randomly choose the question, select the first available one, follow their gut feeling, or consistently select the same question(s). Four respondents (n = 4, 1.4%) reported that they would select questions whose answers are known to people close to them (e.g., trusted family and friends): “Choosing ones, my spouse will know.” [Man, 60 y.o., White]. This could be because they might need help remembering their answer, or because they might want people close to them to be able to recover access to their accounts (e.g., in case of an emergency). Although this makes sense, this weakens the security of SQs from the perspective of a close relative or intimate partner being the adversary [30].

Interestingly, the responses covered all the desirable characteristics of SQs listed in OWASP’s recommendations [14]: memorable, consistent, applicable, confidential, and specific.
<figure><img src="image_7.png" /><figcaption>(a) Frequency of providing truthful answers to SQs.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Comfort wrt giving away personal information in answers to SQs.</figcaption></figure>

<caption>Fig. 4. Respondents' strategies for answering SQs.</caption>

## B. User Strategies for Answering

When asked whether they provide different responses to the same SQs across different services (Q14), only 16.1% (n = 47) of the respondents reported at least "frequently" providing different answers, whereas the majority mentioned that they never do so (n = 113, 38.7%), or do so very rarely (n = 61, 20.9%). In terms of answering truthfully vs. untruthfully when respondents registered their accounts (Q15), as shown in Figure 4a, the majority of the respondents usually provide truthful responses ("always": n = 143, 49.0%; "very frequently": n = 43, 14.7%; "frequently": n = 35, 12.0%).

We also asked them to describe (in free text) their reasons for providing either truthful or untruthful answers (Q16 and Q17; see Table II). An overwhelming majority of respondents (n = 236, 80.8%) indicated that they provide truthful answers for better recall: "To paraphrase Mark Twain, if you tell the truth, you do not have to remember anything. [...] If I were using a made-up name [...], that would be harder to remember. That would be almost as bad as having to remember a password." [Woman, 64 y.o., Black]. Some respondents (n = 12, 4.1%) mentioned providing truthful answers due to incorrect mental models. For example, they were unaware that they could provide untruthful answers: "I never thought not to! This is a good question!" [Woman, 42 y.o., Black], or they believed that the service provider would verify the answers, thus potentially leading to negative consequences: "If it's a financial service, like a bank, student loan, or whatnot, they may have access to the real answers from the original creation. Lying makes me look suspicious potentially." [Woman, 38 y.o., White]. This perception, though understandable, is problematic as it suggests a misunderstanding of how SQs operate. Such a belief could be the result of dark patterns, where users might feel pressured to provide truthful answers (thus exposing their privacy), because they mistakenly think that the service already knows the correct response. Some respondents (n = 10, 3.4%) mentioned honesty as their reason for providing truthful

answers, expressing that they do not like to lie or see no reason to do so: "I am a person that does not lie much. My first reaction would be to use the truth." [Woman, 46 y.o., Mixed].

Among those who provide untruthful answers, several respondents (n = 55, 18.8%) do so to make them harder to guess. A substantial proportion of them (n = 58, 19.9%) mentioned that they provide untruthful responses, primarily when they are concerned about security, need more protection, deal with a sensitive service, or do not trust the service provider. "If it is a question that I am concerned about security with, or I think it might be something that someone could figure out easily, I might slightly alter the answer." [Woman, 39 y.o., White]. A smaller group of respondents (n = 12, 4.1%) mentioned providing untruthful answers when necessary or when possible, such as when they do not know the true answer, the answer does not make sense to them, the service provider does not accept specific formatting (e.g., spaces or special characters), or the question allows for ambiguity in responses: "When they're tropey things no one would guess but would make perfect sense to me: 'What is your favorite smell?' and I respond 'Nothing'." [Woman, 42 y.o., White]. Lastly, we observed a notable practice among a few respondents (n = 3, 1.0%) who reported using the same untrue answers for all SQs to enhance ease of recall. Although providing untrue answers might protect them, providing the same (yet untrue) answer on different services compromises their security (e.g., if their answer on one of the services is exposed).

We asked the respondents how comfortable they would be sharing personal information when answering SQs (Q18). Overall, the largest group of respondents (n = 117, 40.1%) expressed some level of comfort, yet a substantial proportion (n = 107, 36.6%) declared being uncomfortable, as shown in Figure 4b. Neutral responses accounted for a substantial portion of the group (n = 68, 23.3%), indicating a mixed overall sentiment.

## C. Answer Recall

About 20.5% (n = 60) of the respondents thought that it was difficult to remember their responses to SQs (Q19; Figure 5a), with the majority finding it "easy" (n = 110, 37.7%), or "extremely easy" (n = 41, 14.0%). We then examined the frequency with which respondents forget their responses to s (Q22; Figure 5b). Note that people might remember the answer to their SQs but not their exact response because of the variety of ways to spell it (e.g., upper/lower cases, spaces, hyphens, spelling of non-Latin characters in Latin). Note that we did not distinguish between the case where people forget the answer (i.e., the true response) and the case where they forget *their* response. The results indicated that only a few respondents forget the responses to SQs ("always": n = 5, 1.7%; "very frequently": n = 8, 2.7%; "frequently": n = 12, 4.1%).

We asked the respondents whether they use any mechanism to help them recall their responses (Q20), and how frequently they do so (Q21); Table III summarizes these findings. Password managers (n = 107, 36.6%) and regular written notes
<figure><img src="image_9.png" /><figcaption>Fig. 5. Memorability of SQs (reported by the respondents).</figcaption></figure>

(n = 67, 22.9%) are the two most common mechanisms. The relatively high usage of password managers may be explained by our sample's higher technical awareness, as reflected in their SeBIS score, education level, and perceived tech-savviness. However, to the best of our knowledge, though password managers often offer the option to store generic notes (that can accommodate answers to SQs), none of the popular ones appear to offer the possibility to automatically save and/or fill the answers to SQs directly on webpages (as done for passwords). This is probably because there is no web standard associated with SQs (unlike for passwords, for which HTML elements such as `<input type="password">` can be used). Notably, 22.9% (n = 67) of respondents did not use any specific mechanism. Regarding the extent to which they rely on such mechanisms, we observed a tendency toward the lower end of the scale ("none": n = 21, 9.3%; "almost none": n = 40, 17.8%; "slightly less": n = 38, 16.9%).

## D. Perceived Effectiveness

We assessed the respondents' perceptions of SQ protection level compared to passwords (Q23). More respondents (n = 117, 40.1%) indicated that SQs provided better protection than passwords, whereas 31.5% (n = 92) believed that passwords offered better protection (the rest perceived them as equivalent). Respondents were asked (Q24) to elaborate on the reasons why they think the level of protection provided by SQs is better/same/worse than provided by passwords. Some respondents who perceived SQs as more secure than passwords highlighted various advantages (n = 49, 16.8%). These advantages included SQs being more resistant to hacking, such as brute-force attacks, and the requirement of multiple SQs compared to a single password, which makes passwords more vulnerable. Conversely, some respondents pointed out the security advantages of passwords over SQs (n = 47, 16.1%), such as the predictability of SQ answers, the increased complexity of passwords due to password managers, and the limitations imposed by most websites on the use of special characters in SQs. Additionally, they noted that SQs often lack the strength requirements typically enforced for passwords.

<figure><img src="image_10.png" /><figcaption>Fig. 6. Levels of difficulty for different entities to guess someone's answers to SQs (assessed by the respondents).</figcaption></figure>

While these two groups primarily focused on system dynamics and mechanisms, other respondents emphasized user behavior in their reasoning. Several respondents (n = 37, 12.7%) believed that SQs are more effective because users often employ them securely. They noted that users provide personal, unique, long, and complex answers, making it difficult for others to guess. In contrast, a larger group of respondents (n = 68, 23.3%) believed that passwords are superior due to weak user practices associated with SQs. They pointed out that close relatives and friends might know the answers, or that the answers could be accessible through users' social media due to poor privacy practices. "Most answers are easily found on information I've willingly shared on social media" [Man, 38 y.o., White]. Additionally, they mentioned that scammers could easily find these answers through quick icebreaker questions or social engineering tactics, such as phishing e-mails, phone calls, or conversations. "[Answers] can easily be figured out by online scammers in a quick 'innocent' conversation by asking icebreaker questions." [Genderflux, 29 y.o., White]. Lastly, several respondents (n = 58, 19.9%) perceived passwords and SQs as providing an equivalent level of protection, with mechanisms sharing similar vulnerabilities.

We also asked respondents how effective they find SQs for protecting their online accounts (Q25). Most respondents (n = 190, 65.1%) believed that SQs were effective, with "slightly effective" being the most selected option (n = 88, 30.1%).

Beyond overall effectiveness, an important factor influencing SQ security is the ease with which different entities can guess users' answers (Q28; results shown in Figure 6). Respondents reported that it would not be too difficult for their close relatives and intimate partners to guess their answers. Yet, interestingly, they reported similar levels of difficulty for friends and for e-mail/SMS providers, and higher levels for people with access to their social profiles.

## E. Security Mechanisms Preferences

We asked respondents to consider both security and usability and rank the security mechanisms when used as a second factor of authentication—on top of a password (Q29). The results are shown in Figure 7a. SMS was the most preferred authentication method with a mean rank of 2.8 (Mdn = 2),
<figure><img src="image_11.png" /><figcaption>(a) Ranking of security mechanisms (used as a second factor in addition to a password); A rank of “1” denotes that a mechanism is “*the most preferred*” mechanism. A rank of “7” denotes the “*least preferred*” mechanism.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Ranking of security questions types. A rank of “1” denotes that a question type is “*the most preferred*” type. A rank of “6” denotes “*least preferred*” type.</figcaption></figure>

<caption>Fig. 7. Respondents' preferences regarding different security mechanisms, considering both security and usability. The orange lines indicate the median values.</caption>

while Hardware Tokens were the least preferred with a mean rank of 5.0 (Mdn = 6). SQs are ranked after authenticator mobile apps and e-mails with a mean rank of 3.9 (Mdn = 4). SMS was the most preferred authentication method, likely due to its excellent usability, particularly the seamless autofill feature that automatically pastes the verification code, making the process quick and convenient. However, despite its usability advantages, SMS has security issues that respondents may not be fully aware of, such as susceptibility to shoulder surfing attacks [31], SIM swap attacks [32], and the lack of encryption, which can allow adversaries to read the messages.

Similarly, we inquired about the kinds of SQs respondents preferred to be asked about (Q30). Figure 7b shows the results. While Firsts were the most preferred type with having the lowest mean rank at 3.1, overall, Firsts, Names, Favorites, and Locations were all preferred with having the same median rank (Mdn = 3), possibly suggesting that these types of questions are generally perceived as easier to remember and answer. Aspirations were the least preferred type with the highest mean rank at 4.3 (Mdn = 5), likely because they involve more subjective and potentially changing information, which may introduce uncertainty or difficulty in providing consistent answers over time.

# V. METHODOLOGY OF WEBSITE ANALYSIS

To address RQ3, which investigates the types of SQs and answer requirements currently implemented by websites and their alignment with best practices, we collected and analyzed data on the use of SQs on a selected set of websites.

## A. Data Collection Procedure

We built our dataset of websites, which rely on SQs, from two sources: (1) the findings of Q5 and Q6 from our user survey and (2) manual web searches. For the web search, we relied on Google, using the following templated search string: "$term $category websites online services", where $term was one of terms used to denote SQs (e.g., security question, personal knowledge question; five in total) as identified in our literature review (see Section II), and $category was either empty or picked from the list of typical services that rely on SQs (e.g., insurance, air travel; 32 in total) as identified in our survey results. We manually handled results within the first five pages for each search string, checking whether we could set up SQs after creating accounts. For website categorization, we used Similarweb [33]. However, Similarweb did not provide categories for 23 websites, hence we had to manually categorize them.

For searching, we used a fresh instance of Firefox v125.0.2 for Windows 11 with no plugins/extensions, using a commercial VPN service in the US. For each search, we cleared the state (e.g., cookies) of the browser and used a new private window. For each identified website, we tried to register an account in order to collect data on the ways SQs are used on the website. We excluded websites that require real-world IDs (e.g., US SSN, client ID) or that do not involve SQs during registration. From the websites where we could register an account, we collected the list of SQs they used and other relevant aspects, those matching the questions asked in our survey: e.g., the purpose of SQs such as password recovery (Q8) and the numbers of SQs *offered upon registration* (Q9), *required upon registration* (Q10), and usage (Q12).

## B. Data Analysis

We labeled the collected SQs, based on the categorization of Golbeck and Li [17]. We also assessed whether the collected SQs and the way they were used by the analyzed websites met a number of requirements. For these requirements, we relied on the dedicated guidelines from OWASP [14]. In particular, we assessed whether the SQs possess the desired characteristics (i.e., memorable, consistent, applicable, confidential, specific) and whether they were used properly in the “authentication” and “forgotten password” flows. The assessment was done by the first author. To limit subjectivity, it was then checked by another author and conflicts were resolved through discussions. For the complete list of requirements, see Table VIII in Appendix C. In addition to OWASP’s recommendations, we included a few additional requirements—some borrowed from password research [34], notably how special characters are handled and how responses to SQs are validated (e.g., case sensitivity, handling of spaces, punctuation, and symbols).

## C. Data Collection Limitations

First, we were unable to analyze the requirements of SQs on many websites because they required real-world identification, such as Social Security Numbers. The most prob-
lematic categories were: Banking Credit and Lending, Law and Government, Investing, and Insurance. These requirements posed significant technical, but also legal and ethical, barriers to accessing websites for a thorough analysis. Although we identified 146 websites offering SQs from the survey, only 35 out of 146 were ultimately suitable for detailed analysis. A detailed breakdown of the categories of websites from our survey and web searches—where we successfully created accounts and where we failed—is provided in *Supplementary 3*.

Second, OWASP outlines several characteristics that SQs should meet to be effective. However, assessing some of these requirements is inherently subjective, as there is no standardized evaluation method.

## VI. RESULTS OF WEBSITE ANALYSIS

We collected 146 unique<fnref n="11" /> websites from our user survey (from the 292 URLs provided by the respondents), and 86 unique websites from our web search; thus totaling 212 unique websites. As we included website categories in our web searches, thus biasing the results, we report only on the categories of the 146 websites collected through our user survey (see Table VI). For the rest of the analysis, however, we report on the characteristics of all the collected websites (on which we were able to register accounts, i.e., 73 in total).

The five most frequent categories for the collected websites—excluding those from the web search—are “Banking Credit and Lending” (n = 39, 26.7%), “Law and Government” (n = 16, 11.0%), “Insurance” (n = 8, 5.5%), “Investing” (n = 8, 5.5%), “Financial Planning and Management” (n = 6, 4.1%), “Telecommunications” (n = 6, 4.1%), and “Finance” (n = 6, 4.1%).

Aside from 42.5% websites (n = 31) that did not implement re-authentication, passwords (n = 24, 32.9%) were the most common method. A website also required simultaneously current SQs and date of birth. Other methods included current SQs (n = 9, 12.3%) and SMS (n = 2, 2.7%). A few websites (n = 11, 15.1%) designed a security flow in login authentication based on OWASP. For forgotten password flow, 28.8% of websites (n = 21) complied with OWASP guidelines (authenticated with SQs and e-mail, see Table VII). Contrary to the suggestion that the user's question on the usage page should remain the same until they answer it correctly, users can answer different questions if they input incorrect answers on 12.3% of websites (n = 9, see Table VII).

Our analysis revealed further weaknesses in the implementation of SQs. For example, nearly half of the websites (n = 36, 49.3%) allowed single-character answers, and 6.8% (n = 5) enforced case-sensitive ones. Moreover, only 11.0% of the websites (n = 8) allowed for customized questions, thus limiting the flexibility and personalization of SQs.

Figure 8 illustrates the cumulative distribution of the number of SQs offered, selected, and used across different services. About the usage of SQs on websites, most websites (n = 33,

45.2%) offered between five to ten questions to choose from, while only 4.1% websites (n = 3) provided fewer than five questions. Regarding the number of SQs required during registration, the most websites allowed users to set one question (n = 28, 38.4%), followed by three (n = 21, 28.8%) and two (n = 12, 16.4%). We found that 26.0% of the websites (n = 19) allowed setting SQs but did not use SQs in usage (i.e., login and password reset operations). Thus, we eventually analyzed 54 websites for testing the number of questions required for usage; the distribution is as follows: answering one question is required by 39.7% of the websites (n = 29), two questions by 17.8% (n = 13), three questions by 13.7% (n = 10), and four questions by 1.4% (n = 1).

As for SQs, we collected a total of 1913 questions; after merging equivalent questions, we obtained 627 unique questions. The frequencies of the different categories of SQs, across the entire set of SQs and the set of unique SQs, are provided in Table IV. Specifically, the most frequent questions are “What was the name of your first pet?” (n = 65, 10.4%), “What was the maiden name of your mother?” (n = 46, 7.3%), and “What was the middle name of your father?” (n = 39, 6.2%).

As for the desired characteristics of the SQs, based on our assessment, the proportions of SQs that satisfied these characteristics are summarized in Table V. These findings show that whereas 61.9% of the unique questions (n = 388) were considered applicable, only 3.8% were confidential (n = 24). This indicates a significant reliance on easily discoverable information. Furthermore, only 15.9% of unique questions (n = 100) were deemed specific, thus highlighting a potential weakness in the design of SQs.

### A. Responsible Disclosure

Lastly, we conducted responsible disclosure for any identified security concerns with the websites involved, following best practices for ethical research [35]. We contacted 44 websites to report SQs weaknesses but received four non-automated and 17 automatic responses, none of which addressed the reported weaknesses. For instance, an airline company disregarded the reported weaknesses, responding with (“I understand that you want to know more about our website”) and providing a phone number and a link to their FAQ instead. Similarly, a public administration referred us to a document about the national cybersecurity strategy for the next decade but avoided discussing the weaknesses, citing concerns about disclosing sensitive information. An online service redirected us to its bug bounty program. A template of the responsible disclosure letter is provided in *Supplementary 4*.

## VII. DISCUSSION

Our findings highlight critical security and usability concerns surrounding SQs, emphasizing their impact on both users and website authentication practices. For developers, our results demonstrate the need for clearer SQ policies, stronger security requirements, and better communication with users about how SQs function. For researchers, our study reveals gaps in user understanding that require further exploration,

<footnote n="11">¹¹By “unique”, we mean without duplicates. Throughout the analysis, we report proportions with respect to the total number of unique websites/SQs.</footnote>
particularly regarding untruthful answers and their trade-offs between security and usability. Additionally, we examine how website design choices shape user misconceptions and influence security behaviors. Our findings indicate that unclear communication and misleading interface designs contribute to incorrect mental models of how SQs function, leading users to provide truthful answers even when doing so increases their risk. Next, we discuss the implications of these findings. Before proceeding, we emphasize that we do not advocate for the continued or renewed use of SQs as a preferred authentication mechanism. Our goal is to assess and reduce the risks they pose in contexts where SQs are still in use.

## A. User Misconceptions and Security Weaknesses

One key finding from our user survey is that the misconceptions about the ways SQs operate contribute to insecure behaviors. Some users mistakenly believe that online services already possess the correct answers to personal SQs, which prompts them to provide truthful answers out of fear of being flagged for suspicion. This behavior, combined with users' reluctance to provide false information due to concerns about honesty, reflects how misleading web-design choices or unclear communication can lead to insecurity. Users can feel pressured to reveal truthful information, unaware that providing untruthful answers can be a valid and effective security strategy. This pressure also increases their exposure to social engineering attacks, e.g., via casual conversations or phishing attempts, particularly from friends and relatives, and even strangers.<fnref n="12" /> To mitigate these risks, users should be informed about the importance of varying their answers across services and be encouraged to use password managers for securely storing answers, especially when using more complex responses. Educating users about the flexibility of SQs and promoting these practices could help reduce their weaknesses. These user-centric mitigations could be strengthened by cryptographic frameworks such as K-pop [36], eliminating service providers' access to plaintext answers and reducing the risk of social engineering.

Our findings also highlight insider threats, such as intimate-partner violence [37], as a significant concern. SQs often rely on personal information accessible to those closest to the user, making them vulnerable to exploitation by individuals with social proximity. Protecting users from these insider threats is especially important in cultural contexts where shared-device use is common. To address these vulnerabilities in contexts where SQs are still in use, these mechanisms might need to be redesigned or supplemented/replaced with other options that are more resistant to insider threats, such as biometric authentication. Lastly, websites should consider the type of personal information users share publicly on their social media profiles and should propose SQs that rely on more personalized, less predictable, and non-public behavioral information rather than static, easily accessible details.

## B. Challenges and Regulatory Considerations

Our analysis of website SQ requirements reveals that many websites fail to adhere to established guidelines, such as those from OWASP, thus increasing the risk of brute-force attacks. They commonly allow users to set short easily guessable answers, and their requirements for SQ answers remain weak, with many of them allowing single-character answers or not enforcing complexity standards as they do for passwords. Although increasing the complexity of SQ answers might enhance security, this approach conflicts with the main advantages of SQs: simplicity and memorability. Requiring users to create highly complex answers for SQs introduces usability challenges, such as difficulties in recalling answers. This finding emphasizes the trade-off between security and usability in the use of SQs. Although password managers can help with storing complex answers securely, their use also introduces reliance on external tools, which complicates the balance between security and usability. Additionally, to better understand the broader SQ landscape, future research should explore web developers' perspectives on integrating SQs into authentication systems, their awareness of security risks, the decision-making processes involved, and the challenges they face in adopting alternative mechanisms.

Our study suggests that policymakers and data-protection authorities could play a more active role in evaluating the security of existing SQ implementations, given that these mechanisms remain widely used despite NIST discouragement. We are not proposing changes to laws such as the General Data Protection Regulation (GDPR) or the California Consumer Privacy Act (CCPA). Rather, we highlight that these existing regulatory frameworks already obligate organizations to implement strong data protection measures. In cases where SQs are still used, regulators and/or certification authorities (e.g., for ISO/IEC 27001) may assess whether their design and implementation meet the standard of reasonable security.

# VIII. CONCLUSION

We have examined the ongoing use of SQs for online authentication, despite recommendations against them. Our survey of N = 292 respondents and analysis of 26 security requirements across 73 websites reveal that users often select predictable, easy-to-recall answers, sometimes due to misconceptions about SQs, such as believing that service providers already know the correct answers. This misunderstanding, along with users' tendency to provide accurate but easily guessable answers, increases their vulnerability to exploitation by individuals with social proximity. Our analysis highlights significant gaps in website practices, with many services allowing easily guessable answers and failing to enforce best practices. This emphasizes the trade-off between usability and security, as the simplicity of SQs often compromises defenses.

<footnote n="12">¹²While this misconception was observed in a minority of respondents, we discuss it here due to its potential security implications, as it has not been addressed in prior SQ research.</footnote>
ACKNOWLEDGMENTS

We thank Holly Cogliati for proofreading this arti-
cle. We also thank James Arnéra, Lahari Goswami, Tariq
Hawis, Lamisa Kazi, Philippe Mangeard, Abdelrahman Ragab,

Fahimeh Rezaei, Jiawei Yao, and Xiufen Yu for their par-
ticipation in the cognitive pre-tests of the survey. Lastly,
we acknowledge the support of the Natural Sciences and
Engineering Research Council of Canada (NSERC) for the
contributions of the fourth and fifth authors.
REFERENCES

[1] S. W. Shah and S. S. Kanhere, "Recent Trends in User Authentication – A Survey," *IEEE Access*, vol. 7, pp. 112 505–112 519, 2019. [Online]. Available: https://ieeexplore.ieee.org/document/8784263/

[2] C. Campanile, "Dem Pol's Son Was 'Hacker'," 2008. [Online]. Available: https://web.archive.org/web/20150516052714/http://nypost.com/2008/09/19/dem-pols-son-was-hacker/

[3] P. A. Grassi, M. E. Garcia, and J. L. Fenton, "Digital identity guidelines: revision 3," National Institute of Standards and Technology, Gaithersburg, MD, Tech. Rep. NIST SP 800-63-3, Jun. 2017.

[4] W. J. Haga and M. Zviran, "Question-and-answer passwords: An empirical evaluation," *Information Systems*, vol. 16, no. 3, pp. 335–343, 1991. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/030643799190005T

[5] J. Bonneau, E. Bursztein, I. Caron, R. Jackson, and M. Williamson, "Secrets, Lies, and Account Recovery: Lessons from the Use of Personal Knowledge Questions at Google," in *Proc. of the Int'l Conf. on World Wide Web (WWW)*. Florence, Italy: International World Wide Web Conferences Steering Committee, 2015, pp. 141–150.

[6] S. Höltervennhoff, N. Wöhler, A. Möhle, M. Oltrogge, Y. Acar, O. Wiese, and S. Fahl, "A Mixed-Methods Study on User Experiences and Challenges of Recovery Codes for an End-to-End Encrypted Service," in *USENIX Security Symp*. Philadelphia, PA, USA: USENIX Association, 2024, pp. 7267–7284. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity24/presentation/h%C3%B6ltervennhoff

[7] S. Schechter, A. B. Brush, and S. Egelman, "It's No Secret. Measuring the Security and Reliability of Authentication via "Secret" Questions," in *Proc. of the IEEE Symp. on Security and Privacy (S&P)*. Oakland, CA, USA: IEEE, May 2009, pp. 375–390.

[8] A. Rabkin, "Personal knowledge questions for fallback authentication: security questions in the era of Facebook," in *Proc. of the Symp. on Usable privacy and security (SOUPS)*. Pittsburgh, PA, USA: ACM, 2008, pp. 13–23.

[9] M. Just and D. Aspinall, "Personal choice and challenge questions: a security and usability assessment," in *Proc. of the Symp. on Usable Privacy and Security (SOUPS)*. Santa Clara, CA, USA: Association for Computing Machinery, 2009, pp. 1–11.

[10] L. Lassak, P. Markert, M. Golla, E. Stobert, and M. Dürmuth, "A Comparative Long-Term Study of Fallback Authentication Schemes," in *Proc. of the ACM Conf. on Human Factors in Computing Systems (CHI)*. Honolulu, HI, USA: Association for Computing Machinery, 2024, pp. 1–19.

[11] R. AlHusain and A. Alkhalifah, "Evaluating knowledge-based security questions for fallback authentication," *PeerJ Computer Science*, vol. 8, p. e903, 2022. [Online]. Available: https://peerj.com/articles/cs-903

[12] R. Dhekane, "Towards a usable fallback authentication mechanism," Ph.D. dissertation, CSU, Sacramento, CA, USA, 2020.

[13] N. Micallef and N. A. G. Arachchilage, "Understanding users' perceptions to improve fallback authentication," *Personal and Ubiquitous Computing*, vol. 25, no. 5, pp. 893–910, 2021. [Online]. Available: https://doi.org/10.1007/s00779-021-01571-y

[14] OWASP, "Choosing and Using Security Questions," 2018. [Online]. Available: https://cheatsheetseries.owasp.org/cheatsheets/ Choosing_and_Using_Security_Questions_Cheat_Sheet.html

[15] V. Distler, M. Fassl, H. Habib, K. Krombholz, G. Lenzini, C. Lallemand, L. F. Cranor, and V. Koenig, "A Systematic Literature Review of Empirical Methods and Risk Representation in Usable Privacy and Security Research," *ACM Trans. Comput.-Hum. Interact.*, vol. 28, no. 6, pp. 43:1–43:50, 2021. [Online]. Available: https://dl.acm.org/doi/10.1145/3469845

[16] J. Bonneau, C. Herley, P. C. v. Oorschot, and F. Stajano, "The Quest to Replace Passwords: A Framework for Comparative Evaluation of Web Authentication Schemes," in *Proc. of the IEEE Symp. on Security and Privacy (S&P)*. San Francisco, CA, USA: IEEE, 2012, pp. 553–567.

[17] J. Golbeck and S. Li, "A Large Crowdsourced Dataset of Security Questions," in *Proc. of Who Are You?! Adventures in Authentication Workshop*, Virtual Event, 2020, p. 4. [Online]. Available: https://wayworkshop.org/2020/papers/way2020-golbeck.pdf

[18] M. Golla and M. Dürmuth, "Analyzing 4 Million Real-World Personal Knowledge Questions (Short Paper)," in *Technology and Practice of Passwords*, ser. PASSWORDS'15, vol. 9551. Cambridge, UK: Springer International Publishing, 2016, pp. 39–44.

[19] A. Senarath, N. A. G. Arachchilage, and B. B. Gupta, "Security Strength Indicator in Fallback Authentication: Nudging Users for Better Answers in Secret Questions," 2017, arXiv:1701.03229 [cs].

[20] S. Palan and C. Schitter, "Prolific.ac—A subject pool for online experiments," *Journal of Behavioral and Experimental Finance*, vol. 17, pp. 22–27, 2018. [Online]. Available: https://linkinghub. elsevier.com/retrieve/pii/S2214635017300989

[21] Fairwork, "Fairwork Cloudwork Ratings 2022: Work in the Planetary Labour Market," 2022. [Online]. Available: https://shorturl.at/kgULB

[22] Prolific, "Prolific's payment principles," 2025. [Online]. Available: https://researcher-help.prolific.com/en/article/2273bd

[23] S. Egelman and E. Peer, "Scaling the Security Wall: Developing a Security Behavior Intentions Scale (SeBIS)," in *Proc. of the ACM Conf. on Human Factors in Computing Systems (CHI)*. Seoul, Republic of Korea: ACM, 2015, pp. 2873–2882.

[24] A. Al Abdulwahid, N. Clarke, S. Furnell, I. Stengel, and C. Reich, "The Current Use of Authentication Technologies: An Investigative Review," in *Proc. of the Int'l Conf. on Cloud Computing*. Riyadh, Saudi Arabia: IEEE, 2015, pp. 1–8.

[25] X. Wang, Z. Yan, R. Zhang, and P. Zhang, "Attacks and defenses in user authentication systems: A survey," *Journal of Network and Computer Applications*, vol. 188, p. 103080, 2021. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1084804521001028

[26] C. Gilsenan, F. Shakir, N. Alomar, and S. Egelman, "Security and Privacy Failures in Popular 2FA Apps," in *USENIX Security Symp*. Anaheim, CA, USA: USENIX Association, 2023, pp. 2079–2096. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity23/presentation/gilsenan

[27] T. Matsuura, A. A. Hasegawa, M. Akiyama, and T. Mori, "Careless Participants Are Essential for Our Phishing Study: Understanding the Impact of Screening Methods," in *Proc. of the European Symp. on Usable Security (EuroUSEC)*. Karlsruhe, Germany: Association for Computing Machinery, 2021, pp. 36–47. [Online]. Available: https://dl.acm.org/doi/10.1145/3481357.3481515

[28] A. Lucero, "Using Affinity Diagrams to Evaluate Interactive Prototypes," in *Human-Computer Interaction (INTERACT)*, J. Abascal, S. Barbosa, M. Fetter, T. Gross, P. Palanque, and M. Winckler, Eds. Bamberg, Germany: Springer International Publishing, 2015, pp. 231–248.

[29] M. Humbert, B. Trubert, and K. Huguenin, "A Survey on Interdependent Privacy," *ACM Computing Surveys*, vol. 52, no. 6, pp. 1–40, 2020. [Online]. Available: https://dl.acm.org/doi/10.1145/3360498

[30] K. Levy and B. Schneier, "Privacy threats in intimate relationships," *Journal of Cybersecurity*, vol. 6, no. 1, p. tyaa006, 2020. [Online]. Available: https://academic.oup.com/cybersecurity/article/doi/ 10.1093/cysec/tyaa006/5849222

[31] M. Eiband, M. Khamis, E. von Zezschwitz, H. Hussmann, and F. Alt, "Understanding Shoulder Surfing in the Wild: Stories from Users and Observers," in *Proc. of the ACM Conf. on Human Factors in Computing Systems (CHI)*. Denver, CO, USA: Association for Computing Machinery, 2017, pp. 4254–4265.

[32] K. Lee, B. Kaiser, J. Mayer, and A. Narayanan, "An Empirical Study of Wireless Carrier Authentication for SIM Swaps," in *Proc. of the USENIX Conf. on Usable Privacy and Security (SOUPS)*. Virtual Event: USENIX Association, 2020, pp. 61–79. [Online]. Available: https://www.usenix.org/conference/soups2020/presentation/lee

[33] Similarweb, "All Categories of Similarweb Website Ranking," 2024. [Online]. Available: https://www.similarweb.com/category/

[34] J. Tan, L. Bauer, N. Christin, and L. F. Cranor, "Practical Recommendations for Stronger, More Usable Passwords Combining Minimum-strength, Minimum-length, and Blocklist Requirements," in *Proc. of the ACM SIGSAC Conf. on Computer and Communications Security (CCS)*. Virtual Event: ACM, 2020, pp. 1407–1426. [Online]. Available: https://dl.acm.org/doi/10.1145/3372297.3417882

[35] A. Efremov, "Ethical principles of vulnerability disclosure," May 2020. [Online]. Available: https://www.kaspersky.com/blog/ vulnerability-disclosure-ethics/35581/

[36] R. Little, L. Qin, and M. Varia, "Secure Account Recovery for a Privacy-Preserving Web Service," in *USENIX Security Symp*. Philadelphia, PA, USA: USENIX Association, 2024, pp. 1993–2010. [Online]. Available: https://www.usenix.org/conference/usenixsecurity24/presentation/little

[37] D. Freed, J. Palmer, D. Minchala, K. Levy, T. Ristenpart, and N. Dell, ""A Stalker's Paradise": How Intimate Partner Abusers Exploit Technology," in *Proc. of the ACM Conf. on Human Factors in Computing Systems (CHI)*. Montréal, Canada: Association for Computing Machinery, 2018, pp. 1–13.
# APPENDIX

## A. Additional User Survey Results

<table><caption>TABLE II
SUMMARY OF ANSWERS FOR THREE OPEN-ENDED QUESTIONS.</caption><thead><tr><th>General approach to selecting SQs</th><th>Frequency</th></tr></thead><tbody><tr><td>Questions with easy to remember answers</td><td>n = 153, 52.4%</td></tr><tr><td>Questions that cannot be guessed</td><td>n = 89, 30.5%</td></tr><tr><td>Questions with unique and definitive answers</td><td>n = 52, 17.8%</td></tr><tr><td>No specific approach</td><td>n = 15, 5.1%</td></tr><tr><td>Questions that do not compromise privacy</td><td>n = 6, 2.1%</td></tr><tr><td>Looking for unique or new questions</td><td>n = 5, 1.7%</td></tr><tr><td>Questions that my acquaintances can also answer</td><td>n = 4, 1.4%</td></tr><tr><th>Reasons for providing (un)truthful answers</th><th>Frequency</th></tr><tr><td>Truthful answers for better recall</td><td>n = 236, 80.8%</td></tr><tr><td>Untruthful answers when sensitive context</td><td>n = 58, 19.9%</td></tr><tr><td>Untruthful answers to make answers difficult to guess</td><td>n = 55, 18.8%</td></tr><tr><td>Untruthful answers when necessary or possible</td><td>n = 12, 4.1%</td></tr><tr><td>Truthful answers due to incorrect mental model</td><td>n = 12, 4.1%</td></tr><tr><td>Truthful answers due to honesty</td><td>n = 10, 3.4%</td></tr><tr><td>Truthful answers remain secure</td><td>n = 10, 3.4%</td></tr><tr><td>Risky practices in providing untruthful answers</td><td>n = 3, 1.0%</td></tr><tr><th>Justify why SQs vs. password</th><th>Frequency</th></tr><tr><td>Prefer passwords bc. of users' weak security practices</td><td>n = 68, 23.3%</td></tr><tr><td>Equivalence of SQs and passwords</td><td>n = 58, 19.9%</td></tr><tr><td>Prefer SQs due to their secure mechanisms</td><td>n = 49, 16.8%</td></tr><tr><td>Prefer password due to weak mechanisms of SQs</td><td>n = 47, 16.1%</td></tr><tr><td>Complementary roles of SQs and passwords</td><td>n = 39, 13.4%</td></tr><tr><td>Prefer SQs bc. of users' robust security practices</td><td>n = 37, 12.7%</td></tr><tr><td>Prefer passwords due to SQs' lack of ease of use</td><td>n = 7, 2.4%</td></tr><tr><td>Prefer SQs due to their ease of use</td><td>n = 3, 1.0%</td></tr></tbody></table>

<table><caption>TABLE III
MECHANISMS USED BY THE RESPONDENTS FOR RECALLING THEIR ANSWERS TO SQS.</caption><thead><tr><th>Mechanism</th><th>Frequency</th></tr></thead><tbody><tr><td>Password manager</td><td>n = 107, 36.6%</td></tr><tr><td>Regular written notes (e.g., a physical notebook)</td><td>n = 67, 22.9%</td></tr><tr><td>Regular digital notes (e.g., simple file)</td><td>n = 53, 18.2%</td></tr><tr><td>Secure digital notes (e.g., Standard Notes, Joplin)</td><td>n = 34, 11.6%</td></tr><tr><td>Secure written notes (e.g., a physical note in a vault)</td><td>n = 26, 8.9%</td></tr><tr><td>None</td><td>n = 67, 22.9%</td></tr></tbody></table>

## B. Additional Website Analysis Results

<table><caption>TABLE IV
CATEGORIES OF THE COLLECTED SQS.</caption><thead><tr><th>Category</th><th>Frequency (all)</th><th>Frequency (unique)</th></tr></thead><tbody><tr><td>Aspirations</td><td>n = 32, 1.7%</td><td>n = 12, 1.9%</td></tr><tr><td>Favorites</td><td>n = 439, 22.9%</td><td>n = 160, 25.5%</td></tr><tr><td>Firsts</td><td>n = 605, 31.6%</td><td>n = 175, 27.9%</td></tr><tr><td>Locations</td><td>n = 329, 17.2%</td><td>n = 103, 16.4%</td></tr><tr><td>Names</td><td>n = 1373, 71.8%</td><td>n = 420, 67.0%</td></tr><tr><td>Numbers</td><td>n = 50, 2.6%</td><td>n = 29, 4.6%</td></tr><tr><td>Others</td><td>n = 23, 1.2%</td><td>n = 19, 3.0%</td></tr><tr><td>Seconds</td><td>n = 5, 0.3%</td><td>n = 3, 0.5%</td></tr><tr><td>Times</td><td>n = 44, 2.3%</td><td>n = 25, 4.0%</td></tr></tbody></table>

<table><caption>TABLE V
FIVE DESIRED OWASP CHARACTERISTICS ANALYZED IN THE COLLECTED SQS.</caption><thead><tr><th>Characteristic</th><th>Frequency (all)</th><th>Frequency (unique)</th></tr></thead><tbody><tr><td>Memorable</td><td>n = 789, 41.2%</td><td>n = 181, 28.9%</td></tr><tr><td>Consistent</td><td>n = 841, 44.0%</td><td>n = 227, 36.2%</td></tr><tr><td>Applicable</td><td>n = 1190, 62.2%</td><td>n = 388, 61.9%</td></tr><tr><td>Confidential</td><td>n = 39, 2.0%</td><td>n = 24, 3.8%</td></tr><tr><td>Specific</td><td>n = 539, 28.2%</td><td>n = 100, 15.9%</td></tr></tbody></table>

<table><caption>TABLE VI
ALL CATEGORIES OF WEBSITES FROM THE SURVEY (ACCORDING TO SIMILARWEB). ONLY CATEGORIES WITH $n \ge 4$ ARE SHOWN HERE; THE FULL TABLE IS AVAILABLE IN SUPPLEMENTARY 5.</caption><thead><tr><th>Category</th><th>Frequency</th></tr></thead><tbody><tr><td>Banking Credit and Lending</td><td>n = 39, 26.7%</td></tr><tr><td>Law and Government</td><td>n = 16, 11.0%</td></tr><tr><td>Insurance</td><td>n = 8, 5.5%</td></tr><tr><td>Investing</td><td>n = 8, 5.5%</td></tr><tr><td>Financial Planning and Management</td><td>n = 6, 4.1%</td></tr><tr><td>Telecommunications</td><td>n = 6, 4.1%</td></tr><tr><td>Finance</td><td>n = 6, 4.1%</td></tr><tr><td>Health</td><td>n = 5, 3.4%</td></tr><tr><td>Computers Electronics and Technology</td><td>n = 5, 3.4%</td></tr><tr><td>Universities and Colleges</td><td>n = 5, 3.4%</td></tr><tr><td>Video Games Consoles and Accessories</td><td>n = 5, 3.4%</td></tr><tr><td>Government</td><td>n = 5, 3.4%</td></tr><tr><td>Air Travel</td><td>n = 4, 2.7%</td></tr></tbody></table>

<table><caption>TABLE VII
ALLOWANCE OF DIFFERENT FEATURES BASED ON REQUIREMENT ANALYSIS (†) AND THE OWASP GUIDELINES (‡).</caption><thead><tr><th>Requirement</th><th>Not allowed</th><th>Allowed</th></tr></thead><tbody><tr><td>Customized questions†</td><td>n = 65, 89.0%</td><td>n = 8, 11.0%</td></tr><tr><td>Same answer†</td><td>n = 20, 27.4%</td><td>n = 25, 34.2%</td></tr><tr><td>Digits†</td><td>n = 2, 2.7%</td><td>n = 71, 97.3%</td></tr><tr><td>Case-sensitive†</td><td>n = 51, 69.9%</td><td>n = 5, 6.8%</td></tr><tr><td>Symbols in the settings†</td><td>n = 28, 38.4%</td><td>n = 45, 61.6%</td></tr><tr><td>Symbols in the usage†</td><td>n = 3, 4.1%</td><td>n = 27, 37.0%</td></tr><tr><td>Space characters in the settings†</td><td>n = 4, 5.5%</td><td>n = 69, 94.5%</td></tr><tr><td>Space characters in the usage†</td><td>n = 7, 9.6%</td><td>n = 39, 53.4%</td></tr><tr><td>Forgotten password flow‡</td><td>n = 15, 20.5%</td><td>n = 21, 28.8%</td></tr><tr><td>Fixed questions‡</td><td>n = 9, 12.3%</td><td>n = 4, 5.5%</td></tr></tbody></table>

<figure><img src="image_13.png" /><figcaption>Fig. 8. Cumulative distribution function (CDF) of the numbers of SQs offered, selected, or used (by the service, e.g., for authentication) from analyzed websites.</figcaption></figure>
### C. List of requirements for security questions

<table><caption>TABLE VIII
DESCRIPTION OF THE CRITERIA USED IN THE SQ REQUIREMENT ANALYSIS.</caption><thead><tr><td>Requirements (i.e., criteria)</td><td>Description</td></tr></thead><tbody><tr><td>Allowance of customized questions</td><td>Does the website allow users to write their own questions?</td></tr><tr><td>Allowance of digits</td><td>Does the website allow users to input digits in an answer?</td></tr><tr><td>Allowance of punctuation marks</td><td>Does the website allow users to input punctuation marks?</td></tr><tr><td>Allowance of same answer</td><td>Does the website allow users to set same answer if multiple questions needed?</td></tr><tr><td>Allowance of same question</td><td>Does the website allow users to set same question if multiple questions needed?</td></tr><tr><td>Allowance of space characters in the settings</td><td>Does the website allow users to input space characters on the settings page (registration)?</td></tr><tr><td>Allowance of space characters in the usage</td><td>Does the website allow users to input space characters on the usage page (e.g. login page)?</td></tr><tr><td>Allowance of symbols in the settings</td><td>Does the website allow users to input symbols on the settings page (registration)?</td></tr><tr><td>Allowance of symbols in the usage</td><td>Does the website allow users to input symbols on the usage page (e.g. login page)?</td></tr><tr><td>Authentication flow</td><td>Does the website follow login flow in OWASP guidance?</td></tr><tr><td>Case-sensitive</td><td>Does the website treat uppercase and lowercase letters as distinct?</td></tr><tr><td>Fixed questions</td><td>Do the questions remain consistent until users complete verification if multiple questions are set?</td></tr><tr><td>Forgotten password flow</td><td>Does the website follow forgotten password flow in OWASP guidance?</td></tr><tr><td>Maximum length</td><td>What is the maximum number of characters in an answer a user can input?</td></tr><tr><td>Minimum length</td><td>What is the minimum number of characters in an answer a user can input?</td></tr><tr><td>Minimum number of digits</td><td>What is the minimum number of digits in an answer a user can input?</td></tr><tr><td>Minimum number of letters</td><td>What is the minimum number of letters in an answer a user can input?</td></tr><tr><td>Minimum number of letters or digits</td><td>What is the minimum number of letters or digits in an answer a user can input?</td></tr><tr><td>Minimum number of lower case letters</td><td>What is the minimum number of lower case letters in an answer a user can input?</td></tr><tr><td>Minimum number of upper case letters</td><td>What is the minimum number of upper case letters in an answer a user can input?</td></tr><tr><td>Number of the questions offered in the source</td><td>How many questions does the website offer for users to select?</td></tr><tr><td>Number of the questions in the settings</td><td>How many questions does the website allow users to set up on the settings page (registration)?</td></tr><tr><td>Number of the questions in the usage</td><td>How many questions do users answer on the usage page (e.g. login page)?</td></tr><tr><td>Re-authentication methods</td><td>What method does the website implement for authentication when users edit their security questions?</td></tr><tr><td>Tries threshold</td><td>How many attempts does the website allow for an answer?</td></tr><tr><td>Usage</td><td>What is the usage of security questions on the website?</td></tr></tbody></table>
# D. Survey Transcript

Note: Coding rules are colored in gray (not visible to respondents)

## 1) Main Survey:

### Sec. A. Introduction and Consent

**Q1.** [Text Entry.] What is your Prolific ID? Please note that this response should **auto-fill** with the correct ID. [ ] (text)

### STUDY

You are invited to participate in a research study about people's perceptions of and responses to security questions on online services (e.g., "What is your mother's maiden name?", which could be used to secure access to users' accounts). This study is conducted by researchers from Concordia University (Montreal, Canada) and the University of Lausanne (Switzerland). We ask you to take a survey that takes approximately **15 minutes to complete**.

### PARTICIPATION CRITERIA

To be eligible for this study, you must be a user of at least one online service that relies on security questions.

### YOUR RIGHTS

You will be paid £2.75 (~USD 3.5) for your complete participation in the study. You may choose to terminate your participation in this study at any time and for any reason. However, if you choose to terminate your participation, you will not be compensated, and your data will be deleted. If you participate, your answers will be kept **confidential**. Also, we do not collect any personally identifying information, such as your name and e-mail address. All data will be **stored on a secured server**, and only the researchers participating in this study can access it.

The results of this research study will be used solely for academic research and might be published in scientific journals and conferences. Any published information will be **aggregated** and/or **de-identified**.

As a precautionary measure, in your responses to this survey, please **refrain from providing any responses that you usually give to the security questions you receive online**.

**Q2.** [Single Selection.] **CONSENT**

If you wish to participate in this research study and you meet the participation criteria, please select the **“Agree”** option to continue. It will indicate that you fulfill the participation criteria, that you will answer all questions truthfully, and that you consent to our use of the collected data under the conditions stated above. If you select the **“Disagree”** option, you will not participate in this research study and will not be paid.

a) Agree

b) Disagree [Terminate.]

### Sec. B. Sample Website that use Security Question

**Q3.** [Single Selection.] Does any of the online services, where you have an account, rely on **security questions** (e.g., "What is your mother's maiden name?")?

a) Yes

b) No [Terminate.]

**Q4.** [Single Selection] Approximately what proportion of the online services where you have an account rely on security questions?

a) 1 - None [Terminate.]

b) 2 - Almost None

c) 3 - Slightly Less

d) 4 - About Half

e) 5 - Slightly More

f) 6 - Almost All

g) 7 - All

**Please note:** For the following questions, only accurate URLs and relevant screenshots of security questions will be accepted. Respondents who submit incorrect or irrelevant URLs/screenshots **will not be compensated**.

**Q5.** [Text Entry.] Please provide the URL (link) of the website of one such online service where you have an account. [ ] (text)

**Q6.** [File Upload.] Please provide a screenshot showing the security questions (**but not your responses**) for one such online service where you have an account.

Only PDF and graphic files (e.g., PNG, JPG) are accepted. Here is an example (from Allstate).

<figure><img src="image_14.png" /></figure>

For instructions on how to take a screenshot on your device, click on the appropriate link: Android, iOS, Windows, macOS, Linux. [Drop files or click here to upload] (file upload)

### Sec. C. User Practices and Experiences

**Q7.** [Single Selection.] Based on your experience, the proportion of online services that rely on security questions has recently

a) 1 - Drastically Decreased

b) 2 - Decreased

c) 3 - Slightly Decreased

d) 4 - About the Same

e) 5 - Slightly Increased

f) 6 - Increased

g) 7 - Drastically Increased

**Q8.** [Multiple Selection.] For what purpose do these online services, where you have an account, rely on security questions?
(Select all that apply)

a) To access my account (e.g., login)

b) To regain access my account (e.g., password recovery)

c) To perform sensitive operations (e.g., transferring large amounts of money, modifying important account settings)

d) Other (please specify) [ ] (text)

**Q9.** [Single Selection.] Approximately how many different security questions do the online services you use typically **offer you to select from, when registering your account**?

a) Less than 5

b) 5-10

c) 11-15

d) 16-20

e) More than 20

**Q10.** [Single Selection.] Approximately how many security questions do your online services typically **require you to select when registering your account**?

a) 1

b) 2

c) 3

d) 4

e) More than 4

**Q11.** [Open-ended.] What is your **general approach for selecting** security questions when **registering** your account? [ ] (text)

**Q12.** [Single Selection.] Approximately how many security questions are typically **required when you use your online services** (e.g., to login)?

a) 1

b) 2 [Display this option if Q10$=2 or Q10$=3 or Q10$=4 or v$=More than 4]

c) 3 [Display this option if Q10$=3 or Q10$=4 or Q10$=More than 4]

d) 4 [Display this option if Q10$=4 or Q10$=More than 4]
e) More than 4 [Display this option if Q10$=More than 4]

**Q13.** [Single Selection.] Among your services that rely on security questions, approximately what proportion of them rely on questions based on your use of that service?

Examples: from your bank: "approximate amount of the last transaction on your bank account", from your mobile operator: "phone number you call most often", etc.

a) 1 - None

b) 2 - Almost None

c) 3 - Slightly Less

d) 4 - About Half

e) 5 - Slightly More

f) 6 - Almost All

g) 7 - All

Sec. D. User Strategies for Answering

**Q14.** [Single Selection.] How often do you use different responses to the **same security questions** across multiple online services (e.g., declaring your favorite food as "Pizza" on one service and as "Pasta" on another)?

a) 1 - Never

b) 2 - Very Rarely

c) 3 - Rarely

d) 4 - Occasionally

e) 5 - Frequently

f) 6 - Very Frequently

g) 7 - Always

**Q15.** [Single Selection.] In general, how frequently do you provide the **truthful answers** (e.g., the **actual** maiden name of your mother) to security questions **when registering your account**?

a) 1 - Never

b) 2 - Very Rarely

c) 3 - Rarely

d) 4 - Occasionally

e) 5 - Frequently

f) 6 - Very Frequently

g) 7 - Always

[Display Q16 if Q15$!=1 - Never]

**Q16.** [Open-ended.] Please briefly describe your reasons for providing **truthful** answers to security questions, when registering your account. [ ] (text)

[Display Q17 if Q15$!=7 - Always]

**Q17.** [Open-ended.] Please briefly describe your reasons for providing **untruthful** answers to security questions, when registering your account. [ ] (text)

**Q18.** [Single Selection.] How comfortable are you with giving away personal information to an online service when providing responses for the security questions?

a) 1 - Extremely Uncomfortable

b) 2 - Uncomfortable

c) 3 - Slightly Uncomfortable

d) 4 - Neither Comfortable Nor Uncomfortable

e) 5 - Slightly Comfortable

f) 6 - Comfortable

g) 7 - Extremely Comfortable

Sec. E. Answer Recall

**Q19.** [Single Selection.] How difficult/easy do you find it to remember your responses to security questions?

a) 1 - Extremely Difficult

b) 2 - Difficult

c) 3 - Slightly Difficult

d) 4 - Neither Difficult Nor Easy

e) 5 - Slightly Easy

f) 6 - Easy

g) 7 - Extremely Easy

**Q20.** [Multiple Selection.] Which of the following mechanisms do you use to help you recall the responses (to the security questions) you provided when registering the account?

Select all that apply.

a) Password manager

b) Regular digital notes (e.g., simple file)

c) Secure digital notes (e.g., Standard Notes, Joplin)

d) Regular written notes (e.g., a physical notebook)

e) Secure written notes (e.g., a physical notebook stored in a vault)

f) Other (please specify) [ ] (text)

g) None

[Display Q21 if Q20$!=None]

**Q21.** [Single Selection.] To what extent do you use these mechanisms (i.e., to help you recall the responses) on the online services that rely on security questions?

a) 1 - None

b) 2 - Almost None

c) 3 - Slightly Less

d) 4 - About Half

e) 5 - Slightly More

f) 6 - Almost All

g) 7 - All

**Q22.** [Single Selection.] How frequently do you forget the response you provided to a security question (when registering the account)?

a) 1 - Never

b) 2 - Very Rarely

c) 3 - Rarely

d) 4 - Occasionally

e) 5 - Frequently

f) 6 - Very Frequently

g) 7 - Always

Sec. F. Perceived Effectiveness

**Q23.** [Single Selection.] In your opinion, **compared to** the level of protection provided by **passwords**, the level of protection provided by **security questions** is:

a) Much Worse

b) Worse

c) Slightly Worse

d) About the Same

e) Slightly Better

f) Better

g) Much Better

**Q24.** [Open-ended.] Please briefly describe why you think the level of protection provided by security questions is [selected choice at Q23] than that provided by passwords. [ ] (text)

**Q25.** [Single Selection.] How effective do you find security questions for protecting your online accounts?

a) 1 - Strongly Ineffective

b) 2 - Ineffective

c) 3 - Slightly Ineffective

d) 4 - Neither Effective Nor Ineffective

e) 5 - Slightly Effective

f) 6 - Effective

g) 7 - Strongly Effective

**Q26.** [Single Selection.] Please select "3 - Slightly Disagree" to show you are paying attention to this question. [Attention check]

a) 1 - Strongly Disagree

b) 2 - Disagree

c) 3 - Slightly Disagree

d) 4 - Neither Agree Nor Disagree

e) 5 - Slightly Agree

f) 6 - Agree

g) 7 - Strongly Agree

**Q27.** [Single Selection.] Do you have any online social network accounts (e.g., Facebook, Twitter)?

a) Yes

b) No

When answering the following question, please note that a **public** social media profile means that anyone can see your posts/stories, and a **private** social media profile means that only your friends/contacts/followers can see them.
**Q28.** [Grid question.] How difficult/easy do you think it is for the following entities to guess — in a few tries (1-3) — your responses to security questions?
For entities that do not apply, please imagine and let us know.

Row options:

- Strangers

- Close relatives (parents, siblings)

- Intimate partners

- Colleagues

- Friends

- Anyone with access to your **PUBLIC** social network profiles (information such as gender and age), posts, photos, etc. [Display this option if Q27$=Yes]

- Anyone with access to your **PRIVATE** social network profiles (information such as gender and age), posts, photos, etc. [Display this option if Q27$=Yes]

- Your e-mail provider (e.g., Gmail)

- Your mobile operator (e.g., AT&T)

a) Extremely Difficult

b) Moderately Difficult

c) Slightly Difficult

d) Neither Easy Nor Difficult

e) Slightly Easy

f) Moderately Easy

g) Extremely Easy

## Sec. G. Preference of Security Mechanisms

**Q29.** [Rank Order. Order randomized.] Taking into account both **security and usability**, please rank the following security mechanisms — when used as a second factor of authentication (on top of a password) — from the most preferred to the least preferred:

a) SMS

b) Security questions

c) Third-party authenticator mobile app (e.g., Google Authenticator)

d) Hardware Tokens (e.g., bank smartcard and HSBC Secure Key)

e) E-mail

f) Voice call

g) Mobile app (i.e., the mobile app provided by the online service provider, such as HSBC Mobile Banking App)

**Q30.** [Rank Order. Order randomized.] Considering both **security and usability**, which kind of questions would you prefer to be asked about? Please rank them from the most preferred to the least preferred one.

a) Firsts (e.g., What was the first movie you saw in the theater?)

b) Names (e.g., What is your mother's maiden name?)

c) Favorites (e.g., What is your favorite music?)

d) Numbers (e.g., What was your childhood phone number, including area code?)

e) Aspirations (e.g., What is your dream job?)

f) Locations (e.g., In which city did you meet your spouse/significant other?)

## Sec. H. General Security Behavior (SeBIS) [23]

Please indicate how often you use the following technology-related practices.

**Q31.** [Single Selection.] I set my computer screen to automatically lock if I don't use it for a prolonged period of time.

**Q32.** [Single Selection.] I use a password/passcode to unlock my laptop or tablet.

**Q33.** [Single Selection.] I manually lock my computer screen when I step away from it.

**Q34.** [Single Selection.] I use a PIN or passcode to unlock my mobile phone.

**Q35.** [Single Selection.] I do not change my passwords, unless I have to.

**Q36.** [Single Selection.] I use different passwords for different accounts that I have.

**Q37.** [Single Selection.] When I create a new online account, I try to use a password that goes beyond the site's minimum requirements.

**Q38.** [Single Selection.] I do not include special characters in my password if it's not required.

a) 1 - Never

b) 2 - Rarely

c) 3 - Sometimes

d) 4 - Often

e) 5 - Always

## 2) Follow-up Survey (Demographics):

### Sec. I. Introduction and Consent for the Follow-up Survey

**Q39.** [Text Entry.] What is your Prolific ID? Please note that this response should **auto-fill** with the correct ID. [ ] (text)

#### STUDY

Recently, you participated in our survey about users' perceptions of security questions on online services (e.g., "What is your mother's maiden name?"). Thank you! This study was conducted by researchers from Concordia University (Montreal, Canada) and the University of Lausanne (Switzerland).

Now, we would like to collect **further information about your educational, technology background, and gender identity** as it may be required to be reported in our paper (in aggregated form).

We ask you to take a survey with **three questions** (max 2 min). You will be paid GBP 0.70 (~USD 0.91) for your complete participation in the study.

**Q40.** [Single Selection.] **CONSENT**

If you wish to participate in this research study, please select the "Agree" option to continue. It will indicate that you will answer all questions truthfully and that you consent to our use of the collected data under the conditions stated above. If you select the "Disagree" option, you will not participate in this research study and will not be paid.

a) Agree

b) Disagree [Terminate.]

### Sec. J. Education, Tech-savviness, Gender

**Q41.** [Single Selection.] What is the highest level of education you have completed?

a) None - No formal schooling

b) Some elementary or middle school (1st - 8th grade)

c) Some high school (9th - 11th grade)

d) High school graduate

e) Some college, no degree

f) Associate's degree (occupational or academic)

g) Bachelor's degree

h) Master's degree

i) Professional degree (e.g., JD, MD)

j) Doctoral degree (e.g., PhD, EdD)

k) Prefer not to answer

**Q42.** [Single Selection.] To what extent do you agree with this statement? "I am considered **tech-savvy** (i.e., knowing a lot about modern technology, especially computers) by myself and others."

a) Strongly Disagree

b) Disagree

c) Neither Agree Nor Disagree

d) Agree

e) Strongly Agree

**Q43.** [Multiple Selection.] What is your gender?

a) Woman

b) Man

c) Non-binary

d) Prefer to self-describe [ ] (text)

e) Prefer not to disclose