# What Goes Around Comes Around; Effects of Unclear Questionnaire Items in Information Security Research

Marcus Gerdin, Åke Grönlund, Ella Kolkowska

► To cite this version:

Marcus Gerdin, Åke Grönlund, Ella Kolkowska. What Goes Around Comes Around; Effects of Unclear Questionnaire Items in Information Security Research. 17th International Symposium on Human Aspects of Information Security and Assurance (HAISA), Jul 2023, Kent, United Kingdom. pp.470-481, 10.1007/978-3-031-38530-8_37. hal-05462086

HAL Id: hal-05462086

https://inria.hal.science/hal-05462086v1

Submitted on 16 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# What Goes Around Comes Around; Effects of Unclear Questionnaire Items in Information Security Research

Marcus Gerdin(✉), Åke Grönlund, and Ella Kolkowska

Örebro University, Örebro, Sweden

Marcus.gerdin@oru.se

**Abstract.** The credibility of research on information system security is challenged by inconsistent results and there is an ongoing discussion about research methodology and its effect on results within the employee non-/compliance to information security policies literature. We add to this discussion by investigating discrepancies between what we claim to measure (theoretical properties of variables) and what we actually measure (respondents' interpretations of our operationalized variables). The study asks: (1) How well do respondents' interpretations of variables correspond to their theoretical definitions? (2) What are the characteristics and causes of any discrepancies between variable definitions and respondent interpretations? We report a pilot study including interviews with seven respondents to understand their interpretations of the variable Perceived severity from the Protection Motivation Theory (PMT).

We found that respondents' interpretations differ substantially from the theoretical definitions which introduces error in measurement. There were not only individual differences in interpretations but also, and more importantly, systematic ones; When questions are not well specified, or do not cover respondents' practice, respondents make interpretations based on their practice. Our results indicate three types of ambiguities, namely (i) Vagueness in part/s of the measurement item causing inconsistencies in interpretation between respondents, (ii) Envision/Interpret 'new' properties not related to the theory, (iii) 'Misses the mark' measurements whereby respondents misinterpret the fundamentals of the item. The qualitative method used proved conducive to understanding respondents' thinking, which is a key to improving research instruments.

**Keywords:** Non-/compliance research · Validation of measurement instruments · Interpretation of measurement items

## 1 Introduction

The credibility of research on information system security compliance research is challenged by inconsistent results. As a result, there is an ongoing discussion about research methodology and its effect on research results within the employee non-/compliance
to information security policies (ISP) literature. For example, researchers such as Mou et al. (2022) and Cram et al. (2019) have observed significant effect on research results depending on the level of specificity of the content of a security policies when operationalized into measurement instruments. Similarly, Johnston et al. (2015) exemplified the grave importance of contextualizing behavioral variables into non-/compliance settings, while Gerdin et al. (2021) and Sommestad et al. (2014) noted inconsistencies between variable definitions and measurement items within and across studies. Siponen and Vance (2014) illustrated the need of contextualizing measurement items for practitioners. The common denominator is that all authors have highlighted problems with the conceptualization and operationalization of behavioral variables and several attempts to solve these problems have been made. Despite these, however, the problems with inconsistent results largely remain (Cram et al. 2019).

This paper contributes to this line of research by examining yet another fundamental aspect of validation procedures, namely, the extent to which key variables have been operationalized into questionnaire measurement items in ways that are not open for multiple interpretations by respondents, but if, are their interpretation in consonance with the variables stated property/characteristic (Desimone and Le Floch 2004; Mackenzie et al. 2011; Luft and Shields 2003).

To investigate that, we critically analyze respondents’ interpretations of questionnaire measurement items commonly used in the field in order to understand how much variation there is, and what causes that variation. Drawing from influential papers discussing variable measurement and validation procedures (e.g., Boudreau et al. 2001; MacKenzie et al. 2011; Luft and Shields 2003) we investigate if there are any discrepancies between what we claim to measure (a theoretical property of the variable as stated in our variable definition) and what we actually measure (which is respondents’ interpretations of our operationalized variables). If there is a gap between the two, there is noise in the measurement as there is a high risk that we do not measure what we think we do. Specifically, the study asks:

* How well do respondents' interpretations of variables correspond to their theoretical definitions?

* What are the characteristics and causes of any discrepancies between variable definitions and respondent interpretations?

The full study will include variables from the two most used theories in the field, Protection Motivation Theory and Theory of Planned Behavior. The present paper reports a pilot study where we use a small respondent sample ($n = 7$) to test (1) how useful our method is for investigating respondents’ interpretations of questionnaire question and (2) if there are any discrepancies to find.

The pilot tests only one variable, “perceived severity”, under the logic that (1) if we find discrepancies in this variable, it is likely that there are such also in other variables; (2) the nature of the variables and the causes for them will give us clues that are useful to adjust the methodology for the full study.

This study, hence, investigates the nature of the problem in order to prepare for a study covering also the extent of it.
# 2 Related Research and Protection Motivation Theory

## 2.1 Related Research

Behavioral research involves very complex phenomena which are complicated to empirically measure. For this reason, issues related to the validation of questionnaire measurement instruments has been discussed in the literature of behavioral management information systems (MIS) for over 30 years. Along these lines, for example, Straub (1989) raised concerns about the lack of systematic validation of measurement instruments in MIS research and called for more methodological rigor in the field. Since then, important papers such as that of Boudreau et al. (2001) and MacKenzie et al. (2011) have noted that even though the research field as a whole has improved in important respects, many problems related to measurement instrument validation remain.

Accordingly, researchers in non-/compliance disciplines have recognized the importance of a good measurement practice and taken vital steps to improve it. Within behavioral MIS literature, two alternative perspectives on how to improve methodological rigor in research has been discussed. The first perspective is based on theoretical considerations and emphasizes the need for clear and uniform specifications of variables' theoretical properties/characteristics, which is required to identify consistent cause-and-effect relationships between them (Luft and Shields 2003; MacKenzie et al. 2011). Along these lines within non-/compliance field Johnston et al. (2015) proposed an enhanced version of the theory (PMT) where variable properties had been tailored to better fit the ISP non-/compliance context. See Karjalainen, et al. (2019) for example of similar type of study.

Hence, according to the first perspective researchers first and foremost must clearly specify the theoretical properties/characteristics of variables based on the context of ISP non-/compliance, i.e., make sure that the variable properties are adapted to this 'new' context. This is absolutely vital since many of the theories within the research area are 'borrowed' from other research disciplines (Johnston et al. 2015; Moody et al. 2018).

In contrast to the first perspective, the second one takes variables' theoretical properties largely for given and instead focuses on how to successfully operationalize them into questionnaire measurement items. For example, Siponen and Vance (2014) illustrated the importance of developing measurement items tailored to the intended respondents' organizational perspective to minimize the risk of measurement errors and to increase to possibility for researchers to draw correct conclusions about cause-and-effect relationships between variables. They also stressed the importance of testing and validating the measurement instrument with the target population before doing to actual study. If this is not done, a measurement item may have an increased risk of being subjected to multiple interpretations which can imply that it only captures parts of the intended theoretical property, captures content that does not correspond to the intended property, and/or capture several properties, some of which may not be in line with the theoretical definition (Luft and Shields 2003) See Karlsson et al. (2017) and Li et al. (2021) for examples of similar type of studies.

Notwithstanding their important contributions, however, these two perspectives have to our knowledge essentially only been studied in isolation. Thus, there is no study which have investigated whether variable definitions in the non-/compliance literature clearly
specify their theoretical properties/characteristics and whether these properties are fully
translated into adequate measurement items that are not subject to multiple interpreta-
tions (Desimone and Le Floch 2004; Mackenzie et al. 2011; Luft and Shields 2003).
We believe that these two perspectives could beneficially be combined, as successful
research is dependent on both. As effectively put by Desimone and Le Floch (2004, p.4),
"An important aspect of validity is that the respondent has a similar understanding of the
questions as the survey designer; that the questions do not omit or misinterpret major
ideas or miss important aspects of the phenomena being examined".

## 2.2 Perceived Severity

This pilot study uses the variable Perceived severity from Protection Motivation Theory (PMT). Originating from Rogers (1975, 1983), PMT was initially developed to explain the effects of fear appeals on health attitudes and behaviors (Prentice-Dunn and Rogers 2000). We have two reasons for specifically focusing on the variable Perceived Severity. First, it has been a central focus of research on non-/compliance methodology in the past (e.g., Cram et al. 2019; Gerdin et al. 2021; Sommestad et al. 2015; Haag et al. 2021; Johnston et al. 2015).

Second, the variable has a high level of specificity in the conceptualization of its the-
oretical property/characteristic, namely *the perceived consequences of an information
security threat* (Sommestad et al. 2015; Gerdin et al. 2021) but the notation concern-
ing whom the consequences refer to differs between different studies.. Some refer to
*consequences related to the organization* (e.g., Aigbefo et al. 2020; Hooper and Blunt
2020), others to *consequences related to the individual* (e.g., Aurigemma and Matt-
son 2019; Vrhovec and Mihelic 2021), and some fail to *specify to whom the conse-
quences refer to* (e.g., Burns et al. 2017; Rajab and Eydgahi 2019). Perceived severity
is well-suited for this study due to its extensive research history in the context of non-
/compliance research, and interesting characteristics in terms of conceptualization and
operationalization (Sommestad et al. 2015; Gerdin et al. 2021).

# 3 Method

## 3.1 Data Collection

We conducted seven semi-structured interviews with professionals from the Swedish public sector. Of the respondents, five worked with home care within municipalities, one worked in the Swedish Transport Agency, and one in the Agency of Digital Government. All respondents handled classified information and were required to adhere to an information security policy in their respective organizations.

The choice of questionnaire items was based on the above mentioned inconsistencies
in the variable's conceptualization and operationalization, especially regarding whom
the consequences referred to (see Sect. 2.2) differed in the literature (Sommestad et al.
2015; Gerdin et al. 2021). We incorporated a mix of items specifically related to whom
the consequences of non-compliance were directed towards in order to make sure that
all differences in the variable conceptualization identified in previous studies where
covered. For example, item 6 referred to consequences in general, item 7 referred to consequences related to the organization and item 8 referred to consequences related to the individual. Moreover, following Cram et al. (2019), we also made sure that different level of specificity of threats where covered. For example, item 1 is considered general as it refers to threats to the security of an organization, item 2 is instead specifying which segment that becomes affected and the impact of it. See full list of items in Table 1.

The interviewees were asked to think-out-loud and explain 1) how they interpret the question, 2) how they would have answered the question using a 5-point Likert-scale and 3) the reason for choosing the specific number. When necessary, probing follow-up questions were asked.

Six interviews were conducted online, using Zoom or Microsoft Teams, while one was conducted in person. Two researchers were present at each interview, both of whom were familiar with the interview protocol. While one researcher was responsible for following the interview protocol the other could focus on unclarities in the answers and ask follow-up questions. Each interview lasted about one hour. We adopted an iterative approach, whereby the interview protocol was modified (if needed) after each interview.

<table><caption>**Table 1.** List of items and original studies</caption><tr><td>Item 1</td><td>"Threats to the security of my organizations information are harmful" (Ifinedo 2012)</td></tr><tr><td>Item 2</td><td>"If my computerized data were temporarily not available, serious information security problems would result" (Barlette et al. 2015)</td></tr><tr><td>Item 3</td><td>"If my work device were infected by malware, it would be severe" (Blythe and Coventry 2018)</td></tr><tr><td>Item 4</td><td>"In terms of information security violations, attacks on my organization's information and information systems are severe" (Posey et al. 2015)</td></tr><tr><td>Item 5</td><td>"If my password was stolen, the consequences would be severe" (Johnston et al. 2015)</td></tr><tr><td>Item 6</td><td>"Threats to the security of my organization's information and information systems are severe" (Ma 2022; Posey et al. 2015)</td></tr><tr><td>Item 7</td><td>"An information security breach in my organization would be a serious problem for my organization." (Siponen et al. 2014)</td></tr><tr><td>Item 8</td><td>"An information security breach in my organization would be a serious problem for me" (Vance et al. 2012)</td></tr></table>

## 3.2 Method for Analysis

The coding process comprised of three steps. First, we transcribed, compiled, and coded each interview separately. During the transcription and the subsequent readings, we identified numerous interesting words, phrases, terms and concepts offered by the respondents and coded them (Bazeley 2013).

Second, we re-read all transcripts and devoted our focus to combining words, phrases, terms, and concepts from each interview into categories representing similar ideas and/or
issues shared among the respondents. This process consisted of comparisons of tran-
scripts and subsequent analysis to over time detect conceptual patterns shared between
the transcripts (Bazeley 2013).

Third, we formulated overarching labels that subsumed the previously identified
categories on a more abstract level, representing a more aggregated analytical dimension
of each category (Bazeley 2013). The focus here was on identifying different types of
ambiguity in the operationalization of variables. Six categories were identified.

To ensure confidence in our identified categories and subsequent labels, we continu-
ously worked back and forth with the material, literature, and the emerging table (Table 2)
and two authors were involved in the process. Continuous discussion and modification
formed the basis for achieving agreement. Note however, the interviews, transcripts and
succedent coding was done in Swedish, thus the emergent findings have been translated
into English.

4 Emergent Findings

We set out to investigate how the respondents interpreted questionnaire measurement
items, and if their interpretations were in line with the theoretical conceptualization of
the variable as suggested in ISP non/compliance research literature (see Sect. 2). Overall,
we found three types of ambiguities in the formulation of these items which increased
the risk of misinterpretations the among the respondents, namely:

(i) *Vagueness* in words or concepts used in the measurement items causing inconsistencies in interpretation among respondents,

(ii) Envisioning, where respondents are unable to discern the intended meaning of a variable and therefore – based on their understanding of the actual work situation – interpret ‘new’ properties/characteristics which are outside of the theoretical conceptualization in order for them to make sense of the measurement item.

(iii) ‘Misses the mark’ measurements whereby the respondent misinterprets the theo-
retical property/ies which the items seek to measure.

**4.1 Ambiguity (i) -Vagueness**

The first type of ambiguity refers to vagueness in part/s of the measurement item causing inconsistencies in interpretation between respondents. Especially in relation to items 1, 2 and 6, we found that the respondents interpreted items differently due to vagueness and/or imprecise wording. This type of ambiguity is not related to any type of (mis)interpretation of properties/characteristics of a variable per se, rather other types of ‘external’ aspects of the measurement items which cause differences in interpretation between respondents.

For example, regarding questions referring to ‘seriousness of security threats’ (items
1 and 6 in Table 1), respondents were not confident in how they should understand
the level of seriousness of the threat. Consequently, some respondents answered the
question from a worst-case-scenario perspective while others had a normal, “run-of-
the-mill” scenario in mind. This made them choose, respectively, very low or very high
options on the 5-point Likert-scale. That is, the respondents would grade their responses
differently on the Likert scale depending on how they interpreted the level of seriousness.
Note, however, that even if the respondents would grade similarly on the Likert-scale concerning different levels of seriousness of threat, the fact that they interpret the level of seriousness differently is still noteworthy as they did in fact not answer the same question. This type of findings can lead to increased theoretical specificity in future research. That is, we have the possibility to not only validate, but also make rational improvements to, our measurement instrument to be able to draw more fine-tuned conclusion of our work, i.e., better understand the conceptual range and boundaries of a variable.

Another telling example of vagueness concerns the aspect of time (see illustrative example from category 1 in Table 2). Not only did the respondents envision different time periods, but they also mentioned that they would change their answered drastically (e.g., going from a 2 to a 5 on the Likert scale) if the time aspect changed. This type of imprecise wording therefore introduces noise in the measurement whereby the interpretation of a crucial concept (time) may differ between the respondents, but also between the respondents and the research which may cause misinterpretation of results.

## 4.2 Ambiguity (ii) – Envisioning Unintended Properties

The second type of ambiguity refers to respondents envision/interpret ‘new’ properties/characteristics not related to the conceptualization of a variable. This type of ambiguity differed from vagueness (ambiguity (i)) in the sense that respondents added new information to the item in question. Vagueness means that respondents interpret important words differently due to imprecise wording, but their interpretations still operate within the conceptualization of the variable. To contrast further, when respondents envision/interpret new content, they *add* new information to the item in question which is not necessary in line with the conceptualization of the variable, nor anticipated by the researcher.

For example, for respondents working in the municipality, when faced with measurement items regarding consequences, they relate the consequences to the citizen/clients to whom they served in their role as a public organization. As described in Sect. 2.2, the variable perceived severity has in previous research been found to sometimes refer to *consequences related to the organization* (e.g., Aigbefo et al. 2020; Hooper and Blunt 2020), others to *consequences related to the individual* (e.g., Aurigemma and Mattson 2019; Vrhovec and Mihelic 2021), and some failing to *specify to whom the consequences refer to* (e.g., Burns et al. 2017; Rajab and Eydgahi 2019). However, never have consequences for the citizen/client been defined as a property/characteristic of the variable. On one hand, even though differences in interpretation regarding whom the consequences refer to may seem arbitrary as they all refer to *consequences* in some sort. The Sommestad et al. (2015) meta-study observed that the weighted mean correlation was substantially higher when the threat targeted individuals ($r = 0.3$) compared to when the threat targeted the organization as a whole ($r = 0.17$). Consequently, there are empirical evidence that suggest that effect on intention by consequences fluctuates according to whom the consequences refer to.

On the other hand, if we consider that the respondents grading is similar on the Likert scale might be similar independently to whom the consequences refers, their actual interpretation of the question (consequences for the citizen/client) is unknown to the researcher and may not correlate with the conceptualization of the variable. In line
with MacKenzie et al. (2011) and Luft and Shields (2003) papers, this type of ambiguity poses a significant risk of drawing invalid conclusions and not correctly understanding the phenomenon being studied.

## 4.3 Ambiguity (iii) – ‘MiSses the Mark’

The third type of ambiguity refers to what we call ‘misses the mark’ measurement items. This refers to situations where the respondents simply interpreted important content, which did not correlate with the conceptual definition of the variable. For example, when facing question related to the consequences of a threat, some respondents referred to properties such as *vulnerability and probability*. While these properties share some characteristics with the conceptualization of Perceived severity, they actually belong to another variable from the Protection Motivation Theory (PMT), namely, Perceived Vulnerability (see example in Table 2).

Another type of category within this ambiguity refers to types of threats. That is, even though the respondents were aware of the context (information security) in which the questionnaire operates within, some respondents still referred to other types of threats such as shooting, or fire not related to IS when interpreting the question.

This type of ambiguity can lead to invalid conclusions if not addressed properly, at least if this interpretation is common among the respondents. Similar to above-mentioned ambiguities, the results of these types of (mis)interpretation from the respondents may result in that the item captures content that does not correspond to the intended property, and/or capture several properties, some of which may not be in line with the conceptual definition.

<table><caption>**Table 2.** Note* the representative examples have been modified to be more specific.</caption><thead><tr><td>Type of ambiguity</td><td>Category</td><td>Representative example (respondent)</td></tr></thead><tbody><tr><td>Vagueness in part/s of the measurement item</td><td>Timeframe</td><td>The word “temporarily” makes it very difficult to interpret this question…What does it mean. 5 days or an hour? (Respondent 2- Item 4) For me it depends on how temporary the break is. If it is a couple hours, then it is not a problem but if it is longer than that, well there is definitely a problem then. (Respondent 4- Item 4)</td></tr></tbody></table>

(continued)
<table><caption>**Table 2.** (continued)</caption><thead><tr><th>Type of ambiguity</th><th>Category</th><th>Representative example (respondent)</th></tr></thead><tbody><tr><td rowspan="2">Envision/Interpret 'new' properties not related to the conceptualization</td><td>Seriousness of the threat; Run-of-the-mill or Worst-case-scenario</td><td>If they steal all my passwords, they can do anything with my computer and the consequences will be unimaginable. (Respondent 6 - Item 5)</td></tr><tr><td>Level of analysis - Organizational, department or unit</td><td>Well, it depends on what level in the organization the breach happens. (Respondent 4 - Item 8)</td></tr><tr><td rowspan="2">'Misses the mark' measurements</td><td>Consequences related to the citizens/clients</td><td>For us who work within the health care sector, everything is always serious. In the end, the clients can die or be injured. (Respondent 7 - Item 5)<br>Our mission is to always look out for the best interests of the citizens and care citizen/clients. So that nothing can happen to them, and that is what we must try to avoid. (Respondent 3 - Item 3)</td></tr><tr><td>Other types of threats which is not information security related</td><td>It could be a school shooter and create chaos at the school, or perhaps share suicide clips to other students. (Respondent 2 - Item 8)</td></tr><tr><td></td><td>Vulnerability and probability</td><td>These types of threats occur on a daily basis. (Respondent 4 - Item 3)</td></tr></tbody></table>

# 5 Discussion and Conclusion

This pilot study investigates how well respondents' interpretations of variables correspond to the theoretical definitions of them as well as the characteristics and causes of any discrepancies.

The short answers to these questions are:

(i) Respondents interpret variables so differently from the theoretical definitions that it is a problem. In some cases, they respond to something else than the question asked which introduces error in measurement.
(ii) There are not only individual differences in interpretations but also, and more impor-
tantly, systematic ones; When questions are not well specified, or do not cover
respondents' practice, respondents make interpretations based on their practice.

The pilot study uses a small respondent sample (n = 7) to test (1) how useful our method is for investigating respondents’ interpretations of questionnaire question and (2) if there are any discrepancies to find.

Moreover, the pilot tests only one variable, “Perceived severity”, under the logic that (1) if we find discrepancies in this variable, it is likely that there are such also in other variables; (2) the nature of the variables and the causes for them will give us clues that are useful to adjust the methodology for the full study.

This study, hence, investigates the nature of the problem in order to prepare for a
study covering also the extent of it.

We found three different types of ambiguities, namely (i) Vagueness in part/s of the
measurement item causing inconsistencies in interpretation between respondents, (ii)
Envision/Interpret ‘new’ properties not related to the conceptualization, (iii) ‘Misses
the mark’ measurements whereby the respondent miss-interpret the fundamentals of the
item.

We conducted interviews with seven respondents to see if their interpretation of the variable was suitable given the established definition of the variable, but also to see if and how the interpretation varied among the respondents. Overall, our study sheds light on the challenges of interpreting questionnaire items in non-/compliance research and highlights the importance of improving measurement instruments to achieve more theoretical specificity and avoid invalid conclusions.

Moreover, we discovered that respondents from the public sector and from healthcare identified relevant aspects of ‘perceived severity’ that have hitherto never been included in research, namely consequences for the client to the organization. In healthcare, patient security and privacy are important concerns, and GDPR has added legal restrictions to information handling not only for the public sector but for every organization. This increasingly important concern for consequences on part of the clients is not only one factor for confusing the interpretation of items in a questionnaire but also an issue that should be considered for inclusion in research.

Even though we investigated only one variable with relatively few respondents, the findings suggest a need for further validation of measurements as recommended by researchers such as Straub (1989), MacKenzie et al. (2011).

Our study adds to the discussions about validating variable measurements by showing that respondents indeed interpret important content in measurement instruments differently than intended by the researchers, and that there are systematic causes for important parts of this discrepancy. Most importantly the discrepancies stem from what could be called theory/practice mismatch; either measurement items do not clearly measure the variables of the theory, or they do not clearly relate to practice, or both.

We believe the method we used was conducive to investigating our research questions as it both revealed a number of ambiguities and proved effective in making respondents explain their reasoning.
References

Aurigemma, S., Mattson, T.: Generally speaking, context matters: making the case for a change from universal to particular ISP research. J. Assoc. Inf. Syst. **20**(12), 7 (2019)

Barlette, Y., Gundolf, K., Jaouen, A.: Toward a better understanding of SMB CEOs' information security behavior: Insights from threat or coping appraisal. J. Intell. Stud. Bus. **5**(1) (2015)

Bazeley, P.: Qualitative Data Analysis Practical Strategies, 2nd edn. Sage, London (2013)

Blythe, J.M., Coventry, L.: Costly but effective: comparing the factors that influence employee anti-malware behaviours. Comput. Hum. Behav. **87**, 87–97 (2018)

Burns, A.J., Posey, C., Roberts, T.L., Lowry, P.B.: Examining the relationship of organizational insiders' psychological capital with information security threat and coping appraisals. Comput. Hum. Behav. **68**, 190–209 (2017)

Boudreau, M.C., Gefen, D., Straub, D.W.: Validation in information systems research: a state-of-the-art assessment. MIS Q. 1–16 (2001)

Cram, W.A., D'arcy, J., Proudfoot, J.G.: Seeing the forest and the trees: a meta-analysis of the antecedents to information security policy compliance. MIS Q. **43**(2), 525–554 (2019)

Desimone, L.M., Le Floch, K.C.: Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educ. Eval. Policy Anal. **26**(1), 1–22 (2004)

Gerdin, M., Grönlund, Å., Kolkowska, E.: Use of protection motivation theory in non-compliance research (2021)

Haag, S., Siponen, M., Liu, F.: Protection motivation theory in information systems security research: a review of the past and a road map for the future. ACM SIGMIS Database: DATABASE Adv. Inf. Syst. **52**(2), 25–67 (2021)

Hooper, V., Blunt, C.: Factors influencing the information security behaviour of IT employees. Behav. Inf. Technol. **39**(8), 862–874 (2020)

Ifinedo, P.: Understanding information systems security policy compliance: an integration of the theory of planned behavior and the protection motivation theory. Comput. Secur. **31**(1), 83–95 (2012)

Johnston, A.C., Warkentin, M., Siponen, M.: An enhanced fear appeal rhetorical framework. MIS Q. **39**(1), 113–134 (2015)

Karjalainen, M., Sarker, S., Siponen, M.: Toward a theory of information systems security behaviors of organizational employees: a dialectical process perspective. Inf. Syst. Res. **30**(2), 687–704 (2019)

Karlsson, F., Karlsson, M., Åström, J.: Measuring employees' compliance – the importance of value pluralism. Inf. Comput. Secur. **25**(3), 279–299 (2017)

Li, H., Luo, X.R., Chen, Y.: Understanding information security policy violation from a situational action perspective. J. Assoc. Inf. Syst. **22**(3), 7398–7772 (2021)

Luft, J., Shields, M.D.: Mapping management accounting: graphics and guidelines for theory-consistent empirical research. Acc. Organ. Soc. **28**(2–3), 169–249 (2003)

Ma, X.: IS professionals' information security behaviors in Chinese IT organizations for information security protection. Inf. Process. Manage. **59**(1), 102744 (2022)

MacKenzie, S.B., Podsakoff, P.M., Podsakoff, N.P.: Variable measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS Q. **35**, 293–334 (2011)

Moody, G.D., Siponen, M., Pahnilä, S.: Toward a unified model of information security policy compliance. MIS Q. **42**(1) (2018)

Mou, J., Cohen, J.F., Bhattacharjee, A., Kim, J.: A test of protection motivation theory in the information security literature: a meta-analytic structural equation modeling approach. J. Assoc. Inf. Syst. **23**(1), 196–236 (2022)
Posey, C., Roberts, T.L., Lowry, P.B.: The impact of organizational commitment on insiders' motivation to protect organizational information assets. J. Manag. Inf. Syst. **32**(4), 179–214 (2015)

Rajab, M., Eydgahi, A.: Evaluating the explanatory power of theoretical frameworks on intention to comply with information security policies in higher education. Comput. Secur. **80**, 211–223 (2019)

Rogers, R.W.: A protection motivation theory of fear appeals and attitude change. J. Psychol. **91**(1), 93–114 (1975)

Rogers, R.W.: Cognitive and physiological processes in fear-based attitude change: a revised theory of protection motivation. In: Cacioppo, J., Petty, R. (eds.) Social Psychophysiology. A Source Book, pp. 153–176. Guilford, New York (1983)

Siponen, M., Vance, A.: Guidelines for improving the contextual relevance of field surveys: the case of information security policy violations. Eur. J. Inf. Syst. **23**(3), 289–305 (2014)

Siponen, M., Mahmood, M.A., Pahnila, S.: Employees' adherence to information security policies: an exploratory field study. Inf. Manag. **51**(2), 217–224 (2014)

Sommestad, T., Karlzén, H., Hallberg, J.: The sufficiency of the theory of planned behavior for explaining information security policy compliance. Inf. Comput. Secur. **23**(2), 200–217 (2015)

Sommestad, T., Hallberg, J., Lundholm, K., Bengtsson, J.: Variables influencing information security policy compliance: a systematic review of quantitative studies. Inf. Manag. Comput. Secur. **22**(1), 42–75 (2014)

Straub, D.W.: Validating instruments in MIS research. MIS Q. **147–169** (1989)

Vance, A., Siponen, M., Pahnila, S.: Motivating IS security compliance: insights from habit and protection motivation theory. Inf. Manag. **49**(3–4), 190–198 (2012)

Vrhovec, S., Mihelič, A.: Redefining threat appraisals of organizational insiders and exploring the moderating role of fear in cyberattack protection motivation. Comput. Secur. **106**, 102309 (2021)