# Are People with Cyber Security Training Worse at Checking Phishing Email Addresses? Testing the Automaticity of Verifying the Sender's Address

Daniel Conway, Marcus Butavicius, Kun Yu, Fang Chen

► To cite this version:

Daniel Conway, Marcus Butavicius, Kun Yu, Fang Chen. Are People with Cyber Security Training Worse at Checking Phishing Email Addresses? Testing the Automaticity of Verifying the Sender's Address. 17th International Symposium on Human Aspects of Information Security and Assurance (HAISA), Jul 2023, Kent, United Kingdom. pp.310-323, 10.1007/978-3-031-38530-8_25. hal-05462081

HAL Id: hal-05462081

https://inria.hal.science/hal-05462081v1

Submitted on 16 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Are People with Cyber Security Training Worse at Checking Phishing Email Addresses? Testing the Automaticity of Verifying the Sender's Address

Daniel Conway<fnref n="1" />(✉), Marcus Butavicius<fnref n="2" />(✉), Kun Yu<fnref n="1" />(✉), and Fang Chen<fnref n="1" />(✉)

**Abstract.** Cyber security training emphasises checking the sender's email address to identify phishing emails. Dual process theories of cognition suggest that with practice such tactics can transition from effortful, analytic processes to involuntary heuristics and become 'automatic'. We tested the automaticity of this email habit by developing a scale for cyber security experience and then deployed an interference task where participants (n = 61) had to make a decision about text colour and ignore sender's addresses from either legitimate or phishing emails. A surprising result emerged: the more cyber security training participants had, the less interference they exhibited in the colour selection task and the more they were able to ignore the content of the sender's addresses. This suggests that evaluating sender's addresses does not fulfill the criterion for 'automatic' processes when practiced and that more experienced people seem to be able to ignore this important cue when extraneous task goals are present.

**Keywords:** Cyber Security · Human Factors · Decision Making · Dual Process Models · Automaticity · Phishing Attacks

## 1 Introduction

Phishing emails remain the most common method of attack in a cyber-attacker's arsenal with globally an estimated 29 billion such emails being sent every day [22]. Successful attacks can have extremely deleterious effects on peoples' personal lives and is known to have cost industry in the U.S.A alone $2.7 billion in 2018 [12].

While many Internet attacks can be thwarted via technical measures, as long as there is a human within a system who is capable of receiving emails - a single click on a malicious link can compromise that entire system [29]. This means that organisations that attempt to reduce their attack surface must grapple with the challenges of 'human in the loop' systems in order to reduce their risk of breaches. This involves training of employees in order to improve their cyber hygiene behaviours as well as to increase

<footnote n="1">¹ University of Technology, Sydney, Australia</footnote>

<footnote>daniel.conway@student.uts.edu.au, {kun.yu,fang.chen}@uts.edu.au</footnote>

<footnote n="2">² Department of Defence, Defence Science and Technology Group, Canberra, Australia

marcus.butavicius@defence.gov.au</footnote>
their ability to detect attacks such as phishing emails. A number of Human-Computer Interaction solutions have also been fielded in response to this problem such as alerts that attempt to provide users information about possible attacks [26], however attackers have also used existing internet/software capabilities to increase the effectiveness of their attacks via approaches such as spoofing email addresses [14].

For this work we define phishing emails as generically constructed, mass-distributed emails with either an attachment containing malicious code, or a link to an external website designed to trick the reader into divulging information such as authorisation credentials [1]. These emails are constructed to appear like legitimate requests for information or a call to action. Users are therefore faced with the prospect of trying to detect these attacks during their day-to-day use of their email systems [20].

While phishing victimisation has been studied using a number of theoretical frameworks, dual process models of cognition provide a number of advantages over other theories. Most importantly they are causal, in that they actually explain the underlying processes that result in the emergent behaviour rather than just noting associations between stimuli and outcomes. They therefore provide a theoretically-based account of the cognitive processes that play out during decision-making within an individual.

Dual process models of cognition are based on a long-established body of evidence that there are two distinct systems involved human decision-making [18]. Both these systems can contribute to a decision, but are quite different in their characteristics. System 1 thinking is characterised as fast, shallow, unconscious and effortless. It consists of the use of heuristics or rules-of-thumb and we rely on this type of thinking most of the time and in particular when we perceive that there is little consequence to the decision [8]. An example of this is the recognition heuristic, where we ascribe a higher value to something that we have encountered before and recognise – than something new [13]. The fact that this happens effectively instantaneously, with no effort, usually with little self-awareness and is extremely difficult to compensate for, exemplifies the automatic nature of this type of thought. The other process described by this model is characterised as slow, more effortful thought and has been dubbed System 2 thinking. This is suggested to be more exhaustive, rational and analytical but is less frequently engaged in since it is more cognitively demanding [2].

There are many theories about the way in which System 1 and 2 interact in order to provide a single answer to a problem or situation the decision-maker is faced with. The principle of the cognitive miser appears to apply well here in that System 1 usually dominates most decision-making because System 2 requires more cognitive effort and we generally don't like (and it would be highly inefficient) to think very deeply about everything. There are variants of dual process models that describe different relationships between the two models, but there appears to be consensus that while System 1 thought happens almost all of the time (the particular information processing strategy depending on cue characteristics), System 2 type thinking is less frequently engaged in and is often overcome by the outputs from the shallow, immediate, 'intuitive' System 1 [11].

This human tendency to rely mostly on simple decision-making heuristics has important implications in the context of phishing emails. Dual process theories suggest that if users allow fast, automatic, and shallow processing to dominate their decision-making, they are more likely to fall victim to malicious emails [20, 31]. In particular, because
attackers now include cues that deliberately provoke a strong System 1 response in order to bias the decision make away from deeper, elaborative thought that may cast doubt on message source. These types of deliberate manipulations of the weaknesses of our information processing systems are referred to as social engineering tactics [5]. Common examples of these tactics are messages that use persuasive messages characteristics such as fear appeals, urgency, scarcity and calls to authority [30]. These potent cues are weaponised in phishing emails in order to provoke a fast, intuitive response of responding to the incoming message. However users engaging in more analytical, System 2 processing to carefully evaluate the incoming message have been shown to be more likely to recognise malicious intent and therefore not respond to the attack [31].

One phenomena noted within dual process theories, and which is central to the work presented here, is the observation that certain processes, with practice, can move from System 2 to System 1. From originally requiring effortful thought, processes which are practiced can become more automatic, and can, in some circumstances, then be carried out quickly and with little effort [24].

This phenomena is important within cyber security since staff capacities to respond appropriately to email attacks is one of the most important determinants of the vulnerability of most modern organisations [9]. As such many institutions engage in regular and repeated cyber security training efforts of staff as well as constant security reminders via intra-office communications, ‘awareness campaigns’ and exercises such as Phishing drills. This exposure to training materials is noted to significantly improve participants’ ability to identify phishing emails [19]. Within such training regimes one of the commonest diagnostic tools emphasised in these efforts is that of examining the sender’s email address [3]. In this paper we examine this particular tactic of identifying suspicious emails and attempt to see if this method can be practiced to the point where it can be considered a System 1 process.

Such research has implications not only for the theoretical application of dual process models in this area but also practical benefit in the design of cyber security awareness programs in that it would demonstrate that such practices can be not only transmitted but become ‘second nature’. Furthermore, understanding the automaticity of this highly diagnostic tactic has important ramifications for the interaction design of email systems.

## 2 Background

Within corporate cyber security training a central tactic that is taught to help users identify fraudulent senders is that of evaluating the sender's email address [3]. Attributes of sender's addresses that are taught as predicting malicious intent are: letter substitution (for example: noreply@mazon.com), unusual domain names (E.g.: mailer.srvvscust-yaoelauzwb9446325@mntapjwaku.com), and domains that do not match the content of the email [3, 17].

According to dual process theory, one of the primary characteristics of System 1 processes is that they are ‘automatic’ [28]. This implies that, once stimuli are perceived that match and trigger a System 1 process, they run to completion, impervious to efforts to stop them. Furthermore, dual process theories suggest that some processes that can transition from effortful, System 2 processes to automatic and effortless System 1 processes – through practice [24].
Based on the findings presented in Conway [7] we therefore hypothesised that repeated training may well result in users having practiced this particular process – of immediately looking at the sender's address to ascertain authenticity – to some form of automaticity – and potentially an effortless System 1 process. This current paper investigates whether a process such as looking for the sender's address can be practiced to the point of automaticity and whether such automaticity complies with the expectations that dual process theories attribute to a System 1 heuristic.

## 2.1 Testing for Automaticity

A varying number of attributes, ranging in number from 2 to 14 have been ascribed as to be necessary for automatic processing [25] but one of the most commonly referred to attributes is that of obligatoriness, or not being able to consciously control or stop the execution of such processes once triggered [25]. This attribute is therefore what we sought to test for in the experiment that follows.

We hypothesised that there would be an individual difference between people in how they processed an email's senders address, and that this difference would be based on the amount of cyber security training they have received. We expected that those people with extensive training would have practiced the process of email identification to the point of automaticity and would therefore be unable to disregard the content of a sender's email address – even when instructed to do so.

An experiment was devised, in many ways similar to the widely-used Stroop task [27], where the primary measure was the difference in reaction times between congruent and non-congruent trials that we will refer to as the ‘Phishing Email Sender’s Address Interference Task’.

The Stroop task typically presents two sources of information, one being the colour of the stimuli, and other being the text of stimuli. Participants are instructed to ignore the text presented and respond only to the colour. Trials where the colour of the text is the same as the text itself are considered congruent trials, and ones in which they are different are considered incongruent. The fundamental finding here is that regardless of instructions, people typically respond slower to incongruent cues. This suggests that regardless of attempts to attenuate attention to the information presented in the text, reading and then some level of subsequent conceptual priming is carried out automatically. The slower response for non-congruent cues is said to be the result of conflict between the two sources of information, where the precept of the colour extracted from the text causes interference with the response to the colour stimuli.

The experiment presented here was similar in structure to the classic Stroop task in that participants were told that they were to make a decision based on the colour of two on-screen elements and should respond with one button if these two cues were the same colour, and another button if they did not match. One of the two cues was a sender's email address, and the other a solid block of colour. In order to strongly cue the precepts of phishing emails and encourage semantic processing of the text – we labeled the two response buttons ‘Phishing email’ and ‘Legitimate email’. Participants were, however, told to ignore this labeling as well as the content of the sender's address and respond only in response to whether the colours of the two cues matched.
We therefore formulated trials as being either congruent or incongruent. Congruent trials were those where the label of the correct response button matched the status of sender's address regardless of stimuli colour, i.e., pressing the 'Phishing email' button when the sender's address was a phishing email or pressing the 'Legitimate email' button when the sender's address was a legitimate email. Incongruent responses were those where the correct button to be pressed (according to the colour matching task) did not match status of the sender's address, i.e.: pressing 'Phishing email' when the sender's address was legitimate or pressing 'Legitimate' email when the sender's address was a phishing email.

The experiment that follows therefore presented trials of the Stroop-like attention-
based task, as well as then assessing participants cyber security training level via the
instrument presented above. In doing so we expected to see:

**Hypothesis 1:** Participants would exhibit slower reaction times for incongruent trials than for congruent trials.

**Hypothesis 2:** This effect would be greater in those participants with extensive cyber security training.

**3 Method**

A single task and a number of questions measuring individual differences were presented sequentially within the same session using an online experiment coded in PyschoPy [23].

**3.1 Participants**

Participants were recruited via the web-based recruitment platform Prolific (www.prolific.co). 66 participants were selected with the aim of collecting at least 60 results and allowing for incomplete results or error. Participation criterion was speaking fluent English and therefore we made the participants only available to those countries with English as a first language. Participants from the USA (17), UK (16), Ireland (1), Australia (2), NZ (3) and SA (21) took part. 33 were fulltime employed in 'business management and administration' or 'finance' according to the user data provided by Prolific. Removing incomplete responses, 61 responses remained (W: 31, M: 30), age: (M = 31.14, SD = 12.55). Mean experiment duration was 9 min 36 s.

**3.2 Materials**

**Assessing Experience.** The experiment needed to measure individual differences in cyber security training and experience between participants. Accordingly, we developed a ‘Cyber Security Training and Experience’ measure using a small number of constructs that have been associated with cyber security experience in the literature.

Previous studies have shown e-mail knowledge and experience as protective against
individual phishing victimization [10, 17] while Halevi [15] found that users who were
more aware of cyber risks will protect themselves more. As such we asked Q1: "I *have*
read about or heard about how to identify phishing emails."
Training users to examine sender's email addresses and whether they match the content of the email is central to many existing corporate training platforms [3]. We expected this particular 'experience' to be one of the more powerful determinants of whether people had practiced the procedure of checking sender's addresses to automatic levels. Therefore we asked Q2: "I have had one or more training sessions on how to identify phishing emails at work."

However we were cognisant of the fact that people may have encountered formal training in situations other than at work (Jakobsson, 2007) and therefore wanted to capture this eventuality. Therefore we asked Q3: "I have had one or more training sessions on how to identify phishing emails elsewhere (not at work)."

Apart from awareness training in the forms of educational materials, many corporate entities now run regular phishing 'drills' with their staff where they (or a third party) send emails to staff (un-announced) emulating phishing attacks in order to assess whether staff can identify them [9, 19]. We therefore asked Q4: "My company regularly sends out phishing drills / emails."

Self-efficacy has been shown to strongly relate to ability in terms of being cyber secure. In a Health Belief Model based study, Ng (2009) found that self-efficacy (in this case of being able to adopt protective security measures) was the most important determinant of intention and behaviour of all the variables they examined. As such we asked Q5: "I know how to identify phishing emails."

Previously being victimised has a fraught relationship with subsequent good cyber security practices. Downs [10] found that people who had previously been victimised did not consider the consequences of fraud differently to those who had not. Therefore Q6 asked: "I have fallen victim to one or more phishing emails in the past."

And finally, there has been some preliminary evidence [6] that those who are confident and experienced with cyber security have a tendency to share their knowledge and teach others. We were therefore again interested if this particular self-efficacy based trait was associated with such a practice. Q7 asked: "I teach others how to identify phishing emails."

All questions were presented as statements and participants would nominate as how much they agreed with each statement, with responses gathered via a five point Likert scale with all anchors labeled ('Strongly disagree', 'Disagree', 'Neutral', 'Agree', 'Strongly agree'). See Table 1 for the text of all the questions.

**Sender's Email Addresses.** The sender email addresses presented were taken from real emails, both phishing (e.g., accounts@mazon.com) and legitimate (e.g., ejection@parliament.nsw.gov.au). A small collection was made of phishing emails received by the author and colleagues in the nine months before the experiment was began. These were then analysed as to characteristics noted by other authors [1, 4] as indicative of phishing emails. We then selected emails that were most representative of these characteristics. As such we believe that the phishing email sender addresses presented were broadly representative of phishing emails in circulation at the time of writing. The legitimate email addresses presented were selected from legitimate emails similar in format to phishing emails in that they were not personal communications and contained a call to action.
## 3.3 Procedure

After gathering basic demographic information (age, gender and English speaking ability) instructions were provided as to the nature of the main task as well as two practice trials with explicit feedback. This was followed by all 16 trials of the phishing email sender's address interference task which were presented in a different random order for each participant, with eight consisting of a sender's address from a phishing email, and eight from legitimate emails. The colour congruency of the two cues were also distributed randomly throughout the trials. Then finally, the seven cyber security training and experience survey items were presented.

The phishing email sender's address congruency task consisted of trials where a sender's email address and a solid block of colour were presented. Both address and colour block could be either brown or blue. Participants were instructed to press the left hand button when the colours did not match and the right hand button when they did. Of primary interest however was whether participants could ignore the semantic information contained in the email addresses. In order to provoke as much conflict as possible between the email content and the response, the left response button was labeled 'Phishing email' and the right 'Legitimate email'. Participants were, however, instructed to ignore these labels and just respond to the colours of the two cues.

**Measures.** Reaction time was recorded for each trial along with the congruency of the colours presented and the response button pressed. Where the colours of the two cues matched and therefore the correct response was the 'Legitimate email' button, this was coded as congruent when the content of the sender's address was from a legitimate email and incongruent when from a phishing email. For trials where the colours were different and participants should have pressed the 'Phishing email' button, where the content of the sender's address was a phishing email, trials were coded as congruent, and where the address was legitimate, coded as incongruent (Fig. 1).

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Stimuli and target response combination examples (not all combinations shown).</figcaption></figure>

At the conclusion of the email/colour trials, participants were asked a series of questions to quantify their experience with or training in cyber security via the seven
questions of the cyber security training and experience measure. The resulting one to five scores for the seven questions were summed resulting in a total experience score with a range of 7 to 35.

# 4 Results

Trials where response time was longer than five seconds (five trials) were removed as outliers (double the traditional $1.5 \times \text{IQR}$ in order to remove less trials) leaving in 487 congruent trials and 484 incongruent trials.

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Response times by whether the button pressed label was congruent to the status of the sender's address.</figcaption></figure>

A one-sided Welch's t-test of reaction times for all congruent and non-congruent trials revealed that the mean reaction times of non-congruent trials, $M = 1.327$ s, was not significantly higher than the mean reaction times of congruent trials, $M = 1.305$ s, $t = -0.635$, $p = .262$, (see Fig. 2).

## 4.1 Interference Score

A total interference score was derived for each participant. This score was calculated by subtracting each participant's mean reaction time for non-congruent trials from their mean reaction time for congruent trials ($M = 0.105$, $SD = 0.47$). Positive values on this score indicated that the participant was slower for non-congruent trials and the magnitude of the score indicated the size of this effect. This interference score is used for much of the analysis that follows.

To examine the effect of experience score on interference a non-parametric, Rank Based regression analysis was carried out between participants' interference score and total experience scores. Simple rank-based linear regression indicated that for every increase of 1 in experience scores participants' interference score was 0.012 lower, $\beta =$
-0.012, *p* = .043. The experience score explained a statistically significant proportion of the interference score, *R*²(multiple robust) = 0.08, although only accounted for 8% of the variation. See Fig. 3.

<figure><img src="image_6.png" /><figcaption>Fig. 3. Summed, total experience score by interference, for each participant.</figcaption></figure>

Since the individual difference questions included a neutral response and we were interested in the difference between those responded ‘yes’ and ‘no’ to these questions for subsequent analysis, we discarded neutral responses and divided the responding answers into two bins where answers of 1 (‘strongly disagree’) and 2 (‘disagree’) were classified as ‘no’ answers and responses of 4 (‘agree’) and 5 (‘strongly agree’) were classified as ‘yes’ answers.

# 5 Discussion

We hypothesised that examining sender's address to identify malicious emails has been practiced to the point of automaticity in people with extensive cyber security training. We tested this automaticity via one of it's the most commonly discussed attributes, namely the attribute of 'obligatoriness' or the inability to ignore the content of sender's address when presented with a task not requiring the processing of this information. However, our results did not support our first hypothesis, i.e., we found no evidence that responding to email addresses was automatic in our phishing email sender's address interference task. Specifically, there was no difference in response times between trials where the sender's address status matched the required response button (phishing/legitimate email) and when it did not match the response button. This implies that there was, overall, no cognitive interference between the content of the sender's email address and the required response, suggesting that participants were able to ignore this information.
<table><caption>**Table 1:** Results of one-sided Welch's t-tests and effect sizes for all seven experience questions.</caption><thead><tr><th>Q</th><th>Item</th><th>Mean Interference 'No' responses</th><th>n of 'No' responses</th><th>Mean Interference 'Yes' responses</th><th>n of 'Yes' responses</th><th>T-score</th><th>P value <sup>a</sup></th><th>Cohen's d</th><th>Effect size</th></tr></thead><tbody><tr><td>1</td><td>I have read about or heard about how to identify phishing emails</td><td>0.433</td><td>9</td><td>0.007</td><td>46</td><td>1.474</td><td>.177</td><td>.678</td><td>Large</td></tr><tr><td>2</td><td>I have had one or more training sessions on how to identify phishing emails at work</td><td>0.35</td><td>22</td><td>-0.041</td><td>32</td><td>2.618</td><td>.015</td><td>.781</td><td>Large</td></tr><tr><td>3</td><td>I have had one or more training sessions on how to identify phishing emails elsewhere (not at work)</td><td>0.162</td><td>33</td><td>-0.073</td><td>19</td><td>2.484</td><td>.017</td><td>.631</td><td>Large</td></tr><tr><td>4</td><td>My company regularly sends out phishing drills / emails</td><td>0.174</td><td>34</td><td>0.014</td><td>22</td><td>1.424</td><td>.162</td><td>.356</td><td>Medium</td></tr><tr><td>5</td><td>I know how to identify phishing emails</td><td>0.684</td><td>8</td><td>0.0169</td><td>48</td><td>1.797</td><td>.114</td><td>.883</td><td>Large</td></tr><tr><td>6</td><td>I have fallen victim to one or more phishing emails in the past</td><td>0.32</td><td>46</td><td>-0.064</td><td>12</td><td>1.337</td><td>.187</td><td>.321</td><td>Medium</td></tr><tr><td>7</td><td>I teach others how to identify phishing emails</td><td>0.197</td><td>16</td><td>-0.062</td><td>35</td><td>2.404</td><td>.02</td><td>.603</td><td>Large</td></tr></tbody></table>

<footnote><sup>a</sup> In order to control the family-wise error rate when conducting multiple statistical tests, we used the Bonferroni adjustment for *p* values, resulting in a significance threshold of *α* > .007 – which none of tests achieved</footnote>
However, further analysis revealed a surprising phenomenon. Total cyber security training and experience scores, exhibited a weak, but significant correlation with the amount of interference that non-congruent trials presented. However, this correlation was in the opposite direction of that expected under our second hypothesis and instead participants with higher self-reported experience were in fact better able to suppress the sender's address information when responding to the primary task.

A number of observations can be made to support this conclusion. Firstly, regardless of statistical significance, all seven questions trended in the same direction, suggesting that the main correlation presented is not the result of chance, but some kind of phenomena related to cyber experience. Secondly, the relative magnitudes of the effect sizes for each question correlate well with previous research. Here, questions with less stringent requirements such as Q1 ('*I have read about...*') show a smaller effect size than does a question more likely to be more predictive of cyber training (Q2, '*I have received training...*'). And thirdly, questions involving constructs where previous research has shown a strong relationship with cyber security competence, such as self-efficacy [16, 21] demonstrated a larger effect size than those where the variable has been shown to have less effect, such as previous victimisation [10]. All of this taken together provides evidence that the phenomena described is closely related to cyber experience.

Our finding that those who are more experienced with cyber security are more adept at controlling their attention and ignoring task-irrelevant stimuli – even when this stimuli is strongly cued to interfere with the primary task – was counter to our predictions based on dual process theories of cognition. One possible explanation of this is that perhaps those with good attentional control are more likely to succeed in corporate environments and therefore receive more frequent and better-quality cyber security training. However, this would mean that corporate training needs to take this tendency into account. Alternatively, those who have had more cyber security training and experience are also likely to have also had more experience in processing emails in a work environment. Such experience might have led to an increased ability to suppress detailed processing of email addresses when such processing is not relevant to the task requirements such as quickly understanding and responding to emails within the context of ongoing email conversations. Again, this has implications for current training regimes. It is clear that more research is required to examine exactly how sender email addresses are processed in email tasks using a variety of experimental paradigms including, but not limited to, dual-task and eye-tracking experiments.

The implications of this finding are manifold. If some aspect of cyber security training results in people who are more able to ignore what they consider task-irrelevant stimuli, it suggests that this tendency would lead to increased victimisation. Within the HCI context, this suggests that new methods of providing visual emphasis to the Sender's address field such as highlighting may be required to influence users to better identify phishing emails targeting organisations.

# 6 Limitations

It should be acknowledged that one of the limitations of our study was that our measure of training experience was only based on self-report. Our results may have been different had this measure been more rigorously assessed either by assessing experience via a
questionnaire or using a training manipulation (and subsequent practice sessions), and a separate control group, prior to testing. We propose that future research in this area should use a more direct measure of training and seek to replicate our results. Future research should also repeat similar experiments with a larger and more diverse sample in order to determine the generalisability of the results we found in this study.

# 7 Conclusions

Since staff members' ability to identify phishing emails is an important determinant of an organisations' vulnerability to attack, a common training method is to teach staff to examine the sender's address of incoming emails to ensure they are from legitimate sources. We hypothesised that this simple protocol could be practiced to the point of automaticity and therefore fall within the domain of System 1 thought as suggested by dual process models of cognition. We assessed experience by developing and then implementing a seven-question survey to capture how much cyber security training and experience participants had. We then tested the automaticity of this habit by asking participants to make a decision about the colour of two on-screen elements and used the colour of sender's email addresses (both legitimate and phishing) as a task-irrelevant cue. Similar to a Stroop task we were interested in whether participants who had received cyber security training and presumably had practiced this particular tactic would find it more difficult to ignore the content of sender's address and therefore respond more slowly when required to click a button where the label was non-congruent to the status of the sender's address.

We found that this hypothesis was not supported with response times for non-congruent trials not being longer than for congruent trials across all participants. This indicates that this process does not fulfill one of the most cited requirements for consideration as a System 1 task, namely that of automaticity. We then went on to calculate interference scores for each participant – being the total time difference between all congruent and non-congruent trials – with positive values indicating more interference when responding to non-congruent trials. We then carried out a rank-based regression of the experience scores for each participant and their interference scores and found that there was a statistically significant relationship between experience and interference where those with higher experience scores exhibited less interference from non-congruent cues, meaning that these participants found it easier to ignore the task irrelevant cue of the sender's email address. This is a surprising finding and we speculate that good attitudinal control may be a predictor of success in the corporate world where cyber security training is often administered. This has implications for cyber security training as well as for the design and personalisation of email systems and is an intriguing result clearly calling for further research.

# References

1. Akbar, N.: Analysing persuasion principles in phishing emails (2014). http://essay.utwente.nl/66177/. Accessed 29 May (2021)
2. Barrett, L.F., Tugade, M.M., Engle, R.W.: Individual differences in working memory capacity and dual-process theories of the mind. Psychol. Bull. **130**(4), 553–573 (2004). https://doi.org/10.1037/0033-2909.130.4.553

3. Burita, L., Klaban, I., Racil, T.: Education and training against threat of phishing emails. Int. Conf. Cyber Warfare Secur. 17(1), 7–18 (2022). https://doi.org/10.34190/iccws.17.1.28

4. Butavicius, M., Taib, R., Han, S.J.: Why people keep falling for phishing scams: The effects of time pressure and deception cues on the detection of phishing emails. Comput. Secur. **123**, 102937 (2022). https://doi.org/10.1016/j.cose.2022.102937

5. Cialdini, R.B.: The Psychology of Persuasion. New York (1993)

6. Conway, D., Taib, R., Harris, M., Yu, K., Berkovsky, S., Chen, F.: A qualitative investigation of bank employee experiences of information security and phishing, pp. 115–129 (2017). https://www.usenix.org/conference/soups2017/technical-sessions/presentation/conway. Accessed 8 Mar 2021

7. Conway, D., Yu, K., Butavicius, M., Chen, F.: Are phishing emails conflict problems? Dual process theory applied to an email identification task (2022)

8. De Neys, W.: Automatic-heuristic and executive-analytic processing during reasoning: chronometric and dual-task considerations. Q. J. Exp. Psychol. **59**(6), 1070–1100 (2006). https://doi.org/10.1080/02724980543000123

9. Dodge, R., Coronges, K., Rovira, E.: Empirical benefits of training to phishing susceptibility. In: Gritzalis, D., Furnell, S., Theoharidou, M. (eds.) SEC 2012. IAICT, vol. 376, pp. 457–464. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30436-1_37

10. Downs, J.S., Holbrook, M.B., Cranor, L.F.: Decision strategies and susceptibility to phishing. In: Proceedings of the Second Symposium on Usable Privacy and Security (SOUPS 2006), pp. 79–90 (2006). https://doi.org/10.1145/1143120.1143131

11. Evans, J.S.B.T., Stanovich, K.E.: Dual-process theories of higher cognition: advancing the debate. Perspect. Psychol. Sci. **8**(3), 223–241 (2013). https://doi.org/10.1177/1745691612460685

12. FBI. Internet Crime Complaint Center (IC3)—Annual Report 2018. Federal Bureau of Investigations. https://www.ic3.gov/Home/AnnualReports. Accessed 14 Jan 2023

13. Gigerenzer, G., Goldstein, D.G.: The recognition heuristic: a decade of research. Judgm. Decis. Mak. **6**(1), 100–121 (2011)

14. Gupta, A., Sharda, R., Greve, R.A.: You’ve got email! Does it really matter to process emails now or later? Inf. Syst. Front. **13**(5), 637–653 (2011). https://doi.org/10.1007/s10796-010-9242-4

15. Halevi, T., Memon N., Nov, O.: Spear-phishing in the wild: a real-world study of personality, phishing self-efficacy and vulnerability to spear-phishing attacks. Social Science Research Network, Rochester, NY (2015). https://doi.org/10.2139/ssrn.2544742

16. Ifinedo, P.: Understanding information systems security policy compliance: an integration of the theory of planned behavior and the protection motivation theory. Comput. Secur. **31**(1), 83–95 (2012). https://doi.org/10.1016/j.cose.2011.10.007

17. Jakobsson, M.: The human factor in phishing. Priv. Secur. Consum. Inf. **7**, 1–19 (2007)

18. Kahneman, D.: Thinking, Fast and Slow. Farrar, New York (2011)

19. Kumaraguru, P., Rhee, Y., Acquisti, A., Cranor, L.F., Hong, J., Nunge, E.: Protecting people from phishing: the design and evaluation of an embedded training email system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2007), pp. 905–914 (2007). https://doi.org/10.1145/1240624.1240760

20. Musuva, P.M.W., Getao, K.W., Chepken, C.K.: A new approach to modelling the effects of cognitive processing and threat detection on phishing susceptibility. Comput. Hum. Behav. **94**, 154–175 (2019). https://doi.org/10.1016/j.chb.2018.12.036
21. Ng, B.Y., Kankanhalli, A., Xu, Y.C.: Studying users' computer security behavior: a health belief perspective. *Decis. Support Syst.* **46**(4), 815–825 (2009). https://doi.org/10.1016/j.dss.2008.11.010

22. Norris, G., Brookes, A., Dowell, D.: The psychology of internet fraud victimisation: a systematic review. *J. Police Crim. Psychol.* **34**(3), 231–245 (2019). https://doi.org/10.1007/s11896-019-09334-5

23. Peirce, J., et al.: PsychoPy2: experiments in behavior made easy. *Behav. Res. Methods* **51**(1), 195–203 (2019). https://doi.org/10.3758/s13428-018-01193-y

24. Purcell, Z.A., Wastell, C.A., Sweller, N.: Domain-specific experience and dual-process thinking. *Think. Reason.* **27**(2), 239–267 (2021). https://doi.org/10.1080/13546783.2020.1793813

25. Rawson, K.A.: Exploring automaticity in text processing: syntactic ambiguity as a test case. *Cogn. Psychol.* **49**(4), 333–369 (2004). https://doi.org/10.1016/j.cogpsych.2004.04.001

26. Siu, N., Iverson, L., Tang, A.: Going with the flow: email awareness and task management. In: *Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (CSCW 2006)*, pp. 441–450 (2006). https://doi.org/10.1145/1180875.1180942

27. Stroop, J.: Studies of interference in serial verbal reactions. *J. Exp. Psychol. Gen.* **18**(6), 643–662 (1935)

28. Thompson, V.A., et al.: The role of answer fluency and perceptual fluency as metacognitive cues for initiating analytic thinking. *Cognition* **128**(2), 237–251 (2013). https://doi.org/10.1016/j.cognition.2012.09.012

29. Vishwanath, A., Herath, T., Chen, R., Wang, J., Raghav Rao, H.: Why do people get phished? *Testing individual differences in phishing vulnerability within an integrated, information processing model*. *Decis. Support Syst.* **51**(3), 576–586 (2011). https://doi.org/10.1016/j.dss.2011.03.002

30. Workman, M.: Wisecrackers: a theory-grounded investigation of phishing and pretext social engineering threats to information security. *J. Am. Soc. Inform. Sci. Technol.* **59**(4), 662–674 (2008). https://doi.org/10.1002/asi.20779

31. Yan, Z., Gozu, H.Y.: Online decision-making in receiving spam emails among college students. *Int. J. Cyber Behav. Psychol. Learn. (IJCBPL)* **2**(1), 1–12 (2012). https://doi.org/10.4018/ijcbpl.2012010101