**Towards Understanding Security Education**

Markus Jakobsson

Markus.Jakobsson@parc.com

*This note argues that security education is not as straightforward as many of us might have thought. At the same time, we suggest that by approaching things in a different way than has traditionally been done, we can hope for much better impact of the efforts. We describe an approach for determining how to address a given problem, based on user experiments, surveys, statistics, reported incidence rates and economical aspects. A full paper is in the process of being produced; contact the author for more information or for advice of how to practically apply the techniques described herein.*

It is the lifeblood of scientists to hypothesize, measure and evaluate. All established sciences do, computer science and social sciences not being exceptions. But the intersection of these two sciences – as embodied by anti-fraud education – has not reached that level of maturity yet. Institutions with a need to educate their clients do not have clear motivations backing their educational campaigns. Little effort is spent on evaluating the success of the campaigns, and campaigns are hardly ever tested before being deployed. 

The lack of measurements is not due to a disagreement that knowledge is beneficial. Rather, measuring is very difficult, and is fraught with pitfalls. For example, one cannot ask a person “Have you been a victim to fraud” without having to worry that the confrontational style – associated with the stigma of victimization – makes him or her answer “no” even if that is not the truthful answer. (See http://portal.acm.org/citation.cfm?doid=1290958.1290968 for evidence that stigma substantially affects the results of surveys.) Moreover, it has not been evident how the outcome of the measurements should best guide the educational efforts. 

*Example of survey pitfall:*

Our experiments show that 83% of people who were asked  “Have you been a victim to fraud” select the answer “No, I am very cautious to avoid being tricked.” In contrast, if the person is first told “Online crime is rampant.” – to suggest that they are not the only one to potentially have lost money – followed by the question “Based on your own experience with online fraud, do you think that users like you might lose money to online criminals?” then only 35% of them select the answer “No, crime is not a big concern to me and users like me.” While one may argue that the two questions measure slightly different phenomena, the very different outcomes still serve to highlight the importance of formulation. 

The above example does not mean that we believe that surveys are not meaningful, but it *does* say that special care has to be taken when designing the questions. The design, ultimately, should be informed by knowledge of actual fraud rates (which may be difficult to obtain) or by *naturalistic* experiments, also referred to as *sting* experiments. Such experiments, which can be thought of as ethical versions of real fraud incidents, have the benefit of reporting the truth – at least when performed correctly. (An added benefit of using naturalistic experiments – in comparison to observing real incidence rates – is that naturalistic experiments allow us to assess the risk associated with abuse types that have not yet become prevalent threats.) Thus, knowledge of the correct answer – whether obtained by observing incidence rates or by experimenting – can be used to fine-tune how survey questions are designed; this design can then be extended to topics and demographics where no naturalistic experiments have been – or can be – performed. 

Performing naturalistic experiments is not trivial. Apart from requiring technical and social insights, it also poses both ethical and branding dilemmas.  We refer to http://www2.computer.org/portal/web/csdl/doi/10.1109/MSP.2008.52 for an overview of the former; the latter problem corresponds to the understandable unwillingness for the affected brands to taint themselves by performing experiments that look like fraud to their users. It is possible to largely avoid this problem by having the research done by third parties (see e.g., http://portal.acm.org/citation.cfm?id=1135853 ), by performing the experiment in controlled environments,  or by performing the experiment using a hypothetical brand – although the latter two could affect the correctness of the outcome if care is not taken. 

***Measuring education.***

There are several things worth measuring. The first and most obvious is the *need for education* – for a given type of fraud and a particular demographic group. This is a measure of how well people understand what to do to avoid being victimized. If we compute the ratio between needed education and the known losses for various topics, we can then see what topics are most important to educate consumers about. (This can be done from the perspective of a given organization or society as such.) Such a *normalized* measurement helps us identify where educational efforts have the greatest potential of making a difference in terms of losses.

*Example: phishing vs. Nigerian scams*

Without going into details of how the numbers below were derived, or even what they mean, let’s look at an example of how to identify what educational campaign is most needed. In one large-scale study run by the author of this note, subjects were determined to have an understanding of how to avoid Nigerian scams estimated at 267, and an understanding of how to avoid phishing estimated at 230 (where larger numbers means a better understanding). The losses of these two crimes to a collection of organizations considered were 0.57 resp. 1.44. This results in education-to-losses ratios of 470 resp 160. This approach normalizes the degree of education to the observed losses, and tells us that people understand how to defend themselves against Nigerian scams much better than against phishing – relative to the need for them to have this understanding. This may not be a surprise given the nature of the defenses against the two attacks, but the principle helps us identify such relations for problems where this is much less clear a priori. 

***Measuring perceived education.***

The degree of existing education is not the only important aspect to quantify. It is also important to measure the *perceived* need for education. That is the degree to which a given group believes they need (vs. do not need) education about a given topic. If there is a large need for education of how to deal with a given problem, but very little *perceived* need for the same, then it will be difficult to make the target group pay attention to the efforts to educate them. Using a similar approach as in the example above, we can rank topics by their perceived need for education, normalized by the actual estimated needs for education and the losses. This shows us a small perceived need for education on lost pet scams (a type of fraud in which a criminal reports to have found a lost pet, and asks for a payment to ship it from where he supposedly is located), and a much greater perceived need for knowledge of how to deal with malware. This is in spite of the fact that many consumers think that having AV software – whether with an active subscription or not – is sufficient to be protected. Again, while this result may not be surprising, being able to rank all threats allow us a comparison of closely related threats. 

***Measuring awareness.***

A third important aspect to measure is the degree of awareness a target group has of a given type of fraud. We note that it is much less expensive to mount an awareness campaign than an educational campaign. Awareness campaigns can be used to increase the perceived need for education, and may sometimes suffice by themselves to address the problem. An easily described example of such a situation is the typical warning of pickpockets in a crowded market – this is an awareness campaign with no need for an associated educational campaign. The target audience would know already what it means to be careful. For many other types of crime, and online fraud in particular, this may not be the case. If a given topic needs improved education, but there is a very small perceived need for this, an awareness campaign may be a first step towards educating consumers.

Continuing our example from above, we can see that among the types of fraud considered in our study, the problem that deserves most attention to be given to it in the form of an awareness campaign – again, normalized by losses – is auction fraud, while the topic with the least need is the Nigerian scam. This makes intuitive sense given how often people see Nigerian spam in their mailboxes, while attempts to auction fraud might not be as easy to spot for average consumers. 

***Understanding costs and making decisions.***

It is important to consider the costs of a given type of fraud, along with the cost and anticipated impact of the appropriate campaign to address the problem. Given estimates of these, and of all of the previously described quantities, it is possible to rank all the types of fraud in terms of their relative needs to be addressed using educational campaigns and awareness campaigns. When we have this ranking, we can determine how to proceed. The same rankings can be used to identify what technical strategies to prioritize as well. The beauty of our approach is that it allows us to develop strategies for hypothetical scenarios – based on measurements of how consumers would be likely to react to a give threat, were it to start being commonplace.

***Why imperfection is not a show-stopper.***

One may argue that the estimates one may base the strategy decisions on are not entirely accurate, and depend intrinsically on how we assign numeric values to opinions and expressions of these. Similarly, while the functions involved in computing the ranking most certainly are monotonic, there is no clear evidence that they are linear functions. While this suggests the need for further refinements, it does not mean that it is not useful to compute such rankings based on simple models of relationships: The alternative is to base a decision of what to do without considering these measures at all – clearly a less desirable strategy. 

**Author bio:**

Dr. Markus Jakobsson is a Principal Scientist at PARC, and a member of the PARC security group. He is also running a consulting company aimed at improving security education, and serves on the advisory boards of several organizations concerned with fraud. While he is maybe best known for his research on phishing and crimeware, he has also made significant contributions to online payment schemes, applied security, security education, and privacy-preserving cryptographic protocols.

Markus believes in taking a holistic approach to security, in which everything is measured, modeled and considered in the final design. This belief has compelled him to study the human aspect of security, and has guided his work on phishing, crimeware, user authentication, and user messaging.

Before joining PARC, Markus held positions at Bell Labs, RSA Labs, New York University, Indiana University, and RavenWhite, the anti-fraud startup that he co-founded. He holds over 100 patents and has published two books and over a hundred papers. He has a PhD in Computer Science from University of California at San Diego, a Masters degree in engineering from Lund Institute of Technology, and a Scottish Terrier named Zero.