Qualitative Adaptation: Informing Design forRisk-based Decision Making
1112

3

Andrew M’manga , Shamal Faily , John McAlaney , Chris Williams , Youki Kadobayashi and Daisuke Miyamoto
1
Bournemouth University
Poole, UK{ammanga, sfaily, jmcalaney}@bournemouth.ac.uk
2
Defence Science and Technology Laboratory
Porton Down, UK
cwilliams@mail.dstl.gov.uk
3
Nara Institute of Science and Technology
Nara, Japan{youki-k, daisu-mi}@is.naist.jp

3

Research on decision making during risk and uncertainty facilitates risk-based decision making by understanding
techniques decision makers use to arrive at informed decisions. Approaches to the research usually involve a mixof cognitive techniques for information discovery and sense-making; these were methodologically not intended toinform design. We detail our experience in applying qualitative techniques to elicit persona characteristics fromrisk-based decision making data.
Cross-language research, Thematic analysis, Personas, Risk-based decision making, Security

1. INTRODUCTION

Persona Case technique (PCT) (Faily and Flechais,2011). As a final step, the spreadsheet wasimported into CAIRIS (Computer Aided Integrationof Requirements and Information Security), a tool
supporting the Persona Case technique formodelling personas (Faily, 2018).

Risk-based decision making (RBDM) is anendeavour to make informed decisions underconditions of risk and uncertainty. The conditions
may be the result of time limitations, insufficient
information, and dynamic environments to mentiona few. Understanding how these decisions are
made is a step towards understanding how tofacilitate RBDM through design.

3. THEMATIC ANALYSIS
In this section, we present a detailed overview of
the thematic analysis approach taken during theresearch (illustrated in Figure 1). Details here are
independent of tool usage and aim at highlighting
the methodological approach.

To achieve this, the research aimed at identifying
how persona characteristics may be grounded inrisk-based decision making data.
2. APPROACH

The first round of thematic analysis wasincorporated into the study’s response form bydividing it into eight steps of RRP. Participants
were expected to present the rationale for theirdecision in accordance to the eight steps. For
example; Situation assessment information wasprovided for step 1 and Option validation
information for step 7.

A data elicitation exercise was carried out in Japan
with a group of 30 industrial participants
undertaking a cyber security course. Cyber security
was chosen as its activities exemplify RBDM inaction. Participants were drawn from 11 different
sectors including Transport, Oil, Electricity andManufacturing, and with experience ranging from 1
to 20 years.

After responses were grouped into the eight RRP
steps (themes), the second round of thematic
analysis aimed at identifying sub-themes to formrisk rationalisation variable. For example, thefollowing risk rationalisation variables were culledfrom Situation assessment; analysing security
policies, analysing security trends, seeking
clarification, analyse past security trends, andmonitoring network activity. Due to the translated
nature of the data (discussed in Section 4.1), ourapproach to coding was based on similarities instatements (conveying meaning) as opposed tosimilarities in words (Saldana, 2015). For example,

Participants
weretrainedontheRisk
Rationalisation Process (RRP) (M’manga, 2018),
our adaptation of Boyd's (1996) Observe Orient
Decide Act (OODA) and provided with a cybersecurity decision making scenario containing
elements of risk and uncertainty. From this, they
were asked to provide rationales for their decisions
using RRP. Responses were thematically clusteredinto risk rationalisation variables using Nvivo, whichwere then exported to a spreadsheet andcategorised as behaviour variables following the© 2018 M’manga et al.
Published by BCS Learning andDevelopment Ltd. Proceedings of British
HCI 2018. Belfast, UK.

1

Qualitative Adaptation: Informing Design for Risk-based Decision Making
Andrew M’manga ● Shamal Faily ● John McAlaney ● Chris Williams ● Youki Kadobayashi ● Daisuke Miyamoto

the sub-theme analysing security policies was
based on statements such as “Identify information
security policies”, “confirm with the department
which developed the information security policies”,
and “identify the history of information securitypolicies”

secretary proficient in the two languages. The
secretary was a regular translator at the university,
held a university degree, and possessed twelve
years working experience in the USA. Because ourdata collection approach took the form of written
responses to the decision making scenario, thesecretary was given the responsibility of translating
both the scenario and its questions from English toJapanese, and vice-versa for the responses. The
secretary had limited cyber security knowledge, but
we believe that the use of a single translator
increased the secretary’s immersion in the dataand improved contextual translation. A Japanesecyber security lecturer assisted the secretary incases where the security terminology was notunderstood, and the main researcher wascontacted when English words were unclear. Forexample, the main researcher was asked to clarify
the term “rationalisation” in relation to the scenario.

Unlike Grounded Theory (Corbin and Strauss,2008), where a core category emerges during thefinal round of coding, our adoption of PCT (Failyand Flechais, 2011) for the third round of thematic
analysis was meant to use the risk rationalisation
variables in an argumentation (Toulmin, 2003) form
that represents behaviour variables. To do this, therisk rationalisation variables were categorised asrisk assessment activities, validation activities andgoals. For example, “analysing security policies”
was argued by the goal “provide security advice”
and validation activity “policy effectiveness”.

The disadvantage of the approach was that thedata immersion process - essential in thematic
analysis - was slow due to the main researcher’s
reliance on translation. Becoming lost in translation
is inevitable in cross-language research, and we
encountered a few false positives. For example,
one of the participants professed experience as a
SOC (security operations centre) analyst in a
printing company, which we believed was an error
in translation. Verification of this experience
showed that this was a false positive.
4.2 Methodological differences
Figure 1: Thematic analysis

The second challenge the research faced, relates
to the methodological gap between techniques
used in decision making research and techniques
usedinspecifying
designrequirements.
Traditionally, decision making research has
followed a descriptive to prescriptive approach; thisis the modelling of how decisions are made torecommend facilitation techniques. In contrast, theapproach to design is to elicit requirements; these
are then modelled to inform design decisions(Fischer, 1991).

4. CHALLENGES FACED
4.1 Cross-language research
Cross-language research occurs when there is alanguage barrier between researchers andparticipants (Squires, 2009). The language barrier
was the first research challenge faced as the study
was undertaken in Japan, with non-English
speaking participants, and an English speaking
main researcher.

While there is undoubtedly more modelling in the
two approaches than our simplified explanation, themain point is the difference in model use. Decision
makingmodels
areprimarilyaimedat
understanding the problem domain, while designmodels are aimed at supporting the solution. In a
perfect world, this would simply mean that one
serves as input to the other, however,methodologicaldifferences
present
incompatibilities. For example, RRP aims at
understanding the rationale behind risk-based
decision making; decision making may, however,be influenced by biases such as theoverconfidence in one’s own abilities (Kahneman,2011). For example, none of our participants
identified personal inabilities (experience or

Cross-language research is rare in computing but
common in health and ethnographic studies wherethe objective includes studying cases related tominority groups (Wallin and Ahlstrom, 2006). We
followed this approach due to the lack of local
cyber security analysts willing to participate inresearch due to privacy and others issues; thisissue was previously reported by Kotulic and Clark(2004). On the other hand, the study in Japanbenefitted from multiple participants in a single
place, with diverse experiences from multiple
industrial sectors.
To tackle the language problem, we worked with aJapanese university; they provided a Japanese2Qualitative Adaptation: Informing Design for Risk-based Decision Making
Andrew M’manga ● Shamal Faily ● John McAlaney ● Chris Williams ● Youki Kadobayashi ● Daisuke Miyamototraining) as a reason for erroneous assumptions intheir responses to the scenario. While decision
making research may bring this to light, relating it todesign may not be as straightforward.

qualitative research, but the wider researchcommunity as a whole. In summary, the areas ofconcern raise some of the following questions;what approaches should researcher take whensubject matter experts are unavailable for empiricalresearch? How should cross-discipline research
approaches be adapted to provide meaningful
findings? How should tool dependent research beapproached when recommended tools areunavailable?

This problem is evident in decision making
research, where design recommendations areusually presented in a textual format with no clear
relation to design models (Groenewald et al., 2017;
Gerber et al., 2016). We are, however, not statingthat our approach is an all-encompassing solution,
but rather that it provides the means to relate
decision making to design models through the RRP
to Persona mapping (Figure 1).

6. ACKNOWLEDGEMENT
The research was funded by Bournemouth
Universitystudentship
DSTLX1000104780R_BOURNEMOUTH_PhD_RB
DM, with the initial collaborative meeting between
UK/Japan researchers facilitated by support fromthe Great Britain Sasakawa Foundation. We arealso grateful to DSTL for their sponsorship of thiswork.

4.3 Seamful tool support
The final significant challenge we encounteredconcerned tool support in relation to seamfulinteraction. Seamless interaction is explained as atool’s ability not to intrude on one’s consciousness
but allows the user to focus on the task at hand(Weiser, 1994). Conversely, seamfulness is thelack of tool invisibility. While invisibility refers to theseamless integration of system components or theirinteraction (Chalmers, 2003), our experience refers
to the work overhead resulting from passing data
between tools during the research.

7. REFERENCES
Boyd, J. (1996) The essence of winning and losing.
Unpublished lecture notes.
Chalmers, M. (2003). Seamful design: showing theseams in wearable computing. Presented at theIEE Eurowearable conference, Evanston, IL,USA, pp 11–16.

As indicated in Section 2, we used Nvivo (version11) to thematically analyse the participant’s
responses to risk rationalisation variables. Thefindings were then exported to a spreadsheet (PCTtemplate) for behaviour classification. However, inFaily and Flechais (2011) proposal, PCT relied on
the qualitative data analysis tool ATLAS.ti; licensing
restricted our research to Nvivo.

Corbin, J. M, and Strauss, A. L. (2008) Basics ofqualitative research: techniques and proceduresfor developing grounded theory. 3rd ed. Los
Angeles, Calif: Sage Publications, Inc.

We identified that although Nvivo presents several
data export functions, none adequately satisfied thePCT requirements. For example, one of the mainfeatures of PCT is the use of concept relationships
forpersonacharacteristics
argumentation.
However, it was not possible to export concepts
and their relationships from Nvivo to a single
spreadsheet. In light of this, we manually populated
the spreadsheet. This, in turn, was a laborious
process due to a large amount of data collected.
Our experience illustrates how differences in
Computer-Assisted Qualitative Data Analysis
Software (CAQDAS) may affect qualitative
research and lead to a lack of seamless interaction
in tool use.

Faily, S. (2018) Designing Usable and Secure
Software with IRIS and CAIRIS. Springer.
Faily, S., and Flechais, I. (2011) Persona cases: atechnique for grounding personas. Proceedingsof the SIGCHI Conference on Human Factors in
Computing Systems, Vancouver Canada, pp.
2267–2270, ACM.
Fischer, G. (1991) The importance of models inmaking complex systems comprehensible. In:
Human Factors in Information Technology.
Elsevier.
Gerber, M., Wong, B., and Kodagoda, N. (2016)
How Analysts Think: Decision Making in theAbsence of Clear Facts. Presented at theEuropean Intelligence and Security Informatics,
Uppsala Sweden, pp 132–135, IEEE.

5. CONCLUSION
In this paper, we presented our experiences
adapting qualitative methods to inform persona
design for risk-based decision making. We
presented
cross-language
research,methodological differences and seamful tool
support as our three main areas of concern. Theconcerns raised are however not limited toGroenewald, C., Wong, W., Attfield, S., Passmore,P., and Kodagoda, N. (2017) How Analysts
Think:
Navigating
Uncertainty–Aspirations,
Considerations and Strategies. In: The 13th biannual international conference on Naturalistic
Decision Making. Bath United Kingdom, pp 56.

3

Qualitative Adaptation: Informing Design for Risk-based Decision Making
Andrew M’manga ● Shamal Faily ● John McAlaney ● Chris Williams ● Youki Kadobayashi ● Daisuke Miyamoto

Kahneman, D. (2011) Thinking, fast and slow.
Macmillan.

Toulmin, S. E. (2003) The uses of argument.
Cambridge, U.K.; New York: Cambridge
University Press.

Kotulic, A., and Clark, J. (2004) Why there aren’t
more information security research studies.
Information & Management, 41 (5), pp 597–607.

Wallin, A., and Ahlstrom, G. (2006) Cross-culturalinterview studies using interpreters: systematic
literature review. Journal of Advanced Nursing,55 (6), pp 723–735.

M’manga, A. (2018) Risk Rationalisation Process.
[online].
Availablefrom:https://doi.org/10.5281/zenodo.1255383.
Saldana, J, (2015) The coding
qualitative researchers. Sage.

manualWeiser, M. (1994) The world is not a desktop.
interactions, 1 (1), 7–8.

for4