# Reflections on Measuring the Trust Empowerment Potential of a Digital Environment

Natasha Dwyer, Anirban Basu, Stephen Marsh

► To cite this version:

Natasha Dwyer, Anirban Basu, Stephen Marsh. Reflections on Measuring the Trust Empowerment Potential of a Digital Environment. 7th Trust Management (TM), Jun 2013, Malaga, Spain. pp.127-135, 10.1007/978-3-642-38323-6_9. hal-01468736

HAL Id: hal-01468736

https://inria.hal.science/hal-01468736

Submitted on 15 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Reflections on Measuring the Trust
Empowerment Potential of a Digital
Environment

Natasha Dwyer¹, Anirban Basu², and Stephen Marsh³

¹ Victoria University, Melbourne, Australia
natasha.dwyer@vu.edu.au

² University of Sussex, Brighton, UK
a.basu@sussex.ac.uk

³ University of Ontario Institute of Technology, Oshawa, Canada
stephen.marsh@uoit.ca

**Abstract.** We claim that the digital trust research area has tended to privilege the act of trusting while considering distrust a negative outcome. However, from the users perspective distrust might be as valid an option as trust (it may not be a good idea to trade, collaborate or exchange in a particular context). How do we evaluate digital environments that aim to empower trust for the user? This position paper explores some of the complexities. The traditional approach of measuring for an increase in trust is not appropriate. Testing for whether the user has made the ‘right’ trust choice is also unsuitable as trust is personal and idiosyncractic and can only be understood via the perspective of the user. We suggest that the measuring of the reduction of uncertainty before and after a user interacts has potential because it can allow access into whether a user has received the benefit shared by both trust and distrust, a reduction in uncertainty.

**Keywords:** trust, evaluation, measuring trust, reduction of uncertainty

# 1 Introduction

Currently the trust research area privileges the fostering of trust within interactions [1] and evaluation methods follow this bias, measuring for the presence and intensity of trust. For instance, standard productivity tools are, as the name implies, likely to aim to encourage as much work between people as possible and are introduced by organisations in order to increase output. An environment of trusting behaviour is valued because the result is a higher level of production, even if it is not in the best interests for the individual worker to trust (perhaps the organisation has little consideration for an individual’s moral rights over what they produce). Measuring a user’s propensity to trust before and after interaction is the obvious way to evaluate, (although how this is done is complex).

But what if you wish to create a digital environment that instead works in an individual user’s interests (trust and distrust may be equal choices for that
user)? The aim of this paper is to outline the challenges when evaluating digital
applications that assist users to negotiate on the user's own terms with other
people and organisations. We describe these type of digital environments as trust
empowering and wish to know how to determine whether a digital environment
is really empowering a user and, as a future research step, to what extent. Recent
research has described the need for applications following the trust empowering
design approach (for instance, [2-5]). Examples of these types of trust empower-
ing applications could be a dating site that helps users negotiate fraught relations
(e.g. the work described in Basu et al [3]) or an application that allows individu-
als to self-select and come together to create a project (for instance, the creative
initiatives described in Möllering [6] or the more commercial projects outlined in
Kaur et al [7]). It may seem premature to tackle evaluation before these types of
projects are widespread, but if we are to develop trust empowering applications,
we need to know the measures by which to judge success. The motivation of
this position paper is a starting point to assist designers of trust empowering
environments to know what to aim for. Potential funders of digital research need
to know how to measure impact before development can start. Projects need to
be evaluated in order for us to know their impact and how to improve current
and future initiatives. As Zender [8] claims, it is irresponsible to design some-
thing irrelevant in the current recession, given the rigorous scrutiny of funding
expenditure and return on investment [9]. Design has also become an influential
research area within public policy and decision makers require evidence if they
are to change their practice.

So new means of evaluating applications in the trust research area need to
be developed. Traditionally, an increase in trust is regarded as success [10]. But
from the perspective of an individual user, distrust as well as trust may be valid
options. Most researchers agree that trust and distrust can have certain out-
comes, including cooperation, clarity and confidence (as documented in [4]). In
this paper, we focus on one: the effect of the reduction of uncertainty of atti-
tude. By uncertainty we mean a lack of clarity about how a particular situation
or interaction may unfold. We propose an approach that taps into an established
area of research, attitude measurement, to investigate how uncertainty may be
reduced using Likert scale surveys.

## 2 Background

Traditionally, the trust research area uses two main approaches to evaluate in-
teractions between participants; surveys and games. There are many variations
to these two approaches as researchers adapt tools for their own purposes. As
Langheinrich [11] argues, the definition of trust held by a researcher depends on
the outcome sought by that researcher. McEvily and Tortoriello [12] reviewed
96 studies and found 42 different measurements of trust and often researchers
bemoan the lack of well-validated trust measures [13]. The approach described
in this paper sidesteps this problem, as we focus on an effect of trust and distrust
(reduction of uncertainty) that is agreed upon in the research area.
In the first approach, surveys used to access respondents’ level of trust by directly asking them. In 1948, Noelle-Neumann developed one of the most commonly used questions to determine trust: ‘Generally speaking, do you believe that most people can be trusted or can’t you be too careful in dealing with people?’ [14]. Other early work includes McCroskey [15] who measured the trustworthiness of speakers. In more recent times, Cummings and Bromiley [16] created the Organizational trust inventory (OTI) to calculate how much trust is in an organization. Reflecting contemporary interest in visual design and branding, Corritore et al [17] designed a model to test how a design can express trustworthiness.

The validity and reliability of surveys to measure trust have often been challenged. For instance the Noelle-Neumann 1948 question is arguably unclear, the possible answers to the question are not mutually exclusive and basing a research conclusion on one answer is unreliable [18]. As Boyer et al [19] argue attempts to measure trust often miss the nuances of local context, such as economic attitudes. Recent developments, according to [14], demonstrate that surveys should use a scale, rather than a dichotomous measure. A scale that allows ‘shades of grey’ is more reflective of how trust functions between people and institutions. However, a problem with scales is that respondents become overwhelmed by choice or follow their own personal pattern and provide responses that tend towards one point of the scale (otherwise known as ‘clumping’).

In the second approach, trust games are used to detect the presence of trust. Rather than asking questions of a participant, this mode is indirect and behaviour is regarded as a signal of trust or distrust. Underlying the choice of research tool is the assumption that cooperation is a reflection of the presence of trust. Researchers who want people to work together (such as in an organisation) would tend to choose this tool and emphasise the collaborative aspect of trust. An early model of this tool was the Prisoners Dilemma (Deutsch 1958, Loomis 1959 in [20]). Later researchers used a trust game within which participants were asked to exchange amounts of money with an anonymous stranger who may provide a return or may abscond [20]. Some researchers have tested whether there is a relationship between trust as measured by the survey and trust measured by performance in the trust game. Results have been mixed [18].

There are advantages to the trust game model. The result of the game is a number that can be calculated and compared. In the game, players deal with money, which is universally understood as an incentive. The framework of the game encourages participants to deliver quick responses. This model allows researchers to create variations that can tap into specific concerns [20]. The research method also attracts criticism. It is difficult to embed the game in a context for participants, so it is debateable whether the game is measuring what participants might really do in a situation and how they might calculate risk [21]. The use of this tool indicates most clearly how choices made by trust researchers reflect their agenda. Those who are focused on the promotion of collaboration choose a tool that emphasises the dimension of trust that facilitates people working together. For instance, Jarvenpaa and Leidner’s study of trust
in a social network [22] was driven by how leaders can access the intellectual
property of those using a network.

We have alluded to the fact that the area of trust research is shaped by a
positivist view of trust: that trust is beneficial and distrust is a negative state [1].
Funded by companies or influenced by commercial agendas, a trust researcher
studying virtual work may choose the result of cooperation as the focus when
measuring trust [14]. Trust research in the context of e-commerce looks to the
speed of transactions as an indication of trust. Arguably some of this research
is not centered on the interests of the user. For instance, there are very good
reasons why an employee should not collaborate in the virtual workplace, even
if it is in the interest of the company. When focusing on what is in the user's
interests, the judgments around trust versus distrust become less clear and more
dependent on context. In some cases it may be advantageous for the individual to
distrust. Too much trust can lead to complacency and an individual not taking
the path that is most advantageous because it might offend or damage trust [20].

The area of human computer interaction (HCI) research also has a bias to-
ward business goals. The links with commercial industry have been strong. Often
the success of HCI has been determined by the achievement of interface products
that support ‘better’ business imperatives; evaluation is based on how productive
an application allows a user to be [23]. Thus an application is judged successful
if it does not frustrate and is efficient. [23] points out that these standards are
inappropriate for the relatively new forms of applications such as games. Games
are intended to cause players to spend time and sometimes feel frustration in
balance with a challenge. Thus new and more appropriate forms of evaluation
are required. Bargas-Avila et al [24] add that as computers expand into more
areas of everyday life, more and more factors must be considered in their eval-
uation. We argue that metrics with a productivity bias are also not appropriate
for interfaces that seek to empower trust for the user. Trust is not either entirely
rational or subjective. It is a peculiar combination that is idiosyncratic, context
bound and sometimes does not conform to a timeframe.

3 The challenge

How can we measure the success of digital applications that empower users to
negotiate trust on their own terms? As mentioned, measuring the existence of
trust is not appropriate; distrust may be a valid option for a certain user in a
specific context. For instance, a user may be considering whether to trust the
advice of an online provider. The provider could be a company that may not
be working in the interest of the user. Additionally, we cannot test that users
have made the ‘correct’ trust choice, as the capacity to judge another’s trust
perception is limited. For instance, perhaps it is in the user’s best interest at
a particular moment in time to follow the advice of a company. A trust choice
is personal and idiosyncratic made in relation to an individual’s sense of risk,
gain and moral viewpoint. In this paper, the authors tend towards the word
‘choice’ and avoid the word ‘decision’ when considering the output of trust. The
word ‘decision’ emphasizes the rational aspect of trust. However, trust is neither entirely subjective nor objective. As argued by [25], there is something ‘mystical’ and inexplicable about trust, otherwise the concept under consideration is not trust – and could more aptly be described by other terms such as ‘calculation’.

## 4 Our approach

To measure the impact of an interface that attempts to empower users to negotiate trust on their own terms, we propose that researchers measure the level of uncertainty before and after interaction. We are not attempting to measure trust, but the presence of an effect of trust. There are many different outcomes of trust and distrust listed in the research, including cooperation, a willingness for vulnerability, confidence, and a reduction of uncertainty (as documented in [4]). Measuring how much a user is willing to cooperate or be vulnerable focuses on what the user might be agreeable to, or arguably, how much a user can be exploited. The notion of confidence does centre more on the user, but is difficult to define and measure [4, p. 53]. We focus on the reduction of uncertainty, because we argue that it is the dimension that is most orientated to the user's interests and is accessible to measurement.

Focusing on the reduction of uncertainty of attitude, allows researchers to investigate whether a trustee is able to reap the benefits of trust and distrust, which is clarity, the ability to move forward (whether the choice is to trust or distrust) with more certainty [2]. Without trust and distrust, the user is caught up in the cycle of exploring possibilities. With trust and distrust, some future possibilities are foreclosed. This is why trust is often viewed as a type of confidence, even when the confidence is not well placed. Distrust is at least as important in this respect also. So, although often seen as a negative state, distrust can in fact resolve a complex scenario, closing down possible paths for the individual to choose as well as protecting the individual from negative consequences.

Uncertainty, that is not being confident about understanding of the past, present or future, can either be epistemic, when we know more current knowledge or consideration might help or on the other hand, aleatory, which is due to chance [26]. The perception of uncertainty is highly personal; two people will rarely be incomplete agreement. “It is not a matter of the uncertainty but your uncertainty” [27]. Uncertainty is core in a trust interaction. Möllering [25] describes trust as compensating for uncertainty. Luhmann [28] argues that trust is the reducer of complexity for individuals in everyday life, helping individuals understand dangers but also allowing individuals to function with confidence (for instance, we drive on the roads confident that others will obey traffic lights). The trust research area has debated whether trust and distrust are separate judgments. Traditionally researchers have tended to conflate the two; trust is on the positive end of the spectrum, while distrust is on the opposite side. More recent research, argue [29], see trust and distrust as separate entities that can co-exist. For instance, I may trust my friend to drive me to the station but not
to be able to fix the car or arrive on time. The approach proposed in this pa-
per sidesteps the need to consider the relationship between trust and distrust
because it focuses on the effect: more certain attitudes.

The way we suggest attitude strength might be measured is to draw on the
field of the psychology of attitudes, which dates back to the 1920s, and has
well-developed survey techniques to measure how strongly a respondent holds
an opinion via self-report. The act of evaluation is a core part of what defines an
attitude [30]. Several fields have drawn on this research area, including marketing
(consumer behaviour) and political science [30]. Rensis Likert is the most notable
of these early attitude researchers, and together with his colleague Thurstone
they developed the Likert scale to quantifiably measure attitudes. A common
argument within the discipline is that strong attitudes are more likely to exist
across time, influence behaviour, and predict behaviour than are attitudes that
are not as certain [30]. The construct validity of attitude strength, certainty and
uncertainty is arguably stronger than that of trust. There is a clear definition of
what certainty and uncertainty mean across the research area of the psychology
of attitudes: the expression of an evaluative judgment about an object, event or
person. However, the construct validity of experiments attempting to measure
trust is often queried because trust is so difficult to definitively define [31].

In our approach, we draw on the technique of surveys, which are an estab-
lished tool in the trust research area as described earlier in this paper. However,
rather than using surveys to test for trust, we wish to investigate whether a
respondent's strength of attitude has changed. Participants are asked to under-
take the same survey before and after interacting in a digital environment and
the results are compared. The survey can be written around the factors in the
trust research literature that researchers claim are central to how users form
trust-based perceptions. Cofta has summarized the factors into 3 dimensions:
competence, motivation, and continuity. The dimension of competence refers to
whether the trustee has the ability and skill to fulfil the requirements of the
interaction [4, p.111]. Competence can include the ability to competently ne-
gotiate a relationship. This involves the trustee as in providing indications of
understanding the trustor's perspective. Kydd [32] describes trust as a 'confi-
dence game'. The dimension of motivation has to do with shared interest [4,
p.111]: Does the trustee have an interest in working towards the welfare of the
trustor? The dimension of continuity is shared interest beyond the current en-
counter; what is the 'shape' of the 'shadow of the future' [33]? A judgment using
continuity uses the length of time the trustee has existed in a certain community
and the amount of associated 'social capital'. Social capital refers to the ways in
which one's social relationships provide access to needed information, resources
and supports. Each survey item is scripted to the relevant subject matter of
the digital environment. The language used to express the survey items can be
tested on the user group targeted by the project to ensure that familiar con-
cepts, words, and expressions are used in the surveys. The important point to
note here, is that we are not wanting to test for the presence and strength of
continuity, competence and motivation but how clear a respondent is about their
perception and conviction regarding these dimensions.

In our model, there are 2 survey items for each of the three dimensions of competence, motivation and continuity. Firstly, the survey asks the respondent to rate the trustee's level of competence, motivation and continuity using separate Likert scales (using language that translates academic theory into more understandable and accessible expression). Secondly, for each one of these ratings, the respondent is asked to nominate on a Likert scale how certain s/he is of the rating allocated. The researchers involved in this paper are most interested in this second set of ratings as we wish to monitor the change in attitude before and after the respondent has interacted with a trust-empowering digital environment. A certainty change level is produced from comparing the two surveys.

The first round of the survey is administered once the respondent is orientated
to the context but has not had time to become familiar with the interaction.
Distinguishing between when a respondent is orientated as distinct from when
the respondent is informed by familiarity is an issue to be explored in further
research. In the research project described in this paper, the participants will be
briefed about the area under consideration, before they have had the opportunity
to interact in the workshop. They will be asked to complete the initial survey.
At the end of the experience, participants will be asked to complete the post-
questionnaire.

5 Problems and further research

The approach suggested above, has problems and raises questions that could
be refined and solved through further research. How many survey items can
gather a suitable amount of data about an interaction? What is the threshold
criteria? How much reduction of uncertainty is considered a success and does this
change across contexts? If so, why? In some situations, individuals are unable
to report their attitudes. An increase in attitude strength may be a result of a
participants becoming more familiar with the context they are placed in for the
research. Experimental researchers often find this problem. Can an intervention
really change behaviour in the fashion intended or are the results the effect of
the participants simply being involved in a study which has primed them to
think in certain ways [34]? However, familiarity is part of the trust equation. As
[28] points out, familiarity breeds trust. However, there is a predictive validity
issue here. Is the approach measuring the effect of trust and distrust or other
variables entering into the equation that could interfere with the results? Refining
the boundaries of trust, familiarity, and attitude strength is a task for future
research.

As the survey items are tailored for a specific context, the capacity to compare
results across studies is reduced. Bijlsma-Frankema and Rousseau [31] debate the
balance between gathering results that are specific to a context and arguably
more precise versus results that are generalisable. They argue that sometimes
research findings do not ring true because as readers we intuitively know that the results may not hold up in the 'real world' (even if the results could be replicated in a laboratory). 'If a theory's representation is accurate, then it is either general-accurate (and not simple) or simple-accurate (but not general)'. Trust is a concept embedded in a context and the participant's understanding of that context, arguably more than other concepts. We argue that data collection should err on the side of context specific at the expense of generalisability.

## 6 Conclusion

In this paper, we propose measuring a reduction in the user's uncertainty as reflection of trust empowerment, using the established technique of Likert scale surveys. Certainty, whether it is tainted by either trust or distrust, allows the individual clarity, which is a benefit of trust argued by the research area. We argue that distrust can also be useful, allowing a user to proceed instead of constantly reviewing options.

New forms of evaluation are required for trust empowering applications that assist users to negotiate trust on their own terms. Measuring for the presence of trust is not appropriate because if an interaction is considered from the user's point of view, trust and distrust are equally valid options. Perhaps it is in the user's best interests not to collaborate, trade or exchange. Appropriate evaluation approaches require debate in the research community so that developers know what the measures of success are for the design of a digital environment.

## References

1. Ashleigh, M., Meyer, E.: Deepening the understanding of trust: combining repertory grid and narrative to explore the uniqueness of trust. Handbook of Research Methods on Trust (2012) 138

2. Marsh, S., Basu, A., Dwyer, N.: Rendering unto cæsar the things that are cæsar's: Complex trust models and human understanding. Trust Management VI (2012) 191-200

3. Basu, A., Dwyer, N., Naicken, S.: A concordance framework for building trust evidences. In: Privacy, Security and Trust (PST), 2012 Tenth Annual International Conference on, IEEE (2012) 153-154

4. Cofta, P.: Trust, complexity and control: confidence in a convergent world. Wiley (2007)

5. Dwyer, N.: Traces of digital trust: an interactive design perspective. PhD thesis, Victoria University (2011)

6. Möllering, G.: Trusting in art: Calling for empirical trust research in highly creative contexts. Journal of Trust Research 2(2) (2012) 203-210

7. Kaur, P., Ruohomaa, S., Kutvonen, L.: Enabling user involvement in trust decision making for inter-enterprise collaborations. International Journal On Advances in Intelligent Systems 5(3 and 4) (2012) 533-552

8. Zender, M., Ecker, K., York, A.: Responsible design for social change: Designing hiv/aids prevention curriculum in southern Africa. In: Design Responsibility: AIGA design education conference, Toledo, OH. May. (2010)
9. Whicher, A., Raulik-Murphy, G., Cawood, G.: Evaluating design: Understanding the return on investment. Design Management Review **22**(2) (2011) 44–52

10. Fukuyama, F.: Trust: The social virtues and the creation of prosperity. Free Press
New York (1995)

11. Langheinrich, M.: When trust does not compute-the role of trust in ubiquitous computing. In: Workshop on Privacy at UBICOMP. Volume 2003. (2003)

12. McEvily, B., Tortoriello, M.: Measuring trust in organisational research: Review and recommendations. Journal of Trust Research **1**(1) (2011) 23–63

13. Gillespie, N.: Measuring trust in organizational contexts: an overview of survey-based measures. Handbook of Research Methods on Trust (2012) 175

14. Uslaner, E.: Measuring generalized trust: in defense of the ‘standard’ question. Handbook of Research Methods on Trust (2012) 72

15. McCroskey, J.: Scales for the measurement of ethos. Speech Monographs **33** (1966) 65–72

16. Cummings, L., Bromiley, P.: The organizational trust inventory (oti): Development and validation. In Kramer, R.M., Tyler, T.R., eds.: Trust in organizations: Frontiers of theory and research. Sage Publications, Inc, Thousand Oaks, CA. (1996) 302–327

17. Corritore, C., Kracher, B., Wiedenbeck, S.: On-line trust: concepts, evolving themes, a model. International Journal of Human-Computer Studies **58**(6) (2003) 737–758

18. Naef, M., Schupp, J.: Measuring trust: Experiments and surveys in contrast and combination. Technical report, IZA Discussion Paper No. 4087 (2009)

19. Boyer, P. ad Lienard, P., Xu, J.: Cultural differences in investing in others and in the future: Why measuring trust is not enough. PloS one **7**(7) (2012) e40750

20. Lewicki, R., Brinsfield, C.: Measuring trust beliefs and behaviours. Handbook of Research Methods on Trust (2012) 29

21. Rad, P.: Measuring trust in online social networks. In Krempels, K.H., Cordeiro, J., eds.: WEBIST. SciTePress (2011)

22. Jarvenpaa, S., Leidner, D.: Communication and trust in global virtual teams. Journal of Computer-Mediated Communication **3**(4) (1998) 0–0

23. Moreno-Ger, P., Torrente, J., Hsieh, Y., Lester, W.: Usability testing for serious games: Making informed design decisions with user data. Advances in Human-Computer Interaction **2012** (2012)

24. Bargas-Avila, J., Hornbæk, K.: Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In: Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011) 2689–2698

25. Möllering, G.: Trust, institutions, agency: towards a neoinstitutional theory of trust. Handbook of trust research (2006) 355

26. Bjornar, S., Stolen, K.: Uncertainty, subjectivity, trust and risk: How it all fits together. In Meadows, C., Fernandez-Gago, C., eds.: Proceedings STM 2011: LNCS 7170, Berlin Heidelberg, Springer-Verlag (2011)

27. Lindley, D.: Understanding uncertainty. Wiley-Interscience (2006)

28. Luhmann, N., Davis, H., Raffan, J., Rooney, K.: Trust and power: two works by Niklas Luhmann. Wiley Chichester (1979)

29. Gargiulo, M., Ertug, G.: The dark side of trust. Handbook of trust research (2006) 165

30. Maio, G., Haddock, G.: The psychology of attitudes and attitude change. Sage Publications Limited (2010)
31. Bijlsma-Frankema, K., Rousseau, D.: It takes a community to make a difference: evaluating quality procedures and practices in trust research. Handbook of Research Methods on Trust (2012) 259

32. Kydd, A.: Overcoming mistrust. Rationality and Society 12(4) (2000) 397–424

33. Axelrod, R.: The Evolution of Cooperation. Basic Books, New York (1984)

34. Klasnja, P., Consolvo, S., Pratt, W.: How to evaluate technologies for health behavior change in hci research. In: Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011) 3063–3072