# Hard-to-Get-at Data from Difficult-to-Access Users

Bob Fields, Andy Bardill, Lisa Marzano, Kate Herd

► To cite this version:

Bob Fields, Andy Bardill, Lisa Marzano, Kate Herd. Hard-to-Get-at Data from Difficult-to-Access Users. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.409-416, 10.1007/978-3-319-22723-8_32. hal-01610835

HAL Id: hal-01610835

https://inria.hal.science/hal-01610835

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Hard-to-get-at data from difficult-to-access users

Bob Fields, Andy Bardill, Lisa Marzano, Kate Herd

Middlesex University, London, UK

`{bfields,a.bardill,l.marzano,k.herd}@mdx.ac.uk`

**Abstract.** This paper reports on the development of a suite of tools to collect, analyze and visualize a diverse range of data from sufferers of mental ill health. The aim is to allow researchers and ultimately sufferers and clinicians to better understand ‘individual signatures’ of factors that indicate or identify episodes of ill health. The tools have been applied in a study working with clients of a mental health service that demonstrates their applicability and acceptability in developing a better understanding of the factors surrounding self-harm behavior.

**Keywords.** Apps, wearable technology, mental health, visualization

## 1 Introduction

The importance of context on behaviors and experiences has long been recognized, and many attempts by the HCI community have been made to devise methods for understanding what people do, in the settings they do it. Approaches to collecting data in-situ have yielded tools for mobile ‘experience sampling’ [2], that regularly collect data from the user, for example to gauge emotional responses to specific locations.

This paper reports an approach to collecting ‘live’ data ‘in the field’ from users who often fall outside of the ‘standard’ population, and in situations that are not typically the focus of more traditional user research methods such as evaluations, diary studies etc. An array of technologies are employed: mobile devices, wearable computing, web applications, to allow participants to engage in a data collection process that is both autonomous and active, providing an enriched understanding of the way that physical, physiological, emotional and environmental factors can influence maladaptive behaviors, in this instance, that of self-harm. Qualitative and quantitative approaches are combined to construct an understanding of both the perception of activity, behavior and context as well a quantitative underpinning of what is actually occurring.

## 2 Background

This research project sought to create an ‘ecological model of self harm’, by better understanding the lives and actions of young men who have a history of self-harm. The literature on self-harm has identified a number of factors that may be implicated
in the changes in mental state leading to self-harm episodes. However, such studies have tended to rely heavily on self-reports that require after-the-fact recollection. The case for reliably monitoring symptoms and signs, and the potential for novel mobile technology is made forcefully in a recent report of the Chief Medical Officer [3].

Several experimental attempts have been made to collect and analyze data that can afford a patient or clinician a better understanding of relevant contextual factors. Typically, however, such projects have been quite narrow in scope – either focusing only on a single condition, or employing only limited sensing, monitoring and analysis technologies. In some cases, projects have been unable to engage with actual patients – an important class of users – and have relied instead on experts and proxy test users.

The Trajectories of Depression project focused on the effects of mobility (as a proxy for activity) on depression [12]. In addition to subjective self-reports, only a single type of more objective data was captured (GPS location). Darzi and others [4] propose a similar approach – using data that can be readily collected by modern smartphone sensors (physical activity and location) to build up a picture of peoples’ lifestyle to support better weight loss strategies. Mappiness [10] demonstrates similar ideas in an attempt to study the effects of place and location on happiness and well-being. Again, a range of subjective data and GPS location is collected, and again, the capabilities of a smartphone provide a convenient platform, but also limit what data is available. The unCUT app [9] is similar in spirit to Insight, though it focuses only on self reporting, rather than the broader range of sensing and data collection used in the initial Insight prototype, and has not, as yet been tested with real patients.

In the current study, we recognized the need for a methodological approach and appropriate tools to collect a diverse range of data from real patients, allowing us to build up an understanding of both the broad context in which self-injury takes place and the individual perspective of experience and personal history on the behaviors taking place.

# 3 Design objectives

Difficulties in affect regulation, poor sleep and lack of physical activity are all markers of poor physical and mental health. Accurate assessment and monitoring of these and other key variables (e.g. compulsive and impulsive behaviors) have long been recognized as important, particularly for relapsing psychological problems. They can help us to understand individual symptom trajectories and potentially identify personalized relapse signatures. To collect a broad range of multi-dimensional data that are not reliant only on self-report, a suite of smartphone apps and other devices were brought together. The system (Fig. 1) includes a combination of off-the-shelf and bespoke hardware and software, and supports the collection of digital diary content, location, and activity and physiological data, that is uploaded to a secure server, and made accessible to the research team via a web-based interface.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Insight system structure</figcaption></figure>

## 3.1 Digital Diary

Participants are prompted twice-daily to complete a multi-media diary ('My Diary') on moods and activities; intensity, duration and contextual features of any self-injurious thoughts and behavior; other risk-taking and impulsive behaviors (e.g. binge eating and drinking); flashbacks; and nightmares. Diary entries include closed questions consisting of check-boxes, rating scale sliders or free text entry fields (Fig. 2).

Participants conclude the diary entry with a free text account of events, thoughts and feelings that can incorporate audio, photographs and video. This content is uploaded to a private 'blog' that can be used for personal recollection and reflection. Participants are also encouraged to post about their broader life histories and experiences ('My Story'). The blog-diary and questionnaire responses are also available, via the web server, to project researchers for later analysis.

Important ethical questions surrounds the effect that using an app like this may have on users. Several studies have shown that individuals participating in research about suicide and self-harm (including qualitative interview studies such as the one used here) do not appear to be negatively affected, and many derive benefit from participation [e.g., see 11]. In addition, previous research has shown that maintaining contact with those who self-harm (through letters, postcards, text messaging, etc.) does not increase their risk of suicide and has potential to reduce repetition. For example, studies have suggested that suicide attempters who receive a follow-up letter or postcard including a simple message of concern are less likely to engage in further self-harm than those who do not receive such contact [8].

## 3.2 Location tracking

Location and physical mobility have been implicated as factors affecting mental state [e.g. see 12], and there is evidence that specific locations may have a causal effect on self-harm behavior (e.g. because of memories triggered in a particular place).

The Insight app tracks location, and uploads location data to the secure server each time the participant completes a diary entry. To manage the desire for privacy, location tracking can be disabled or the tracking accuracy adjusted by the user.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** The Insight App: open and closed diary question entries</figcaption></figure>

## 3.3 Activity and physiological data

Research suggests heart rate variability (HRV) is a reliable biomarker for stress [e.g. 1, 7]. Obtaining reliable data for assessing HRV over an extended period (around 3 weeks in this project), outside of the laboratory in naturalistic settings using non-invasive methods is a non-trivial task. Many commercial heart rate measurement products exist, either as ‘lifestyle devices’, sports training aids, or medical monitoring devices (e.g. cardiological diagnostic tools). However, none (at the time of this study) was intended to unobtrusively capture heart data over a prolonged period: many do not monitor continuously, and those that do tend to be bulky and uncomfortable.

A bespoke heart beat data logger was developed to collect data wirelessly from a commercial chest strap heart sensor. The inter-beat intervals are used to calculate heart rate and HRV. Many ways exist of calculating HRV [e.g. see 5], and the one employed here was the commonly-used approach of using standard deviation of inter-beat intervals over fixed time windows. This sensing method enables reliable data to be collected in a relatively non-invasive and power efficient way.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Data collection using Jawbone UP, chest-strap heart monitor, data-logger and app</figcaption></figure>
While collecting continuous heart beat data required bespoke hardware, the measurement of other indicators of behavior was made simpler by the range of readily available activity tracking devices. The project made use of Jawbone UP wristbands (http://jawbone.com), to sense physical activity, and sleep quality and duration (see Fig. 3).

## 4 Making sense

A key aim has been to provide a means of analyzing data so researchers can gain insights into factors and variables influencing a person's mental state. Of particular interest is the identification of *individual signatures*, indicative of a person's mental state, and which may be connected with recurrence of self-harming behavior. The aim is therefore not (only) to make data amenable to statistical analysis across a cohort of participants. Rather the aim is to support investigations in which patterns, possibly highly individual in nature, may be discerned.

Key questions for a researcher concern the events leading up to an instance of self harm. For instance, was the person in a particular location? Was the person's sleep of poor quality? Were there indications of stress? Do subjective reports of affective state appear to correlate with physiological and environmental data?

To this end, the Insight system provides two main visualization tools: a map and a timeline, which are linked so that selections in one view (e.g. to select only a portion of the timeline data) filters data displayed in the other view (e.g. to show only locations logged during the selected time window). This approach has proved successful in other contexts where an analyst attempts to make sense of related temporal and geographical data [13]. The Geo-view (Fig. 4) shows the locations logged by the smartphone app, overlaid on a map.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Geo-view showing user's movement</figcaption></figure>

The second visualization is a timeline that plots a broad range of data over time, including measured variables (e.g. activity, sleep quality, heart rate and variability) and subjective reports (e.g. responses to questions about affective state) as well as discrete events (e.g. reports of self harm). Data in this fairly raw form presents a confusing
and complex picture, and is likely to be of limited use in discerning any pattern. However, the view provides filtering and zooming, allowing the user to interact with the data to investigate more specific patterns and relationships (e.g., see Fig. ). Currently, data visualizations are at a prototype stage, and suitable for researchers rather than patients. Future development will create visualizations that enable patients to explore and better understand their own behavioral signatures.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Data zoomed and filtered to explore link between HRV and thoughts of self-harm</figcaption></figure>

## 5 An ecological study of self-harm

The technology setup was trialed with 5 users who had a history of self-injury, and who were recruited through a mental health service of which they were clients. People who self-injure may feel uncomfortable discussing their feelings and behavior in a one-to-one interview situation, or find it difficult to verbalize what triggers self-harming behavior. They can however be more willing to engage with a well-designed digital diary or blogging study (for example, ‘Day in the Life’ Projects – see https://dayinthelifemh.org.uk/).

The research was approved by the Middlesex University Psychology Department's Ethics Committee, which is subject to the University research governance and the code of conduct of the British Psychological Society and Health and Care Professions Council. Participant were asked to take part in the study for around 3 weeks, though two participants chose to take part for considerably longer. **Table 1** summarizes participation in the study, indicating the volume and variety of data collected.

<table><caption>**Table 1.** Summary study data</caption><thead><tr><th>Participant no.</th><th>Days in study</th><th>No days made My Diary</th><th>Total no. My Diary entries</th><th>Total no. My Story entries</th><th>Thoughts of Self Harm</th><th>Self Harm</th></tr></thead><tbody><tr><td>1</td><td>79</td><td>65</td><td>99</td><td>128 (+4 videos/photos)</td><td>36</td><td>15</td></tr><tr><td>2</td><td>21</td><td>21</td><td>43</td><td>60 (+30 videos/photos)</td><td>13</td><td>0</td></tr><tr><td>3</td><td>21</td><td>13</td><td>15</td><td>4 (text)</td><td>6</td><td>0</td></tr><tr><td>4</td><td>21</td><td>18</td><td>36</td><td>12 (+1 photo)</td><td>8</td><td>1</td></tr><tr><td>5</td><td>49</td><td>35</td><td>37</td><td>5 (text)</td><td>29</td><td>5</td></tr></tbody></table>

A thorough analysis of the data, and presentation of a ‘model of self harm’ is beyond the scope of the current paper. However, it is worth noting that all participants continued in the study for the expected duration, persisting in making diary entries
throughout (averaging around 1.3 entries per day). No significant drop-off was observed, with participants continuing to contribute throughout the study. The aim here is not to evaluate usability or study the participants' experience directly, but to explore issues such as participants' motivation and willingness to engage, as a way of understanding the potential of the approach as an effective research tool.

Post-study, participants were de-briefed in an interview that explored, among other things, the experience of participation and use of the tools. Reactions to the technology, its usability, and reflections on the value of participation, were generally positive. For example, one participant (P5) reported that the study had helped him “*express some of what I'm going through that's in my head down as data. So it has, yeah, I've found it beneficial. [especially] after I've self-harmed because...*”. Another participant reported using the digital diary app to vent frustrations in a safe way, “*I'm alone but I got my diary to keep me company. I can rant on here and not get told off or nicked...*” (P1); whilst P2 told us that he planned to invest in his own Jawbone wristband at the end of the research, as this had helped him with “*...keeping me in touch with my sleep patterns and when to go to bed, which is fantastic*”. The same participant also reported watching his own video-diaries back and then showing them to his therapist “*so they can see what I am actually like when I'm feeling depressed and down ... So again doing the research is helping me again.*”

# 6 Conclusions

This research-in-progress has developed a novel suite of software and hardware tools to support the collection, analysis and visualization of a range of self reports and sensed data from people with a history of self-harming behavior. The approach has been trialed on a small but substantial study in which data was collected over a period of more than 3 weeks from 5 users. Initial analysis of the data, as well as post-study interviews suggest a very positive response to the approach, with the experience of the technology being generally positive, and the ability to record and reflect being regarded as highly beneficial.

Work is progressing in several areas. The analysis process, and the interactive visual analytic tools needed to make sense of a mass of complex, heterogeneous data is progressing towards the original aim of informing a model of self-harm, that explores links between contextual variables and behavior.

The aim of this project was certainly not to develop a therapeutic tool (although the study indicated a positive effect in taking part and in reflection and gathering of personal commentary). Indeed, the intended users of the data and visualizations of it are mental health researchers, rather than patients or clinicians (and, apart from the blog elements, users don't have access to the recorded data other than through the project team). However, giving patients easy access to their data, and exploring the use of such data in personal and clinical settings is a natural direction for the project to take.

A further set of developments under active investigation concern making the technology more configurable (e.g. to easily create bespoke content for a particular study), personal (e.g. identifying data relevant to an 'individual signature' of a particular
patient), and general (e.g. allowing a range of sensors and visualization tools to be incorporated). On this latter point we are exploring the use of extensions of quite general 'm-health' frameworks such as *Open mHealth* [6].

**Acknowledgements.** The work described in this paper was supported by the Richard Benjamin Trust, to whom the authors are grateful. We would also like to thank our study participants for their participation, and the service through which they were recruited for their support.

# 7 References

1. Brosschot, J.F. et al.: Daily worry is related to low heart rate variability during waking and the subsequent nocturnal sleep period. Int. J. Psychophysiol. 63, 1, 39–47 (2007).

2. Cherubini, M., Oliver, N.: A Refined Experience Sampling Method to Capture Mobile User Experience. Workshop on Mobile User Experience Research - CHI’2009. (2009).

3. Chief Medical Officer: Annual Report of the Chief Medical Officer 2013, Public Mental Health Priorities: Investing in the Evidence, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/351629/Annual_report_2013_1.pdf, (2013).

4. Darzi, A.: Quantified-self for obesity: Physical activity behaviour sensing to improve health outcomes from surgery for severe obesity. EPSRC project EP/L023814/1, http://gow.epsrc.ac.uk/NGBOViewGrant.aspx?GrantRef=EP/L023814/1.

5. Electrophysiology, Task Force of the European Society of Cardiology and the North American Society of Pacing: Heart Rate Variability: Standards of Measurement, Physiological Interpretation, and Clinical Use. Circulation. 93, 5, 1043–1065 (1996).

6. Estrin, D., Sim, I.: Open mHealth Architecture: An Engine for Health Care Innovation. Science. 330, 6005, 759–760 (2010).

7. Horsten, M. et al.: Psychosocial Factors and Heart Rate Variability in Healthy Women. Psychosom. Med. 61, 1, (1999).

8. Kapur, N. et al.: Postcards, green cards and telephone calls: therapeutic contact with individuals following self-harm. Br. J. Psychiatry. 197, 1, 5–7 (2010).

9. Lederer, N. et al.: unCUT: Bridging the Gap from Paper Diary Cards Towards Mobile Electronic Monitoring Solutions in Borderline and Self-Injury. 3rd Int. Conf. on Serious Games and Applications for Health. IEEE (2014).

10. MacKerron, G., Mourato, S.: Happiness is greater in natural environments. Glob. Environ. Change. 23, 5, 992–1000 (2013).

11. Muehlenkamp, J.J. et al.: Emotional and Behavioral Effects of Participating in an Online Study of Nonsuicidal Self-Injury: An Experimental Analysis. Clin. Psychol. Sci. (2014).

12. Musolesi, M.: Trajectories of Depression: Investigating the Correlation between Human Mobility Patterns and Mental Health Problems by means of Smartphones. EPSRC project EP/L006340/1, http://www.cs.bham.ac.uk/research/projects/tod.

13. Xu, K. et al.: Visual analysis of streaming data with SAVI and SenseMAP. 2014 IEEE Conference on Visual Analytics Science and Technology (VAST). pp. 389–390 (2014).