UBICOMP/ISWC '15 ADJUNCT, SEPTEMBER 7–11, 2015, OSAKA, JAPAN

Quantifying the Mental State on theBasis of Physical and Social Activities
Abstract

Shoya Ishimaru
Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japanishimaru@m.cs.osakafu-u.ac.jp

We demonstrate the idea of "Thermometer for the Mind":the mental state estimating system by a user’s activity log
derived from wearable devices. While it is well known thatphysical and social activities are correlated with the mentalstate, our aim is to build an application which tracks dailyactivities automatically, estimates the mental state and visualizes it in an easily understandable way. As a preliminary
experiment, we investigated how information about physicalactivity from a smartphone (step counts) and social activity from a Web service (Twitter post counts) can be used toestimate a user’s mental state. The method is evaluated on
one participant’s 5 months recording. The classification accuracy for 3 classes (mood is low, middle and high) is 60%.

Koichi Kise
Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japan
kise@cs.osakafu-u.ac.jp

Author Keywords
Activity recognition; depression; mental illness; personal
health system; mobile application
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from Permissions@acm.org.
UbiComp/ISWC’15 Adjunct, September 7–11, 2015, Osaka, Japan.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3575-1/15/09...$15.00
DOI: http://dx.doi.org/10.1145/2800835.2807934

1217

ACM Classification Keywords
H.5.m [Information interfaces and presentation (e.g., HCI)]:
Miscellaneous

Introduction
Mental illness, especially depression is one of the mostpressing concerns all over the world. According to the research by WHO in 2012, more than 350 million people

UBICOMP/ISWC '15 ADJUNCT, SEPTEMBER 7–11, 2015, OSAKA, JAPAN

could suffer from depression [1]. If the recognition of depression is late, the symptom will be serious and the duration of therapy will be long towards several years. Knowing
our mental state is important. Yet, the appearance of symptoms of mental illnesses is hard to detect compared withsymptoms of common physical illnesses. You can check
whether you have caught a cold or not by measuring thebody temperature with a thermometer. However, there is nosimple tool which can quantify your mental state. You can’tmeasure how much you feel depressed in an objective way.

Figure 1: A screen shot ofThermometer for the Mind App.

The goal of our research is 1) estimating the mental stateby the user’s biological information and activity log derived
from several wearable devices and 2) visualizing it to makeit easy to understand. Contributions of this paper are twofolds. First, we present the concept of Thermometer for
the Mind, which is an idea of estimating and visualizing
the mental state. Second, we show the preliminary data
analysis of mental state estimation. We have recorded one
participant’s step counts (as one of the measurements ofphysical activities) and Twitter post counts (as one of themeasurements of social activities) with his self-assessment
ground truth for 5 months. We have obtained the classification accuracy of 60% for 3 classes, which might not beenough. Yet, the result and the feature plot show the potential of physical and social activities as features for estimating the mental state.

The second is to develop a personal health monitoring system like MONARCA by Bardram et al. [2] and MoodRhythm
by Voida et al. [9]. These applications are designed forbipolar. They track the user’s daily rhythms by active andpassive methods (e.g. mood by self-assessment, physicaland social activities by sensor logs on a smartphone) andvisualize them to the user and the physician.
The last is to detect depression by functional brain imaging.
Some researchers have investigated functional changes ofbrain oxygenation in prefrontal brain area during a verbal
fluency task by using near-infrared spectroscopy (NIRS).
They have proved that depressive patients had significantly
lower brain activation during the task although there are no
significant differences in behavioral tasks [5] [8].
The aim of "Thermometer for the Mind" is combining them.
It records a user’s activities that are related to the mental
state with wearable sensors, and detects the symptom of
depression easily compared with functional brain imaging.

Concept – Thermometer for the Mind

Related work
There is a large corpus of work focusing on mental illnesses,
and these approaches can be grouped into 3 categories.

Figure 2: A flow image to estiamte
the mental state

depressive symptoms in the general population and depressive symptoms in early adulthood can be a barrier to activity [7]. There is also an opinion survey in U.S. which showssocial activities and emotional wellbeing is related. According to Gallup-Healthways Well-Being Index, Americans feel
happy on days when they spend 6 to 7 hours socializing [4].

The first category is to find the relation between mentalstates and activities. Pereia et al. have proved that the relation between physical activity and depressive symptoms
is bidirectional. Physical activity can help reduce the risk of1218

Figure 1 shows a screenshot of "Thermometer for the Mind"
App. The application scores a user’s mental state and map
it onto a single scale like a temperature. We use the analogy of thermometer for visualization because everyone
should have gone through hardships with fever, and they
can understand how much tired the patient is by this format.
The system estimates the mental state according to theWORKSHOP

flow as shown in Figure 2. First of all is detecting biologicalinformation (e.g. step counts, blinks, heart rate) by raw sensor signals of wearables (e.g. accelerometer, electrooculography sensor). We see the potential of J!NS MEME asa detector of blinks and eye movements, Apple Watch as aheart rate monitor and smartphones as a location logger.
They also have tough batteries which run almost full of one
daytime. They are designed for everyday everywhere use.
The next step is to estimate the user’s activity (e.g. reading,talking, cycling) by biological information. Activity recognition by the combination of blinking frequency and headmotion can be applied in this step [3] [6]. If the system canrecognize almost all of our daily activities, our lifestyle (e.g.
work-life balance, social-private balance) will be appearedin activity log. The system finally estimates the mental stateby monitoring the change of lifestyle.

Preliminary Data Analysis
Estimating Mental States by Steps and Tweets per Day
To evaluate the potential of physical and social activities, weestimate the mental state with the data we can record every
day at the moment. Two features are calculated for eachday, and mental states of days are classified into 3 classes(mood is low, middle, high) by k -nearest neighbor (k =7).
The first feature is step counts derived from a smartphone
as a measurement of physical activity. We use Moves 1 ,which is an activity tracking App. We export summaries foreach day. Then the moving mean (an average of the day
and last 6 days) is calculated and used for a feature.
The second feature is Twitter post counts 2 as one of themeasurements of social activities. We use data on the Web
for the preliminary analysis because logs on the Web can12

https://www.moves-app.com
https://twitter.com

1219

Figure 3: An overview of one participant’s 5 months recording.

be collected easily and it’s still difficult to record all conversation in the real life. We fetch all tweets for each day. Thentweets which contain URLs are removed because most ofthem are not conversation and sometimes they are posted
automatically by other services. Finally, the moving mean iscalculated and used as a feature.
Experimental Setup
We recruited one participant to track his mental state, steps
and tweets. He had suffered from depression since October 2014 and was in the convalescent stage during therecording. The experimental task was to report his mentalstate (self-assessment, from 1 to 5). The daily average x
is used as ground truth. We remapped it to 3 classes (low:1.0 <= x <= 2.5, middle: 2.5 < x < 3.5 and high: 3.5 <= x<= 5.0) for the classification. the period of the experiment
was 5 months, and actually evaluated day was 131 days.
The classification method was evaluated by 10-fold crossvalidation. Figure 3 shows an overview of ground truth andfeatures by physical and social activities.

UBICOMP/ISWC '15 ADJUNCT, SEPTEMBER 7–11, 2015, OSAKA, JAPANPredicted classmiddlehighmiddle

53% 47% 0%
20% 75% 5%highActual class

low

low

0% 93% 7%

Figure 4: A confusion matrix of theevaluation. The classification
accuracy is 60%.

Results and Discussions
Figure 4 is a confusion matrix of the classification result.
The accuracy of classification is 60%. Low and middle conditions are confused, and a lot of high conditions are classified as the middle. Yet, no low condition and high condition
are confused with each other. All samples are classified
into correct or neighboring classes. Figure 5 is a featureplot for each samples. Low mood samples tend to be plotted at few steps and few tweets position. There is no lowmood sample where the step counts are over 6000.

Conclusion and Future WorkWe showed the concept of "Thermometer for the Mind"and proposed the mental estimating method by step andtweet counts per day. The classification accuracy 60% for
3 classes is not enough. Yet, the feature plot shows the potential of physical and social activities as features to estimate the mental state. A possible improvement of thissystem is to estimate higher level information (activitiesand lifestyles) and using them for mental state estimation.
We are also going to get feedbacks from specialists about
which feature can be related to our mental state and whatkind of information can be useful for the diagnosis.

REFERENCES
1. 2012. WHO | Depression. (2012).
http://www.who.int/mediacentre/factsheets/fs369/en/.

Figure 5: A plot of physical andsocial features for each day.

2. Jakob E Bardram, Mads Frost, Károly Szántó, Maria
Faurholt-Jepsen, Maj Vinberg, and Lars Vedel Kessing.
2013. Designing mobile health technology for bipolar
disorder: a field trial of the monarca system. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems. ACM, 2627–2636.
3. M Haak, S Bos, S Panic, and LJM Rothkrantz. 2009.
Detecting stress using eye blinks and brain activity from1220

EEG signals. Proceeding of 1st Driver Car Interaction
and Interface (2009).
4. James Harter and Raksha Arora. 2008. Social timecrucial to daily emotional wellbeing in U.S. GALLUP(2008).
5. MJ Herrmann, A-C Ehlis, and AJ Fallgatter. 2004.
Bilaterally reduced frontal activation during a verbal
fluency task in depressed patients as measured bynear-infrared spectroscopy. The Journal ofneuropsychiatry and clinical neurosciences (2004).
6. Shoya Ishimaru, Jens Weppner, Kai Kunze, Andreas
Bulling, Koichi Kise, Andreas Dengel, and Paul
Lukowicz. 2014. In the Blink of an Eye – Combining
Head Motion and Eye Blink Frequency for Activity
Recognition with Google Glass. In Proceedings of 5th
Augmented Human International Conference. 150–153.
7. Snehal M Pinto Pereira, Marie-Claude Geoffroy, andChristine Power. 2014. Depressive symptoms andphysical activity during 3 decades in adult life:
bidirectional associations in a prospective cohort study.
JAMA psychiatry 71, 12 (2014), 1373–1380.
8. Tomohiro Suto, Masato Fukuda, Makoto Ito, ToruUehara, and Masahiko Mikuni. 2004. Multichannel
near-infrared spectroscopy in depression andschizophrenia: cognitive brain activation study.
Biological Psychiatry 55, 5 (2004), 501–511.
9. Stephen Voida, Mark Matthews, Saeed Abdullah,
Mengxi Chrissie Xi, Matthew Green, Won Jun Jang,
Donald Hu, John Weinrich, Prashama Patil, Mashfiqui
Rabbi, and others. 2013. Moodrhythm: tracking andsupporting daily rhythms. In Proceedings of the 2013
ACM conference on Pervasive and ubiquitous
computing adjunct publication. ACM, 67–70.