# HeadsUp: Keeping Pedestrian Phone Addicts from Dangers Using Mobile Phone Sensors

Authors: Zhengjuan Zhou
Journal: International Journal of Distributed Sensor Networks
 (2015)

---

## Abstract

Walking while staring at the mobile phone is dangerous, and the danger mainly arises from distraction. While watching the mobile phone, one could fall into a deep well without noticing the manhole cover was missing, one could be hit by a rushing car without observing the traffic light, and so forth. Some mobile phone users are already aware of the crisis, and they keep looking up and down to allocate some focus to danger spying; however, the statistics data revealed by US government make such efforts frustrating: about 1,152 pedestrians are injured in US in the year 2010, while they were using mobile phones, and the number doubled in the year 2012. This paper identified the possibility of using mobile phone sensors to develop a walk pattern recognition system. By sampling from embedded sensor, such as accelerometer and gyroscope, the movement pattern of mobile phone users can be computed. We design and implementHeadsUp, a system that warns pedestrian and locks the screen when one looks at the mobile phone while walking. Evaluation results from experiments of 20 testers in real life situation show that, on average, the false negative rate is less than 3%.


---

## Body

## 1. Introduction

The rapid growth of needs for smart mobile terminals expands the mobile phone industry; each 100 persons own 82.6 mobile phones in China by the end of the year 2012, according to MIIT (the Ministry of Industry and Information Technology) of China [1]. People carry around their phones anywhere and anytime, not only for communication, but also for entertaining. People read news using mobile phones while they are on subways and buses, people watch movies using mobile phones while they are lying on couches and beds, people chat with friends using mobile phones while they are walking and driving. We call these people as “phone addicts” or “phubbings” [2]. Some researches [3] report that “phone addiction” makes people indifferent with each other, and traditional social behaviours are breaking down. In fact, other than social problems, “phone addicts” are suffering safety crisis and some of them are not even aware of that.The US Consumer Product Safety Commission reports that 1,152 pedestrians were treated in emergency rooms after being injured while using a mobile phone or some other electronic devices in 2010, and the number had doubled since the year before [4]. The number of pedestrian deaths and injuries has risen dramatically as a result of texting while walking, the number of pedestrians who died rose from 4,109 in 2009 to 4,432 in 2011 and an additional 69,000 injuries, according to a study published by the National Highway Traffic Safety Administration [5].Recent research works have already seen the potential of mobile phone to enhance the safety of people. Several works focus on the safety of drivers: to forbid a driver to use the mobile phone while driving, regulation is not always effective, how to recognize the phone belongs to the driver, and blocking the incoming call and message is a research challenge. Reference [6] use car speakers, which are fixed, to periodically send beacon messages to detect the location of every mobile phone inside the car; therefore, the one that belongs to the driver is recognized. To infer whether a user enters the vehicle from left side or right side of the vehicle or sits in front or rear seats [7] exploit the unique patterns mined from the rich dynamics in the accelerometer and magnetometer data observed during respective actions and make cognitive decision based on machine learning techniques. Some research works tend to protect elderly people from certain accidents [8–12], like accidental falls. Reference [13] use accelerometer to constantly monitor the fall event, and when such event is triggered, the event message will be routed [14] to the nearby cameras which can be added to assist to find and locate the fallen elder; [15] designs type-2 fuzzy models [16–20] that utilize smart phone triaxial accelerometer signals to detect fall incidents and identify abnormal gaits among elderly. Once a fall incident is detected, an alarm is sent to notify the medical staff for taking any necessary treatment. However, only a few researches try to enhance pedestrian safety [21–23] in the perspective of pedestrian itself through carry-on mobile phones. Reference [24] points out that mobile phone conversations distract users, presenting a significant impact to pedestrian safety. It uses the back camera of the mobile phone to detect vehicles approaching the user while he is making a phone call, alerting the user of a potentially unsafe situation.In this paper, we introduce and evaluateHeadsUp, a system that warns pedestrian and lock the screen when one looks at the mobile phone while walking. Different from existing research works which warn users of upcoming rushing cars only while they are making phone calls,HeadsUp detects the behaviours of mobile phone user. Two key behaviours are Walking and Watching.HeadsUp only gives the mobile phone user a warning when a combination of behaviours of walking and watching occurs. Mobile phones are equipped with various of sensors, like accelerometer, gyroscope, magnetometer, and so forth [25], and the sensors provide the ability of sensing the physical world; therefore, the mobile phone has the possibility to sense the behaviour of user. Although some issues arise from sensing with mobile phones, such as noise filtering, a number of research works underpin the design ofHeadsUp. Reference [26] gives a detailed understanding of how well pedometry works when applied to mobile phones and evaluates common walk detection and step counting algorithms applied to mobile phone sensor data. However, the design ofHeadsUp still faces several challenges. (
1
) Unlike most research works that keep wearable devices and mobile phones fixed, we assume that users will place their mobile phones randomly. (
2
) A mobile phone could be in different space for different usage, it could be in a pocket for carrying, could be held in hand for watching, could be lifted for calling. It’s a challenge to judge which space the mobile phone is in and tell if the user is watching the phone.We solve these challenges by taking advantage of gravity. The data generated by accelerometer embedded in a mobile phone is three-dimensional; take android system for example, see Figure1. Due to the noise and accuracy of device, the movement of the mobile phone can be approximately depicted, meanwhile constant gravity will also impact the sensing of accelerometer. In a common sense, gravity will be treated as noise and be cancelled; however, it could be a hint to discover the state of mobile phone. For instance, a mobile phone is placed as Figure 1 shows; the reading of the accelerometer is “x-axis: −0.22984336; y-axis: 9.423578; z-axis: 1.7238252”, the y-axis reading reflects gravity; hence, the placement of mobile phone can be deduced via gravity readings.Figure 1
Coordinate system of mobile phone.We note thatHeadsUp needs not to keep the mobile phone fixed which is in accord with users’ daily usage habits, and it will not block any incoming calls and messages; it only give warning and locks the screen as the user is walking and watching the phone, which keeps the inconvenient experience to the minimum. Our contributions may be summarized as follows.(i)
We designHeadsUp,a pedestrian phone addicts warning system to prevent phubbings from dangers. Mobile phone sensors, including accelerometer and gyroscope, are used to detect the gait pattern.
(ii)
We identified the possibility to detect users’ gait pattern without keeping the mobile phone fixed. It makesHeadsUp practical.
(iii)
We implementHeadsUpon Android platform.The rest of the paper is organized as follows. Section2 presents a high level system overview, followed by the challenges and solutions in Section 3. The system evaluation is presented in Section 4, while summary is given in Section 5.


## 2. System Overview

HeadsUp periodically measures the accelerometer and gyroscope readings from user-carried mobile phones. However, gait recognition is somehow a computational work, and mobile phone battery power is very limited. As Figure 2 shows,HeadsUp divides detection mode into two stages: (
1
) phone usage detection; (
2
) gait recognition. Only when a mobile phone is being held and watched, the gait recognition will be triggered, which makesHeadsUp more energy efficient.Figure 2
HeadsUp architecture.HeadsUp starts with aSampling stage, both accelerometer and gyroscope will be constantly read to get enough sample data. The data will be sent to a module calledUsage Detection; this module will determine of which state the mobile phone is. There are three states to be determined as Figure 3 shows:Pocket state as Figure 3(a) shows is a random state of mobile phone, it could be rolling, could be lying flat, could be in an erect position.Watching as Figure 3(b) shows is that the mobile phone may be held by the user or lying flat on a table, and the user is using it which also means the user is watching it.Calling is that the mobile phone is lifted for calling and it is in an erect position as Figure 3(c) shows.Figure 3
Three phone states: (a) mobile phone is left in the pocket, (b) mobile phone is held in hand and being watched and played, and (c) mobile phone is held for calling.
(a)
Pocket
(b)
Watching
(c)
CallingWhenWatching state is detected, the system will progress toGait Recognition. In this stage, there are two steps. The first step is to determine and transform the coordinate system, because the mobile phone could be in any position, a universe coordinate system is important for gait recognition. The second step is walk pattern recognition, the sampled data will be compared with a walking pattern data set to recognize the gait style of the user.AfterGait Recognition, the pedestrian phone addict will receive a warning and the mobile phone screen will be locked. The user could choose to stop walking for a few seconds to unlock the screen or abruptly abortHeadsUp.


## 3. Challenges and System Design

The design ofHeadsUp in real environments presents practical challenges. This section characterizes the nature of these challenges and presents our approaches in addressing them.


### 3.1. Usage Detection

As mentioned before, there are three states of mobile phone:Pocket,Watching, andCalling, which are to be determined byHeadsUp. Each state has distinguishable features as Figure 4 shows;Pocket state could be in any position as Figure 4(a) shows;Watching state is always horizontal or only a few degrees off horizontal as Figure 4(b) shows;Calling state is always vertical or a few degrees off vertical as Figure 4(c) shows. DistinguishingWatching fromCalling is simple. Using gravity as a reference, it is easy to determine the two states through coordination system readings. However, it would be a little difficult to differentiateWatching fromPocket. AsPocket could be in any position, it may be placed horizontally which is the same position asWatching. We noticed that when a user is using and watching the mobile phone, the screen will be tapped and touched intermittently; although these actions maybe subtle to other actions like walking, gyroscope is sensitive enough to capture these slight actions, also it is observed that the gyroscope is not affected by the walking movement, mainly because there is no spin or rotation of mobile phone during the walking.Figure 4
Distinguishable features of three phone states: (a)Pocket may be in random position, (b)Watching is horizontal or a few degrees off horizontal, and (c)Calling is vertical or a few degrees off vertical.
(a)
Pocket
(b)
Watching
(c)
Calling


#### 3.1.1. Watching versus Calling

The direction of gravity is constant, we can determine the state ofWatching andCalling by accelerometer readings. Suppose the reading of the accelerometer is a vector, which is defined as S
a
=
(
A
x
,
A
y
,
A
z
), where A
x is the reading of axis x, A
y is the reading of axis y, and A
z is the reading of axis z. The reference vector, which refers to an ideal reading of the accelerometer when the mobile phone is placed perfectly horizontal, is defined as R
a
=
(
0,0
,
g
). Normally, g equals 9.8
m
/
s
2; however, it varies in different areas. To eliminate the variance of g, we normalize R
a as R
a
N
=
R
a
/
R
a
=
(
0,0
,
1
), S
a as S
a
N
=
S
a
/
|
S
a
|. Now we give the equation for determining theWalking state:(1)
θ
=
arccos
S
a
·
R
a
S
a
R
a
=
arccos
S
a
N
·
R
a
N
.In (1), θ equals the inner product of S
a
N and R
a
N, and it refers to how close the mobile phone approaches horizontal position, the smaller the closer. If θ
<
δ, where δ is an adjustable parameter used as a threshold to indicate the state ofWatching, we set the current state of the mobile phone asWatching. InHeadsUp, we usually set δ to 4
5
°.


#### 3.1.2. Watching versus Pocket

If the mobile phone is occasionally placed horizontally in a pocket or bag, it may be confused as in theWatching state inHeadsUp. A gyroscope is a device for measuring or maintaining orientation, based on the principles of angular momentum [27]. The gyroscope is very sensitive that it can capture subtle movements which refer to rotation and spin. When a user is using the mobile phone, the user may tap the screen to open a web page, slide the screen to unlock it, or scroll a news page. Although these actions are very subtle, they can still be captured by the gyroscope. Also, the movement of walking involves few or very slow spin or rotation action, it hardly affects the reading of gyroscope. Therefore, we can use gyroscope to distinguishWatching fromPocket.Figure5 gives readings of gyroscope in two situations: one is that a user held the mobile phone horizontally and kept walking; the other one is similar except that the user was using the mobile phone. We can see from Figures 5(a) to 5(c) that the readings of gyroscope are hardly fluctuating due to the walking movement; the peak fluctuation is around 0.06 radians per second. From Figures 5(d) and 5(e) which refer to the readings components of x-axis and y-axis, we can see some obvious fluctuations of which the peak is around 0.25 radians per second. Hence, theWatching can be distinguished fromPocket as long as avg
(
abs
(
S
g
)
)
>
ϕ where S
g
=
(
G
x
,
G
y
,
G
z
) is a vector which is the reading of gyroscope, G
x to G
z are the component readings of coordination system of gyroscope, and ϕ is a threshold which indicates theWatching state.Figure 5
The readings of gyroscope in two different cases: the first three subfigures refer to whether the user is walking and watching the mobile phone; the last three subfigures refer to the user is walking, meanwhile watching and using the mobile phone.
(a)
x-axis reading of gyroscope while walking
(b)
y-axis reading of gyroscope while walking
(c)
z-axis reading of gyroscope while walking
(d)
x-axis reading of gyroscope while walking and using phone
(e)
y-axis reading of gyroscope while walking and using phone
(f)
z-axis reading of gyroscope while walking and using phone


#### 3.1.3. Watching versus All

Usually, we sample a series of sensor readings for analysis. Suppose we getn samples of sensor readings; A
x
i represents ith sample of accelerometer readings, so as G
x
i. The samples are organized as the form of underlying matrixes. M
S
a is a series of accelerometer readings; M
S
g is a series of gyroscope readings, M
S is the combination of M
S
a and M
S
g,(2)
M
S
a
=
A
x
1
A
x
2
⋯
A
x
n
A
y
1
A
y
2
⋯
A
y
n
A
z
1
A
z
2
⋯
A
z
n
,
M
S
g
=
G
x
1
G
x
2
⋯
G
x
n
G
y
1
G
y
2
⋯
G
y
n
G
z
1
G
z
2
⋯
G
z
n
,
M
S
=
M
S
a
M
S
g
,to distinguishWalking fromCalling; (1) needs to be expanded to matrix operations as (3) shows: (3)
θ
=
∑
i
=
1
n
arccos
⁡
M
S
a
i
·
R
a
/
M
S
a
i
R
a
n
=
∑
i
=
1
n
arccos
M
S
a
i
N
·
R
a
N
n
,where M
S
a
i
=
A
x
i
A
y
i
A
z
i
′.To distinguishWalking fromPocket, the underlying condition needs to be satisfied: (4)
G
=
∑
i
=
1
n
avg
abs
M
S
g
i
n
>
ϕ
,where M
S
g
i
=
G
x
i
G
y
i
G
z
i
′.Therefore, we can give (5) to determine whether the mobile phone is in aWatching state:(5)
“Watching"
=
θ
<
δ
,
G
>
ϕ
.


### 3.2.  Coordinate System Transformation

For the convenience of Pattern Recognition, we need to make coordinate transformation to a universal coordinate system. We choose an ideal situation where a mobile phone is placed horizontally and the Earth’s acceleration due to gravity equals the standard one which is9.8
m
/
s
2, and we set the coordinate system as the universal one.As shown in Figure6, the coordinate system of the mobile phone needs to rotate to fit the universal one. Moreover, there exists measurement errors in every mobile phone which needs to be eliminated. We assume the coordinate system transformation is in a affine space; then, we get the underlying equation as follows:(6)
A
x
=
B
y
+
ε
,where A is the base vectors of the universal coordinate system; it is equal to 1
0
0
0
1
0
0
0
1; x is the ideal reading of accelerometer in the universal coordinate system; B is the base vectors of the mobile phone coordinate system which is unknown; y is the reading of accelerometer which equals to S
a; ε is the measure error of the mobile phone.Figure 6
Coordinate system transformation.Now we give a theorem for coordinate system transformation when the user is standing still.Theorem 1.
Letx
=
0
0
x
3
′ and let y
=
y
1
y
2
y
3
′; then, x
3
=
|
y
|
=
y
1
2
+
y
2
2
+
y
3
2.Proof.
(7)
A
x
=
B
y
+
ε
⟹
A
x
=
B
y
+
ε
⟹
x
′
A
′
A
x
=
y
′
B
′
B
y
+
y
′
B
′
ε
+
ε
′
B
y
+
ε
′
εbecause lim
ε
→
0
⁡
y
′
B
′
ε
+
ε
′
B
y
+
ε
′
ε
=
0 then x
′
A
′
A
x
=
y
′
B
′
B
y because A and B are base vectors, they are orthogonal matrixes, then A
′
A
=
B
′
B
=
I:(8)
⟹
x
=
y
x
=
0
0
x
3
′, then |
x
3
|
=
y.For Walking Pattern Recognition, the componentx
3 of x is enough; however, when the user starts to move, x
1 and x
2 will no longer be zero which makes Theorem 1 not fit. From the proof we know that |
x
|
=
|
y
| and |
x
|
2
=
x
1
2
+
x
2
2
+
x
3
2; then, |
x
3
|
=
|
y
|
2
-
x
1
2
-
x
2
2. What we need to get is the sum of x
1
2
+
x
2
2.We all know the underlying famous equation which reveals the relation between distance and acceleration, whered is distance traveled in a certain amount of time (t), v
0 is starting velocity, a is acceleration, and t is time: (9)
d
=
v
0
t
+
1
2
a
2
t
.If we can get the value of d and v
0, we can get the value of a
2. Now that the accelerometer is not available for measuring, we turned to another common sensor, GPS. GPS gives the exact location of the mobile phone; suppose the sampling interval is t, then we can get the value of d by d
=
d
t
+
1
-
d
t and get the value of v
0 by v
0
=
Δ
d
/
Δ
t. In the universe coordinate system, a
2 equals x
1
2
+
x
2
2; then, we have (10) for coordinate transformation:(10)
x
3
=
y
2
-
2
d
t
-
v
0
.


### 3.3. Walking Pattern Recognition

Figure7 gives the readings of accelerometer in five typical situations, including walking, running, jumping, sitting, and riding. We can observe that the fluctuation of readings of each axis component is different; axis-z is dominant. To verify the observation, we depict all the sampled data on a 3D figure as Figure 8 shows. It is obvious that the data points expand along axis-z. PCA (Principal Component Analysis) is a statistical procedure that uses orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components [28]. Using PCA, we can get the principal component of the data set. Suppose the data set is A, the underlying equation is how to get eigenvectors of the covariance of A; then, the first eigenvector ofcoefficient is the principal component which is (
-
0.0613,0.1338,0.9891
). Therefore, we choose axis-z as the feature to perform Walking Pattern Recognition: (11)
coefficient
=
PCA
A
.Figure 7
The readings of accelerometer in five typical situations: including Walking, Running, Jumping, Sitting, and Riding.
(a)
axis-x
(b)
axis-y
(c)
axis-zFigure 8
3D presentation of the data sets.Now that we leave only one feature, we can then use a simple and effective way to perform the walking pattern recognition. In signal processing, cross-correlation is a measure of similarity of two waveforms as a function of a time-lag applied to one of them. This is also known as a sliding dot product or sliding inner-product. It is commonly used for searching a long signal for a shorter, known feature [29]. We establish a data set which contains several readings of accelerometer related to different users while they are walking. Once the Walking Pattern Recognition module is triggered, the system will compare current readings of accelerometer with each record in the data set using cross correlation.Suppose the known symbol pattern of walking ist
s of length L, X
(
n
) is the complex number representing the n
th received symbol, then the cross-correlation at a shift position p is as (12) shows:(12)
C
t
s
,
X
,
p
=
∑
k
=
1
L
‍
t
s
∗
k
X
k
+
p
.t
s
∗ is the complex conjugate of t
s and C
(
t
s
,
X
,
p
) is the correlation coefficient. Once the X
[
n
] is aligned with t
s, there will be a sudden spike in the correlation as Figure 9 shows.Figure 9
Cross-correlation.OnceC
(
t
s
,
X
,
p
)
>
ξ, where ξ is a threshold which indicates the walking pattern, we assume the user is in a walking state.


## 3.1. Usage Detection

As mentioned before, there are three states of mobile phone:Pocket,Watching, andCalling, which are to be determined byHeadsUp. Each state has distinguishable features as Figure 4 shows;Pocket state could be in any position as Figure 4(a) shows;Watching state is always horizontal or only a few degrees off horizontal as Figure 4(b) shows;Calling state is always vertical or a few degrees off vertical as Figure 4(c) shows. DistinguishingWatching fromCalling is simple. Using gravity as a reference, it is easy to determine the two states through coordination system readings. However, it would be a little difficult to differentiateWatching fromPocket. AsPocket could be in any position, it may be placed horizontally which is the same position asWatching. We noticed that when a user is using and watching the mobile phone, the screen will be tapped and touched intermittently; although these actions maybe subtle to other actions like walking, gyroscope is sensitive enough to capture these slight actions, also it is observed that the gyroscope is not affected by the walking movement, mainly because there is no spin or rotation of mobile phone during the walking.Figure 4
Distinguishable features of three phone states: (a)Pocket may be in random position, (b)Watching is horizontal or a few degrees off horizontal, and (c)Calling is vertical or a few degrees off vertical.
(a)
Pocket
(b)
Watching
(c)
Calling


### 3.1.1. Watching versus Calling

The direction of gravity is constant, we can determine the state ofWatching andCalling by accelerometer readings. Suppose the reading of the accelerometer is a vector, which is defined as S
a
=
(
A
x
,
A
y
,
A
z
), where A
x is the reading of axis x, A
y is the reading of axis y, and A
z is the reading of axis z. The reference vector, which refers to an ideal reading of the accelerometer when the mobile phone is placed perfectly horizontal, is defined as R
a
=
(
0,0
,
g
). Normally, g equals 9.8
m
/
s
2; however, it varies in different areas. To eliminate the variance of g, we normalize R
a as R
a
N
=
R
a
/
R
a
=
(
0,0
,
1
), S
a as S
a
N
=
S
a
/
|
S
a
|. Now we give the equation for determining theWalking state:(1)
θ
=
arccos
S
a
·
R
a
S
a
R
a
=
arccos
S
a
N
·
R
a
N
.In (1), θ equals the inner product of S
a
N and R
a
N, and it refers to how close the mobile phone approaches horizontal position, the smaller the closer. If θ
<
δ, where δ is an adjustable parameter used as a threshold to indicate the state ofWatching, we set the current state of the mobile phone asWatching. InHeadsUp, we usually set δ to 4
5
°.


### 3.1.2. Watching versus Pocket

If the mobile phone is occasionally placed horizontally in a pocket or bag, it may be confused as in theWatching state inHeadsUp. A gyroscope is a device for measuring or maintaining orientation, based on the principles of angular momentum [27]. The gyroscope is very sensitive that it can capture subtle movements which refer to rotation and spin. When a user is using the mobile phone, the user may tap the screen to open a web page, slide the screen to unlock it, or scroll a news page. Although these actions are very subtle, they can still be captured by the gyroscope. Also, the movement of walking involves few or very slow spin or rotation action, it hardly affects the reading of gyroscope. Therefore, we can use gyroscope to distinguishWatching fromPocket.Figure5 gives readings of gyroscope in two situations: one is that a user held the mobile phone horizontally and kept walking; the other one is similar except that the user was using the mobile phone. We can see from Figures 5(a) to 5(c) that the readings of gyroscope are hardly fluctuating due to the walking movement; the peak fluctuation is around 0.06 radians per second. From Figures 5(d) and 5(e) which refer to the readings components of x-axis and y-axis, we can see some obvious fluctuations of which the peak is around 0.25 radians per second. Hence, theWatching can be distinguished fromPocket as long as avg
(
abs
(
S
g
)
)
>
ϕ where S
g
=
(
G
x
,
G
y
,
G
z
) is a vector which is the reading of gyroscope, G
x to G
z are the component readings of coordination system of gyroscope, and ϕ is a threshold which indicates theWatching state.Figure 5
The readings of gyroscope in two different cases: the first three subfigures refer to whether the user is walking and watching the mobile phone; the last three subfigures refer to the user is walking, meanwhile watching and using the mobile phone.
(a)
x-axis reading of gyroscope while walking
(b)
y-axis reading of gyroscope while walking
(c)
z-axis reading of gyroscope while walking
(d)
x-axis reading of gyroscope while walking and using phone
(e)
y-axis reading of gyroscope while walking and using phone
(f)
z-axis reading of gyroscope while walking and using phone


### 3.1.3. Watching versus All

Usually, we sample a series of sensor readings for analysis. Suppose we getn samples of sensor readings; A
x
i represents ith sample of accelerometer readings, so as G
x
i. The samples are organized as the form of underlying matrixes. M
S
a is a series of accelerometer readings; M
S
g is a series of gyroscope readings, M
S is the combination of M
S
a and M
S
g,(2)
M
S
a
=
A
x
1
A
x
2
⋯
A
x
n
A
y
1
A
y
2
⋯
A
y
n
A
z
1
A
z
2
⋯
A
z
n
,
M
S
g
=
G
x
1
G
x
2
⋯
G
x
n
G
y
1
G
y
2
⋯
G
y
n
G
z
1
G
z
2
⋯
G
z
n
,
M
S
=
M
S
a
M
S
g
,to distinguishWalking fromCalling; (1) needs to be expanded to matrix operations as (3) shows: (3)
θ
=
∑
i
=
1
n
arccos
⁡
M
S
a
i
·
R
a
/
M
S
a
i
R
a
n
=
∑
i
=
1
n
arccos
M
S
a
i
N
·
R
a
N
n
,where M
S
a
i
=
A
x
i
A
y
i
A
z
i
′.To distinguishWalking fromPocket, the underlying condition needs to be satisfied: (4)
G
=
∑
i
=
1
n
avg
abs
M
S
g
i
n
>
ϕ
,where M
S
g
i
=
G
x
i
G
y
i
G
z
i
′.Therefore, we can give (5) to determine whether the mobile phone is in aWatching state:(5)
“Watching"
=
θ
<
δ
,
G
>
ϕ
.


## 3.1.1. Watching versus Calling

The direction of gravity is constant, we can determine the state ofWatching andCalling by accelerometer readings. Suppose the reading of the accelerometer is a vector, which is defined as S
a
=
(
A
x
,
A
y
,
A
z
), where A
x is the reading of axis x, A
y is the reading of axis y, and A
z is the reading of axis z. The reference vector, which refers to an ideal reading of the accelerometer when the mobile phone is placed perfectly horizontal, is defined as R
a
=
(
0,0
,
g
). Normally, g equals 9.8
m
/
s
2; however, it varies in different areas. To eliminate the variance of g, we normalize R
a as R
a
N
=
R
a
/
R
a
=
(
0,0
,
1
), S
a as S
a
N
=
S
a
/
|
S
a
|. Now we give the equation for determining theWalking state:(1)
θ
=
arccos
S
a
·
R
a
S
a
R
a
=
arccos
S
a
N
·
R
a
N
.In (1), θ equals the inner product of S
a
N and R
a
N, and it refers to how close the mobile phone approaches horizontal position, the smaller the closer. If θ
<
δ, where δ is an adjustable parameter used as a threshold to indicate the state ofWatching, we set the current state of the mobile phone asWatching. InHeadsUp, we usually set δ to 4
5
°.


## 3.1.2. Watching versus Pocket

If the mobile phone is occasionally placed horizontally in a pocket or bag, it may be confused as in theWatching state inHeadsUp. A gyroscope is a device for measuring or maintaining orientation, based on the principles of angular momentum [27]. The gyroscope is very sensitive that it can capture subtle movements which refer to rotation and spin. When a user is using the mobile phone, the user may tap the screen to open a web page, slide the screen to unlock it, or scroll a news page. Although these actions are very subtle, they can still be captured by the gyroscope. Also, the movement of walking involves few or very slow spin or rotation action, it hardly affects the reading of gyroscope. Therefore, we can use gyroscope to distinguishWatching fromPocket.Figure5 gives readings of gyroscope in two situations: one is that a user held the mobile phone horizontally and kept walking; the other one is similar except that the user was using the mobile phone. We can see from Figures 5(a) to 5(c) that the readings of gyroscope are hardly fluctuating due to the walking movement; the peak fluctuation is around 0.06 radians per second. From Figures 5(d) and 5(e) which refer to the readings components of x-axis and y-axis, we can see some obvious fluctuations of which the peak is around 0.25 radians per second. Hence, theWatching can be distinguished fromPocket as long as avg
(
abs
(
S
g
)
)
>
ϕ where S
g
=
(
G
x
,
G
y
,
G
z
) is a vector which is the reading of gyroscope, G
x to G
z are the component readings of coordination system of gyroscope, and ϕ is a threshold which indicates theWatching state.Figure 5
The readings of gyroscope in two different cases: the first three subfigures refer to whether the user is walking and watching the mobile phone; the last three subfigures refer to the user is walking, meanwhile watching and using the mobile phone.
(a)
x-axis reading of gyroscope while walking
(b)
y-axis reading of gyroscope while walking
(c)
z-axis reading of gyroscope while walking
(d)
x-axis reading of gyroscope while walking and using phone
(e)
y-axis reading of gyroscope while walking and using phone
(f)
z-axis reading of gyroscope while walking and using phone


## 3.1.3. Watching versus All

Usually, we sample a series of sensor readings for analysis. Suppose we getn samples of sensor readings; A
x
i represents ith sample of accelerometer readings, so as G
x
i. The samples are organized as the form of underlying matrixes. M
S
a is a series of accelerometer readings; M
S
g is a series of gyroscope readings, M
S is the combination of M
S
a and M
S
g,(2)
M
S
a
=
A
x
1
A
x
2
⋯
A
x
n
A
y
1
A
y
2
⋯
A
y
n
A
z
1
A
z
2
⋯
A
z
n
,
M
S
g
=
G
x
1
G
x
2
⋯
G
x
n
G
y
1
G
y
2
⋯
G
y
n
G
z
1
G
z
2
⋯
G
z
n
,
M
S
=
M
S
a
M
S
g
,to distinguishWalking fromCalling; (1) needs to be expanded to matrix operations as (3) shows: (3)
θ
=
∑
i
=
1
n
arccos
⁡
M
S
a
i
·
R
a
/
M
S
a
i
R
a
n
=
∑
i
=
1
n
arccos
M
S
a
i
N
·
R
a
N
n
,where M
S
a
i
=
A
x
i
A
y
i
A
z
i
′.To distinguishWalking fromPocket, the underlying condition needs to be satisfied: (4)
G
=
∑
i
=
1
n
avg
abs
M
S
g
i
n
>
ϕ
,where M
S
g
i
=
G
x
i
G
y
i
G
z
i
′.Therefore, we can give (5) to determine whether the mobile phone is in aWatching state:(5)
“Watching"
=
θ
<
δ
,
G
>
ϕ
.


## 3.2.  Coordinate System Transformation

For the convenience of Pattern Recognition, we need to make coordinate transformation to a universal coordinate system. We choose an ideal situation where a mobile phone is placed horizontally and the Earth’s acceleration due to gravity equals the standard one which is9.8
m
/
s
2, and we set the coordinate system as the universal one.As shown in Figure6, the coordinate system of the mobile phone needs to rotate to fit the universal one. Moreover, there exists measurement errors in every mobile phone which needs to be eliminated. We assume the coordinate system transformation is in a affine space; then, we get the underlying equation as follows:(6)
A
x
=
B
y
+
ε
,where A is the base vectors of the universal coordinate system; it is equal to 1
0
0
0
1
0
0
0
1; x is the ideal reading of accelerometer in the universal coordinate system; B is the base vectors of the mobile phone coordinate system which is unknown; y is the reading of accelerometer which equals to S
a; ε is the measure error of the mobile phone.Figure 6
Coordinate system transformation.Now we give a theorem for coordinate system transformation when the user is standing still.Theorem 1.
Letx
=
0
0
x
3
′ and let y
=
y
1
y
2
y
3
′; then, x
3
=
|
y
|
=
y
1
2
+
y
2
2
+
y
3
2.Proof.
(7)
A
x
=
B
y
+
ε
⟹
A
x
=
B
y
+
ε
⟹
x
′
A
′
A
x
=
y
′
B
′
B
y
+
y
′
B
′
ε
+
ε
′
B
y
+
ε
′
εbecause lim
ε
→
0
⁡
y
′
B
′
ε
+
ε
′
B
y
+
ε
′
ε
=
0 then x
′
A
′
A
x
=
y
′
B
′
B
y because A and B are base vectors, they are orthogonal matrixes, then A
′
A
=
B
′
B
=
I:(8)
⟹
x
=
y
x
=
0
0
x
3
′, then |
x
3
|
=
y.For Walking Pattern Recognition, the componentx
3 of x is enough; however, when the user starts to move, x
1 and x
2 will no longer be zero which makes Theorem 1 not fit. From the proof we know that |
x
|
=
|
y
| and |
x
|
2
=
x
1
2
+
x
2
2
+
x
3
2; then, |
x
3
|
=
|
y
|
2
-
x
1
2
-
x
2
2. What we need to get is the sum of x
1
2
+
x
2
2.We all know the underlying famous equation which reveals the relation between distance and acceleration, whered is distance traveled in a certain amount of time (t), v
0 is starting velocity, a is acceleration, and t is time: (9)
d
=
v
0
t
+
1
2
a
2
t
.If we can get the value of d and v
0, we can get the value of a
2. Now that the accelerometer is not available for measuring, we turned to another common sensor, GPS. GPS gives the exact location of the mobile phone; suppose the sampling interval is t, then we can get the value of d by d
=
d
t
+
1
-
d
t and get the value of v
0 by v
0
=
Δ
d
/
Δ
t. In the universe coordinate system, a
2 equals x
1
2
+
x
2
2; then, we have (10) for coordinate transformation:(10)
x
3
=
y
2
-
2
d
t
-
v
0
.


## 3.3. Walking Pattern Recognition

Figure7 gives the readings of accelerometer in five typical situations, including walking, running, jumping, sitting, and riding. We can observe that the fluctuation of readings of each axis component is different; axis-z is dominant. To verify the observation, we depict all the sampled data on a 3D figure as Figure 8 shows. It is obvious that the data points expand along axis-z. PCA (Principal Component Analysis) is a statistical procedure that uses orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components [28]. Using PCA, we can get the principal component of the data set. Suppose the data set is A, the underlying equation is how to get eigenvectors of the covariance of A; then, the first eigenvector ofcoefficient is the principal component which is (
-
0.0613,0.1338,0.9891
). Therefore, we choose axis-z as the feature to perform Walking Pattern Recognition: (11)
coefficient
=
PCA
A
.Figure 7
The readings of accelerometer in five typical situations: including Walking, Running, Jumping, Sitting, and Riding.
(a)
axis-x
(b)
axis-y
(c)
axis-zFigure 8
3D presentation of the data sets.Now that we leave only one feature, we can then use a simple and effective way to perform the walking pattern recognition. In signal processing, cross-correlation is a measure of similarity of two waveforms as a function of a time-lag applied to one of them. This is also known as a sliding dot product or sliding inner-product. It is commonly used for searching a long signal for a shorter, known feature [29]. We establish a data set which contains several readings of accelerometer related to different users while they are walking. Once the Walking Pattern Recognition module is triggered, the system will compare current readings of accelerometer with each record in the data set using cross correlation.Suppose the known symbol pattern of walking ist
s of length L, X
(
n
) is the complex number representing the n
th received symbol, then the cross-correlation at a shift position p is as (12) shows:(12)
C
t
s
,
X
,
p
=
∑
k
=
1
L
‍
t
s
∗
k
X
k
+
p
.t
s
∗ is the complex conjugate of t
s and C
(
t
s
,
X
,
p
) is the correlation coefficient. Once the X
[
n
] is aligned with t
s, there will be a sudden spike in the correlation as Figure 9 shows.Figure 9
Cross-correlation.OnceC
(
t
s
,
X
,
p
)
>
ξ, where ξ is a threshold which indicates the walking pattern, we assume the user is in a walking state.


## 4. Evaluation

We implementHeadsUp on an Android platform and it is compatible with 2.2 and up versions of Android operation system. The test devices include 20 mobile phones with android 4.2, such as HTC one, Sumsung S4, and Lenovo S2. All the devices are embedded with accelerometer, gyroscope, and GPS. There are 20 users involved in the test, each one is in charge of one mobile phone. The test performs in four different scenes: (
1
) walking and watching; (
2
) walking and not watching; (
3
) sitting and watching; (
4
) riding and watching. Each scene tests for 100 times in random places.


### 4.1. Scene 1: Walking and Watching

Twenty users randomly choose walking paths and walk in a usual way with the intermittently stop; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure10 shows.Figure 10
Walking and watching test.


### 4.2. Scene 2: Walking and Not Watching

Twenty users randomly choose walking paths and walk in an usual way with intermittently holding up the mobile phone and watching; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure11 shows.Figure 11
Walking and not watching test.


### 4.3. Scene 3: Sitting and Watching

Twenty users sit with random postures and use the mobile phone for internet surfing or game playing; they would intermittently stand up and walk for a while; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure12 shows.Figure 12
Sitting and watching test.


### 4.4. Scene 4: Riding and Watching

Twenty users randomly choose a riding tool, including bus, taxi, and subway, with intermittently leaving the car or train and walking away while using the mobile phone. Each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure13 shows.Figure 13
Riding and watching test.Through the test results from four-scene test, we notice that the false positive rate is lower that 2% on average, which shows thatHeadsUp seldom interferes users’ normal usage of the mobile phone. The false negative rate is around 3% on average which guarantees thatHeadsUp will duly warn and keep the pedestrian phone addicts from watching the screen then make them stop and use the mobile phone, therefore, reducing the occurrence rate of potential dangers.


## 4.1. Scene 1: Walking and Watching

Twenty users randomly choose walking paths and walk in a usual way with the intermittently stop; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure10 shows.Figure 10
Walking and watching test.


## 4.2. Scene 2: Walking and Not Watching

Twenty users randomly choose walking paths and walk in an usual way with intermittently holding up the mobile phone and watching; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure11 shows.Figure 11
Walking and not watching test.


## 4.3. Scene 3: Sitting and Watching

Twenty users sit with random postures and use the mobile phone for internet surfing or game playing; they would intermittently stand up and walk for a while; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure12 shows.Figure 12
Sitting and watching test.


## 4.4. Scene 4: Riding and Watching

Twenty users randomly choose a riding tool, including bus, taxi, and subway, with intermittently leaving the car or train and walking away while using the mobile phone. Each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure13 shows.Figure 13
Riding and watching test.Through the test results from four-scene test, we notice that the false positive rate is lower that 2% on average, which shows thatHeadsUp seldom interferes users’ normal usage of the mobile phone. The false negative rate is around 3% on average which guarantees thatHeadsUp will duly warn and keep the pedestrian phone addicts from watching the screen then make them stop and use the mobile phone, therefore, reducing the occurrence rate of potential dangers.


## 5. Summary

In this paper, we design and implementHeadsUp, a practical system that warns pedestrian and locks the screen when one looks at the mobile phone while walking. Evaluation results show that, on average, the false negative rate is less than 3%, which guarantees thatHeadsUp will duly warn and keep the pedestrian phone addicts from watching the screen then make them stop and use the mobile phone, therefore, reducing the occurrence rate of potential dangers.



---

*Source: 279846-2015-05-18.xml*