Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 265–269___________________________________________________________________________________________________________Fall Risk Reduction for the Elderlyby Using Mobile Robots Based on Deep Reinforcement Learning
Takaaki Namba*

Graduate School of Engineering, Nagoya University,
Furo-cho, Chikusa-ku, Nagoya City, 464-8603, Japan
Yoji Yamada

Graduate School of Engineering, Nagoya University,Furo-cho, Chikusa-ku, Nagoya City, 464-8603, Japan
E-mail: namba.takaaki@b.mbox.nagoya-u.ac.jp, \RML\DPDGD#PDHQDJR\DXDFMS
www.nagoya-u.ac.jp

Abstract
Slip-induced fall is one of the main factors causing serious fracture injuries among the elderly. In this paper, we
propose a fall risk reduction measures for the elderly, based on deep reinforcement learning using mobile assistantrobots. Our method involves online real-time risk analysis and risk reduction. The results suggest that our method isapplicable to the prevention of not only slip-induced fall, but also other factors related to falling and other cases ofaccidents.
Keywords: Safety, risk reduction, mobile robot, deep learning, reinforcement learning.

1. Introduction
In some countries, including Japan, there is an increasing
tendency in the number of accidents involving the elderly.
Slip-induced falls are one of the main factors causing
serious fracture injuries among the elderly, subsequently
leading to them being bed ridden. To reduce theincreasing fall accidents among the elderly, we propose
an approach using mobile assistant robots. We consider
safety in environments wherein humans and assist robots
coexist to enhance the fields of medical welfare and assist
living1-3.
This paper proposes deep reinforcement learning(hereinafter called ‘DQN’)-based fall risk reduction
measures by employing mobile assistant robots for theelderly. First, we collect preparatory data regarding pastFig. 1. Reinforcement learning for fall risk reduction ofthe elderly.

incidents and accidents as the input data to analyze fallrisks, and to assess the examples of risk reduction
measures. Second, we use a deep convolutional neuralnetwork4 to analyze the fall risks of the elderly. Third, weapply DQN5-6 to control mobile robots according to the*Panasonic Advanced Technology Development Co., Ltd, Nagoya Mitsui Building New Building 11F, 1-24-20 Meieki-minami, Nakamura-ku, Nagoya

City, 450-0003, Japan. namba.takaaki@jp.panasonic.com

Copyright © 2018, the Authors. Published by Atlantis Press.
This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
265

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 265–269___________________________________________________________________________________________________________results of risk analysis, and consequently reduce the slipinduced fall risks of the elderly.

Table 1. Definitions of designed parameters for deep
reinforcement learning.
DesignedparameterAction2. Data Preparation
We collect the images and movies, including informationof the location where the real incidents and accidents hadoccurred, and where similar hazards exist. In addition tothese data, we use open datasets, such as Caltech101
datasets. Moreover, we expand the data by 14 times using
data augmentation such as additional noise, rotation,zoom-in, zoom-out, foreground, and background. We set
128 labels as risk levels3 based on the risk classificationof the international standard ISO12100 for all collecteddata.

StateReward

3. Method for risk analysisExperiment

Risk analysis is carried out through methods such as
Fault Tree Analysis (FTA),7 Hazard and Operability
Study (HAZOP),8-9 and Failure Modes Effects andCriticality Analysis (FMECA).10 Conventional risk
analyses conducted manually or partially use aprobability calculation tool when employed inapplications such as system safety requirement analysis
or design. This type of analysis has the following
limitations: Failure in identifying that hazards may occur owing
to differences in the proficiency of the safety
management officers, that is, their experience andcapability,
 Risk analysis procedures are complex and require a
specific number of man-hours depending on thescale of the assessment.

Components
34 actions (32 directions (including avoid hazard) +1 stop + 1 remove hazard).
Current sensing data by a multi-spectrum camera on
the agent (e.g., the distance to the object by a stereo
camera and by a laser range finder),current position and posture of the agent,current goal position (toilet or hospital room),time (current step count), and previous action.
Reachability (reach goal), transfer efficiency(approach the goal; minimum steps, timeout, andinertia), and safety (change in the risk: remove
hazard, avoid hazard).
Previous state, previous action, current reward, andcurrent state.

The main output is the result of risk analysis and thelevel of risk as quantitative information. The risk ispresented using the following formula:
𝑅𝑖𝑠𝑘 = 𝑅(𝑆, 𝑃ℎ).

(1)

𝑃ℎ = 𝑃(𝐹, 𝑃𝑠, 𝐴).

(2)

Eq. (1) shows that the risk is a product of the severityof the harm (S) and the probability of the harm (Ph). Thisis one of the principal criterion of ISO1210012. Eq. (2)indicates that the probability of the harm (Ph) is a productof the following: the frequency of exposure to hazards(F), probability of occurrence of hazardous event (Ps),and possibility of avoiding or limiting the harm (A). We
classify the risk level in 128 categories. Furthermore,according to the result of risk analysis, we generate animage of the future by blending a representative
hazardous image per risk category with the current image.

Most analyses and assessments are carried out offline
based on prior information, and require time to beconducted; the results are not immediately available.
Therefore, the measures of risk reduction are often
delayed, and it becomes difficult to respond promptly andflexibly in continually changing situations.
We develop an online real-time risk analysis based on
the deep convolutional neural network. The input isenvironmental sensing data aimed at detecting Hazard
Elements (HEs), Initiating Mechanisms (IMs, triggers of
the accident), and Target and Threat (T/T) with reference
to Ericson’s Hazard Theory11. According to this theory,an accident (mishap) occurs when HE, IMs, and T/Tappear simultaneously at a specific time. In our study, thesensing data are mainly time-series images of theenvironment and distance information to the objects andparts of human body.

4. Method for risk reduction
In general, regarding the safety of machinery, risk
reduction is implemented by using methods such as
inherently safe design measures, safeguarding
implementation of complementary protective devices,and information for use.12-14 In addition, risk reduction
can be performed by learning from previous incidents
and accidents15, and this can further improve theenvironment; for example, removing hazardous elements,
avoiding hazardous situations, and ceasing the triggeringof incidents and accidents.

266

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 265–269___________________________________________________________________________________________________________In this paper, we propose online real-time risk
reduction measures based on DQN. Figure 1 portrays ourconcept. The inputs are the environmental data involving
the elderly individual’s outer and inner factors. The riskof the outer factors is estimated by a variety of sensing
data, for example, images and distance information
obtained from a stereo camera and a laser range finder.
The risk of the inner factors is estimated by the elderly
individual’s physical condition; for example, changes inthe breathing obtained from radio sensors on the mobile
robots, and stability measures calculated from the signalsof the accelerometers mounted in the individuals.
Another input is the risk level, which is the result of risk
analysis as one of the received rewards. The risk
reduction policy selects an action using a deep neuralnetwork. The output is one of the actions including therisk reduction measures.
Table 1 lists the definitions of the parameters thatmust be determined by humans for DQN. Eachcomponent of the parameters is normalized from 0 to ±1.00. This DQN is trained with the Q-learning algorithm,
5-6
by the epsilon-greedy strategy, and uses Adam16 forthe parameter optimization. To accelerate theconvergence rate of parameters, we apply experience
replay, and reduce epsilon according to the learning
status. Furthermore, to avoid over-fitting, we apply early
stopping, use the generated model for our other scenes,
and use dropout in the neural network.

Fig. 2 Simulation environment for experiments.

Fig. 3 A deep neural network model generating risk
reduction measures.

Figure 3 depicts a feasible example of the deep neuralnetwork model that estimates the Q-value, and decides
the risk reduction measures. We apply a deep fully
connected network with nine layers. The deep
convolutional neural network is not necessarily required
to be applied to the scenes depicted in Figure 2, because
we can determine the features of the environment by thecombination of the components of the input data, without
dimension reduction. The data of the input layercomposes a normalized state vector. A state parameter
includes the sensing data, robot’s position, posture, goalposition, and time information as environmental data.
The data of the output layer shows each estimated Qvalue for actions. Each activation function is a leaky
ReLU. The initial value of epsilon ε, which indicates theprobability of random action, is set as 0.2. The reduction
rate of epsilon is 0.00005. The discount rate of the future
reward is γ=0.9.
Figure 4 describes the process explaining how thereward changes during training on the simulation. An
increase in the reward indicates well-designed rewards
and risk reduction. We validate the effects of our5. Experiments
We performed experiments using a computer simulation.
Figure 2 illustrates the conditions of the environment andagents. Mainly four types of scenes are shuffled andexperimented upon. The hospital room and the toilet arelocated at random within a certain range. As a DQNbased agent, the red circle “R” is an autonomous mobile
robot that assists the elderly. The scenario is that agent“R” goes from the hospital room to the toilet at a certain
interval, in the hospital. The floor is partially wet aroundthe toilet at random spatiotemporally, and agent “R”
detects those area. The fan-shaped area shows the sensing
range of agent “R”. Agent “R” assists the prevention ofslip-induced falls for the elderly. Moreover, we compare
the DQN-based agent with conventional rule-based agent
according to the potential method.
6. Results and discussion

267

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 265–269___________________________________________________________________________________________________________more positive reward to the action with higher priority.
Finally, the input data of our deep risk reduction
network suggests that the features of our model include
the local map, global map, trajectory planning, andscreening for risk assessment.

Fig. 4. Change in reward (including risk reduction)
during training on the simulation. Two peaks indicate
the rewards when an agent reaches the toilet or hospitalroom, in other words, achieves goals in two episodes.

7. Conclusion
In this paper, we propose fall risk-reduction measures forthe elderly, by using mobile robots based on deep
reinforcement learning, and present its usability. Theresults suggest that our method is applicable to not only
fall prevention, but also to other factors influencing falland other cases of accidents.
References
1. T. Namba, and Y. Yamada, A study on an assistance bymobile robots in preventing the elderly from falling, inProc. Mechanical Engineering Congress, 2015 Japan(MECJ-15), J1630205, (in Japanese).
2. T. Namba, and Y. Yamada, A study on an assistance bymobile robots in preventing the elderly from falling (DeepConvolutional Neural Network for Fall Risk Analysis), inProc. Mechanical Engineering Congress, 2016 Japan(MECJ-16), J1640304, (in Japanese).
3. T. Namba, and Y. Yamada, Risk Analysis Method
Preventing the Elderly from Falling, in Proc. 22nd
Robotics Symposia, 3A1, (2017) (in Japanese)
4. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet
classification with deep convolutional neural networks, In
Advances in Neural Information Processing Systems 25,(2012), pp.1106–1114.
5. C. J. Watkins and P. Dayan, Q-learning, In Machine
Learning, vol. 8, (1992), pp. 279-292.
6. V. Mnih et al. Playing Atari with Deep Reinforcement
Learning". In:CoRR abs/1312.5602,(2013).
7. C. Ericson, Fault tree analysis – a history, in:17th Int.
System Safety Conference. Unionville, VA, (1999), pp. 1-9.
8. F. I Khan, and S. A Abbasi, Techniques andmethodologies for risk analysis in chemical processindustries, J. Loss Prevention in the Process Industries,Vol.11, Issue 4, (July 1998), pp. 261–277
9. J. Dunjóa, V. Fthenakis, J. A. Vílchez, and J. Arnaldos,Hazard and operability (HAZOP) analysis, A literature
review, J. Hazardous Materials, Vol. 173, Issues 1–3, (15January 2010), pp. 19–32
10. MIL-STD-1629A, Military Standard, procedures for
performing a Failure Mode, Effects and Criticality
Analysis, (Department of Defense, Washington, DC,1980).
11. C. A. Ericson II, Hazard Analysis Techniques for System
Safety, Second Edition, (John Wiley & Sons, Inc., 2016).
12. ISO12100, Safety of machinery – General principles forFig. 5. Relationship between episodes and transfer
efficiency as a learning curve.

(a)(b)Fig. 6. Trajectories of rule-based (a) and DQN-based (b) agentsapproach through the variations in the reward graph andconvergence of the loss function. We noted that the deep
reinforcement learning-based agent not only surpasses
the rule-based agent but also automatically reduces slipinduced fall risks.
Figure 5 shows the relationship between the numberof goal steps of an agent and the number of episodes. Itindicates enhanced convergence of steps and stability ofactions by the reinforcement learning.
Figure 6 discloses the result of comparison of the twomethods. The agent in Figure 6(a) took 489 steps with thepotential method, whereas the agent in Figure 6(b) took
113 steps with DQN to reach the goal. The results
indicate that the transfer efficiency of the DQN agent isequal or better than that of the rule-based agent.
Furthermore, the highest priority of risk reduction
measures is to “remove hazard,” followed in order by“avoid hazard”, and “post a warning.” We verified thatthese policy decisions were able to control actions byadjusting the safety reward design, such as assigning a268

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 265–269___________________________________________________________________________________________________________design – Risk assessment and risk reduction, (2010).
13. IEC61508-ed2, Functional safety of electrical /electronic /programmable electronic safety-related systems, 2010-04.
14. ISO13482, Robots and robotic devices -- Safety
requirements for personal care robots, (2014).
15. D. L. Cooke, Learning from Incidents, System Dynamics
Society, (2013)
16. D. P. Kingma and J. L. Ba., Adam: A Method for
Stochastic Optimization, 3rd Int. Conf. for Learning
Representations (ICLR), San Diego, (2015)269