Entire-Body Capturing System using a Monitoring Robot in a Context of Passing
a PedestrianKazuyuki Hayashide a , Masaki Takahashi ba School of Science for Open and Environmental Systems, Graduate School of Science and Technology, Keio University, 3-14-1 Hiyoshi,

Kohoku-ku, Yokohama, 223-8522, Kanagawa, Japanb Department of System Design Engineering, Faculty of Science and Technology, Keio University, 3-14-1 Hiyoshi,

Kohoku-ku, Yokohama, 223-8522, Kanagawa, Japan

1. Introduction

2. Overview of the proposed system
2.1. Assumed environment

In a society struggling with an aging population ordealing with a pandemic, a shortage of hospital staﬀ
will place an increasing burden on medical staﬀ and reduce the quality of life for patients[1]. To address theseconcerns, automation of transportation and monitoring
operations is gaining traction[2]. Robots are replacing
transportation services not only in hospitals but also ina wide range of public spaces. Building on this automation trend, it is possible to leverage mobile robots, which
are already being introduced in various settings, for patient monitoring tasks as well.
In this paper, monitoring is deﬁned as taking place ina corridor of a hospital ward or a nursing home. There
are two reasons for this. First, in hospital corridors withmany blind corners, it is diﬃcult for nurses to constantly
monitor for wandering or falling patients[3]. Second,
physical therapists need to evaluate the gaits of patientsin everyday life, not in a rehabilitation room[4][5][6].
For these reasons, the monitoring system is used for gaitmeasurement or monitoring to detect patients who fallor wander around the ward.
This study proposes a mobile robotic system thatsafely captures full-body images of passing pedestrians,
with a view to application for gait measurement or ward
monitoring. Also, by exhibiting the motion of the robot
and measurement data, the future prospect of the proposed method is discussed.

∗ Corresponding author

Email addresses: kazuyuki.hayashide@keio.jp (Kazuyuki
Hayashide ), takahashi@sd.keio.ac.jp (Masaki Takahashi )Preprint submitted to International Conference on Robotics and Automation

In this study, a straight corridor with a width of 2.4
m and a length of 15 m is assumed, with no obstaclesor other pedestrians, as shown in Fig. 1. The width ofthe corridor was deﬁned so that the robot could capture
a full-body image of a pedestrian passing through, even
if the pedestrian walked in the middle of the corridor.
Therefore, the proposed system can be used in narrower
corridors by using a sensor with a wider angle.
The pedestrian was assumed to be less than 2 m talland to walk at a speed of 0.3 m/s to 1.2 m/s. In thisstudy, the applicability of the proposed method was also
conﬁrmed by experiments with pedestrians who bent at
the waist or wore a paralysis experience kit.
The robot that was employed in this study was theTOYOTA HSR-D (height: 1.1 m, radius: 0.3 m, maximum speed: 0.4 m/s) with a ZED-2i stereo camera(angle of view: 110° × 70°, range: 0.2 m to 8.0 m)
mounted on its head in a vertical orientation. Motion
planning and image processing were performed by awirelessly connected computational server(Intel Corei7-11700 @2.5GHz).
2.2. Assumed task
There are three requirements for the robot’s behavior.
First, the robot measures the gait of the pedestrianby walking past each other, expecting that the robot
can measure his or her gait from multiple viewpoints.
Second, the robot optimizes its motion to capture thefull-body images of the pedestrian. This is because observing the entire body of the pedestrian is essential forphysical therapists to diagnose the characteristics of hisor her gait. Finally, the robot ensures the safety of its
movement, a crucial requirement to avoid frightening
April 25, 2024

or colliding with the pedestrian in case of his or her abnormal movements or sudden falls. Therefore, the evaluation of the movement of the robot includes determining when to yield and how to behave when passing eachother.

value when the pedestrian is in the center of the viewing angle. By deﬁning the evaluation function (5) as thesum of Fr Fφ over all times, the robot is able to pass eachother while keeping the entire body of the pedestrian in
the center of the measurable range. Equation (6) represents the dynamics of the robot. Equations (7) and (8)denote the state quantity and control input constraints,respectively. The maximum norm of velocity vector isconstrained as deﬁned in (9). Equation (10) outlines thecollision avoidance constraint, and (11) constrains thestarting position and target positions of the robot.
The robot follows the trajectory planned by the motion planner using the model predictive controller. Even
after the robot starts moving, the pedestrian’s positionand velocity are constantly observed and used for replanning of the trajectory. The update frequency in thisresearch was about 3Hz, depending on the computational speed of the motion planning and image processing. During the execution of the motion, the positionand velocity of the person are re-measured and the pathis updated at a rate of 3 Hz. This means that the path isupdated every 0.4 m for a pedestrian walking at 1.2 m/s.
This is thought to be eﬀective in improving the safety ofthe motion.

2.3. Structure of the proposed system
The system architecture diagram is shown in Fig. 2.
The motion planning part is based on the following
formulation proposed in [7] and [8]. The decision variables are a time series of accelerations with respect tothe robot’s position and neck angle, deﬁned as in (1)and (2).
[]TuR = a x , ay , αang
(1)
UR = [uR [k0 ] , . . . , uR [kF ]](2)

The time series of the state quantity consisting of therobot’s position and velocity is deﬁned as in (3) and (4).
[]zR = xR , yR , θR , ẋR , ẏR , ωR T
(3)
Z R = [zR [k0 ] , . . . , zR [kF ]](4)

Similarly, the time series of the state quantity of the person is deﬁned as Z H . The optimization problem is deﬁned as follows:

3. Experiments.t. zR [k + 1] = fdynamics (zR [k] , uR [k])(6)This experiment was conducted to clarify features ofthe movements of the robot, ﬁndings derived from thedata obtained, and the issues to be addressed in order tomake the system more practical for clinical use.
The experiment was approved by the Bioethics Committee of the Faculty of Science and Technology, Keio
University. (Approval No. 2023-030)

zR low ≤ zR [k] ≤ zR upp(7)

3.1. Conditionsulow ≤ u [k] ≤ uupp

(8)∥vR [k]∥ ≤ vR max(9)r [k] ≥ r personal(10)[]zR k0,F = zR0,F(11)max J (ZR , UR ) =kF∑Fr (zR [k] , zH [k]) Fφ (zR [k] , zH [k])

k=k0

(5)

The experiment was conducted under the conditions
described in Section 2. 36 trials were obtained from 9
healthy subjects (5 males and 4 females, 23.3±0.8 years
old, height 1.68 ± 0.11 m, maximum height was 1.88m), excluding failures such as cases in which the robot
lost sight of the pedestrian. The subjects were instructedto walk (i) in a normal posture, (ii) with a bent waist,and (iii) with the paralysis experience kit attached. The
subjects were instructed to walk slowly since the main
target of this system is a person who has a walking disability. Although this is omitted from this paper, it has
been conﬁrmed that the method is applicable to a person
walking at a speed of 1.2 m/s in the simulation.

Note that r is the relative distance between the person
and the robot and φ is the angle of the person from therobot. The term Fr and Fφ in (5) takes its maximum
2

Fig. 1: Assumed environment

Fig. 2: System architecture

3.2. Results
3.2.1. Motion of the robot
An example of the situation during the experiment isshown in Fig. 3. The robot ﬁrst detected the pedestrian
coming from the other side and gave way to him. Afterthe pedestrian passed by the robot, the robot resumed its
path to the target point.
The motion of the robot exhibited three characteristics. First, the robot made a move to yield the rightof-way early. The robot maintained a distance greaterthan the collision avoidance constraint in order to capture full-body images of the pedestrian. As a result, therobot was able to give way to pedestrians earlier andthey were able to pass smoothly. Second, the robot almost stopped at the edge of the road at the moment passing each other. This was achieved by trying to capture
the pedestrian for as long as possible, which is one of theadvantages of adding measurement to the robot requirements. Third, the robot stared at pedestrians passing by.
This is thought to be due to the sensor’s limited angle ofview.
The merits and demerits of these three features needFig. 3: Motion of the robot and pedestrian3Fig. 4: Data obtained in a trial of scenario (ii)Fig. 5: Data obtained in a trial of scenario (iii)

to be discussed, because it is necessary to investigate
what impressions medical personnel and patients haveabout the movements of the robot. Some people may
ﬁnd it reassuring to know that the robot is acknowledging their presence through its motions, while others may
ﬁnd it uncomfortable because they may feel that they arebeing watched by the robot.

reﬂect the characteristics of these gaits.
As for future work, it is necessary to improve the accuracy of pose estimation. In the proposed scenario, it isinevitable that half of the pedestrian’s body is occluded,
which is one of the causes of reducing the accuracy.
Also, the vibration noise of the robot should be one of
the main factors for this problem. Therefore, estimation of hidden joint positions and reduction of vibrationnoise should be addressed.
Once the problem is solved, the proposed system
could be useful in that it can automatically collect thegaits of admitted patients. Furthermore, taking advantage of the fact that the sensor itself moves, it may beable to diagnose the athletic or cognitive abilities of thesubject by having the robot perform some action on himor her.

3.2.2. Obtained data
The robot driven by the proposed method was able tocapture the passing pedestrian through an RGB-D sensor on its head. The measurement data exhibited threemain characteristics.
First, the robot can record gaits of long distances. As
shown in Table 1, it was able to measure the gait of morethan 5 m on average. By using time stamps and walking distance, the average walking speed was obtained.
In clinical practice, the calculation of walking speed ina similar way is also widely practiced as 5-Meter Walk
Test[9], which is one of the most standard methods todiagnose or predict the health status of subjects. Therefore, the proposed system could be used as an automatic
testing tool in the future.
Second, as shown in Figs. 4 and 5, the robot was able
to capture the entire body of the pedestrian from every
angle. It captures front, side and back of the pedestrian in about 30 seconds, so the data contains a variety
of information such as hunchback, stability, face direction, limb movement, and so on. Although the eﬀects
of noise remain, the pose estimation of Detectron2[10]and center-of-mass trajectories shown in the ﬁgures also

4. Conclusion
This paper proposes a novel robotic system that captures images of a pedestrian passing each other in a corridor with a view to applying it to gait measurement
and monitoring. The robot ensures the safety of its motion by yielding earlier and stopping when it detects thepedestrian, and resuming its path after the pedestrianpasses. The system appears promising as a tool for gaitmeasurement in daily life, as it can capture the entire
body of the pedestrian over long distances from any angle. By improving the accuracy of pose estimation andextending its overall capabilities, the proposed system is4

Table 1: Results of the experiment(i) walk normally (n = 17)(ii) walk with back bent (n = 8)(iii) walk with paralysis experience kit (n = 6)
expected to increase its practicality for automating someof the monitoring tasks in hospitals or nursing homes.

Time [s]32.7 ± 4.2
34.0 ± 2.5
32.2 ± 8.3

Distance [m]9.67 ± 2.89
8.84 ± 1.40
5.55 ± 2.74

Speed [m/s]0.30 ± 0.10
0.26 ± 0.04
0.17 ± 0.06

[9] T. Van Criekinge, C. Heremans, J. Burridge, J. E. Deutsch,
U. Hammerbeck, K. Hollands, S. Karthikbabu, J. Mehrholz,
J. L. Moore, N. M. Salbach, et al., Standardized measurementof balance and mobility post-stroke: Consensus-based core recommendations from the third stroke recovery and rehabilitation roundtable, Neurorehabilitation and neural repair (2023)
15459683231209154.
[10] Y. Wu, A. Kirillov, F. Massa, W. Lo, R. Girshick, Detectron2,
https://github.com/facebookresearch/detectron2(2019).

5. Acknowledgement
This work was supported by Core Research for Evolutional Science and Technology (CREST) of the Japan
Science and Technology Agency (JST) [grant number
JPMJCR19A1]. We would also like to thank Editage(www.editage.com) for English language editing.
References[1] K. Minato, J. Shin, S. Kunisawa, K. Fushimi, Y. Imanaka, The
total number of patients with any of four major fragility fractures decreased during the ﬁrst wave of the covid-19 epidemic
in japan, commencing before the state of emergency declaration,
which was not as enforceable as lockdown, Archives of Osteoporosis 18 (1) (2023) 86.
[2] J. Holland, L. Kingston, C. McCarthy, E. Armstrong, P. O
Dwyer, F. Merz, M. McConnell, Service robots in the healthcare sector, Robotics 10 (1) (2021) 47.
[3] L. Magnuszewski, A. Wojszel, A. Kasiukiewicz, Z. B. Wojszel,
Falls at the geriatric hospital ward in the context of risk factors offalling detected in a comprehensive geriatric assessment, International Journal of Environmental Research and Public Health
19 (17) (2022) 10789.
[4] A. M. Barthuly, R. W. Bohannon, W. Gorack, Gait speed is a
responsive measure of physical performance for patients undergoing short-term rehabilitation, Gait & posture 36 (1) (2012)
61–64.
[5] C. Zampieri, A. Salarian, P. Carlson-Kuhta, J. G. Nutt, F. B. Horak, Assessing mobility at home in people with early parkinsons disease using an instrumented timed up and go test, Parkinsonism & related disorders 17 (4) (2011) 277–280.
[6] V. V. Shah, J. McNames, M. Mancini, P. Carlson-Kuhta, R. I.
Spain, J. G. Nutt, M. El-Gohary, C. Curtze, F. B. Horak, Laboratory versus daily life gait characteristics in patients with multiple
sclerosis, parkinson s disease, and matched controls, Journal ofneuroengineering and rehabilitation 17 (2020) 1–12.
[7] K. Hayashide, M. Takahashi, Generation of walking measurement motion for mobile robot passing each other based on nonlinear programming method, in: The Proceedings of JSME
annual Conference on Robotics and Mechatronics (Robomec)
2023, The Japan Society of Mechanical Engineers, 2023, pp.
2P2–H03.
[8] K. Hayashide, M. Takahashi, Nonlinear programming-based
robot motion planning for gait measurement when passing
pedestrians, in: Intelligent Autonomous Systems 18: Proceedings of the 18th International Conference IAS-18, Springer, toappear.

5