SUGISAKAMASANORI

Journal of Robotics, Networking and Artificial Life
Vol. 9(4); March (2023), pp. 349–357
ISSN (Online):2352-6386; ISSN (Print):2405-9021
https://alife-robotics.org/jrnal.html

Daigo Katayama1, Kazuo Ishii1, Shinsuke Yasukawa1, Yuya Nishida1, Satoshi Nakadomari2,
Koichi Wada2, Akane Befu2, Chikako Yamada2
1

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu-shi, Fukuoka,
808-0196, Japan
2
NEXT VISION, Kobe Eye Center 2F, 2-1-8, Minatojima-Minamimachi, Chuo-ku, Kobe-shi, Hyogo, 650-0047, Japan

Received 25 November 2021
Accepted 04 May 2023

Electronic travel aids
Negative obstacle detection
Point cloud
LiDAR sensor

We have developed a fall risk alert system using a smartphone with a LiDAR sensor, Apple’s
iPhone 12 Pro, as an electronic travel aid (ETA) for visually impaired to reduce fall risks fromthe edge of station platform and the stairs. The proposed system generates the height grid map
generated from 3D point clouds and posture from the smartphone, and classify the walking area
to four areas, walkable, positive obstacle, negative obstacle and occlusion areas. In this paper, we
describe the algorithm and the experimental results of the environmental recognition system forthe fall risk alert. In the experiments, the accuracy, stability, and time delay are evaluated in approaching a fall risk area, such as stairs. The results show the smartphone has the possibility to
be the ETA to support the visually impaired and reduce the fall risks.
© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

on ETAs which detect and emit a warning of entering fallrisk areas, such as stairs and gaps. However, the primary
purpose of commercially available ETAs such as
Miniguide [4], UltraCane [5], WeWALK [6], Sunu band[7], BuzzClip [8] are positive obstacle detection (walls,
pillars, etc.) and warn approaching obstacles. Few devices device have been commercialized to make a warningon approaching of fall risk areas. Due to the developmentof information technology and computers, smartphones
with high computational power and sensors are available
and widespread recently. Smartphones have the big possibility to be the supporting tools for the visually impaired as the smartphones have necessary functions suchas voice recognition, image processing, GPS positioning
and navigation, internet services, LiDAR and so on.
In this research, an ETA, whose main component is asmartphone with LiDAR sensor, has been proposed andevaluated. The ETA obtains the distance data of surroundings using the LiDAR sensor and estimate the fall

1. Introduction
Fall accidents on station platforms are one of severe
problems for the visually impaired during walk alone.
According to statistics data from the Ministry of Land,
Infrastructure, Transport, and Tourism in Japan [1], theaverage of approximately 76 fall accidents occurs every
year. One of factors of the fall accidents is the difficulty
for the visually impaired to recognize road conditions
further away than detecting with a cane (until approx. 1
m ahead), and they don’t have enough time to find andavoid obstacles with their walking speed [2]. To reduce
the fall accidents, the visually impaired needs to identify
areas likely to fall down including further area than notified area with a cane. To solve this problem, the support
devices called Electronic Travel Aids (ETAs) have beendeveloped, which inform alert using ultrasonic or infrared sensors to the visually impaired in non-visual information formats such as sound and vibration [3]. In recent
years, a lot of research and development have been done

Corresponding author’s E-mail: katayama.daigo696@mail.kyutech.jp, ishii@brain.kyutech.ac.jp, s-yasukawa@brain.kyutech.ac.jp,
y-nishida@brain.kyutech.ac.jp, satoshin1@mac.com, wakichi@nextvision.or.jp, befu@nextvision.or.jp, yamada@nextvision.or.jp
URL: www.lsse.kyutech.ac.jp349

risk of walking area, and labels the risk level. The algorithm of ETA is proposed and evaluated through experiments.

3. Fall Risk Estimation Method
3.1. Basic Concept
We have developed an ETA using a smartphone with
LiDAR to inform the fall risk areas ahead of the visually
impaired where he/she cannot recognize with a cane asshown in Fig. 1, and offers safety checks in advance toreach those area if used with the cane. We assume that
the ETA (iPhone) is attached on the user's chest with atilt downwards to measure the distances of walking surface. The ETA estimates the surface height and comparewith current ground level, and the height is not within thesafety region, ETA vibrates and the connected smart
watch also make alarms and vibrations. As shown in Fig.
2, the ETA categorizes the front surface to four areas,
positive obstacle area (blue), walkable area (green), negative obstacle area (orange) and occlusion area (red)based on the surface height R𝑧R in coordinate ΣR whose
𝑧-axis is in the gravity direction. The coordinate ΣC is the
local coordinate corresponding to the ETA. Comparing
the surface height with that of ETA 𝐻chest , system determines the category of front surface. Here, 𝑋min is the
distance to the nearest line of mapping area of therecognition space to the ground surface, 𝑋max is that of
the farest line, 𝑥edge is that of the nearest mesh of
occlusion or negative obstacle areas. 𝑋th1 , 𝑋th2 , 𝑋th3 aredistance thresholds for the fall risk estimation.

2. Related Works
2.1. Electronic Travel Aids to Make Warning ofFall Risk
Most of commercially available ETAs alert their users
for the fall risk based on the distances to the walking floor.
The distance to the floor in the earth coordinate is calculated using 3D posture information from the IMU sensor,
and make a warning if the distance to the floor exceeds
the threshold value. The ETAs have the appearance of awearable device, such as a belt type [9], [10], a cart type[11], a cane-integrated type [12], [13], [14] and a headgear type [15], [16]. Also, negative obstacle detection algorithms for the ETAs have been developed: the edge detection from stereo vision using seed growing and dynamic programming [17], the staircase detection using
image processing and support vector machine (SVM)from RGB and depth (RGB-D) images [18] and theground and hole detection from disparity image [19].
Many ETAs and algorithms have been developed, however, few devices have achieved the commercialization
process.

3.2. Algorithm of Fall Risk Estimation
The flow of the algorithm for fall risk estimation isshown in Fig. 3, and the area 𝑠risk of the fall risk area at
the recognition space. The details of the process in eachstep are shown below.

2.2. Smartphone-based Electronic Travel Aids
The smartphones consist of computers, cameras, IMU,
GPS and so on, and also many advanced applications
have been implemented on them for the visually impaired.
For examples, Seeing AI [20] (Microsoft Corp.) and Envision AI [21] (Envision Technologies B.V.) were already available as commercial applications for object
recognition and notification of recognition results. Highend models of Apple's smartphones after iPhone 12 Pro
have a LiDAR sensor [22] which enables iPhone to acquire high accuracy and wide range of distance information to the surroundings and is often used for navigation by combining the distance information, IMU data,
heading, height and GPS position. For examples, those
applications offer the functions such as the guidance on
aisles [23], recognition to assist in following a line [24],positive obstacle detection and intersection detection forindoor walking assistance [25].

Fig. 1 Concept of electronic travel aid

350

(1) Generate Point CloudA 3D point C𝒑 = [ C𝑥C , C𝑦C , C𝑧C ]𝑇 ∈ ΣC (Camera coordinate) is calculated from the depth 𝑑𝑢𝑣 [26] from the
LiDAR sensor and P𝑢, P𝑣 , which are in the coordinates
on the depth image plane ΣP , 𝑐𝑢 and 𝑐𝑣 are the center position on ΣP , 𝑓𝑢 and 𝑓𝑣 are the focus parameters, as
shown in Eq. (1).
P𝑢 − 𝑐𝑢
C
𝑥C
𝑓𝑢
C
(1)
𝒑 = [ C𝑦C ] = P𝑣 − 𝑐𝑣 𝑑𝑢𝑣
C
𝑧C
𝑓𝑣
[ 1 ]
The point C𝒑 is transformed from ΣC to ΣR (Recognition space coordinate). R𝒑 = [ R𝑥R , R𝑦R , R𝑧R ]𝑇 ∈ ΣR byusing gravity direction from IMU. In the coordinate′′transformation, the quaternion C 𝑞̃ , R𝑞̃ , C 𝑞̃ ∗ and R𝑞̃ ∗ .
C′𝑞̃ are used for the calculation of the coordinate transformation. R𝑞̃ is obtained from the smartphone's 3D pos′ture on ΣR . C 𝑞̃ and R𝑞̃ are defined as shown in Eq. (2)and Eq. (3), respectively.
C′

′

′

′

′

′

C′

𝑞̃ = C 𝑞𝑤 + C 𝑞𝑥 𝒊′ + C 𝑞𝑦 𝒋′ + C 𝑞𝑧 𝒌′ = C 𝑞𝑤 + 𝒒 (2)R
𝑞̃ = R𝑞𝑤 + R𝑞𝑥 𝒊 + R𝑞𝑦 𝒋 + R𝑞𝑧 𝒌 = R𝑞𝑤 + R𝒒 (3)
C′

′

′

′

𝑞𝑤 , C 𝑞𝑥 , C 𝑞𝑦 and C 𝑞𝑧 are scalar values on ΣC′whose unit vectors are 𝒊′ , 𝒋′ , and 𝒌′ , and R𝑞𝑤 , R𝑞𝑥 ,R
𝑞𝑦 and R𝑞𝑧 are on ΣR whose unit vectors are 𝒊, 𝒋, and
𝒌.
′
C′ ∗
𝑞̃ and R𝑞̃ ∗ are the conjugate quaternions of C 𝑞̃and R𝑞̃, respectively. The coordinate transformed point′′′C′
𝒑 = [ C 𝑥C′ , C 𝑦C′ , C 𝑧C′ ]𝑇 ∈ ΣC′ is obtained from Eq.
(4), and R𝒑 is obtained from Eq. (5).
′
′
C′
(4)
𝒑 = C 𝑞̃ ∗ C𝒑 C 𝑞̃
′
R
R ∗ C
R
(5)
𝒑 = 𝑞̃ 𝒑 𝑞̃Fig. 2 Classification of front ground into four areas(2) Generate Grid Map
Using the point R𝒑, the grid map is generated to reduce the noise in the points and simplify the calculation.
R
The height ̅̅̅̅̅
𝑧R in each grid is calculated as the averageof 𝑧-axis position in the points R𝒑𝑚𝑛 in each grid, defined in Eq. (6). 𝑀 and 𝑁 are the lengths of each axison the grid map, 𝑚 = 0, 1, … , 𝑀 − 1 and 𝑛 =0, 1, … , 𝑁 − 1 are position on ΣM (grid map coordinate),
𝑋min and 𝑌min are minimum value of R𝑥R and R𝑦R on
grid map, respectively. 𝐷max is maximum depth on thedepth image.

Fig. 3 Flowchart of fall risk estimation

351

R

𝒑𝑚𝑛 ∈
𝑚∆𝑋+𝑋min ≤ R𝑥R < (𝑚 + 1)∆𝑋+𝑋min ,{ R𝒑 | 𝑛∆𝑌+𝑌min ≤ R𝑦R < (𝑛 + 1)∆𝑌+𝑌min , } (6)−𝐷max ≤ R𝑧R ≤ 𝐷maxR̅̅̅̅̅𝑧 is calculated as shown in Eq. (7). 𝑛 is the numberR𝑝of R𝒑𝑚𝑛 .
R̅̅̅̅̅𝑧R = (1/𝑛𝑝 ) ∙

∑

R

𝑧R

(7)

R𝑧 ∈ R 𝒑
R𝑚𝑛(3) Classify Grid Map

Fig. 4 Definition of fall risk level

The system classifies each cell in the grid map by
thresholds areas into four labels, walkable area(𝐴1 ), positive obstacle area(𝐴2 ), negative obstacle area(𝐴3 ), andocclusion area(𝐴4 ). Equation (8) provides classificationR𝑅
of 𝐴(𝑛𝑝 , ̅̅̅̅̅
𝑧R ) using the parameters, 𝑛𝑝 and ̅̅̅̅̅
𝑧R .
R
̅̅̅̅̅
𝐴(𝑛 , 𝑧 ) =𝑝

3.3. Algorithm Implementation into Smartphone
The fall risk estimation algorithm describe in the previous section was implemented as the iOS app for travelaids for visually impaired. The GUI of the application isshown in Fig. 5. The grid map preview in the center is the
top view with the bottom side as the user side. 𝐻chest and
𝐻w settings and start/end of data recording can be operated in the control panel.
The simulator was created to consider the appropriate
parameters in this algorithm using MATLAB R2022a.
The simulation results in the top view of the grid map,color-coded by area, as shown in Fig. 6.

R𝐿w ̅̅̅̅̅
𝐿w≤ R𝑧R ≤ 𝐻chest + )
2
2
𝐿w
R
̅̅̅̅̅
𝐴2 (if 𝑛𝑝 ≠ 0, 𝑧R > 𝐻chest + )
(8)
2
𝐿w
R
̅̅̅̅̅
𝐴3 (if 𝑛𝑝 ≠ 0, 𝑧R < 𝐻chest − )2{𝐴4 (if 𝑛𝑝 = 0)𝐴1 (if 𝑛𝑝 ≠ 0, 𝐻chest −3.4. Experiments at Fall Risk Area

(4) Estimate Fall Risk

Two kinds of experiments to approach a fall risk area
were conducted to compare the transition of labeling bychanging the approach angles into the fall risk area, theaccuracy of the minimum distance estimation, and the responsiveness of the notification as follows.

The fall risk is estimated, the classified cells along 𝑥axis direction in the grid map, and the edge position oneach row of the grid map 𝑚edge is defined as the edgeRcell where label of 𝐴(𝑛 , ̅̅̅̅̅
𝑧 ) changes from walkable𝑝R

area (𝐴1 ) to fall risk area (𝐴3 or 𝐴4 ). The minimum distance from the edge 𝑥edge is calculated by Eq. (9) to therow with the minimum number of cells to the edge. ∆𝑋is the grid width.
(9)𝑥edge = 𝑋min + ∆𝑋 ∙ min(𝑚edge )Next step, this system calculates the size of the fall risk
area 𝑠risk by Eq. (10). 𝑛risk is the number of cells classified into Occlusion area or Negative obstacle area. 𝑁grid
is total number of cells.
(10)𝑠risk = 𝑛risk / 𝑁grid
As the final step, the fall risk level 𝐹𝑅 is calculated by𝑥edge and 𝑠risk as shown in Fig. 4. The level is classified
into 4 levels (0,1,2,3) as shown below. 𝑋th1 and 𝑋th2 , and𝑋th3 are distance thresholds (𝑋th1 > 𝑋th2 > 𝑋th3 ). 𝑆th isarea threshold of fall risk area.

(1) Procedure
The task of the experiments is that the subject with thedevice on the chest walks straight to the stairs, as shownin Fig. 7. The approach angle 𝜃𝑎 are (i) 90[deg] and (ii)45[deg]. Here, subject was a sighted person, and the experiment was conducted without a blindfold for safety
reasons.
Condition (i) assumed that user is getting on the train,
and condition (ii) is moving along the platform and approaching the edge of the platform. Also, the length 𝑙0
from subject to fall risk area was 3[m]. The collected data
during the experiment were depth images, attitudes, andself-position from smartphone.
The smartphone was fixed to the front of the subject's
chest with a harness-type fixture. The smartphone wasalso tilted downward 30[deg].

352

This experiment was done on the outdoor at the Wakamatsu campus of Kyushu Institute of Technology. As
shown in Fig. 8, the space has wide stairs with the heightof step 0.14[m] and the length 0.34[m].

(3) Results
The minimum distance, fall risk area, and the fall risk
level for the approach angle of 90[deg] is shown in Fig.
9 and for an approach angle of 45[deg] in Fig. 10. In allcases, the fall risk level increased in steps as one approached the fall risk area. Figure 11 shows the results ofthe grid map transition. At first, the area is green to showsafety area, and as going to stairs, the red (occlusion) andorange (negative obstacle) areas appear in front of thesubject. In the case of 45[deg] approach, the orange area
appears from right side of subject as we expected.

(2) Analysis Method
The accuracy, stability, and delay of the fall risk estimation are evaluated by using the data recorded in thisexperiment to calculate the fall risk using the algorithm
proposed here and comparing the fall risk at each timewith each parameter used in the calculation. In the analysis, 𝐻chest = 1.32[m], 𝑋th1 = 2.35[m], 𝑋th2 =1.85[m],𝑋th3 = 1.35[m], 𝑆th = 0.5, 𝑌width = 1.00[m], and 𝐻w =0.10[m]. The grid width ∆𝑋 and ∆𝑌 were 0.05[m] in 𝑥
and 𝑦-axis directions.

4. Discussions
4.1. Discussion on Approach Angle
The results of approaches with the angles of 90[deg]and 45[deg] show that the trends are similar except forthe span of the level increase. This result indicates thepossibility to be robust system on approaching angles.
However, the result shows the difference in the timing to

Fig. 5 Overview of App GUI

Fig. 7 Experiment conditions

Fig. 6 Examples of grid map simulations

Fig. 8 Experiment site with stairs353

change to the maximum fall risk level. In the case of
90[deg], the time duration of the fall risk level changes
from 2 to 3 is 0.02〜0.20[s]. The fall risk area increases
at a faster rate than the case of 45[deg] approach. Also,
focusing on 𝑥edge and 𝑠risk in Fig. 6, the example indicates that 𝑥edge becomes shorter (2.00[m] to 1.40[m]) as𝜃𝑎 decreases with the same distance from the fall riskarea. For maximum fall risk estimation, 𝑠risk may be
dominant when 𝜃𝑎 increases and 𝑥edge may be dominant
when 𝜃𝑎 decreases. Therefore, 𝑋th1 to 𝑋th3 and 𝑆th need
to be set with considering the approaching angle 𝜃𝑎 .
4.2. Discussion on Fall Risk Estimation
We discuss the fall risk estimation algorithm, stability,and delay, considering the case of visually impaired usersof this system based on the analysis result.
(1) Accuracy of Fall Risk Estimation
In the experiment, the risk level had changed as we
expected. However, the variation of walking
environment should be considered into the set ofparameters for the fall risk estimation. The experiments
were conducted in an environment with relatively small
gaps around stairs, so that we should conduct a lot ofexperiments with various gaps, such as at the edge of astation platform. Then it needs to carry out experimentswith visually impaired subjects to evaluate whether thisfall risk level is adequately communicated to the visually
impaired and notification of this level will help improvesafety confirmation techniques.

Fig. 9 Analysis results (𝜽𝒂 =90[deg])(2) Stability of Fall Risk Estimation
The fall risk estimation results are stable except aroundthe minimum distance threshold based on the analysis results of this experiment. This result indicates that the fallrisk notification is stable for situations other than near thethreshold of the minimum distance. The algorithm will
reduce the confusion of the visually impaired by notification. A hysteresis addition to the threshold would improve stability for fall risk estimation around the minimum distance threshold. Consider setting the hysteresis
range based on walking speed, grid size.
(3) Delay of Fall Risk Estimation

Fig. 10 Analysis results (𝜽𝒂 =45[deg])In the analysis result, we found that the fall risk level
increased when the threshold was exceeded for each of
the parameters. the results indicate small delay in the algorithm concerning fall risk estimation.
Since we are considering using the smartphone’s vibration function in the actual notification, the design ofthe notification must consider the delay that would result
from using these functions.

5. Conclusion
In this paper, we proposed a fall risk estimation systemfor visually impaired using distance information from LiDAR sensor on the smartphone and evaluated through354

2. M. Tauchi, T. Ohkura, “A Present Situation and Problemsof Independent Travel of Blind and Visual Impaired”,
Journal of the Society of Instrument and Control Engineers,
vol. 34, no. 2, pp. 140-146, 1995. (In Japanese)
3. T. Ohkura et al., “Science of Orientation and Mobility of
Persons with Visual Impairment -Toward Safe and Reliable Independent Travel-”, CORONA PUBLISHING CO.,LTD., 2014. (In Japanese)
4. GDP Research, “The Miniguide mobility aid”,
http://www.gdp-research.com.au/minig_1.htm. (Accessed
22 Apr. 2022)
5. Sound Foresight Technology Ltd., “About the UltraCane”,
https://www.ultracane.com/about_the_ultracane.
(Accessed 22 Apr. 2022)
6. Wewalk, “The Perfect Pairing for Ultimate Visually Impaired Mobility”, https://wewalk.io/en/product/.(Accessed
22 Apr. 2022)
7. Sunu
Inc.,
“THE
SUNU
BAND”,
https://sunu.io/pages/sunu-band. (Accessed 15 Jan. 2023)
8. iMerciv Inc., “BuzzClip User Guide”, https://www.imerciv.com/user-guide/index.shtml. (Accessed 22 Apr. 2022)
9. R. K. Katzschmann, B. Araki, and D. Rus, “Safe Local
Navigation for Visually Impaired Users With a Time-ofFlight and Haptic Feedback Device”, IEEE Transactions
on Neural Systems and Rehabilitation Engineering, vol. 26,
no. 3, pp. 583–593, 2018.
10. E. J. Argüello Prada and L. M. Santacruz Forero, “A beltlike assistive device for visually impaired people: Toward
a more collaborative approach”, Cogent Engineering, vol.
9, no. 1, pp. 2048440, 2022.
11. C. Feltner et al., “Smart Walker for the Visually Impaired”,
2019 IEEE International Conference on Communications(ICC), pp. 1–6, 2019.
12. M. Murakami et al., “E-cane with Situation Presumption
for the Blind”, Transactions of the Institute of Systems,
Control and Information Engineers, vol. 16, no. 6, pp.287294, 2003. (In Japanese)
13. N. S. Ahmad, N. L. Boon, and P. Goh, “Multi-Sensor Obstacle Detection System Via Model-Based State-Feedback
Control in Smart Cane Design for the Visually Challenged”, IEEE Access, vol. 6, pp. 64182–64192, 2018.
14. M. M. Billah et al., “Experimental investigation of a novel
walking stick in avoidance drop-off for visually impaired
people”, Cogent Engineering, vol. 6, no. 1, pp. 1692468,
2019.
15. S. Caraiman et al., “Computer Vision for the Visually Impaired: the Sound of Vision System”, 2017 IEEE International Conference on Computer Vision Workshops(ICCVW), pp. 1480–1489, 2017.
16. M. M. Islam, M. S. Sadi, and T. Bräunl, “AutomatedWalking Guide to Enhance the Mobility of Visually Impaired People”, IEEE Transactions on Medical Robotics
and Bionics, vol. 2, no. 3, pp. 485-496, 2020.
17. S. Fazli, H. M. Dehnavi, and P. Moallem, “A robust negative obstacle detection method using seed-growing and dynamic programming for visually-impaired/blind persons”,
Optical Review, vol. 18, no. 6, pp. 415-422, 2011.

Fig. 11 Grid map simulation of experiments
experiments. The experimental results show that the system has enough possibility to estimate fall risk, which
could categorize the area to four kinds of areas in the approach angle 90 and 45 degrees to the stairs, using only
the information obtained from a smartphone with a LiDAR sensor. The propose algorithm was implemented as
the smartphone application and could make an alert ofsafety confirmation for subject.
Future issues include stabilizing the fall risk estimation by adding hysteresis and examining the parametersof the fall risk estimation through evaluation experiments
under actual operating conditions.

Acknowledgments
This research was supported by the Research Grant of
JR–West Relief Foundation in 2020 (Grant No. 20R032).
References
1. Ministry of Land, Infrastructure and Transport, “Study
Group for Improving Safety at Station Platforms”,
https://www.mlit.go.jp/tetudo/tetudo_fr7_000015.html.
(In Japanese, Accessed 22 Apr. 2022)355

18. R. Munoz, X. Rong, and Y. Tian, “Depth-aware indoor
staircase detection and recognition for the visually impaired”, 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1-6, 2016.
19. P. Herghelegiu, A. Burlacu, and S. Caraiman, “Negative
obstacle detection for wearable assistive devices for visually impaired”, 2017 21st International Conference on
System Theory, Control and Computing (ICSTCC), pp.
564-570, 2017.
20. Microsoft Corp., “Seeing AI App from Microsoft”,
https://www.microsoft.com/en-us/ai/seeing-ai. (Accessed
24 Apr. 2022)
21. Envision Technologies B. V., “Envision App”,
https://www.letsenvision.com/envision-app. (Accessed 24
Apr. 2022)
22. Apple Inc., “Compare iPhone models”, https://www.apple.com/iphone/compare/?modelList=iphone13promax,iphone13pro,iphone13. (Accessed 29 Aug. 2022)
23. V.Bharati, “LiDAR+ Camera Sensor Data Fusion On Mobiles With AI-Based Virtual Sensors To Provide Situational Awareness For The Visually Impaired.”, 2021 IEEE
Sensors Applications Symposium (SAS), pp. 1–6, 2021.
24. M. Kuribayashi et al., “LineChaser: A Smartphone-Based
Navigation System for Blind People to Stand in Lines”,
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, no. 33, pp. 1-13, 2021.
25. M. Kuribayashi et al., “Corridor-Walker: Mobile Indoor
Walking Assistance for Blind People to Avoid Obstacles
and Recognize Intersections”, Proceedings of the ACM on
Human-Computer Interaction, vol. 6, no. MHCI, pp. 1–22,
2022.
26. Y. Wang et al., “Pseudo-Lidar From Visual Depth Estimation: Bridging the Gap in 3d Object Detection for Autonomous Driving.” Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR), pp.
8437-8445, 2019.
27. M. Yatabe, “Handy Note for Quaternions”, MSS Technical Report, vol. 18, pp. 29-34, 2007. (In Japanese)from Eq. (A.2). Also, the quaternion product is noncommutative (𝑞̃𝑝̃ ≠ 𝑝̃𝑞̃).
𝑞̃𝑝̃ = (𝑞0 + 𝒒)(𝑝0 + 𝒑)
= 𝑞0 𝑝0 + 𝑞0 𝒑 + 𝑝0 𝒒 − 𝒒 ∙ 𝒑 + 𝒒 × 𝒑(A. 5)The norm of quaternions is shown as Eq. (A.6).
‖𝑞̃‖ = √𝑞02 + 𝑞12 + 𝑞22 + 𝑞32(A. 6)The quaternion provided the rotation is provided as
Eq. (A.7). 𝒏 is the unit vector in the cartesian coordinate
system. 𝜃 is the rotation angle. This quaternion’s normis ‖𝑞̃‖ = 1.
𝜃𝜃
(A. 7)𝑞̃ = cos + 𝒏 sin
22
By using the quaternions in Eq. (A.7), the rotation of
the position vector 𝒓 = [𝑥, 𝑦, 𝑧]𝑇 and the rotation of thecoordinate system are shown in Eq. (A.8) and Eq. (A.9),
respectively.
(A. 8)
𝒓′ = 𝑞̃𝒓𝑞̃ ∗(A. 9)
𝒓′′ = 𝑞̃ ∗ 𝒓𝑞̃
Equation (A.10) and Eq. (A.11) are expanding Eq.
(A.8) and Eq. (A.9), respectively. Equation (A.10) is
equal to the rotation of the position vector shown in Fig.
A.1; Eq. (A.11) is the reverse order of the vector product in the third term of Eq. (A.10), which provides therotation of the coordinate system.
𝜃𝜃𝜃𝜃
𝒓′ = (cos + 𝒏 sin ) 𝒓 (cos − 𝒏 sin )
2

2

2

2

= (𝒓 ∙ 𝒏)𝒏 + {𝒓 − (𝒓 ∙ 𝒏)𝒏} cos 𝜃 + 𝒏 × 𝒓 sin 𝜃 (A. 10)
𝜃

𝜃

𝜃

𝜃

2

2

2

2

𝒓′′ = (cos − 𝒏 sin ) 𝒓 (cos + 𝒏 sin )= (𝒓 ∙ 𝒏)𝒏 + {𝒓 − (𝒓 ∙ 𝒏)𝒏} cos 𝜃 + 𝒓 × 𝒏 sin 𝜃 (A. 11)

Appendix A. Quaternion [27]
Quaternion is defined by Eq. (A.1). 𝑞0 is the scalar part,𝒒 = 𝑞1 𝒊 + 𝑞2 𝒋 + 𝑞3 𝒌 is the vector part in the cartesian
coordinate system. 𝒊, 𝒋, and 𝒌 are basis vectors in the
Cartesian coordinate system, 𝑥 -axis, 𝑦 -axis, and 𝑧-axis,
respectively. Also, 𝒊, 𝒋, and 𝒌 are provided in Eq. (A.3)
from the vector product 𝒂 and 𝒃 defined in Eq. (A.2).
(A. 1)𝑞̃ = 𝑞0 + 𝒒 = 𝑞0 + 𝑞1 𝒊 + 𝑞2 𝒋 + 𝑞3 𝒌
(A. 2)𝒂𝒃 ≡ −𝒂 ∙ 𝒃 + 𝒂 × 𝒃
𝒊2 + 𝒋2 + 𝒌2 = 𝒊𝒋𝒌 = −1
(A. 3)
𝒊𝒋 = −𝒋𝒊 = 𝒌, 𝒋𝒌 = −𝒌𝒋 = 𝒊, 𝒌𝒊 = −𝒊𝒌 = 𝒋Conjugate quaternion 𝑞̃ ∗ is the inverted sign of thevector part of the quaternion 𝑞̃, as shown in Eq. (A.4).
(A. 4)𝑞̃ ∗ = 𝑞0 − 𝒒 = 𝑞0 − 𝑞1 𝒊 − 𝑞2 𝒋 − 𝑞3 𝒌
The quaternion product of two quaternions 𝑞̃ and 𝑝̃ =𝑝0 + 𝒑 = 𝑝0 + 𝑝1 𝒊 + 𝑝2 𝒋 + 𝑝3 𝒌 is shown in Eq. (A.5)

Fig. A.1 Position vector rotation(Modified from the paper [27])356Mr. Koichi Wada
He is a Vision Park Information
Master at NEXT VISION. He graduated from Acupuncture and Physical Therapy Teacher Training
School of University of Tsukuba.

Authors Introduction
Mr. Daigo Katayama
He is a Doctoral Student at Graduate
School of Life Science and Systems
Engineering, Kyushu Institute of
University. He received Master’s degree from Graduate School of Life
Science and Systems Engineering,
Kyushu Institute of University.

Ms. Akane Befu
She is a Vision Park Information Concierge at NEXT VISION. She graduated from The Department of Domestic Science, Tokushima Bunri University.

Prof. Kazuo Ishii
He is a Professor at Graduate Schoolof Life Science and Systems Engineering, Kyushu Institute of University. He received Ph.D. from Schoolof Engineering, The University of Tokyo.

Ms. Chikako Yamada
She is an Executive Secretary atNEXT VISION. She graduated Graduate School of Sociology, Bukkyo
University. She received Master’s degree from Graduate School of Sociology, Bukkyo University.

Dr. Shinsuke Yasukawa
He is an Associate Professor at
Graduate School of Life Science andSystems
Engineering,
KyushuInstitute of University. He received
Ph.D. from Graduate School ofEngineering, Osaka University.

Dr. Yuya Nishida
He is an Associate Professor at
Graduate School of Life Science andSystem Engineering, Kyushu Institute of Technology. He received
Ph.D. from Graduate School of Life
Science and Systems Engineering,
Kyushu Institute of University.

Dr. Satoshi Nakadomari
He is a Director at NEXT VISION.
He graduated from the Departmentof Medical, The Jikei University. He
received Ph.D. from the Departmentof Medical, The Jikei University.

357