SUGISAKAMASANORI

Journal of Robotics, Networking and Artificial Life
Vol. 9(3); December (2022), pp. 289–294
ISSN (Online):2352-6386; ISSN (Print):2405-9021
https://alife-robotics.org/jrnal.html

Research Article

Detecting Approaching Human Hands in a Human-Robot
Coexistent Food Preparation Work Area for Preventing
Collision
Joo Kooi Tan1, Takaaki Yotsumoto2, Yuta Ono2

Faculty of Engineering, Kyushu Institute of Technology, Sensuicho 1-1, Tobata, Kitakyushu, Fukuoka 804-8550, Japan
Graduate y of Engineering, Kyushu Institute of Technology, Sensuicho 1-1, Tobata, Kitakyushu, Fukuoka 804-8550, Japan

12

ARTICLE INFO
Article History
Received 26 November 2021
Accepted 24 September 2022
Keywords
Recognition
Hand area extraction
Lab color system
Grab cut

ABSTRACTIn Japan, the population of working age between 15 and 64 years old peaked in 1995 at about 87
million and is expected to continue to decline in the future. Therefore, to solve the labor shortage,the introduction of industrial robots that can perform the same level of work as humans is strongly
requested especially in the food preparation industry. In order to prevent danger to workers there,
industrial robots must recognize workers and avoid them when there is fear of collision. In thispaper, we propose a method for extracting hand regions based on the color distribution of a hand
and GrabCut in an experimental environment to recognize human hands and detect theirdirections of approach. The proposed method was examined experimentally and gave satisfactory
results.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/)

proportion to the decline in the working-age population.
For this reason, the introduction of industrial robots, suchas robot arms capable of performing tasks equivalent tothose of humans, will be necessary to solve the labor
shortage.
According to the Industrial Safety and HealthRegulations [2] and the Ministry of Labor Notification
No. 51 of 1983 (machines specified by the Minister ofHealth, Labor and Welfare under the provisions ofArticle 36, Item 31 of the Industrial Safety and HealthRegulations) [3], except for those that meet certain
standards, in order to use industrial robots, it is basicallynecessary to enclose them with fences and take othersafety measures to prevent humans from entering.
However, according to [4], by implementing measures
based on the investigation of danger, etc., joint work
between industrial robots and humans can be performed.

1. Introduction
According to an announcement by the Small and Medium
Enterprise Agency [1], the working-age population (1564 years old) in Japan peaked in 1995 at about 87 million
and has been declining ever since, reaching about 77
million in 2015. This trend is expected to continue, and it
is estimated that the population will decline to about 60%of the 2015 level by 2060. The working population (thetotal of "employed" and "totally unemployed" among thepopulation aged 15 and over) declined by only about
420,000 between 1995 and 2015, due to an increase in
the labor participation rate of women and those people
aged 65 and over, but this decline is not as large as thatof the working-age population. However, it is easy toimagine that the working population will decline in

Corresponding author’s E-mail: etheltan@cntl.kyutech.ac.jp URL: https://www.kyutech.ac.jp/~etheltan/289

In order to prevent danger to the worker, it is necessary
for the industrial robot to recognize the worker and avoid
the worker when necessary.
In this study, as the first step for an industrial robot toavoid collision with workers, we consider a situation
where robots and humans collaborate in an assembly line
such as a lunch-box assembly line. We propose a method
to detect one’s neighbors’ hands that approach his/herwork area by extracting hand areas from ego camera
images.
One of the methods for hand area extraction is to use
a distance image [5], [6], [7] to capture and recognize its
3-D shape. However, the infrared camera used in thismethod has a drawback that it is easily affected bysunlight, which limits the locations where it is used.
Other methods that use multiple sensors [8], [9] or special
sensors [10] have been proposed, but they are considered
to be difficult to use because of the high price of thesystem. On the other hand, extraction or segmentation
techniques are typically based on skin color [11], [12],[13], background subtraction with hand tracking [14], orappearance-based model [13]. Technique [11] uses skincolor and face color: [12] uses two kinds of color spaces,
i.e., HSV and TSL to differentiate between skin color andnon-skin color: Moreover, [13] uses a skin color modelwith fine-tuning to the depth for segmentation of twohands. [14] uses a particle filter for tracking non-skincolor (the background) and detecting skin color by theGaussian Mixture Model (GMM) with a combination ofedge information for segmentation of a hand. Both skincolor and background subtraction-based techniques use
cues of HSV and RGB color space and are limited tocapturing the target from a distant fixed camera. [15]adopts a hand model. However, the technique requires
three specific views of a hand model for calibration.

Fig.1. Head-mounted camera
attached to the user (See Fig. 1) to avoid collision
between the user’s hand and the neighbors’ hands.
Initially, in order to detect the hand approaching fromthe side of the user, the system acquires hand color
information of a hand area on the spot where the system
is used in the given lighting conditions. The skin color
information is acquired from image frames prepared inadvance and integrated as the knowledge for reference.
Then, the system detects the hand area using theknowledge and identifies the hand from the area.
2.1. Acquisition of color information of a hand area
The proposed system acquires the color information of ahand area in a practical environment before judging theapproach of a human hand, and creating a color
distribution map of the hand at the place and the time.
2.2. Judging approach of a human hand
In the first place, the hand region is extracted from theinput image based on the hand region color distribution
map obtained in advance, and noise is removed. Then thesystem identifies whether or not a hand area isapproaching a user’s work area, and if it is true, it
displays a bounding box containing the hand area in theimage as the result of the detection.
3. Acquisition of Hand Area Information

2. Summary of the Research

In this section, we describe a method to acquire color
information of a hand area from the captured video.
The first frame of the video to acquire the hand area
information is the background frame without a hand.
Then, while keeping the orientation of the camera
unchanged, frames containing several patterns of handareas are acquired for a few seconds as shown in Fig.2.
These frames are used as the frames for extracting thehand color information.

This section gives an overview of the system proposed inthis paper. The proposed system detects a human hand ora robot arm approaching the work area in front of a user(a worker) from the images of a head-mounted camera

2903.1. Extraction of candidate hand areas

between the color distribution of the color information
extraction frame and that of the background frame. Thecolor distribution histograms of the two frames arecompared to select the hand area color distribution in theprocessing frame. Finally, the color distributions in thehand area obtained from the color information extraction
frames are combined by logical OR, and the distribution
having the largest area is selected to remove noise in thecase of selecting a color distribution other than the handarea by mistake. In the proposed method, the above
process is used to obtain the color distribution of the handarea dependent on the usage environment.

As shown in Fig. 3(a), most of the image frames acquired
from the head-mounted camera used in the proposedsystem have a large background area except for handareas. Since these areas interfere with the acquisition of
the color information of a hand area, we delete these areas.
We detect candidate hand areas by extracting thedifference region between each color information
extraction frame and the background frame describedabove (See Fig.3(b)). For each pixel, a foreground pixel
is extracted to prevent false extraction due to changes inbrightness. The L∗ a∗ b∗ color system is used for thisprocess. After that, GrabCut and fine line segmentation
are applied to the hand area extraction image for possible
improvement in detection accuracy.
Since the areas which do not have a skin color are notconsidered to be hand areas, the skin color areas are
extracted using the HSV color system.

4. Judgment on the Approach of a Human Hand
In this section, we describe a method of judging theapproach of a human hand from an input image.
To judge the approach of a human hand, the proposed
method identifies if the approach is occurring, and if so,
it displays a bounding box containing a hand area in theimage where the hand approach is occurring.

3.2. Extraction of hand areas
The color information is extracted from the color
information extraction frame using the candidate handarea extracted in Section 3.1 as a mask. To select thecolor distribution of the hand area, we use the difference

4.1. Extraction of hand areas
The hand area is extracted from the input video. The handarea color distributions of the camera wearer and his/herneighbors, which were created in advance by acquiringhand area color information, are used to extract the pixels
having 𝑎𝑎 ∗ and 𝑏𝑏 ∗ values that are included in the color
distribution. Then, by using a median filter as a denoiser,
the actual hand area is finally extracted from the input
video. An example of hand area extraction after
denoising is shown in Fig.4.

Fig.2. Color information extraction frame

4.2. Judging the approach of a human hand
The proposed method judges the approach of neighbors’hands to the work area in front of a camera wearer/robot.
First, labeling of extracted regions on a given image
is performed, and among the obtained labeled regions,judgment is made on the regions with a size larger than aspecified value. Two attention areas are defined at theright-hand side and the left-hand side of the work area. If
a labeled hand area is found in the attention area, it isjudged dangerous, because the found hand area may havea fear of coming into the frontal work area.
There are four patterns of judgments from the camera
wearer's/robot side: no approach, approach from the righthand side, approach from the left-hand side, and approach

(a)

(b)
Fig. 3 Extraction of area candidate: (a) Input frame, (b)candidate hand areas
291

This paper proposed a system for detecting a handapproaching from the side of a center work area in frontof a worker who is a user of the proposed system. For thedetection, a head-mounted camera was used which theuser wears. Advantages of the employment of the headmounted camera include that the system can escape fromcareful camera setting at the work spot and that it can gain
mobility in, e.g., a factory. The effectiveness of theproposed system was shown experimentally.
The system has originality in that it analyzes the workat hand based on the first-person vision. The idea ofdetecting hands approaching from the side toward thecenter work area is also the originality of the paper. The(a)(b)
Fig. 4 Hand area extraction: (a) Input images, (b)extracted hand areas.
from both sides. Note that the proposed system assumes
there are three people; the camera wearer him/herself, theright neighbor, and the left neighbor.

(a)5. Experimental Results
An experiment was conducted to examine theperformance of the proposed method. Since the focus ison the detection of a hand approaching from the side
toward the center work area in front of a user/robot, arather simple scene containing some dishes with foodwas set (See Fig. 5). The scene was captured by a headmounted camera of the user whose play as a robot and thevideo provided by the camera was processed for
detecting an approaching hand.
It is noted that, although robot hands should be included in
the experiment, all the hands for detection are humanhands in this particular experiment for simplicity.
Namely the user and the neighbors in the experiment are
all humans.
The system performed the analysis on 150 images
extracted every 10 frames from the input video. The
accuracy of correct detection was evaluated by creating aground truth image and by calculating three major
indices; recall, precision, and F-measure. As the resultof the experiment, the recall was 81.0%, precision was98.6%, and F-measure was 88.9% as shown in Table 1.
Some results on the detection of approaching hands are
given in Fig. 5.

(b)

(c)
Fig. 5 Experimental results: (a) A handapproaching from the right-side, (b) approachingfrom the left-side, (c) approaching from the bothsides.
Table 1 Accuracy evaluation
Accuracy evaluation
frames[frame]
𝑇𝑇𝑇𝑇 [frame]
𝐹𝐹𝐹𝐹 [frame]
𝐹𝐹𝐹𝐹 [frame]
𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 [%]𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 [%]𝐹𝐹- measure [%]6. Discussion and Conclusion
292

150
681
16
81.0
98.6
88.9

hand detection described in this paper will become
necessary in the future for the line robots working withhuman line workers to avoid mutual hand collision.
Some issues need further investigation to achieve
higher performance of the proposed system. Advanced
hand-part color calibration should be done moreaccurately since people have a variety of skin colors. The
case of wearing colored gloves should also be taken intoaccount. Alternatively, the shape information of a hand
may be useful for hand detection, since it is lessinfluenced by the change in illumination.
In the present system, the detection of a hand area isdone every 10 frames independently. In order to assure
correct detection of a hand area, the detected hand area
needs tracking in the video, so as to compensate for false
or missed detection of the hand area at certain image
frames.

9.
10.

11.
12.

13.

References
1. The Small and Medium Enterprise Agency, “2019 White
Paper on Small and Medium Enterprises in Japan”, The
Small and Medium Enterprise Agency, p.121, 2018. (InJapanese).
2. Ministry of Health, Labour and Welfare, “Ordinance on
Industrial Safety and Health”, Ministry of Health, Labourand Welfare, 1972. (In Japanese).
3. Ministry of Health, Labour and Welfare, “Ministry ofLabor Notification No. 51 of 1983 (Machinery specified
by the Minister of Health, Labour and Welfare pursuant to
the provisions of Article 36, item 31 of the Industrial
Safety and Health Regulations)”, Ministry of Health,Labour and Welfare, 1983. (In japanese).
4. Ministry of Health, Labour and Welfare, “Partial revisionof the enforcement notice of article 150-4 of the industrial
safety and health regulations concerning industrial robots”,
Ministry of Health, Labour and Welfare, 2012. (InJapanese).
5. S. Tanaka, A. Okazaki, N. Kato, H. Hino, K. Fukui,“Spotting fingerspelled words from sign language video
by temporally regularized canonical component analysis”,
2016 IEEE International Conference on Identity, Security
and Behavior Analysis (ISBA), 7pages, 2016.
6. K.O. Rodriguez, G.C. Chavez, “Finger spellingrecognition from RGB-D information using kerneldescriptor”, IEEE Conference on SIBGRAPI, 7 pages,
2013.
7. N. Pugeault, R Bowden, “Spelling it out: Real-time ASLfingerspellingrecognition”,
IEEEInternational
Conference on Computer Vision Workshops (ICCVWorkshops), pp.1114-1119, 2011.
8. H. Brashear, V. Henderson, K. H. Park, H. Hamilton, S.
Lee, T. Starner, “American sign language recognition in14.
15.

game development for deaf children”, Proc. of the 8thInternational ACM SIGACCESS Conference onComputers and Accessibility, pp.79- 86, 2006.
H. Brashear, T. Starner, P. Lukowicz, H. Junker, “Usingmultiple sensors for mobile sign language recognition”,
Proc. of the 7th IEEE International Symposium onWearable Computers, pp.45-52, 2003.
Yu Wang, Seiji Itai, Satoshi Ono, Nakayama Shigeru,“Human Recognition with Ear Image by Principal
Component Analysis”, Journal of Japan Society ofInformation and Knowledge, Vol. 16, No. 1, pp.15-27,
2005. (In Japanese)
A. Bapat, V. Kanhangad, “Segmentation of hand fromcluttered backgrounds for hand geometry biometrics”,
IEEE Region 10 Symposium, TENSYMP, 4 pages, 2017.
L. Gu, X-y. Yuan, T. Ikenaga,“Hand gesture interface
based on improved adaptive hand area detection andcontour signature”, IEEE International Symposium onIntelligent Signal Processing and Communication System,PP.463-468, 2012.
H-S. Shih, C-H. Ma,“Hand segmentation with skin color
Fine-turning using Kinect Depth Sensor”, IEEE Global
Conference on Consumer Electronic, 2Pages, 2017.
V.Spruyt, A. Ledda, S. Geerts, “Real time multicolourspace hand segmentation”, IEEE International
Conference on Image Processing, pp.3117-3120, 2010.
J.J. Kuch, T. S. Huang, “Vision based hand modeling andtracking
forvirtual
teleconferencingandtelecollaboration”, IEEE International Conference on
Computer Vision, PP. 666-671, 2002.

Authors Introduction
Prof. Dr. Joo Kooi TanProfessor Tan is currently with theDepartment of Mechanical andControl Engineering, Kyushu Instituteof Technology, as Professor. Current
research interests include ego-motionanalysis, three-dimensional shape andmotion recovery, human detection
and its motion analysis from videos.
Professor Tan was awarded SICE Kyushu Branch Young
Author’s in 1999, the AROB Young Author’s Award in
2004, Young Author’s Award from IPSJ of Kyushu
Branch in 2004 and BMFSA Best Paper Award in 2008,2010, 2013 and 2015. Professor Tan is a member ofIEEE, The Information Processing Society, The Instituteof Electronics, Information and Communication
Engineers, and The Biomedical Fuzzy Systems Associationof Japan.

293

Mr. Takaaki Yotsumoto
He obtained his B.E. degree in theDepartment of Control Engineering fromKyushu Institute of Technology in 2019,Japan. He is presently a master’s course
student at the same university. His
research interests include humandetection and recognition.
Mr. Yuta Ono

videos.

He received his M.E. from Kyushu
Institute of Technology. He is now in thePh.D. course at the Graduate School ofEngineering, Kyushu Institute ofTechnology. His research includes
computer vision, machine learning, andrush-out pedestrian detection from294