IoH Technologies into Indoor Manufacturing Sites

Takeshi Kurata, Takashi Maehata, Hidehiko Hashimoto, Naohiro Tada,
Ryosuke Ichikari, Hideki Aso, Yoshinori Ito

► To cite this version:

Takeshi Kurata, Takashi Maehata, Hidehiko Hashimoto, Naohiro Tada, Ryosuke Ichikari, et al. IoH Technologies into Indoor Manufacturing Sites. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2019, Austin, TX, United States. pp.372-380, 10.1007/978-3-030-29996-5_43. [hal-02460477](https://hal-inria HAL Id: hal-02460477)

https://inria.hal.science/hal-02460477

Submitted on 30 Jan 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# IoH Technologies into Indoor Manufacturing Sites

Takeshi Kurata¹², Takashi Maehata¹, Hidehiko Hashimoto¹, Naohiro Tada¹

Ryosuke Ichikari², Hideki Aso³, Yoshinori Ito³

¹ IoT R&D Center, Sumitomo Electric Industries, Ltd., Japan

² Human Augmentation Research Center, AIST, Japan

³ IoT Acceleration Lab, J-Power Systems Co. Ltd., Japan

kurata-takeshi@sei.co.jp

**Abstract.** This paper focuses on introducing measurement technologies into manufacturing sites regarding the worker-oriented part of 6M, which consists of Man, Machine, Material, Method, Mother Nature, and Money. First, we introduce indoor positioning and work motion recognition systems that we have developed as key components of Internet of Humans (IoH) technologies. Next, we briefly report on two case examples of manufacturing sites where worker behavior measurement, analysis, and visualization are promoted. Then, we conclude this paper with discussion about the costs and benefits on the introduction of indoor positioning technologies into manufacturing sites.

**Keywords:** IoH, Indoor positioning, Work motion recognition, 6M, Mieruka, Kaizen, Manufacturing site

## 1 Introduction

To comprehensively understand and specifically improve the situation of manufacturing sites, it should be effective to aggregate big data regarding 6M (Man, Machine, Material, Method, Mother Nature, and Money) [1]. This research is currently continuing with the aims of realizing 6M ‘mieruka’, which means visualization or vision control, as well as providing technologies which support continual kaizen and work/safety/health-care management. With the proliferation of Internet of Things (IoT) products and services, visualization in terms of tangible things (Machine and Material) such as facilities, equipment, raw materials (RM), finished goods (FG), etc. is progressing rapidly for grasping the present situation and confirming the result of kaizen (improvement). However, the development of visualization technologies and methodologies is still ongoing when it comes to intangible things (Man, Method) such as worker conditions and workflow processes [2]. The lack of relevant data on human behavior could be considered a major disincentive for progress on this front.

It is relatively easy to collect individual worker data in cases where the work is repetitive in a specific area of mass production. In cellular manufacturing or high-mix low-volume production, however, moving and working are often combined, and it poses a major barrier to data collection on each worker. Therefore, it is crucial to further
develop and utilize Internet of Humans (IoH) technology for collecting data on human
behavior. As there is often a high correlation between worker positions and operation
contents, and also manufacturing sites are mainly occupied by indoor environments,
indoor positioning technology is regarded as one of key IoH components.

Accordingly, this paper especially focuses on measurement technologies of the
worker-oriented part of 6M, and briefly reports on two case examples of manufacturing
sites where worker behavior measurement, analysis, and visualization are promoted. It
also discusses the costs and benefits of the introduction of indoor positioning technol-
ogies into manufacturing sites.

## 2 IoH Technologies

### 2.1 Indoor Positioning System Configuration

Our indoor positioning system shown in Fig. 1-left has two features suitable for intro-
ducing into actual manufacturing/service sites. The first feature is that the system uti-
lizes xDR (Dead Reckoning for X) which includes PDR (Pedestrian Dead Reckoning)
[3] and VDR (Vehicle Dead Reckoning) [4] as a method of relative positioning. The
second feature is that solar-powered BLE beacons are employed as stationary nodes
placed around the area of interest. This system is an integrated positioning system that
consists of xDR, RSSI-based BLE positioning, and map constraints.

Although solar-powered BLE beacons are twice to three times as expensive as typi-
cal battery-powered BLE beacons, the combination of xDR and BLE positioning makes
it possible to reduce the number of BLE beacons to only a fraction of the number it
would take with only BLE positioning. This allows for a battery-free setup without
increase in initial installation costs. Additionally, a battery-free setup eliminates the
need for battery replacement, greatly reducing operational costs. Although the config-
uration inherently has issues on lighting for energy harvesting, we have been develop-
ing a method of sustain positioning performance with uneven time interval of BLE sig-
nal transmission due to insufficient lighting. We will present the details in near future.

Stationary nodes are naturally placed for covering areas of interest. However, sig-
nificant flow lines or activities may occur outside of the expected areas of interest, and
there are cases in which such data are crucial for worker behavior analysis. Although
there is nothing to do with only BLE positioning, we still can keep tracing flow lines
with xDR.

As hardware for mobile nodes, the workers are able to use embedded modules and
smartphones equipped with a ten-axis sensor (3 for acceleration, 3 for angular velocity,
3 for magnetism, 1 for barometric pressure). Especially because of their increasing size
in recent years, smartphones were not used in this research as they could impede worker
operations. Instead, the Android-based terminal BL-02 made by BIGLOBE is adopted
in our manufacturing sites. The same setup can be also applied to forklifts for VDR.

Fig. 1-right shows the cost-effective arrangements of stationary and mobile nodes.
In general, transmitters such as BLE beacons and passive RFID tags are much less ex-
pensive than receivers or transponders such as smartphones and IoT gateways. In cases
with a larger area and fewer number of persons measured, it is better to choose
transmitters as stationary nodes and receivers as mobile nodes (SB-type system) from the aspect of cost. Our manufacturing sites are categorized in this case. In addition, because cameras are often difficult to bring in and install on site due to issues of cost and privacy, vision-based positioning technologies are not integrated in our system.

<figure><img src="image_2.png" /><figcaption>Fig. 1. (Left) indoor positioning system in use at manufacturing sites; (right) cost-effective arrangements of transmitters and receivers.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. Work motion recognition systems the difference of which is the number of IMUs.</figcaption></figure>

## 2.2 Work Motion Recognition

If micro-level understanding of behaviors is required as in the analysis of hospitality in customer service, cooking and assembly work, skills involved, etc., the use of only position data are not enough for such purposes. Inertia Measurement Units (IMUs) as in mobile nodes for xDR go beyond tracking position, as they are also capable of capturing type and size of motions, allowing for micro-level analysis on work motions and for safety management by detecting falling movements. Fig. 2 depicts three examples of work-motion capture systems.

Typically, 10 to 20 IMUs are attached all over the body as in the left example in Fig. 2. Although this sort of setup is usually permitted for short-term collection of data, the time involved in attaching and detaching the system, and its potential to interfere with work tasks make such a system unlikely to be adopted for long-term, everyday use. The system in Fig. 2-center is designed to reduce the number of IMUs to only five, in order to be less cumbersome for workers and reduce hardware costs. In this case, a smartphone is placed inside the ‘obi’ belt as one of IMUs and also as a BLE receiver.
Compared to a configuration where IMUs are attached to the whole body, this kind of configuration results in a precision reduction of around 10 to 20%. The whole-body configuration provides the position and movement of each body part based on a skeleton model. In contrast, the partial body configuration, as in Fig. 2-center, means some more detailed information on work motions are missed, and we must rely on the local movement data for the available sensors only.

To address such problems, an integrated IoT sensor module with a wearable passive RFID tag reader and a ten-axis sensor module have been developed through METI's project to support the advancement of strategic core technologies (represented by Gobi) (Fig. 2-right). This allows for the micro-level information lost in the decreased number of IMUs mentioned above to be partially obtainable once more by taking micro-positional data with RFID tag reading, and improvement in motion recognition precision can be expected.

# 3 Manufacturing Site Case Studies

In this section, we briefly introduce two sample cases of manufacturing sites where worker behavior measurement, analysis, and visualization are promoted. Implications and suggestions through the following cases are not well structured, however, we believe that they fit properly to many cases at actual manufacturing sites, and therefore they would be helpful for a person in charge to develop or install systems for worker behavior measurement, analysis, and visualization.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Example of flow line measurement at a manufacturing site.</figcaption></figure>

## 3.1 Cable Manufacturing Line

J-Power Systems Co., Ltd. (hereafter referred to as JPS) is a member of Sumitomo Electric Group, and is involved in the designing, manufacturing, constructing, and selling of electric power transmission cables, overhead power transmission lines, amongst other things. We are introducing the indoor positioning system depicted in Fig. 1-left and the 6M visualization setup in hopes of increasing equipment utilization rates on cable production lines.

Although not addressed much in this paper, the first step of implementation at JPS involved the Mother-Nature (Environment) aspect of 6M. A 3D model of the indoor
environment was automatically generated using LIDAR and an omnidirectional camera [5]. This 3D model was then converted into a 2D floor map for the site's area of interest. As shown in Fig. 3-left, BLE beacons were placed on the floor map and the beacons' geospatial coordinates on a real-world coordinate system were automatically extracted. Floor maps are necessary for positioning programs and visualization tools, but many sites lack a CAD model or otherwise, have CAD models with outdated information. Furthermore, the process of calculating geospatial coordinates for BLE beacons can be made complicated by inaccurate floor maps. The approach we have taken makes it possible to avoid such issues.

The installation of a safety laser rangefinder (LRF) has also been arranged for preventing accidents in which workers are caught in equipment. Within the range of measurement, the LRF can be used for precise positioning. Another precise position correction by the PLC also becomes possible. However, since the worker ID is unknown in both methods, it is necessary to have a process to assign IDs with PDR.

## 3.2 Assembly Line

At an assembly line in a factory of Company A, motion capture data have been gathered by multiple IMUs attached to each part of workers' bodies, and micro motion recognition on the assembly line is underway. From this, we hope to precisely make standardized procedure for work element, and efficiently evaluate the variation in work involved.

On typical assembly lines, the work of interest (Material) often moves a little bit at a time. Because of this, it is not enough to simply obtain a static floor map, as was the case at JPS. Rather, linking dynamic data on the line with sensor data makes it possible to perform highly accurate work motion recognition. In addition, quirks in movement peculiar to each individual are inevitably included in the machine learning training data for micro-motion recognition. There is a need for development of technologies and methodologies to realize not only efficient training data collection but also motion recognition which is not affected by such individual differences.

# 4 Discussion

## 4.1 Make Time Tangible

We would like to discuss some of the things that should be considered when introducing measurement technologies to manufacturing or service sites. Generally, there is little objection to the importance of visualization when it comes to 6M as a whole or in part. However, thinking that short-term data collection and a one-time "before and after" comparison is sufficient, or that using a single system successively in many different areas can cut costs, rather than opting for continual long-term data collection, are common lines of thinking.

To shed light on these misconceptions, this section uses Fig. 4 to discuss some of the advantages of continual on-site measurement of data. The top and middle portions of the diagram show sites that are not continually collecting data, while the bottom
shows a site that is. Firstly, a common concern heard during on-site interviews is that
workers are not used to their work being monitored, and it will be difficult to determine
whether the resulting data reflect realistic and natural circumstances or not. Continual
data collection would allow such uncertainty to be dispelled.

It is necessary to consider how causes of found issues are analyzed and addressed. Typically (i.e., where continual data collection has not been adopted), the data collection system is only set up after a problem is identified, in which case one must wait for enough data surrounding the issue to be collected (a period that is referred to as the “Before data collection”). This results in a significant time lag between when the problem is first identified and when its causes can begin to be analyzed and addressed. Furthermore, as shown in Fig. 4-top, if the exact issue of concern does not arise once again after beginning Before data collection, there is no data to analyze to identify causes.

Dashboard cameras in cars use accelerometers to detect incidents such as collisions and sudden braking, and are able to keep records of such incidents for future use and, report accidents. Likewise, with continual data collection in manufacturing or service sites, issues can be analyzed at any time, with seamless transition into data collection for confirming the effectiveness of solutions (the “After data collection”). In this way, issues can be swiftly addressed and dealt with. This “Virtual Time Machine” concept [6] is expected to take root in more areas in the future as 6M visualization technology continues to develop and mature.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Benefits of continual data collection.</figcaption></figure>

## 4.2 Cost Comparison

With regard to the costs involved in introducing human behavior measurement tech-
nology, in this research, we considered three-company cases, which are Companies P,
Q, and R, to investigate the costs of time study and work sampling in human-wave
tactics, as well as of automatic data collection with data collection systems. The details
for each case are as follows:
* Company P: Time study by video recording, 1 observer, 1 worker recorded, and macro-positioning granularity [1] (TS-Macro)

* Company Q: Work sampling, 5 observers, 33 workers recorded, and mezzo-positioning granularity (WS-Mezzo)

* Company R: Time study by video recording, 2 observers, 3 workers recorded, and micro-positioning granularity (TS-Micro)

We added one more system configuration for cost comparison. Up to now, each system for the flow-line measurement and analysis discussed here (system #1), for safety management with worker fall detection (system #2), and for health-care management with gait evaluation (system #3) has been individually implemented and adopted at each manufacturing or service site. Because the data collection system, which contains sensors such as accelerometers, used in system #1 can also be used for systems #2 and #3, it is possible for the systems to integrate with one another. If the system cost can be evenly distributed across those three systems #1, #2, and #3, the cost of system #1 can be reduced to a third.

The results of cost comparison are shown in Tab. 1. In all of three companies' cases, we confirmed that the costs of time study and work sampling in human-wave tactics gets higher than the ones of data collection systems usage.

Additionally, in the case of Company Q, because work sampling was conducted, there is inevitable loss of data in human-wave tactics, whereas visualization was possible with the lossless data by adopting automatic measurement. Integrating work analysis with health-care and safety management into a unified system brings additional value and advantages beyond just system cost reduction. For example, it is possible for such a system to send a fall-detection-based emergency call with position information even indoors although conventional fall detection systems do not cover indoor sites.

<table><caption>**Table 1.** Cost comparison of manual observation and automatic measurement with three cases. H: Cost for human-wave tactics, S: Cost of data collection system usage, D: 1/3 cost of data collection system usage.</caption><thead><tr><td>Case</td><td>Num of Workers</td><td>Num of Observers</td><td>H>S (days)</td><td>H>D (days)</td></tr></thead><tbody><tr><td>Company P (TS-Macro)</td><td>1</td><td>1</td><td>35</td><td>3</td></tr><tr><td>Company Q (WS-Mezzo)</td><td>11</td><td>5</td><td>76</td><td>24</td></tr><tr><td>Company R (TS-Micro)</td><td>3</td><td>2</td><td>101</td><td>25</td></tr></tbody></table>

# 5 Conclusion

Needless to say, even if massive amount of 6M data can be gathered by IoT/IoH devices in real manufacturing sites, it does not hold all the answers to comprehensively understand the real sites since big data in general has issues on quality and variety. In-depth surveying such as retrospective interviewing has potential to complement the defect of big data, however, it inevitably requires intensive effort with high work load. In-depth
surveying with subject screening based on big data would alleviate the load, and it
would result in efficient surveying with both breadth and depth [7]. This is consistent
with the idea of “Pier Data” outlined in Fig. 6 in which “Big Data” are integrated with
“Deep Data” [8]. Demonstration of such a methodology in actual sites through further
development of 6M data collection and visualization technology including IoH tech-
nologies is one of our future works.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Screening with 6M big data for obtaining deep data by in-depth survey.</figcaption></figure>

Acknowledgements

This work was supported by JST-OPERA Program Grant No. JPMJOP1612, Japan.

References

1. Kurata, T., et al., Towards Realization of 6M Visualization in Manufacturing Sites, Proc. IEEE VR Workshop on Smart Work Technologies (WSWT), 4pages (2019).

2. Fukuhara, T., et al., Improving service processes based on visualization of human behavior and POS data: a case study in a Japanese restaurant, Proc. ICServ, pp.1-8 (2013). doi: 10.1007/978-4-431-54816-4_1

3. Kourogi, M., Kurata, T., Personal positioning based on walking locomotion analysis with self-contained sensors and a wearable camera, Proc. ISMAR, pp.103-112 (2003). doi: 10.1109/ISMAR.2003.1240693

4. Ichikari, R., et al., Off-Site Indoor Localization Competitions Based on Measured Data in a Warehouse. Sensors, Vol.19, Issue.4, No.763, 26pages (2019). doi: 10.3390/s19040763

5. Kuramachi, R., et al., G-ICP SLAM: An odometry-free 3D mapping system with robust 6DoF pose estimation, Proc. IEEE International Conference on Robotics and Biomimetics (ROBIO), pp.176-181 (2015). doi: 10.1109/ROBIO.2015.7418763

6. Hirose, M., et al., Virtual time machine, Proc. of ICAT, SS1-6 (2004)

7. Nakajima, M., et al., Cognitive chrono-ethnography lite, work: A Journal of Prevention, Assessment and Rehabilitation (IEA: 18th World congress on Ergonomics - Designing a sustainable future), Vol. 41, pp.617-622 (2012). doi: 10.3233/WOR-2012-0219-617

8. Kurata, T., et al., Making pier data broader and deeper: PDR challenge and virtual mapping party, Proc. MobiCASE, pp.3-17 (2018). doi: 10.1007/978-3-319-90740-6_1