<figure><img src="image_1.png" /></figure>

HAL
open science

# Design and Implementation of Smart Home Cloud System Based on Kinect

Xuebin Tang, Jinchuang Zhao, Wenbei Li, Bin Feng

► To cite this version:

Xuebin Tang, Jinchuang Zhao, Wenbei Li, Bin Feng. Design and Implementation of Smart Home Cloud System Based on Kinect. 10th International Conference on Intelligent Information Processing (IIP), Oct 2018, Nanning, China. pp.120-126, 10.1007/978-3-030-00828-4_13. hal-02197793

HAL Id: hal-02197793

https://inria.hal.science/hal-02197793

Submitted on 30 Jul 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Design and Implementation of smart home cloud system based on Kinect

Xuebin Tang*, Jinchuang Zhao*, Wenbei Li*, Bin Feng*

*Guangxi University, Institute of Computer and Electronic Information, Nanning, China

**Guangxi University, Institute of Art, Nanning, China

xbin_tang_gxu@foxmail.com
zhaojch@gxu.edu.cn
137415870@qq.com
binfeng_gxu@hotmail.com

**ABSTRACT.** Most smart home products on the market today only stay at the level of multi-channel control of home appliances and have a single product function. For those parts that are not involved, this article describes a smart home cloud system based on Kinect. The system uses the STM32 as the control core and utilizes Kinect depth sensors to establish an identity and posture recognition system based on human skeletons, combined with support vector machine (SVM) and Internet of Things technology to realize smart home control systems. What is more, intelligent alarm system and child anti-drop window control system truly realize home Intelligence. After verification, the system can greatly improve the current level of intelligent home security capabilities, make up for the vacancy of the existing identity system, real intelligence for the home rather than intelligent control of home appliances is finally achieved.

**KEYWORDS:** Smart home; Kinect somatosensory device; Skeleton recognition; Support vector machine; Internet of Things

## 1 Introduction

Home security is always a focused topic for people. In this field, fast and effective automatic identity verification plays an important role in security entrance control applications. Fingerprint recognition will reduce accuracy due to finger cuts. Exquisite makeup techniques and headscarf cover will cause misjudgment of face recognition. However, the bone information of the human body belongs to the physical properties of the creature, it's unable to fake and its stability is extremely high [1]. In order to improve the smart home intelligence and security level and achieve autonomous control, this article designs a smart home automatic control system based on IOT technology [2] which is relied on wifi data transmission with STM32 as the control core. Smart door used in the system introduces Kinect somatosensory equipment [3], its 3D infrared camera is used to obtain human skeleton information and adopts a new identification method based on human skeleton information so as to control the switch of the door through judging the legitimacy of the visitor. It successfully achieves a higher
skeletal motion information recognition rate. At the same time, the data model is estab-
lished for different home-customized posture control methods. The use of somatosen-
sory interaction technology to achieve intelligent home control. Its Ingenious use of
infrared sensors to complete the anti-drop wear design of infants and young children is
cost-effective, with market value. This article focuses on the composition of the system,
Kinect somatosensory technology applications, data collection and classification iden-
tification, as well as other hardware and software design.

## 2 Overall structure of the system

The smart home cloud system designed in this paper is mainly composed of three parts: control terminal, link end, and client. The control terminal mainly uses the STM32 as the processing center to collect various types of sensor data, perform data processing and analysis according to different pre-set conditions, issue control instructions, and upload the processing results to the client through the link end, which can achieve Real-time feedback to users via mobile app and receive data instructions from the client to achieve home control. The link end mainly uses the wifi module and cloud server platform for data transmission and exchange so that to realize real-time human-computer interaction between the control end and the client. The client is used to receive and display the system data sent by the control terminal, and the status of the home system is intuitively displayed in the form of a graphical interface. The user can remotely control the home through the interactive interface.

3 Kinect model establishment, data classification recognition

3.1 Skeletal structure recognition principle

In Kinect, a skeleton is represented by 25 joint points, as shown in Figure 1. When a person walks into the field of vision of Kinect, Kinect automatically finds the position of the 25 joints of the individual (stationary) [4], and expresses it by three-dimensional coordinates (x, y, z). What we ultimately need in this system is the three-dimensional coordinates of the individual's 25 joint points.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Human skeleton structure defined by Kinect</figcaption></figure>
## 3.2 The selection and optimization of bone features

### 3.2.1 Dataset construction and feature selection

In this paper, several bone data of different heights and genders were collected. In this experiment, 300 human bone data were randomly selected as samples. Due to the instability of Kinect, the identification of some joint points will have a large error, making the distance between the corresponding joint points unstable, such as the distance between the ankle and the sole of the foot [5]. Then, we selected 30 relatively stable bone fragments as features. The coordinates of the skeletal joint point are the coordinates relative to the three-dimensional coordinate system of Kinect [6]. As shown in Figure 2.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Kinect three-dimensional coordinate system</figcaption></figure>

According to the coordinates of the extracted 25 individual bone skeleton joints, each joint point is connected to form a bone fragment, and different bone feature quantities are obtained by calculating different bone fragment distances. Dist i represents the distance between skull joint points and zygomatic joint, which is one of the skeletal features we selected. Assuming that the coordinates of the two joint points a and b are known to be $(x_1, y_1, z_1)$ and $(x_2, y_2, z_2)$ respectively, the distance between these two joint points is calculated as:

$$ \mathrm{Dist}(a, b) = \sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2 + (z_1 - z_2)^2} \quad (1) $$

We arrange and combine 25 joint points and calculate the distance between different joint points to obtain various feature quantities. The skeleton eigenvector we finally get can be expressed as:

$$ F = (\mathrm{Dist}_1, \mathrm{Dist}_2, \dots, \mathrm{Dist}_n) \quad (2) $$

Where $\mathrm{Dist}_n$ represents the distance of a segment of a bone [7].

### 3.3.2 SVM parameter optimization and feature screening

In this paper, SVM (support vector machine) is used as the classification algorithm, and Gaussian radial basis function (RBF) is chosen as the kernel function [8]. In order to make the classification better, we used the 10-fold cross-validation to combine different features freely and found the feature combinations with the highest recognition rate by changing the number of samples, the number of kernel functions, the number of feature points, and the variable parameters c and g. The recognition rate reaches 97.059% under the optimal parameters. At this time, the first 50% feature combinations
are selected as the optimal features. Finally, we will use the optimal characteristics of
the sample as input to obtain the optimal SVM model. The specific process is shown in
Figure 3.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Kinect figure skeleton identification flow chart</figcaption></figure>

4 System implementation

For ease of development and research, the smart home system adopts the C/S architec-
ture, uses wifi networking, and establishes real-time data exchange links between the
control terminal and the client through the home gateway to realize information collec-
tion and control of various homes. Make sure the system is stable and reliable. The
system structure is shown in Figure 4.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Smart home system structure diagram</figcaption></figure>

4.1 Kinect online real-time character recognition system

We use the C# language to develop a Kinect real-time character recognition system based on WPF. The system includes six modules: bone data entry, training model, adjustment parameters, person recognition, gesture recognition, and real-time character image display. Figure 5 is the system character recognition interface diagram. The dots in the figure are the 25 skeleton joints identified by the system.

<figure><img src="image_7.png" /><figcaption>Fig. 5. system character recognition interface</figcaption></figure>
## 4.2 Based on Kinect gesture recognition control home

By the way of using Kinect's 3D somatosensory technology, people's movements and gestures are recognized. We formulate different home control systems according to different users' living habits and action behaviors. Different home appliances have different gestures, which are simple and easy to distinguish. In order to reduce the probability of system misoperation, each action must be maintained for more than 3s, otherwise it is regarded as an invalid instruction. In general, in order to reduce the energy consumption and ensure the service life of the smart home system, the system uses an infrared human body sensor to sense the human body position. When a person stands above Kinect for more than 5s, the system starts Kinect; when Kinect does not detect a matching custom operation command for more than 5s, the system turns on the self-protection mode and enters the standby state. The system custom operation instructions designed in this paper are shown in Table 1.

<table><caption>**Table 1.** Smart Home System Custom Operation Instructions</caption><thead><tr><th>Furniture Type</th><th>Operation Instructions</th><th>Custom gestures</th></tr></thead><tbody><tr><td>Door</td><td>Open the door</td><td>Standing hands straight, naturally sagging, face detection</td></tr><tr><td rowspan="2">Window</td><td>Open the window</td><td>With both hands up and down or children left</td></tr><tr><td>Close the window</td><td>With both hands up and down or up close</td></tr><tr><td rowspan="4">Air Conditioning</td><td>Turn on the air conditioning</td><td>Hands on hips</td></tr><tr><td>Turn off the air conditioning</td><td>Two-handed side lift</td></tr><tr><td>Cooling mode</td><td>Left hand akimbo right hand side lift</td></tr><tr><td>Heating mode</td><td>Right hand akimbo left hand side lift</td></tr></tbody></table>

## 4.3 Analysis of results

In order to verifying the reliability and stability of the system's posture recognition, each instruction was measured 100 times under the same conditions. Table 2 shows the correct number of times for each instruction and its accuracy. Experiments show that the gesture recognition scheme designed in this paper can effectively control the smart home system and has a very high reliability and stability.

<table><caption>**Table 2.** The recognition rate of each instruction</caption><thead><tr><th>Operation instructions</th><th>Correct times/time</th><th>Accuracy/%</th></tr></thead><tbody><tr><td>Open the door</td><td>98</td><td>98</td></tr><tr><td>Open the window</td><td>97</td><td>97</td></tr><tr><td>Close the window</td><td>99</td><td>99</td></tr><tr><td>Turn on the air con</td><td>97</td><td>97</td></tr><tr><td>Turn off air conditioning</td><td>98</td><td>98</td></tr><tr><td>Cooling mode</td><td>98</td><td>98</td></tr><tr><td>Heating mode</td><td>99</td><td>99</td></tr></tbody></table>

For the purpose of facilitating maintenance and function expansion, this system designed a mobile client based on Android operating system and JAVA language. The mobile client control interface is shown in Figure 6.
<figure><img src="image_8.png" /><figcaption>Fig. 6. Mobile client system interface</figcaption></figure>

## 5 Conclusions

The system is based on the Internet of Things smart home, it has successfully introduced Kinect somatosensory sensor into the smart home system for gesture control command information and face information, and used SVM as a classification recognition algorithm, then performs information matching verification with pre-customized posture to obtain control instructions to achieve home Intelligent control. The experimental result shows that with the successful introduction of Kinect, only one gesture can be used to control indoor facilities, which can effectively improve the smart home's level of intelligence and security and get rid of the dependence on the mobile Internet. In the next step, we can further utilize the advantages of Kinect somatosensory recognition technology to combine smart homes with smart medical care to improve people's smart living standards.

## 6 References

1. GONG Yanxue, WU Zhixia, ZHENG Shuquan, et al. Research on IOT architecture for smart community [J]. Computer Engineering and Design, 2014, 35(1):344-349 (2014)

2. Zhang Qin. Mobile internet-based smart home terminal system design and research [D]. Nanjing University of Science and Technology (2014)

3. Ajmera R, Nigam A, Gupta P. 3D Face Recognition using Kinect[C] Indian Conference on Computer Vision Graphics and Image Processing. ACM, 2014:76 (2014)

4. WU Zhiyong, DU Zhen. Research on gesture control system applied on home appliance by Kinect sensor [J]. Video Engineering, 2015, 39(16):51-53 (2015)

5. Anandakrishnan N, Baboo S S. Microsoft Kinect Sensor Technology and its Methods[J] (2016)

6. Meng Ming, Yang Fangbo, She Qingshan, et al. Human motion detection based on the depth image of Kinect[J]. Chinese Journal of Scientific Instrument, 2015, (02): 386-393 (2015)

7. Ijjina E P, Mohan C K. Facial expression recognition using kinect depth sensor and convolutional neural networks[C]. Machine Learning and Applications (ICMLA), 2014 13th International Conference on, 2014: 392-396 (2014)

8. LIANG Liming, ZHONG Zhen, CHEN Zhaoyang. Research and simulation of kernel function selection for support vector machine [J]. Computer Engineering and Science, 2015, 37(6):1135-1141 (2015)