# Development of the Interactive Rehabilitation Game System for Children with Autism Based on Game Psychology

Authors: Xufeng Ma; Jianhua Yang
Journal: Mobile Information Systems
 (2021)

---

## Abstract

Autism is a widespread neurodevelopmental disorder. The cause of autism is still unclear. Once it occurs, it will be accompanied for life. Intervention treatment is currently recognized as an effective method. With the rise of somatosensory technology, Kinect has gradually been applied to the field of autism rehabilitation with its natural human-computer interaction and game rehabilitation psychology. Because there is a great demand for autism rehabilitation training resources in China and the existing somatosensory rehabilitation equipment has some shortcomings, this study designed a set of somatosensory rehabilitation system for autistic children based on Kinect and carried out its rehabilitation effect with experimental demonstration. The specific research work mainly includes the following contents: first, aiming at the rehabilitation needs of children with autism, this article combines mainstream intervention treatment methods, using C# programming language and Windows Presentation Foundation (WPF) development technology to develop Kinect-based somatosensory game modules. The somatosensory rehabilitation game involves the training of the language communication, behavioral movement, daily life, sensory integration, and other comprehensive abilities of children with autism, which improves the shortcomings of the previous rehabilitation training in the scope of intervention. A variety of game types and diversified game modes are designed to meet the individual needs of rehabilitation training for children with autism. Second, experiments were carried out to verify the rehabilitation effect of the system. Using the Childhood Autism Rating Scale (CARS) and Pediatric Evaluation of Disability Inventory (PEDI) scale as the rehabilitation effect evaluation tools of this system, the experimental subjects were divided into the experimental group and control group, and different comparative experiments were conducted, respectively. Through the analysis of the scoring data of the CARS, it is found that the comprehensive ability of the experimental group of children receiving somatosensory rehabilitation training is significantly improved. Through the analysis of the scoring data of the PEDI scale, it is found that the children’s language communication, behavioral activities, and daily life abilities of the experimental group have been significantly enhanced. This article focuses on the design of the somatosensory game module of the autistic somatosensory rehabilitation system and the analysis of the system’s rehabilitation effect. It satisfies the training and rehabilitation needs of children with autism and has strong practical significance.


---

## Body

## 1. Introduction

Autism, also known as autism spectrum disorder (ASD), is a representative disease of generalized neurodevelopmental disorders. It starts in infants and young children. It is more common in men and is often accompanied by language barriers, social communication barriers, and narrow interests and stereotypes. According to the latest edition (5th) of theDiagnostic and Statistical Manual of Mental Disorders (DSM-V) [1], characteristics that attribute towards the core symptoms of autism are communication difficulties, social disorders, and limited, repetitive, and stereotyped behavior [2]. In recent years, the incidence of autism has shown a rapid upward trend. The US Centers for Disease Control and Prevention (CDC) pointed out in a 2010 survey report that the prevalence of autism among 8-year-old children in the United States is 1/68, of which the prevalence rate is 1/42 for men and 1/189 for women [3]. Later, in the research report released by the National Center for Health Statistics (NCHS) in 2015, at least one out of 50 children and adolescents aged 6–17 has autism [4]. It can be seen that the increasing trend of the incidence of children with autism is very obvious. According to the statistics of the World Health Organization, the number of children with autism in China only is about 600–1.8 million. Some scholars believe that the actual number of children with autism in China has reached 2.6–8 million [5–7]. Children with autism have varying degrees of obstacles in language communication and social skills, making them unable to live and learn independently, and there is no specific drug for autism. As a result, patients with autism have brought serious mental pressure to their families and, at the same time, caused a serious economic burden on the society [8]. As the cause of autism is unknown and effective treatment methods are unavailable, moreover, once a patient exhibits symptoms, they will persist indefinitely and cannot be cured; therefore, only active intervention training can effectively improve autistic children’s symptoms and enable them to integrate into their families and society as soon as possible [9–11].Somatosensory technology can directly interact with surrounding equipment or the environment by controlling body behavior without the use of other control devices. In recent years, somatosensory technology has achieved rapid development, and a number of popular products have emerged. Among them, the EyeToy launched by the Japanese company Sony in 2004 mainly uses optical sensors to collect human images. By comparing two adjacent frames of images, it can perceive the changes in human movements and then compare the body movements of the collected human images with the interactive game content. The EyeToy somatosensory sensor recognizes the movement of the human body through the processing of two-dimensional images. It is for this reason that the accuracy of human body recognition is not high, which limits the extensive development of this sensor in the field of medical rehabilitation [12]. The Wii, which was launched by Nintendo in Japan in 2006, is a combined inertial and optical sensing kit. It is necessary to install a gravity sensor and an infrared sensor on the operating handle to collect the triaxle acceleration of the hand and sense infrared signals. Control the spatial mouse by controlling the displacement of the hand in the vertical and horizontal directions [13]. The Kinect somatosensory sensor released by Microsoft in the United States in 2010 captures the posture and motion information of the human body through laser emission and reception. It can easily achieve good somatosensory interaction effects without the aid of other control devices. Kinect uses structured light technology to complete the coding and arrangement of lasers in the field of view space, and then receive the reflected light through the photosensitive chip, process and analyze the reflected light, and extract the human body information in the field of view, thereby realizing the recognition of the human body’s movements and postures, and it has excellent human body recognition accuracy [14]. In summary, the EyeToy lacks in the processing accuracy of optical images. Wii requires external control equipment for human-computer interaction, while Kinect somatosensory sensor can directly recognize human body posture and has excellent recognition accuracy. One’s own body control can meet the needs of human-computer interaction. Therefore, Kinect’s human-computer interaction method can bring a better interactive experience [15]. In recent years, due to the advantages of simple human-computer interaction and excellent virtual reality experience, somatosensory technology has been widely used in education and medical rehabilitation [16–19]. With the continuous development of somatosensory technology and the increasingly prominent problems of children with autism, somatosensory games have been gradually applied to the intervention treatment of children with autism, and good rehabilitation effects have been achieved [20]. In the rehabilitation training of children with autism, somatosensory games have shown their strong plasticity, providing a new way for the rehabilitation of children with autism. The rest of the paper is structured as follows: in Section 2, related work in the similar domain is presented followed by the method, Section 3, presenting the somatosensory game development platform, its infrastructure and environment, somatosensory game development technology, and realization as the fruit pickup game, object recognition game, command operation game, and balloon pop-up game. In Section 4, experiments and discussions, subject group distribution, evaluation scales, and results are discussed. In Section 5, conclusion is given.


## 2. Related Work

The improvement of motor skills can effectively alleviate the condition of children with autism [21]. In the research of Bartoli et al. [22] and Garzotto et al. [23], a number of ball games were designed by simulating the situation of various ball sports. These games are designed for the athletic ability of children with autism. Children with autism need to move their bodies and control their limbs to hit or avoid the spheres in the picture and use this exercise method to enhance the exercise level. In addition, they have developed a game for the motor skills of children with autism. In the game, children with autism need to control their bodies to capture bubbles on-screen. This game can exercise the motor skills of the autistic children’s limbs, physical coordination, and attention and sense of spatial orientation [24]. Dickinson and Place [25] used Wii sensor-based motion somatosensory games to train the athletic ability and physical fitness of children with autism. In their intervention effect test, 50 children with autism were selected as the experimental group and the control group. The experimental group participated in sports and somatosensory game training, while the control group did not participate. Finally, according to the results of physical fitness assessment, it was found that the physical fitness of the children in the experimental group had been significantly improved. Somatosensory games have also been used to train autistic children’s abilities in language communication, social communication, and imitation. Language communication and social communication skills are necessary skills for children with autism to return to the society. Therefore, it is particularly important to intervene in the language and communication skills of children with autism. Garzotto et al. [26] used image overlap games to improve the social interaction ability of children with autism. In the game, children need to cooperate with their peers to adjust their body posture to match the image displayed on the screen. The research results show that, as the number of games for children with autism increases, the time required to complete the game is gradually shortened, and the overall score is also steadily increasing. The self-model autism intervention system based on Kinect designed by Uzuegbunam [27] achieves the purpose of language and behavior intervention through the imitation of actions and language. In the game, the image of a cartoon character appears on the screen, and the cartoon character makes a certain action and pronounces it. Children with autism are asked to imitate the actions and pronunciation of the cartoon character and achieve self-improvement through self-modeling. Boutsika [28] indicated that the Kinect adventure game can be used to intervene and train the cognitive and matching abilities of children with autism. The Lakeside Autism Training Center in the United States uses somatosensory games to train the cognitive and matching abilities of children with autism [29]. In the autistic somatosensory training game designed by Akhan, color matching and picture classification games also use a similar method to intervene training children with autism [30]. The color matching game requires children with autism to move balls of different colors to the corresponding color boxes, while the picture classification game requires children with autism to correctly select the four pictures on the screen that are different from the other three categories. Both are to train the cognitive understanding and hand-eye coordination ability of children with autism, and they are also helpful to improve the daily life skills of children with autism.


## 3. Method

The proprioception rehabilitation system is based on the rehabilitation training of autistic children, and the core is the somatosensory game module. The overall framework of the system is shown in Figure1. This system is application software designed based on the Windows operating system computer. The system uses Microsoft’s Kinect sensor as the input device of the system. Kinect transfers various data streams captured to the computer through the USB port. The system processes the input information and converts it into visual effects to realize games with children with autism interactive function. While children with autism receive somatosensory game training, the system uploads the data of children’s participation in the game to the server’s database and saves it.Figure 1
System framework.


### 3.1. Somatosensory Game Development Platform

This system uses the Kinect for Windows somatosensory sensor device developed by Microsoft. It is a 3D somatosensory device that can realize the functions of human motion capture, image recognition, voice input, and voice recognition. Kinect somatosensory equipment applies structured light measurement principle technology, uses laser to encode and arrange the measurement space, projects the structured laser on the object within the viewing angle, and then receives the reflected light through the photosensitive chip. Three-dimensional spatial information such as the shape and depth of the scene can be collected by the three-dimensional imaging technology of structured light, and the human body information in the field of view can be extracted, so as to realize the recognition of the human body’s movement and posture.


#### 3.1.1. Kinect Hardware Composition

All applications of the proprietary game module are developed and designed on the basis of the Kinect generation sensor, and its appearance structure is shown in Figure2.Figure 2
Kinect sensor’s location diagram.Hardware of the Kinect sensor mainly includes the following parts: (1) the PS1080 chip is the core chip for Kinect to realize the main functions. It controls the capture of color image information and depth information. (2) The color camera is responsible for the shooting of the scene image within the view angle of Kinect and is the unit for generating the color image data stream. (3) Infrared transmitter generates and actively projects infrared light. Speckles will be generated when the spectrum shines on different objects, which are then read by using the infrared camera. (4) The infrared camera receives the reflected speckle and processes and analyzes the speckle information to obtain the depth image under the Kinect perspective. (5) The microphone array includes four microphone units, which can filter background noise when collecting sounds and can locate the sound source through the four microphone units. (6) The elevation angle drive motor is used to control the elevation angle of Kinect to achieve the best visual field effect.


#### 3.1.2. Kinect Development Environment

Kinect for Windows is an external physical device, so when Kinect for Windows is connected to a computer for development, it needs to be used with the Kinect for Windows software development kit (SDK). Kinect for Windows SDK is mainly designed for Windows 7 system, allowing developers to use C++, C#, and other programming languages with Microsoft Visual Studio tools to develop corresponding applications. The Kinect for Windows SDK can realize the functions of processing color image data stream, depth data stream, and audio data stream and also provides underlying APIs that support technologies such as bone tracking, identity recognition, and speech recognition. The stack structure of the Kinect for Windows SDK API is shown in Figure3.Figure 3
Kinect for Windows SDK API stack structure.Kinect, as a somatosensory sensor, is the focus of somatosensory game development for the target object’s gesture and motion recognition and speech recognition.


### 3.2. Somatosensory Game Development Technology

#### 3.2.1. Color Image Display

The previous section introduced the initialization method of the color image data stream and the method of registering the synchronization event of the color image data stream. This section focuses on the display of the color image, that is, the method of displaying the picture captured by the Kinect color camera on the screen. The image data stream of the Kinect device has a variety of image formats to choose from. After the color image data stream is enabled, the image data stream is obtained in the event processing function, and the obtained data stream is sent to the image control for display. The following describes the event processing method in detail: first, call the color camera, copy the color image data stream to the pixel array variable, and set the step size of the color image according to the image format; then, convert the pixel information in the one-dimensional array into a two-dimensional pixel matrix; finally, the pixel information of the two-dimensional matrix is sent to the image control to display the color image.


#### 3.2.2. Extract Bone Coordinates

Kinect can use infrared transmitters and infrared cameras to generate three-dimensional depth data. After the Kinect SDK obtains the infrared data from the infrared camera, it generates depth image data through complex calculation processing and then extracts the human bone information through the depth data. The skeleton information includes the specific identifier of each skeleton node in the three-dimensional space coordinate system and the corresponding three-dimensional coordinate data. The Kinect generation sensor can detect and recognize 20 different bone joint points of the human body. These bone nodes cover the main information of the human body’s posture. The human body motion model can be abstracted through the coordinates of the bone joint points. Kinect acquires color image data.The acquisition of bone data is similar to that of color images. Each frame of data is a collection of human bone information. When bone data appear, a corresponding event will be triggered. Before starting the sensor, one needs to define the frame array of bone data, initialize the data stream, and register the object. At this point, the initialization setting of the event that Kinect recognizes human bones has been completed, and the coordinate information of each bone node can be directly obtained.


#### 3.2.3. NUI Coordinate Conversion

There are three coordinate forms in the API provided by the Kinect for Windows SDK, which are two-dimensional color image coordinates, three-dimensional depth image coordinates, and three-dimensional bone space coordinates. At any position within the field of view, the expressions in the three coordinate systems are different, and the measurement methods are also different. Therefore, the development of applications that use position coordinates requires coordinate conversion of NUI to achieve the unity of the coordinate system. The acquisition of the human bone coordinates introduced in the previous section is to use the NUI coordinate conversion method to convert the three-dimensional coordinates under the depth data into the three-dimensional coordinates under the skeletal space coordinates.The origin of the coordinates of the two-dimensional color image is the center of the color camera, and the unit is pixel. The origin of the coordinates of the three-dimensional depth image is the center of the infrared camera, and the unit is millimeters. In the 3D bone space coordinates, the origin of the coordinates is the center of the infrared camera, and the unit is meters.The Kinect for Windows SDK provides 4 methods to realize the mutual conversion between color image coordinates, depth image coordinates, and bone space coordinates. In addition to the three types of coordinates provided by the Kinect SDK, screen coordinates are also used in the actual application development. They are the two-dimensional plane coordinates of the screen. The origin is the upper left corner, and the unit of measurement is pixels. The difference between the screen coordinate system and the color image coordinate system is determined. The position of the origin is different, so the coordinate conversion between the color image coordinate and the screen coordinate requires a mathematical geometric transformation.


#### 3.2.4. Skeleton Tracking Technology

Skeleton tracking is a necessary method in somatosensory game development, and it is also a basic technical requirement. Skeleton tracking is the basis for somatosensory games and motion recognition. In order to introduce the skeletal tracking technology in a more popular way, this section takes the effect of “carrying basket” as an example to explain the skeletal tracking technology in detail.“Hand Basket” means that the basket (image control) locks and tracks the coordinates of the bone nodes of the hands. First, extract the skeletal node coordinates of both hands; secondly, map the skeletal coordinates of both hands to the coordinates of the color image; again, transform the coordinates under the color image into screen coordinates through geometric transformation; finally, assign the screen coordinates of the hand nodes, respectively. Give two baskets so that the coordinate position of the basket coincides with the position of the hands, and the effect of the “hand-held basket” is realized. The main process of skeletal tracking is shown in Figure4.Figure 4
Process of skeletal tracking.


#### 3.2.5. Touch Detection of the Body and Props in the Screen

In somatosensory sports games, the game is often played by capturing virtual items in the screen, which involves the detection of the touch between the human body and the props in the screen. The following takes fruit (image control) captured by the left hand of the human body as an example to introduce the touch detection between the left-hand bone node and fruit. First, define a private distance threshold, and set the threshold size. If the distance between LeftHand (the coordinates of the left-hand bone node in the screen coordinate system) and applepoint (the fruit’s screen coordinates) on the horizontal axis and vertical axis is less than this threshold, it is determined that the fruit has been captured successfully.


#### 3.2.6. Action Recognition

All movements of the human body are completed through the combination of various bone joints. Therefore, the recognition movement can directly recognize the joint data and other related information, so the movement of the human body can be feature extracted by simplifying the human body action, directly extracting the key features of the action for judgment, and then combining with the node data to form an action recognition method.Obtain the coordinates of two bone nodes through Kinect, and then calculate the corresponding distance between the two points. These distance data can not only directly identify certain human actions but also continue to process these distance data to calculate angle features, which can also achieve the purpose of action recognition. By processing and analyzing the acquired bone data, a more accurate recognition of human movements can be obtained. When the system detects human movements, it uses information such as the three-dimensional coordinates of the bone nodes, the distance, and the angle between the bone joint points to determine the behavior state of the human body. The core idea of human action recognition is by abstracting and simplifying the target action and extracting features and matching the extracted features with the real-time skeletal node information of the human body, so as to identify and judge human actions.This system only considers the behavior recognition of a single human object and does not consider the situation of multiple human objects. For the recognition of limb movements, it can be realized by the method of feature extraction. All the nodes are marked with three-dimensional space coordinates to obtain the position parameters, and the three-dimensional parameters are used as the judgment result based on the node characteristics. For the detection of squat action, from a defined point of view, the leg is bent. This system also mainly detects whether the legs are bent to a certain extent and judges whether the angle between the hips, knees, and ankles is less than 150 degrees. If it is less, then there is a squat because under normal circumstances, between the three points, the included angle is approximately 180 degrees, or check the distance between the hip and the knee on theY-axis to judge. If the distance is less than 0.2 meters, it is considered to have squat behavior. Regarding the detection of jumping action, when the feet leave the ground when jumping, the body position will move up quickly. By detecting whether the height of the feet off the ground exceeds a given threshold or by detecting whether the increment of the body’s center of gravity on the Y-axis in the adjacent two frames of data is greater than a given threshold, the jumping action can be judged. By judging the change in the position of the center of gravity, we can determine whether the child has completed the corresponding instruction task.


#### 3.2.7. Speech Recognition

The reason why Kinect’s microphone array can automatically acquire factors and signals and process them accordingly is mainly realized by the KinectAudioSource object. The main function of KinectAudioSource is to provide audio data streams for speech recognition events. In addition, it also has the ability to record audio files. It mainly introduces the audio data processing method of KinectAudioSource. KinectAudioSource can make the microphone array of the Kinect device work in different modes through relevant configuration and can also detect which direction the voice signal comes from and can set to accept the voice signal from a fixed direction. KinectAudioSource can also perform a series of processing on the received audio information through a specific algorithm to improve audio quality. These processing include echo suppression and cancellation, automatic gain control, beamforming, and noise suppression. Echo cancellation can filter out specific audio from the audio signal and effectively suppress and eliminate echo. Automatic gain control keeps the amplitude and time of the audio signal synchronized. Beamforming is used to determine the direction of the audio signal source. Noise suppression can effectively reduce environmental noise and make the target audio clearer.


### 3.3. Realization of the Somatosensory Game Module

The somatosensory game module of this system includes four games, fruit-picking game, picture-recognizing game, instruction game, and balloon-topping game, which are designed for the training of different aspects of autistic children.


#### 3.3.1. Pickup Fruit

The fruit-receiving game module is an interactive somatosensory game training module designed mainly for the defects of motor skills and sensory integration ability of children with autism. The main purpose of this game module is to train the brains of children with autism to control each body unit. In the game, when the Kinect sensor detects the human body, the two fruit baskets track the children’s hands separately, and the fruits start to fall at an initial speed from a random position on the top of the screen. The child needs to move his hands and body position to pick up the fallen fruits. The parameters such as “drop speed,” “number of drops,” “number received,” and “time” are displayed on the screen. In the course of the game, each round of receiving fruit has only two results: receiving fruit and not receiving fruit. If the child receives fruit, the values of the “number of drops,” “number of received,” and “drop speed” will be increased by 1, and the received fruit will disappear, and a new fruit will be at the top of the screen. Fruits appear randomly and then fall at a faster speed, and the children start a new round of fruit picking games; if the child fails to pick up the fruit before the fruit falls out of the lower end of the screen, then the “number of drops” will increase by 1, “number of received” remains unchanged, “falling speed” is reduced by 1, and then a new fruit appears at a random position at the top of the screen, starts to fall at a slower speed, and starts a new round of receiving fruits.


#### 3.3.2. Image Identification

The game module for viewing pictures and objects is an interactive game training module designed mainly for children with autism in terms of language communication and life skills. The primary goal of this game module is to develop children’s listening, speaking, and other language communication skills, as well as their cognitive, life, and attention abilities. It is also effective in raising IQ levels. When the game starts, a picture of an item is randomly displayed on the screen, displaying two Chinese prompts: “Please tell me the item name in the picture” and “Voice prompt inX seconds;” at the same time, the voice reads “Please speak the item name in the picture,” counting down from 10 seconds. If the system detects the correct pronunciation of the item name within 10 seconds, it will stop the countdown, play the voice encouragement, give a short voice introduction to the item, and then display the next picture of the item. If the system still does not detect the correct pronunciation of the item within 10 seconds, the system will give a voice prompt, read the voice of the item name three times in succession, then give a voice introduction to the item, and then display the next picture.


#### 3.3.3. Listen to Instructions

The instructional game is designed to improve the sensory integration ability, motor ability, and executive ability of children with autism. In this game, the training of sensory integration ability is mainly manifested as the brain’s ability to receive and process visual and auditory signals and then control the body to complete certain behavioral activities. Executive ability is a skill necessary for people with imperfect capacity in life and refers to the ability of a child to complete a certain behavior activity under the guidance of a guardian. After the game starts, the system simultaneously issues commands through subtitles and voice and starts a 60-second countdown at the same time, such as “please raise your hands,” “please jump,” and “please take a step forward”. If the system detects that the child completes the corresponding action through bone tracking, it uses voice and vision to encourage and strengthen, rests for a while, and then issues the next instruction. If the child reacts incorrectly or does not respond after the instruction is given, the guardian will immediately correct or assist in the demonstration until the child completes the corresponding action; if the child has not completed the corresponding action at the end of the countdown, switch to the next instruction.


#### 3.3.4. Top Balloon

In the top balloon game, children need to find the position of the balloon and break the balloon with their head by jumping. The main purpose of designing this game is to train the coordination and balance of the limbs by jumping, promote the healthy development of the left and right brains, maintain muscle vitality, enhance physical resistance, consume children’s excess physical strength, and relieve emotions. In the game, when the Kinect sensor detects the human body, the balloon appears from a random position at the top of the screen, and a 30-second countdown starts. The child needs to track the position of the balloon and then jump to break the balloon with his head. If the balloon is touched within 30 seconds, the balloon disappears, and the corresponding encouragement is given, and then the next balloon randomly appears at the top of the screen. If the child cannot touch the balloon at the top of the screen by jumping, the distance between the child and the Kinect sensor should be appropriately shortened under the guidance of the guardian, so as to shorten the distance between the head and the balloon. If the child has not touched the balloon at the top of the screen by jumping at the end of the countdown, the balloon will disappear, and the next balloon will appear randomly. The two parameters “time” and “balloon number” are displayed at the top of the screen. Within the set time, the value of “time” starts to count down; each time a balloon is touched, the value of “balloons” will increase by 1, and after the final countdown value of “balloons” will be used as the game score.


## 3.1. Somatosensory Game Development Platform

This system uses the Kinect for Windows somatosensory sensor device developed by Microsoft. It is a 3D somatosensory device that can realize the functions of human motion capture, image recognition, voice input, and voice recognition. Kinect somatosensory equipment applies structured light measurement principle technology, uses laser to encode and arrange the measurement space, projects the structured laser on the object within the viewing angle, and then receives the reflected light through the photosensitive chip. Three-dimensional spatial information such as the shape and depth of the scene can be collected by the three-dimensional imaging technology of structured light, and the human body information in the field of view can be extracted, so as to realize the recognition of the human body’s movement and posture.


### 3.1.1. Kinect Hardware Composition

All applications of the proprietary game module are developed and designed on the basis of the Kinect generation sensor, and its appearance structure is shown in Figure2.Figure 2
Kinect sensor’s location diagram.Hardware of the Kinect sensor mainly includes the following parts: (1) the PS1080 chip is the core chip for Kinect to realize the main functions. It controls the capture of color image information and depth information. (2) The color camera is responsible for the shooting of the scene image within the view angle of Kinect and is the unit for generating the color image data stream. (3) Infrared transmitter generates and actively projects infrared light. Speckles will be generated when the spectrum shines on different objects, which are then read by using the infrared camera. (4) The infrared camera receives the reflected speckle and processes and analyzes the speckle information to obtain the depth image under the Kinect perspective. (5) The microphone array includes four microphone units, which can filter background noise when collecting sounds and can locate the sound source through the four microphone units. (6) The elevation angle drive motor is used to control the elevation angle of Kinect to achieve the best visual field effect.


### 3.1.2. Kinect Development Environment

Kinect for Windows is an external physical device, so when Kinect for Windows is connected to a computer for development, it needs to be used with the Kinect for Windows software development kit (SDK). Kinect for Windows SDK is mainly designed for Windows 7 system, allowing developers to use C++, C#, and other programming languages with Microsoft Visual Studio tools to develop corresponding applications. The Kinect for Windows SDK can realize the functions of processing color image data stream, depth data stream, and audio data stream and also provides underlying APIs that support technologies such as bone tracking, identity recognition, and speech recognition. The stack structure of the Kinect for Windows SDK API is shown in Figure3.Figure 3
Kinect for Windows SDK API stack structure.Kinect, as a somatosensory sensor, is the focus of somatosensory game development for the target object’s gesture and motion recognition and speech recognition.


## 3.1.1. Kinect Hardware Composition

All applications of the proprietary game module are developed and designed on the basis of the Kinect generation sensor, and its appearance structure is shown in Figure2.Figure 2
Kinect sensor’s location diagram.Hardware of the Kinect sensor mainly includes the following parts: (1) the PS1080 chip is the core chip for Kinect to realize the main functions. It controls the capture of color image information and depth information. (2) The color camera is responsible for the shooting of the scene image within the view angle of Kinect and is the unit for generating the color image data stream. (3) Infrared transmitter generates and actively projects infrared light. Speckles will be generated when the spectrum shines on different objects, which are then read by using the infrared camera. (4) The infrared camera receives the reflected speckle and processes and analyzes the speckle information to obtain the depth image under the Kinect perspective. (5) The microphone array includes four microphone units, which can filter background noise when collecting sounds and can locate the sound source through the four microphone units. (6) The elevation angle drive motor is used to control the elevation angle of Kinect to achieve the best visual field effect.


## 3.1.2. Kinect Development Environment

Kinect for Windows is an external physical device, so when Kinect for Windows is connected to a computer for development, it needs to be used with the Kinect for Windows software development kit (SDK). Kinect for Windows SDK is mainly designed for Windows 7 system, allowing developers to use C++, C#, and other programming languages with Microsoft Visual Studio tools to develop corresponding applications. The Kinect for Windows SDK can realize the functions of processing color image data stream, depth data stream, and audio data stream and also provides underlying APIs that support technologies such as bone tracking, identity recognition, and speech recognition. The stack structure of the Kinect for Windows SDK API is shown in Figure3.Figure 3
Kinect for Windows SDK API stack structure.Kinect, as a somatosensory sensor, is the focus of somatosensory game development for the target object’s gesture and motion recognition and speech recognition.


## 3.2. Somatosensory Game Development Technology

### 3.2.1. Color Image Display

The previous section introduced the initialization method of the color image data stream and the method of registering the synchronization event of the color image data stream. This section focuses on the display of the color image, that is, the method of displaying the picture captured by the Kinect color camera on the screen. The image data stream of the Kinect device has a variety of image formats to choose from. After the color image data stream is enabled, the image data stream is obtained in the event processing function, and the obtained data stream is sent to the image control for display. The following describes the event processing method in detail: first, call the color camera, copy the color image data stream to the pixel array variable, and set the step size of the color image according to the image format; then, convert the pixel information in the one-dimensional array into a two-dimensional pixel matrix; finally, the pixel information of the two-dimensional matrix is sent to the image control to display the color image.


### 3.2.2. Extract Bone Coordinates

Kinect can use infrared transmitters and infrared cameras to generate three-dimensional depth data. After the Kinect SDK obtains the infrared data from the infrared camera, it generates depth image data through complex calculation processing and then extracts the human bone information through the depth data. The skeleton information includes the specific identifier of each skeleton node in the three-dimensional space coordinate system and the corresponding three-dimensional coordinate data. The Kinect generation sensor can detect and recognize 20 different bone joint points of the human body. These bone nodes cover the main information of the human body’s posture. The human body motion model can be abstracted through the coordinates of the bone joint points. Kinect acquires color image data.The acquisition of bone data is similar to that of color images. Each frame of data is a collection of human bone information. When bone data appear, a corresponding event will be triggered. Before starting the sensor, one needs to define the frame array of bone data, initialize the data stream, and register the object. At this point, the initialization setting of the event that Kinect recognizes human bones has been completed, and the coordinate information of each bone node can be directly obtained.


### 3.2.3. NUI Coordinate Conversion

There are three coordinate forms in the API provided by the Kinect for Windows SDK, which are two-dimensional color image coordinates, three-dimensional depth image coordinates, and three-dimensional bone space coordinates. At any position within the field of view, the expressions in the three coordinate systems are different, and the measurement methods are also different. Therefore, the development of applications that use position coordinates requires coordinate conversion of NUI to achieve the unity of the coordinate system. The acquisition of the human bone coordinates introduced in the previous section is to use the NUI coordinate conversion method to convert the three-dimensional coordinates under the depth data into the three-dimensional coordinates under the skeletal space coordinates.The origin of the coordinates of the two-dimensional color image is the center of the color camera, and the unit is pixel. The origin of the coordinates of the three-dimensional depth image is the center of the infrared camera, and the unit is millimeters. In the 3D bone space coordinates, the origin of the coordinates is the center of the infrared camera, and the unit is meters.The Kinect for Windows SDK provides 4 methods to realize the mutual conversion between color image coordinates, depth image coordinates, and bone space coordinates. In addition to the three types of coordinates provided by the Kinect SDK, screen coordinates are also used in the actual application development. They are the two-dimensional plane coordinates of the screen. The origin is the upper left corner, and the unit of measurement is pixels. The difference between the screen coordinate system and the color image coordinate system is determined. The position of the origin is different, so the coordinate conversion between the color image coordinate and the screen coordinate requires a mathematical geometric transformation.


### 3.2.4. Skeleton Tracking Technology

Skeleton tracking is a necessary method in somatosensory game development, and it is also a basic technical requirement. Skeleton tracking is the basis for somatosensory games and motion recognition. In order to introduce the skeletal tracking technology in a more popular way, this section takes the effect of “carrying basket” as an example to explain the skeletal tracking technology in detail.“Hand Basket” means that the basket (image control) locks and tracks the coordinates of the bone nodes of the hands. First, extract the skeletal node coordinates of both hands; secondly, map the skeletal coordinates of both hands to the coordinates of the color image; again, transform the coordinates under the color image into screen coordinates through geometric transformation; finally, assign the screen coordinates of the hand nodes, respectively. Give two baskets so that the coordinate position of the basket coincides with the position of the hands, and the effect of the “hand-held basket” is realized. The main process of skeletal tracking is shown in Figure4.Figure 4
Process of skeletal tracking.


### 3.2.5. Touch Detection of the Body and Props in the Screen

In somatosensory sports games, the game is often played by capturing virtual items in the screen, which involves the detection of the touch between the human body and the props in the screen. The following takes fruit (image control) captured by the left hand of the human body as an example to introduce the touch detection between the left-hand bone node and fruit. First, define a private distance threshold, and set the threshold size. If the distance between LeftHand (the coordinates of the left-hand bone node in the screen coordinate system) and applepoint (the fruit’s screen coordinates) on the horizontal axis and vertical axis is less than this threshold, it is determined that the fruit has been captured successfully.


### 3.2.6. Action Recognition

All movements of the human body are completed through the combination of various bone joints. Therefore, the recognition movement can directly recognize the joint data and other related information, so the movement of the human body can be feature extracted by simplifying the human body action, directly extracting the key features of the action for judgment, and then combining with the node data to form an action recognition method.Obtain the coordinates of two bone nodes through Kinect, and then calculate the corresponding distance between the two points. These distance data can not only directly identify certain human actions but also continue to process these distance data to calculate angle features, which can also achieve the purpose of action recognition. By processing and analyzing the acquired bone data, a more accurate recognition of human movements can be obtained. When the system detects human movements, it uses information such as the three-dimensional coordinates of the bone nodes, the distance, and the angle between the bone joint points to determine the behavior state of the human body. The core idea of human action recognition is by abstracting and simplifying the target action and extracting features and matching the extracted features with the real-time skeletal node information of the human body, so as to identify and judge human actions.This system only considers the behavior recognition of a single human object and does not consider the situation of multiple human objects. For the recognition of limb movements, it can be realized by the method of feature extraction. All the nodes are marked with three-dimensional space coordinates to obtain the position parameters, and the three-dimensional parameters are used as the judgment result based on the node characteristics. For the detection of squat action, from a defined point of view, the leg is bent. This system also mainly detects whether the legs are bent to a certain extent and judges whether the angle between the hips, knees, and ankles is less than 150 degrees. If it is less, then there is a squat because under normal circumstances, between the three points, the included angle is approximately 180 degrees, or check the distance between the hip and the knee on theY-axis to judge. If the distance is less than 0.2 meters, it is considered to have squat behavior. Regarding the detection of jumping action, when the feet leave the ground when jumping, the body position will move up quickly. By detecting whether the height of the feet off the ground exceeds a given threshold or by detecting whether the increment of the body’s center of gravity on the Y-axis in the adjacent two frames of data is greater than a given threshold, the jumping action can be judged. By judging the change in the position of the center of gravity, we can determine whether the child has completed the corresponding instruction task.


### 3.2.7. Speech Recognition

The reason why Kinect’s microphone array can automatically acquire factors and signals and process them accordingly is mainly realized by the KinectAudioSource object. The main function of KinectAudioSource is to provide audio data streams for speech recognition events. In addition, it also has the ability to record audio files. It mainly introduces the audio data processing method of KinectAudioSource. KinectAudioSource can make the microphone array of the Kinect device work in different modes through relevant configuration and can also detect which direction the voice signal comes from and can set to accept the voice signal from a fixed direction. KinectAudioSource can also perform a series of processing on the received audio information through a specific algorithm to improve audio quality. These processing include echo suppression and cancellation, automatic gain control, beamforming, and noise suppression. Echo cancellation can filter out specific audio from the audio signal and effectively suppress and eliminate echo. Automatic gain control keeps the amplitude and time of the audio signal synchronized. Beamforming is used to determine the direction of the audio signal source. Noise suppression can effectively reduce environmental noise and make the target audio clearer.


## 3.2.1. Color Image Display

The previous section introduced the initialization method of the color image data stream and the method of registering the synchronization event of the color image data stream. This section focuses on the display of the color image, that is, the method of displaying the picture captured by the Kinect color camera on the screen. The image data stream of the Kinect device has a variety of image formats to choose from. After the color image data stream is enabled, the image data stream is obtained in the event processing function, and the obtained data stream is sent to the image control for display. The following describes the event processing method in detail: first, call the color camera, copy the color image data stream to the pixel array variable, and set the step size of the color image according to the image format; then, convert the pixel information in the one-dimensional array into a two-dimensional pixel matrix; finally, the pixel information of the two-dimensional matrix is sent to the image control to display the color image.


## 3.2.2. Extract Bone Coordinates

Kinect can use infrared transmitters and infrared cameras to generate three-dimensional depth data. After the Kinect SDK obtains the infrared data from the infrared camera, it generates depth image data through complex calculation processing and then extracts the human bone information through the depth data. The skeleton information includes the specific identifier of each skeleton node in the three-dimensional space coordinate system and the corresponding three-dimensional coordinate data. The Kinect generation sensor can detect and recognize 20 different bone joint points of the human body. These bone nodes cover the main information of the human body’s posture. The human body motion model can be abstracted through the coordinates of the bone joint points. Kinect acquires color image data.The acquisition of bone data is similar to that of color images. Each frame of data is a collection of human bone information. When bone data appear, a corresponding event will be triggered. Before starting the sensor, one needs to define the frame array of bone data, initialize the data stream, and register the object. At this point, the initialization setting of the event that Kinect recognizes human bones has been completed, and the coordinate information of each bone node can be directly obtained.


## 3.2.3. NUI Coordinate Conversion

There are three coordinate forms in the API provided by the Kinect for Windows SDK, which are two-dimensional color image coordinates, three-dimensional depth image coordinates, and three-dimensional bone space coordinates. At any position within the field of view, the expressions in the three coordinate systems are different, and the measurement methods are also different. Therefore, the development of applications that use position coordinates requires coordinate conversion of NUI to achieve the unity of the coordinate system. The acquisition of the human bone coordinates introduced in the previous section is to use the NUI coordinate conversion method to convert the three-dimensional coordinates under the depth data into the three-dimensional coordinates under the skeletal space coordinates.The origin of the coordinates of the two-dimensional color image is the center of the color camera, and the unit is pixel. The origin of the coordinates of the three-dimensional depth image is the center of the infrared camera, and the unit is millimeters. In the 3D bone space coordinates, the origin of the coordinates is the center of the infrared camera, and the unit is meters.The Kinect for Windows SDK provides 4 methods to realize the mutual conversion between color image coordinates, depth image coordinates, and bone space coordinates. In addition to the three types of coordinates provided by the Kinect SDK, screen coordinates are also used in the actual application development. They are the two-dimensional plane coordinates of the screen. The origin is the upper left corner, and the unit of measurement is pixels. The difference between the screen coordinate system and the color image coordinate system is determined. The position of the origin is different, so the coordinate conversion between the color image coordinate and the screen coordinate requires a mathematical geometric transformation.


## 3.2.4. Skeleton Tracking Technology

Skeleton tracking is a necessary method in somatosensory game development, and it is also a basic technical requirement. Skeleton tracking is the basis for somatosensory games and motion recognition. In order to introduce the skeletal tracking technology in a more popular way, this section takes the effect of “carrying basket” as an example to explain the skeletal tracking technology in detail.“Hand Basket” means that the basket (image control) locks and tracks the coordinates of the bone nodes of the hands. First, extract the skeletal node coordinates of both hands; secondly, map the skeletal coordinates of both hands to the coordinates of the color image; again, transform the coordinates under the color image into screen coordinates through geometric transformation; finally, assign the screen coordinates of the hand nodes, respectively. Give two baskets so that the coordinate position of the basket coincides with the position of the hands, and the effect of the “hand-held basket” is realized. The main process of skeletal tracking is shown in Figure4.Figure 4
Process of skeletal tracking.


## 3.2.5. Touch Detection of the Body and Props in the Screen

In somatosensory sports games, the game is often played by capturing virtual items in the screen, which involves the detection of the touch between the human body and the props in the screen. The following takes fruit (image control) captured by the left hand of the human body as an example to introduce the touch detection between the left-hand bone node and fruit. First, define a private distance threshold, and set the threshold size. If the distance between LeftHand (the coordinates of the left-hand bone node in the screen coordinate system) and applepoint (the fruit’s screen coordinates) on the horizontal axis and vertical axis is less than this threshold, it is determined that the fruit has been captured successfully.


## 3.2.6. Action Recognition

All movements of the human body are completed through the combination of various bone joints. Therefore, the recognition movement can directly recognize the joint data and other related information, so the movement of the human body can be feature extracted by simplifying the human body action, directly extracting the key features of the action for judgment, and then combining with the node data to form an action recognition method.Obtain the coordinates of two bone nodes through Kinect, and then calculate the corresponding distance between the two points. These distance data can not only directly identify certain human actions but also continue to process these distance data to calculate angle features, which can also achieve the purpose of action recognition. By processing and analyzing the acquired bone data, a more accurate recognition of human movements can be obtained. When the system detects human movements, it uses information such as the three-dimensional coordinates of the bone nodes, the distance, and the angle between the bone joint points to determine the behavior state of the human body. The core idea of human action recognition is by abstracting and simplifying the target action and extracting features and matching the extracted features with the real-time skeletal node information of the human body, so as to identify and judge human actions.This system only considers the behavior recognition of a single human object and does not consider the situation of multiple human objects. For the recognition of limb movements, it can be realized by the method of feature extraction. All the nodes are marked with three-dimensional space coordinates to obtain the position parameters, and the three-dimensional parameters are used as the judgment result based on the node characteristics. For the detection of squat action, from a defined point of view, the leg is bent. This system also mainly detects whether the legs are bent to a certain extent and judges whether the angle between the hips, knees, and ankles is less than 150 degrees. If it is less, then there is a squat because under normal circumstances, between the three points, the included angle is approximately 180 degrees, or check the distance between the hip and the knee on theY-axis to judge. If the distance is less than 0.2 meters, it is considered to have squat behavior. Regarding the detection of jumping action, when the feet leave the ground when jumping, the body position will move up quickly. By detecting whether the height of the feet off the ground exceeds a given threshold or by detecting whether the increment of the body’s center of gravity on the Y-axis in the adjacent two frames of data is greater than a given threshold, the jumping action can be judged. By judging the change in the position of the center of gravity, we can determine whether the child has completed the corresponding instruction task.


## 3.2.7. Speech Recognition

The reason why Kinect’s microphone array can automatically acquire factors and signals and process them accordingly is mainly realized by the KinectAudioSource object. The main function of KinectAudioSource is to provide audio data streams for speech recognition events. In addition, it also has the ability to record audio files. It mainly introduces the audio data processing method of KinectAudioSource. KinectAudioSource can make the microphone array of the Kinect device work in different modes through relevant configuration and can also detect which direction the voice signal comes from and can set to accept the voice signal from a fixed direction. KinectAudioSource can also perform a series of processing on the received audio information through a specific algorithm to improve audio quality. These processing include echo suppression and cancellation, automatic gain control, beamforming, and noise suppression. Echo cancellation can filter out specific audio from the audio signal and effectively suppress and eliminate echo. Automatic gain control keeps the amplitude and time of the audio signal synchronized. Beamforming is used to determine the direction of the audio signal source. Noise suppression can effectively reduce environmental noise and make the target audio clearer.


## 3.3. Realization of the Somatosensory Game Module

The somatosensory game module of this system includes four games, fruit-picking game, picture-recognizing game, instruction game, and balloon-topping game, which are designed for the training of different aspects of autistic children.


### 3.3.1. Pickup Fruit

The fruit-receiving game module is an interactive somatosensory game training module designed mainly for the defects of motor skills and sensory integration ability of children with autism. The main purpose of this game module is to train the brains of children with autism to control each body unit. In the game, when the Kinect sensor detects the human body, the two fruit baskets track the children’s hands separately, and the fruits start to fall at an initial speed from a random position on the top of the screen. The child needs to move his hands and body position to pick up the fallen fruits. The parameters such as “drop speed,” “number of drops,” “number received,” and “time” are displayed on the screen. In the course of the game, each round of receiving fruit has only two results: receiving fruit and not receiving fruit. If the child receives fruit, the values of the “number of drops,” “number of received,” and “drop speed” will be increased by 1, and the received fruit will disappear, and a new fruit will be at the top of the screen. Fruits appear randomly and then fall at a faster speed, and the children start a new round of fruit picking games; if the child fails to pick up the fruit before the fruit falls out of the lower end of the screen, then the “number of drops” will increase by 1, “number of received” remains unchanged, “falling speed” is reduced by 1, and then a new fruit appears at a random position at the top of the screen, starts to fall at a slower speed, and starts a new round of receiving fruits.


### 3.3.2. Image Identification

The game module for viewing pictures and objects is an interactive game training module designed mainly for children with autism in terms of language communication and life skills. The primary goal of this game module is to develop children’s listening, speaking, and other language communication skills, as well as their cognitive, life, and attention abilities. It is also effective in raising IQ levels. When the game starts, a picture of an item is randomly displayed on the screen, displaying two Chinese prompts: “Please tell me the item name in the picture” and “Voice prompt inX seconds;” at the same time, the voice reads “Please speak the item name in the picture,” counting down from 10 seconds. If the system detects the correct pronunciation of the item name within 10 seconds, it will stop the countdown, play the voice encouragement, give a short voice introduction to the item, and then display the next picture of the item. If the system still does not detect the correct pronunciation of the item within 10 seconds, the system will give a voice prompt, read the voice of the item name three times in succession, then give a voice introduction to the item, and then display the next picture.


### 3.3.3. Listen to Instructions

The instructional game is designed to improve the sensory integration ability, motor ability, and executive ability of children with autism. In this game, the training of sensory integration ability is mainly manifested as the brain’s ability to receive and process visual and auditory signals and then control the body to complete certain behavioral activities. Executive ability is a skill necessary for people with imperfect capacity in life and refers to the ability of a child to complete a certain behavior activity under the guidance of a guardian. After the game starts, the system simultaneously issues commands through subtitles and voice and starts a 60-second countdown at the same time, such as “please raise your hands,” “please jump,” and “please take a step forward”. If the system detects that the child completes the corresponding action through bone tracking, it uses voice and vision to encourage and strengthen, rests for a while, and then issues the next instruction. If the child reacts incorrectly or does not respond after the instruction is given, the guardian will immediately correct or assist in the demonstration until the child completes the corresponding action; if the child has not completed the corresponding action at the end of the countdown, switch to the next instruction.


### 3.3.4. Top Balloon

In the top balloon game, children need to find the position of the balloon and break the balloon with their head by jumping. The main purpose of designing this game is to train the coordination and balance of the limbs by jumping, promote the healthy development of the left and right brains, maintain muscle vitality, enhance physical resistance, consume children’s excess physical strength, and relieve emotions. In the game, when the Kinect sensor detects the human body, the balloon appears from a random position at the top of the screen, and a 30-second countdown starts. The child needs to track the position of the balloon and then jump to break the balloon with his head. If the balloon is touched within 30 seconds, the balloon disappears, and the corresponding encouragement is given, and then the next balloon randomly appears at the top of the screen. If the child cannot touch the balloon at the top of the screen by jumping, the distance between the child and the Kinect sensor should be appropriately shortened under the guidance of the guardian, so as to shorten the distance between the head and the balloon. If the child has not touched the balloon at the top of the screen by jumping at the end of the countdown, the balloon will disappear, and the next balloon will appear randomly. The two parameters “time” and “balloon number” are displayed at the top of the screen. Within the set time, the value of “time” starts to count down; each time a balloon is touched, the value of “balloons” will increase by 1, and after the final countdown value of “balloons” will be used as the game score.


## 3.3.1. Pickup Fruit

The fruit-receiving game module is an interactive somatosensory game training module designed mainly for the defects of motor skills and sensory integration ability of children with autism. The main purpose of this game module is to train the brains of children with autism to control each body unit. In the game, when the Kinect sensor detects the human body, the two fruit baskets track the children’s hands separately, and the fruits start to fall at an initial speed from a random position on the top of the screen. The child needs to move his hands and body position to pick up the fallen fruits. The parameters such as “drop speed,” “number of drops,” “number received,” and “time” are displayed on the screen. In the course of the game, each round of receiving fruit has only two results: receiving fruit and not receiving fruit. If the child receives fruit, the values of the “number of drops,” “number of received,” and “drop speed” will be increased by 1, and the received fruit will disappear, and a new fruit will be at the top of the screen. Fruits appear randomly and then fall at a faster speed, and the children start a new round of fruit picking games; if the child fails to pick up the fruit before the fruit falls out of the lower end of the screen, then the “number of drops” will increase by 1, “number of received” remains unchanged, “falling speed” is reduced by 1, and then a new fruit appears at a random position at the top of the screen, starts to fall at a slower speed, and starts a new round of receiving fruits.


## 3.3.2. Image Identification

The game module for viewing pictures and objects is an interactive game training module designed mainly for children with autism in terms of language communication and life skills. The primary goal of this game module is to develop children’s listening, speaking, and other language communication skills, as well as their cognitive, life, and attention abilities. It is also effective in raising IQ levels. When the game starts, a picture of an item is randomly displayed on the screen, displaying two Chinese prompts: “Please tell me the item name in the picture” and “Voice prompt inX seconds;” at the same time, the voice reads “Please speak the item name in the picture,” counting down from 10 seconds. If the system detects the correct pronunciation of the item name within 10 seconds, it will stop the countdown, play the voice encouragement, give a short voice introduction to the item, and then display the next picture of the item. If the system still does not detect the correct pronunciation of the item within 10 seconds, the system will give a voice prompt, read the voice of the item name three times in succession, then give a voice introduction to the item, and then display the next picture.


## 3.3.3. Listen to Instructions

The instructional game is designed to improve the sensory integration ability, motor ability, and executive ability of children with autism. In this game, the training of sensory integration ability is mainly manifested as the brain’s ability to receive and process visual and auditory signals and then control the body to complete certain behavioral activities. Executive ability is a skill necessary for people with imperfect capacity in life and refers to the ability of a child to complete a certain behavior activity under the guidance of a guardian. After the game starts, the system simultaneously issues commands through subtitles and voice and starts a 60-second countdown at the same time, such as “please raise your hands,” “please jump,” and “please take a step forward”. If the system detects that the child completes the corresponding action through bone tracking, it uses voice and vision to encourage and strengthen, rests for a while, and then issues the next instruction. If the child reacts incorrectly or does not respond after the instruction is given, the guardian will immediately correct or assist in the demonstration until the child completes the corresponding action; if the child has not completed the corresponding action at the end of the countdown, switch to the next instruction.


## 3.3.4. Top Balloon

In the top balloon game, children need to find the position of the balloon and break the balloon with their head by jumping. The main purpose of designing this game is to train the coordination and balance of the limbs by jumping, promote the healthy development of the left and right brains, maintain muscle vitality, enhance physical resistance, consume children’s excess physical strength, and relieve emotions. In the game, when the Kinect sensor detects the human body, the balloon appears from a random position at the top of the screen, and a 30-second countdown starts. The child needs to track the position of the balloon and then jump to break the balloon with his head. If the balloon is touched within 30 seconds, the balloon disappears, and the corresponding encouragement is given, and then the next balloon randomly appears at the top of the screen. If the child cannot touch the balloon at the top of the screen by jumping, the distance between the child and the Kinect sensor should be appropriately shortened under the guidance of the guardian, so as to shorten the distance between the head and the balloon. If the child has not touched the balloon at the top of the screen by jumping at the end of the countdown, the balloon will disappear, and the next balloon will appear randomly. The two parameters “time” and “balloon number” are displayed at the top of the screen. Within the set time, the value of “time” starts to count down; each time a balloon is touched, the value of “balloons” will increase by 1, and after the final countdown value of “balloons” will be used as the game score.


## 4. Experiments and Discussion

### 4.1. Object

The subjects of this experiment are all autistic children from an education center and Boai Autism Homeland. The criteria for the selection of the experimental subjects are (1) clinically diagnosed as autistic patients by a regular professional hospital; (2) with certain motor and cognitive abilities; (3) with certain attention. This experiment finally screened out 20 children with autism who met the above criteria and randomly assigned the children with autism into two groups, with 10 people in each of the experimental and control group. All children are 6–13 years old. There are 10 girls in total, 5 in the experimental group and 5 in the control group. See Table1 for the statistics of specific experimental object information.Table 1
Statistical distribution of subject information.
GroupQuantityBoysGirlsAgeExperimental group10559.21 ± 4.36Control group10559.09 ± 4.21


### 4.2. Evaluation Method

In this experiment, the Child Autism Rating Scale (CARS) and the Pediatric Evaluation of Disability Inventory (PEDI) were used to evaluate the two groups of children with autism before and after intervention to evaluate the intervention effect of somatosensory rehabilitation games. The CARS is a standardized evaluation scale with clinical diagnostic significance for autistic patients. It provides a basis for the assessment of speech, behavior, perception, and other aspects of autistic children. The PEDI rating scale is a professional evaluation scale for children with poor ability that assesses three components of ability: daily living skills, activity ability, and social communication ability.


### 4.3. Somatosensory Rehabilitation System Usage

The game part of the proprioception rehabilitation system is composed of four game modules: fruit-receiving game, picture and object recognition game, command listening game, and balloon game. According to the results recorded by the data management module, a total of 2800 games have been played, and the statistical distribution of the usage times of each game module is shown in Table2.Table 2
Somatosensory rehabilitation system usage distribution.
GameProportion (%)Pickup fruit35Image identification30Listen to instructions15Top balloon20


### 4.4. Scoring Data Analysis of the CARS

Use the CARS to measure the pre- and posttest score data of the two groups of children. The data analysis results are shown in Table3. The analysis results showed that the posttest scores of the two groups of children with autism were lower than the pretest scores. The posttest score of the experimental group was significantly lower than that of the pretest group, P<0.05; this difference was obviously statistically significant. Although the posttest score of the control group was lower than the pretest score, P>0.05, the difference between the front and back of the control group was not statistically significant.Table 3
Analysis of the CARS score.
GroupBefore testAfter testPExperimental group36.78 ± 6.9533.88 ± 4.600.03Control group36.56 ± 5.7735.98 ± 3.910.16


### 4.5. Scoring Data Analysis of PEDI

The PEDI scale was used to obtain the score data of the two groups of children with autism before and after the intervention in three aspects: social communication ability, activity ability, and daily life skills, and the data were statistically analyzed. The results are shown in Table4. Through the statistical analysis of the PEDI scale scores of the two groups of children, it can be seen that the PEDI scale scores of the experimental group (EG) and the control group (CG) have been improved to varying degrees after the intervention, and the scores of the experimental group have increased expressively. The group effect is significant. The PEDI scale scores of the social communication ability and activity ability of the children in the experimental group have been considerably improved after the intervention, P<0.01, while the PEDI scale scores of the ability of daily living have also increased, 0.01<P<0.05, which is statistically significant. There were no substantial differences in the PEDI scale scores in all aspects of the children in the control group, and P>0.05, which was not statistically significant.Table 4
Analysis of the PEDI score.
ItemIndexEGCGSocial communication abilityBefore test24.10 ± 7.4725.80 ± 6.75After test30.70 ± 9.6727.41 ± 4.68P<0.010.13Activity abilityBefore test44.75 ± 6.3745.67 ± 7.10After test53.01 ± 5.5146.88 ± 6.81P<0.010.17Living abilityBefore test38.11 ± 9.7836.99 ± 8.41After test41.43 ± 9.5237.66 ± 7.59P0.020.23The PEDI scale scores of all aspects in the experimental group have been significantly improved, and the scores of the social communication ability and activity ability have changed more obviously as compared to the control group.


### 4.6. Analysis Result

The analysis results of CARS scale scores and PEDI scale scores indicate that the control group who did not receive somatosensory rehabilitation training had little improvement in all aspects of abilities compared to the experimental group. After 6 months of somatosensory rehabilitation training, the three abilities of social communication ability, activity ability, and daily living ability of the autistic children in the experimental group have been significantly enhanced; among them, the improvement of social communication ability and activity ability is more obvious. Experiments have proved that the proprioception rehabilitation training system has a good intervention effect on children with autism. During the experiment, the system was operating normally without abnormal conditions, indicating that the system has good stability and anti-interference ability.


## 4.1. Object

The subjects of this experiment are all autistic children from an education center and Boai Autism Homeland. The criteria for the selection of the experimental subjects are (1) clinically diagnosed as autistic patients by a regular professional hospital; (2) with certain motor and cognitive abilities; (3) with certain attention. This experiment finally screened out 20 children with autism who met the above criteria and randomly assigned the children with autism into two groups, with 10 people in each of the experimental and control group. All children are 6–13 years old. There are 10 girls in total, 5 in the experimental group and 5 in the control group. See Table1 for the statistics of specific experimental object information.Table 1
Statistical distribution of subject information.
GroupQuantityBoysGirlsAgeExperimental group10559.21 ± 4.36Control group10559.09 ± 4.21


## 4.2. Evaluation Method

In this experiment, the Child Autism Rating Scale (CARS) and the Pediatric Evaluation of Disability Inventory (PEDI) were used to evaluate the two groups of children with autism before and after intervention to evaluate the intervention effect of somatosensory rehabilitation games. The CARS is a standardized evaluation scale with clinical diagnostic significance for autistic patients. It provides a basis for the assessment of speech, behavior, perception, and other aspects of autistic children. The PEDI rating scale is a professional evaluation scale for children with poor ability that assesses three components of ability: daily living skills, activity ability, and social communication ability.


## 4.3. Somatosensory Rehabilitation System Usage

The game part of the proprioception rehabilitation system is composed of four game modules: fruit-receiving game, picture and object recognition game, command listening game, and balloon game. According to the results recorded by the data management module, a total of 2800 games have been played, and the statistical distribution of the usage times of each game module is shown in Table2.Table 2
Somatosensory rehabilitation system usage distribution.
GameProportion (%)Pickup fruit35Image identification30Listen to instructions15Top balloon20


## 4.4. Scoring Data Analysis of the CARS

Use the CARS to measure the pre- and posttest score data of the two groups of children. The data analysis results are shown in Table3. The analysis results showed that the posttest scores of the two groups of children with autism were lower than the pretest scores. The posttest score of the experimental group was significantly lower than that of the pretest group, P<0.05; this difference was obviously statistically significant. Although the posttest score of the control group was lower than the pretest score, P>0.05, the difference between the front and back of the control group was not statistically significant.Table 3
Analysis of the CARS score.
GroupBefore testAfter testPExperimental group36.78 ± 6.9533.88 ± 4.600.03Control group36.56 ± 5.7735.98 ± 3.910.16


## 4.5. Scoring Data Analysis of PEDI

The PEDI scale was used to obtain the score data of the two groups of children with autism before and after the intervention in three aspects: social communication ability, activity ability, and daily life skills, and the data were statistically analyzed. The results are shown in Table4. Through the statistical analysis of the PEDI scale scores of the two groups of children, it can be seen that the PEDI scale scores of the experimental group (EG) and the control group (CG) have been improved to varying degrees after the intervention, and the scores of the experimental group have increased expressively. The group effect is significant. The PEDI scale scores of the social communication ability and activity ability of the children in the experimental group have been considerably improved after the intervention, P<0.01, while the PEDI scale scores of the ability of daily living have also increased, 0.01<P<0.05, which is statistically significant. There were no substantial differences in the PEDI scale scores in all aspects of the children in the control group, and P>0.05, which was not statistically significant.Table 4
Analysis of the PEDI score.
ItemIndexEGCGSocial communication abilityBefore test24.10 ± 7.4725.80 ± 6.75After test30.70 ± 9.6727.41 ± 4.68P<0.010.13Activity abilityBefore test44.75 ± 6.3745.67 ± 7.10After test53.01 ± 5.5146.88 ± 6.81P<0.010.17Living abilityBefore test38.11 ± 9.7836.99 ± 8.41After test41.43 ± 9.5237.66 ± 7.59P0.020.23The PEDI scale scores of all aspects in the experimental group have been significantly improved, and the scores of the social communication ability and activity ability have changed more obviously as compared to the control group.


## 4.6. Analysis Result

The analysis results of CARS scale scores and PEDI scale scores indicate that the control group who did not receive somatosensory rehabilitation training had little improvement in all aspects of abilities compared to the experimental group. After 6 months of somatosensory rehabilitation training, the three abilities of social communication ability, activity ability, and daily living ability of the autistic children in the experimental group have been significantly enhanced; among them, the improvement of social communication ability and activity ability is more obvious. Experiments have proved that the proprioception rehabilitation training system has a good intervention effect on children with autism. During the experiment, the system was operating normally without abnormal conditions, indicating that the system has good stability and anti-interference ability.


## 5. Conclusions

In recent years, as the number of autistic groups continues to rise, it has brought serious spiritual and economic burdens to the family and society. At present, there is a great demand for rehabilitation training resources in the field of autism rehabilitation in China, while the existing somatosensory training equipment has some shortcomings. This study builds on previous research findings and enhances and innovates knowledge by in-depth research on the autistic group based on the clinical manifestations of children with autism, focusing on the prominent obstacles in the lives of children with autism and combining the existing literature and data; this work can integrate the advantage of applied behavior analysis, sensory integration training, structured educational games, and mainstream rehabilitation methods, and use Kinect somatosensory equipment to develop a somatosensory rehabilitation training system for children with autism. The main work of this article is summarized as follows: (1) it independently designed and developed the somatosensory game module and used Kinect’s image acquisition, bone tracking, motion recognition, voice recognition, and other methods in the development process. Proprietary games involve all-round rehabilitation training such as language communication, behavioral movement, daily life skills, and sensory integration which improve the deficiencies of the previous rehabilitation training in the scope of intervention. The system is designed with diversified game mode options to meet the structural and individual needs of rehabilitation training for children with autism. (2) The rehabilitation effect experiment was carried out on the proprioception rehabilitation system. Using professional evaluation tools, a 6-month controlled experiment was conducted on two groups of experimental subjects, of which only the experimental group received somatosensory rehabilitation system training. Experimental data showed that the experimental group’s ability in all aspects improved significantly, while the control group’s ability was not improved. The experimental results prove that the proprioception rehabilitation system has a good intervention effect on children with autism, and the system has good stability.In the future, work needs to be carried out to develop more challenging somatosensory games which further enhance the abilities of autistic children to carry out normal day routine and behavior while testing the system on more rigorous scales.



---

*Source: 6020208-2021-10-13.xml*