# STMMI: A Self-Tuning Multi-Modal Fusion Algorithm Applied in Assist Robot Interaction

Authors: Ya Hou; Zhiquan Feng; Xiaohui Yang; Tao Xu; Xiaoyu Qiu; Xin Zhang
Journal: Scientific Programming
 (2022)

---

## Abstract

While facing complex surroundings, robots need to identify the same intention which is expressed in different ways. In order to solve the problem of assisting robots to get a better intention understanding, a self-tuning multimodal fusion algorithm is put forward in this paper, which is not restricted by the expressions of interacting participants and environment. The multimodal fusion algorithm can be transferred to different application platforms. Robots can own the understanding competence and adapt new tasks by changing the content of the robot knowledge base. Compared with other multimodal fusion algorithms, this paper attempts to transfer the basic structure of feed-forward neural networks on discrete sets, which has strengthened the consistency and perfect the complementary relations between multiple mode, and has realized the simultaneous operation of fusion operator’s self-tuning and intention search. There are three kinds of modes selected in the paper: speech, gesture, and scene objects, where the single modal classifiers are trained separately. This method conducted a human-computer interaction experiment on the bionic robot Pepper platform, which proved that the method can effectively improve the accuracy and robustness of robots in aspects of understanding human intentions, and reduce the uncertainty about intention judgment in a single modal interaction.


---

## Body

## 1. Introduction

Human-computer interaction has stepped the natural human-computer interaction stage. The single modal interaction has been improved greatly with the development of artificial intelligence technology especially with that of deep learning techniques. In the future, we expect that robots can help human beings create a healthier and more convenient life [1]. However, the robot only can be made with a special function at present. For instance, some social scenarios tend to with relatively narrow task domains and specific user interactions, where robots are limited in the flexibility and adaptability in the face of different users, different scenarios, and different robot platforms [2]. Another example is robots for children’s education, it’s also limited in the aspect of the flexibility of interaction and application [3].During the new stage, there is a new background and new needs, a lot of disadvantages of the single modal interaction mode have been exposed. For example, it can not obtain complete information and cannot accurately understand the user’s intention, whose interaction only is reached by asking users frequently, so users can not have a sound interaction experience. Therefore, the study on how to adopt the multimodal information fusion algorithm has become the focus of this paper.MMHCI (multi-modal human-computer interaction) is a more natural way of human-computer interaction [4]. However, the difficulty of the research is mainly how to make an overall plan of the information among multiple modes [5, 6]. It should also meet the requirement of being unrelated to the operator and the environment. In this aspect, Kivrak H et al. [7] designed a route computing model based on collision detection to adapt human-computer interaction under unknown environments, which can be applied to solve other ambient computing with low density or average density such as indoor environments. Hemminahaus et al. [8] assisted robot to adapt society by itself through reinforcement learning and they managed to map advanced human behavior in interactions to lower-level behaviors executable by robots in different situations. Natural interaction should reduce the burden of the interaction participants, whose actions and expression ways should not be limited but determined by himself in the interaction process.The interactive data expressing the real world is quite different in thousands of ways, which varies in length. The existing multimodal fusion methods can not flexibly deal with the intention of expressing large differences in information. Meanwhile, facing the more and more complicated human-machine interaction scenarios, robots should be able to learn new tasks quickly, which requires a new multimodal fusion algorithm to have a good expansiveness. It is difficult for the simple neural network to identify tasks with large changes in modal information. To find the correct relationship in a new form of expression to understand the human intention is the emphasis of this paper.In order to solve the key issue, a self-tuning multimodal interaction (STMMI) is put forward in this paper, which can transfer the basic structure of feed-forward neural networks, and then truly get the intention of the interaction participants. This paper is no longer directly solving the mapping relation from modal data to intention, but conducts a study on a human-computer interaction algorithm of multi-modal fusion level under the background of assist robot application.This paper will be illustrated from the following aspects. The relevant work is discussed in Section 2, while the fusion approach of STMMI is explained from four generalized structures in Section 3. In Section 4, the effect of human-computer interaction is presented in the experimental multi-modal fusion algorithm on the Pepper bionic robot platform, which is also compared with the primary indicators of other methods. At last, we reach a conclusion in Section 7.


## 2. Related Work

The good results have been achieved in the aspect of the interaction of a single mode, such as speech interaction [9], face [10], gesture interaction [11], emotion interaction [12, 13], handwriting pen interaction [14, 15], and eye movement [16]. Multimodal human-computer interaction has become the key problem of human-robot interaction. In this chapter, the previous methods have been focused on, which try to solve the multimodal interaction, problem to varying degrees.At present, there are several methods to deal with the multimodal fusion problem. Kim and Leskovec [17] tried an information fusion method that is based on the figure model. It can be used to predict the features of missing notes or links. They also applied this model to the inference of multimodal information fusion. To order to improve the robot’s understanding competence of instructions, Vanzo et al. [18] established a structural framework for processing operations on multidimensional information within the environment. This method of adding another dimension to the speech interaction has improved the accuracy by 25%. There still are obvious main channels and auxiliary channels in such a multimodal fusion. A classifier fusion was introduced by Trick et al. [19] that is used when human beings express intentions by using multi-models, including four modes: speech, gesture, gaze direction, and scene objects. It helps to reduce the uncertainty of intention understanding in a single mode. Through the Bayesian rule, the method analyzes the return of each single classifier and re-normalizes to the distribution of one. Similarly, based on the decision model of Bayes, Dou et al. [20] combine Neural Network with a seminaive Bayesian classifier to infer user’s intention. KDI algorithm uses CNN to analyze gesture and action information, and YOLOV3 is used for object detection to provide scene information. Ondos et al. [21] realized multimode interaction with users on the NAO platform. They synthesized the combination of voice and gesture. And the extension uses VoiceXML language, which can control robot dialogue, action, and posture [22].In order to improve the recognition accuracy during interaction and solve the incompatibility of different modal datasets at the same time, a kind of data fusion architecture based on biological inspiration can be used to conduct human gesture recognition by integrating the visual and somatosensory data, so as to achieve the goal of interaction. In aspects of assisting multi-modal human-computer action recognition in human-computer interaction, Rodomagoulakis et al. [23] processed multimodal awareness to identify human actions and commands, which includes voice input and visual input from high-definition cameras and deep cameras. At the end, the conclusion has been reached that the multimodal fusion result of weighted combinations is a human intention. Kim et al. [24] used the reward function to make the robot easily adapted to the interaction with humans, which clarified that feedback is beneficial during the interaction process, and there is a correct mapping between the learning of gestures and human actions and robot tasks. Here, they put forward the reward methods. Feedback refers to that for adjusting the robot interaction action selection. The maximized overall reward got from humans can help robots to find the best action strategy, which can be adjusted and learned by robots, and humans can choose gestures and provide feedback to the robot at their will. The research studies of deep learning in multimodal fusion are mainly in the aspects of processing the normalization of multi-modal data sources or the fusion of feature layers. Ortega et al. [25] used the multi-modal fusion method of deep neural networks for emotion recognition. Each modal of the audio, video, and text is first encoded independently by the fully connected layer, which is then merged into a fully connected layer, and finally completed for multimodal fusion in an end-to-end way. However, there is still a limitation of the exiting category in the fusion output of the last layer. Lukasz et al. [26] attempted to build a unified deep learning model to solve adaptively the multiple models of different types of tasks in different domains and different data modes, with no significant loss of performance on specific tasks. A large-scale deep neural network model for multichannel information fusion is trying to be built for processing multichannel information without difference in the fusion stage, which is definitely a good way to explore the rules of information expression in human-computer interaction by such a broadened deep neural networks. A large number of data types and data categories can be classified by using it. An excellent network model can be applied to perfectly solve a class of problems. Even so, this kind of network is relatively “specific” compared to intention. If adding an intention of user, the network parameters will need to be adjusted again, which makes the use of deep neural networks a time-consuming and power-consuming process. Its advantages are largely limited especially in the process of assisting robot interactions. Different from the data features of pictures and audio, a complete and effective human-computer interaction has uncertain steps and takes a longer time, where large sample data are difficult to obtain. What’s more, the uncertain information data length will increase the difficulty of network processing.Generally speaking, the research framework of basic three aspects is currently formed in many interactive research studies of multiple modal fusion [27], which refers to the characteristic layer, model layer, and decision layer. However, the current multimodal fusion algorithm is not well solved in dealing with the flexibility and intention comprehensiveness problem of the expression in the interaction process. Different from the mostly discussed horizontal fusion of neural networks on MMHCI algorithms in the characteristic layer or the fusion in the decision layer under the statistical model. In this paper, the neural network is used in the recognition layer for the first time, whose structure is used again in the decision layer to complete the self-tuning fusion of collection information.Especially for the work of assist robots, STMMI algorithm presents the characteristics of easily operate and high flexibility in interactions. This paper is designed exactly to explore how to transfer and use the basic structure of neural networks in multimodal fusion interactions, then the optimal fusion operator can be obtained through its self-tuning under the supervision of the robot commands database, so the problem of intention understanding under changeable scenarios can be solved.


## 3. Multimodal Fusion Algorithm

The difference between the proposed algorithm and other fusion algorithms is that multi-modal information is regarded as a relationship without fixed combination rules. It is expected that the self-learning idea of neural networks can be used to obtain the self-tuning results of multi-modal fusion operators.Some scholars believe that multi-modal information has a fixed correlation [28, 29]. They divided them into information complementary, information mutual exclusion, and information redundancy, and then fused them according to the characteristics of the information [30], or divided them into early fusion and late fusion. The cognitive model of multimodal fusion is different from the local perception characteristics of image recognition or the time sequence of speech recognition. Such a cognitive model concerns the information relationship between multiple modalities. In fact, the relationship between the modalities and the contribution of each modality to the final intent understanding result is inconsistent and the situation is not uniform. Neural network finds the low dimensional features of a kind of data through repeated learning. Similarly, STMMI can find the solution expressing intention through finite set operations. In this way, the self-learning of multimodal fusion machine is realized, and it can respond to various interactions flexibly.In order to show the focus of multimodal fusion we discussed, Figure1 compares the migration of BP neural network structure to STMMI structure. The gradation of color indicates the importance of elements in intention expression. The heavier the color, the more important the element is. For example, many people often have Adverbs “fast” and “soft,” and many people speak with unconscious gestures. These modal information are collected, but after information coding, they form a more suitable set elements for understanding. In BP neural network, the flow is the data feature, and in STMMI, the flow is the discrete set operation relationship. After repeated tuning, an intention contained in the robot intention library is finally obtained.Figure 1
Structure comparison of feed-forward neural network and multimodal fusion algorithm.This paper proposes a new multimodal fusion algorithm. The algorithm includes three modalities: speech, scene object, and gesture. For speech, an independent single microphone is used to input the user’s speech information. For scene object, a common camera is used to input the environmental information in real-time interaction. For gestures, Kinect camera is used to input the depth image information of the user’s hand. First, the image and audio data are recognized by three neural networks and transformed into discrete sets.


### 3.1. Information Coding

Because the multimodal information content of different interaction intents is different, this paper first preprocesses the uncertain data length. At the same time, data of different dimensions are transformed into the same dimension for representation.In this section, three independent neural networks are used to preprocess the source data. The RGB image is identified by the object-detection network based on the YOLOv3 structure to extract the Scene object we are concerned about. In daily life scenes, the recognition accuracy yolov3 of the experiment can reach more than 97% [31]. The speech information is processed into a text representation by the REST API. Its recognition accuracy exceeds 98% [32]. In terms of perceiving the human body situation, Kinect is used to obtain and segment the human hand information. The Inception v3 network is used to recognize Depth images. Finally, the accuracy of gesture recognition can reach more than 91% [33]. We focus on the expression of human gesture information in interaction (Figure 2).Figure 2
Modal information is converted to a set of elements.Therefore, this paper converts the images and audio into text sets represented by a classifier. Many of these single-mode information cannot fully express people’s actual intention. They may be complementary, consistent, or even opposite. Therefore, we should find their relationship and understand their intention through multimodal fusion algorithm. The unification of different modal information in dimensions is completed by information coding.


### 3.2. Reward Function

In the existing human-computer interaction field, the quality of an interaction is generally measured by the evaluation after the completion of a certain interaction [19]. This evaluation method has a lag, has no meaning for this interaction, and can only modify the next interaction. In order to optimize the interaction through the fusion algorithm, a new evaluation standard must be specified to predict the pros and cons of this fusion operator and adjust itself.In order to clarify the proposed algorithm and avoid ambiguity, the following explanations are made on the set operators. This definition is only applicable to the special applications of this paper and is not completely classified according to general mathematical concepts:Sets(A) Sets(B) : Find the same elements in the two sets A and B.Sets(A) + Sets(B) : Find the superposition of the elements in the two sets of A and B, regardless of whether they are repeated.Sets(A) - Sets(B) : Find that A deletes the same elements as B.In this way, the calculation process of the set is transformed into three directly defined minimum basic arithmetic units. No matter which one can be divided, the algorithm can be directly called, avoiding the ambiguity of the fusion algorithm.According to the knowledge base about the description of the instructions that the robot can complete, we define the effective information rate. It is assumed that the single-modal set of a fusion isS = \{sS=\s1,s2,…,sm\;O=o1,o2,o3,…,on\; G=\g1\that the set obtained after fusion is F\s1,o1,o2,g1\; and the example of the knowledge base that the knowledge base representation set that really expresses this task is Y=\s1,o2,g1\. Where S = \{…\}, O = \{…\}, and G = \{…\} represent the discrete set of speech modal, scene objects modal and gesture modal. The variables si,oi,gi represent elements in a discrete sets. The variables m, n indicates that the number of information elements may be different during each interaction.The effective information rateP1 of the fusion pros and cons at the time t when the fusion machine performs a fusion inside is expressed by:(1)P1t=numF∩YinumYi,where numF∩Yi is the number of the same elements of F and Yi.


### 3.3. Contribution Rate

Information quantification is the key to the transfer learning neural network in this paper. The contribution rate can evaluate the role of each modality in a fusion and correct the weight of different modalities during the fusion.The single-modal effective information contribution rateP2 at the time t when a fusion is performed inside the fusion machine is given by:(2)P2t=numV∣L|G∩FnumVLG.The two indicators of the algorithm in this paper:P1 determines the best goal, and P2 determines the multimodal fusion situation to reach the goal. A good multimodal fusion operator is determined by the weight of each modality and the operator between the modalities. The adaptive process of the fusion algorithm in this paper is to continuously update the weights and combine the operators between the multi modalities.


### 3.4. Positive or Negative Influence

Changing the multimodal operation relationship will lead to a positive or negative impact on the result of intention understanding. For example, if the set elements of a mode are subtracted, and the effective information rateP1 is greatly reduced, then these elements are important. We hope P1 can be improved, so we should do the opposite operation to this action. Such a calculation and feedback process determines some elements in the intended solution.Our definition of intention understanding positive or negative effects is (3) It can be seen here that STMMI will lead to intention understanding results in two directions, and we need to continuously enhance the positive influence direction. Since the collection element is in text form, the value of w here is 0 or 1.(3)inf=positive,P1t−P1t−1>0,negative,P1t−P1t−1≤0.In order to find the optimal intent, the algorithm continuously changes the contribution of each modal information and the operation relationship of the modal. In (4), ⊗ represents three set operations, and the final fusion result is determined by the three kinds of modal information of V or L or G and the combination relationship between them.(4)F=maxP1w1O⊗i=13w2S⊗i=13w3⊗i=13.Table1 illustrates the detailed process of iteratively updating F. Obviously, the final set may form a combined operation such as full intersection, full union, full complement, first intersection and then complement, and first complement and then intersection.Table 1
STMMI algorithm.
STMMI AlgorithmInput: Multi-modal informationOutput: Operator intent1. Initial F:Random initial operator⊗, random initial weightCalculate F according to (4)2. Current F:The fusion operator with the largestP2 value makes one iteration3. Updata F:If inf equals positive, then increase the weight of this set,w=1If inf equals negative, then decrease the weight of this set,w=04. Judge F:If F is repeated or an empty set appearsThen stop fusion5. Otherwise:Repeat steps 2, 3


## 3.1. Information Coding

Because the multimodal information content of different interaction intents is different, this paper first preprocesses the uncertain data length. At the same time, data of different dimensions are transformed into the same dimension for representation.In this section, three independent neural networks are used to preprocess the source data. The RGB image is identified by the object-detection network based on the YOLOv3 structure to extract the Scene object we are concerned about. In daily life scenes, the recognition accuracy yolov3 of the experiment can reach more than 97% [31]. The speech information is processed into a text representation by the REST API. Its recognition accuracy exceeds 98% [32]. In terms of perceiving the human body situation, Kinect is used to obtain and segment the human hand information. The Inception v3 network is used to recognize Depth images. Finally, the accuracy of gesture recognition can reach more than 91% [33]. We focus on the expression of human gesture information in interaction (Figure 2).Figure 2
Modal information is converted to a set of elements.Therefore, this paper converts the images and audio into text sets represented by a classifier. Many of these single-mode information cannot fully express people’s actual intention. They may be complementary, consistent, or even opposite. Therefore, we should find their relationship and understand their intention through multimodal fusion algorithm. The unification of different modal information in dimensions is completed by information coding.


## 3.2. Reward Function

In the existing human-computer interaction field, the quality of an interaction is generally measured by the evaluation after the completion of a certain interaction [19]. This evaluation method has a lag, has no meaning for this interaction, and can only modify the next interaction. In order to optimize the interaction through the fusion algorithm, a new evaluation standard must be specified to predict the pros and cons of this fusion operator and adjust itself.In order to clarify the proposed algorithm and avoid ambiguity, the following explanations are made on the set operators. This definition is only applicable to the special applications of this paper and is not completely classified according to general mathematical concepts:Sets(A) Sets(B) : Find the same elements in the two sets A and B.Sets(A) + Sets(B) : Find the superposition of the elements in the two sets of A and B, regardless of whether they are repeated.Sets(A) - Sets(B) : Find that A deletes the same elements as B.In this way, the calculation process of the set is transformed into three directly defined minimum basic arithmetic units. No matter which one can be divided, the algorithm can be directly called, avoiding the ambiguity of the fusion algorithm.According to the knowledge base about the description of the instructions that the robot can complete, we define the effective information rate. It is assumed that the single-modal set of a fusion isS = \{sS=\s1,s2,…,sm\;O=o1,o2,o3,…,on\; G=\g1\that the set obtained after fusion is F\s1,o1,o2,g1\; and the example of the knowledge base that the knowledge base representation set that really expresses this task is Y=\s1,o2,g1\. Where S = \{…\}, O = \{…\}, and G = \{…\} represent the discrete set of speech modal, scene objects modal and gesture modal. The variables si,oi,gi represent elements in a discrete sets. The variables m, n indicates that the number of information elements may be different during each interaction.The effective information rateP1 of the fusion pros and cons at the time t when the fusion machine performs a fusion inside is expressed by:(1)P1t=numF∩YinumYi,where numF∩Yi is the number of the same elements of F and Yi.


## 3.3. Contribution Rate

Information quantification is the key to the transfer learning neural network in this paper. The contribution rate can evaluate the role of each modality in a fusion and correct the weight of different modalities during the fusion.The single-modal effective information contribution rateP2 at the time t when a fusion is performed inside the fusion machine is given by:(2)P2t=numV∣L|G∩FnumVLG.The two indicators of the algorithm in this paper:P1 determines the best goal, and P2 determines the multimodal fusion situation to reach the goal. A good multimodal fusion operator is determined by the weight of each modality and the operator between the modalities. The adaptive process of the fusion algorithm in this paper is to continuously update the weights and combine the operators between the multi modalities.


## 3.4. Positive or Negative Influence

Changing the multimodal operation relationship will lead to a positive or negative impact on the result of intention understanding. For example, if the set elements of a mode are subtracted, and the effective information rateP1 is greatly reduced, then these elements are important. We hope P1 can be improved, so we should do the opposite operation to this action. Such a calculation and feedback process determines some elements in the intended solution.Our definition of intention understanding positive or negative effects is (3) It can be seen here that STMMI will lead to intention understanding results in two directions, and we need to continuously enhance the positive influence direction. Since the collection element is in text form, the value of w here is 0 or 1.(3)inf=positive,P1t−P1t−1>0,negative,P1t−P1t−1≤0.In order to find the optimal intent, the algorithm continuously changes the contribution of each modal information and the operation relationship of the modal. In (4), ⊗ represents three set operations, and the final fusion result is determined by the three kinds of modal information of V or L or G and the combination relationship between them.(4)F=maxP1w1O⊗i=13w2S⊗i=13w3⊗i=13.Table1 illustrates the detailed process of iteratively updating F. Obviously, the final set may form a combined operation such as full intersection, full union, full complement, first intersection and then complement, and first complement and then intersection.Table 1
STMMI algorithm.
STMMI AlgorithmInput: Multi-modal informationOutput: Operator intent1. Initial F:Random initial operator⊗, random initial weightCalculate F according to (4)2. Current F:The fusion operator with the largestP2 value makes one iteration3. Updata F:If inf equals positive, then increase the weight of this set,w=1If inf equals negative, then decrease the weight of this set,w=04. Judge F:If F is repeated or an empty set appearsThen stop fusion5. Otherwise:Repeat steps 2, 3


## 4. Experiment

### 4.1. System Setup

In this work, we used Pepper humanoid robot. The robot has 20 degrees of freedom, 2 touch sensors in the hands and has 2 sonar sensors, 6 laser sensors, 2 infrared sensors, and 3 touch sensors on the legs when walking. These sensors improve the accuracy of execution and ensure the safety of the robot during execution. The interactive output modes of pepper include voice, movement, arm, and head posture.We also used external sensing devices to improve the accuracy of the captured information. A Kinect was used to obtain depth images of the human body.The system was implemented in a distributed manner on 2 computers, and each computer was equipped with Windows 10 operating system. One computer carried the functions of scene objects and gesture recognition, and the other computer completed the integrated multimodal fusion and robot control module. The robot and the two computers were in the same local area network. The robot action and speech interface could be called through the local area network.


### 4.2. Experimental Environments

In order to evaluate the multimodal fusion algorithm proposed in this paper, we built a laboratory environment that simulated daily life, which contained simple living items. Normal cameras are sensitive to changes in light when collecting data, so the ambient light was illuminated by indoor soft light. The experimental scene could be transferred when the interaction conditions were met.After the human entered the experimental scene, he was free to interact with the robot. The executable intent set under the robot’s preset is shown in Table2. The intent set is divided into 5 categories, and the specific explanation is shown in Table 2. A simple example: someone say “I want this.” He pointed to the apple on the table. The system should understand the intention of “Give him an apple.” When the robot is not activated for a long time, pepper will actively ask “do you need any help?.” At the same time, pepper is always monitoring the whole environment. Once there are changes that meet the conditions of executable intention, pepper will respond immediately.Table 2
Set of experimental executable intents.
Intent numberRobot control levelExample intention1Cancel“Stop,” “cancel” “not”…2Confirm“Go on,” “OK(gesture),” “OK,” “yes”…3Dialogue“What time is it? ,“ “tell a story”…4Move“Come,” “go forward,” “turn around “…5Acts on the scene object“Take the water cup,” “look for a phone”…


### 4.3. Experimental Design and Procedures

In order to verify whether the proposed STMMI algorithm has achieved the purpose of improving interaction efficiency and performance in daily service scenarios, we designed a simulation experiment. The experiment invited 20 volunteers, aged from 20 to 60, and the ratio of male to female was 1 : 1. Each person conducted 5 interaction experiments at intervals. During the process, the interaction instructions and interaction sequence were not preset to maximize the real purpose of the interaction.In the experiment process, the robot acted on the entire scene after executing the instructions, which caused changed the scene information. So, the scene state must be updated after each interaction. The entire system was always in a monitoring and waiting state, and no special wake-up instructions were required during the experiment. All interactions of the system were driven by the robot to cooperate with the experimenter.In order to clearly verify the fusion effect of STMMI, the following requirements were set for the participants:The experimenter should speak and make gestures as naturally as possible during the interactionThe experimenter should not intentionally be too close to the Kinect and the camera, so as not to block too much scene information and cause invalid interactionThe experimenter should put forward regular interaction instructions, and must not put forward instructions that are contrary to common sense, so as not to cause invalid interaction recordsThe experimenter should intersperse the experiment to prevent the experimenter’s interaction behavior from solidifyingDuring the experiment, we randomly select the multimodal fusion algorithm. However, the volunteers who participated in the interaction did not know which fusion algorithm was used. We counted the user ratings and experimental results. The data were used for subsequent analysis.


### 4.4. Experimental Results and Analysis

#### 4.4.1. Interaction Results

The experiment verified the feasibility of the algorithm proposed in this paper, and a good interaction effect was achieved. Taking a water cup as an example, we selected several images to show the process of the experiment. Figure3(a) shows that the experimenter was sending instructions, Figures 3(b) and 3(c) show that the robot was executing instructions after understanding the intent. All the images came from the real experimental environment.Figure 3
The experimenter gave instructions and the robot executed the instructions.
(a)(b)(c)Experiments verified the effectiveness of the STMMI algorithm in this paper, and a better interaction effect was obtained. Especially compared with single-modal interaction, multimodal interaction achieved the interaction of information between modalities and obtained more accurate interaction intent. This method could be used for multimodal interactions that can complete collective coding. Our experiments proved the feasibility of the proposed algorithm in current applications.


#### 4.4.2. Quantitative Results

The experimental data of each experiment are given in Table3, which contains the number of interactive instructions issued under STMMI, KDI [20] and Bayes-based multimodal fusion algorithm, the number of instructions recognized by the algorithm.Table 3
Experimental data.
CancelConfirmDialogueMoveActs on the scene objectThe intention expressed by the interlocutor (STMMI)463594167114Number of recognized instructions (STMMI)372180149106Number of correctly recognized instructions (STMMI)351971125101The intention expressed by the interlocutor (KDI)272379124105Number of recognized instructions (KDI)2113619790Number of correctly recognized instructions (KDI)1812527981The intention expressed by the interlocutor (Bayes)51398411792Number of recognized instructions (Bayes)2716517567Number of correctly recognized instructions (Bayes)2413455764Further analyze the experimental data and calculate the interactive accuracy and recall of the two algorithms. Among them, the interaction accuracy rate is equal to the number of correctly recognized instructions divided by the number of recognized instructions, and the interaction recall rate is equal to the number of recognized instructions divided by the total number of issued instructions. Figure4 shows a bar chart comparing the interactive accuracy and recall of different algorithms. From the data in the chart, it can be seen that STMMI method is obviously superior to other algorithms in the understanding performance of class 3 and 4 intentions. This is because the third and fourth types of intentions are complex, and the experimenter’s ways of conveying intentions vary. In addition, if the interlocutor pauses for a long time in an intention expression, it may cause the system to stop modal information collection and fusion before the interlocutor fully expresses it. At this time, if the key information in the intention is missing, the system will not be able to get an executable intention or misinterpret it as another intention. However, due to the adaptive ability of STMMI algorithm, the correct intention can still be obtained when noncritical information is missing.Figure 4
Accuracy and Recall of interaction.Through the experimental data in Figure4, it can be found that the first kind of intention-“cancel” STMMI has 14.1% higher accuracy and 35.9% higher recall than the Bayesian fusion method. This is because when expressing the intention of “no,” the interactor often uses body gestures such as waving his hand and shaking his head. Similarly, the STMMI of the second kind of intention “confirmation” has 30% higher accuracy and 40.2% higher recall than the Bayesian fusion method. When the interlocutor uses the “OK” gesture or nods to express the “confirm” intention, the Bayesian fusion method depends on the single-mode recognition results. The third kind of intention-“dialogue,” the accuracy of the two methods is similar, but the recall rate of STMMI is 27.5% higher than that of Bayes. And the recall rate of STMMI is 12% higher than that of KDI. Analyzing the experimental process, it is found that the experimenter often appears the information of the scene object in the dialogue, such as “where is my cup” Such a dialogue. The fourth type of intention-“moving,” when the interactor uses gestures to signal the robot to come, there will be false recognition caused by occlusion, which will affect the result of intention understanding. The fifth kind of intention-“acts on the scene objects.” It can be seen that when the recognition accuracy of the two algorithms is similar, the recall rate of STMMI is 22.7% higher than that of the Bayesian fusion method. And the recall rate of STMMI is 7% higher than that of KDI. This kind of intention needs the most complex information, and three kinds of modal information: voice, gesture, and scene recognition play a role.On the whole, STMMI performs significantly better than KDI method and Bayesian fusion method in recall rate. This shows that STMMI can adapt to the variability of interactive scenes. Although the Bayesian fusion method performs well in accuracy, it cannot understand an intention in many cases, resulting in no response of the robot. This kind of intention is weakly dependent on multimodal information, which depends on the comprehensiveness of the robot knowledge base to a greater extent.STMMI shows that it can get the experimenter’s intention more accurately when getting multimodal information. This is because STMMI adjusts its fusion operator in combination with instruction search in the fusion process in order to find the best result. Therefore, STMMI is more flexible to adapt to different information expression methods and find the best combination mode.


#### 4.4.3. Modal Fusion Results

In order to verify the effectiveness of multimodal interaction and the efficiency of the STMMI algorithm in using multimodal information, we recorded the final fusion results of each experiment. Every time the system calculates the interaction intention, there is a set fusion operator. All experimental data are divided according to five kinds of intentions, and the contribution rate of each mode is taken as the average of multiple experiments. This paper summarizes and forms Figure5. The horizontal axis represents the average contribution rate of three modal information in intention fusion, and the vertical axis represents five types of intention. Obviously, due to the different complexity of the five types of intentions, the contribution of voice, gesture, and scene object in the final result is also different. Generally speaking, voice mode is the mode that expresses the most information in interaction. In fact, this is consistent with the characteristics of human interaction. Speaking can convey most of the daily information. Gesture and scene objects play different roles in different interaction intentions. For example, the intent of “confirm” and “dialogue” does not need the modal information of the scene object. Observing the experimental data, we can see that the information sources expressing different intentions are different, so we cannot understand the interaction intention in a fixed paradigm. In intention understanding, patterns cannot be regarded as a fixed, consistent, or complementary relationship. STMMI algorithm can adaptively find the fusion operator according to each expression of the user, and then understand the user tention. Experiments show that STMMI algorithm is a robust multimodal fusion method.Figure 5
The effectiveness of multimodal information.


#### 4.4.4. User Evaluation

For users, interaction experience is very important. The experiment used user evaluation to verify the interaction effect of the proposed method and whether the user experience met the initial needs of the design. In order to better illustrate the advantages of the multimodal fusion algorithm proposed in this paper, we investigated the evaluation of the system by the experimenters. User evaluation indicators include mental demand (MD), physical demand (PD), task reachability complexity (TRC), robot operation performance (ROP), and satisfaction (S). Each indicator is a 5-point system, divided into 5 levels. Specifically, the higher the scores of MD, PD, and TRC, the heavier the burden. The higher the ROP and S scores, the better the performance. Figure6 shows the average score of each interaction of 20 experimenters. From the figure, it could be known that in terms of user experience, STMMI’s multimodal fusion method had the advantages of a small operation burden and no special memory, and STMMI could have a higher task completion evaluation.Figure 6
User evaluation.According to the evaluation of the participants in the experiment, the interaction of the two fusion algorithms has a certain mental and physical burden, but the scores are less than 2.5. On the task reachability complexity index, due to the repeated interaction of the Bayesian fusion method, the complexity of users completing a task is high. The adaptability of the STMMI method reduces the number of interaction repetitions and verifies the impact of user personalization on experimental interaction. STMMI performs better than KDI in terms of robot operation performance and satisfaction. STMMI algorithm can understand the different intentions of different users and get a good evaluation in the dimension of task reachability complexity. Finally, in the scoring system with a full score of 5, STMMI is 0.5 higher than the Bayesian fusion method. However, both can get a score of more than 3 points, indicating that the experimenter has a high acceptance of human-computer interaction under multimodal fusion.


## 4.1. System Setup

In this work, we used Pepper humanoid robot. The robot has 20 degrees of freedom, 2 touch sensors in the hands and has 2 sonar sensors, 6 laser sensors, 2 infrared sensors, and 3 touch sensors on the legs when walking. These sensors improve the accuracy of execution and ensure the safety of the robot during execution. The interactive output modes of pepper include voice, movement, arm, and head posture.We also used external sensing devices to improve the accuracy of the captured information. A Kinect was used to obtain depth images of the human body.The system was implemented in a distributed manner on 2 computers, and each computer was equipped with Windows 10 operating system. One computer carried the functions of scene objects and gesture recognition, and the other computer completed the integrated multimodal fusion and robot control module. The robot and the two computers were in the same local area network. The robot action and speech interface could be called through the local area network.


## 4.2. Experimental Environments

In order to evaluate the multimodal fusion algorithm proposed in this paper, we built a laboratory environment that simulated daily life, which contained simple living items. Normal cameras are sensitive to changes in light when collecting data, so the ambient light was illuminated by indoor soft light. The experimental scene could be transferred when the interaction conditions were met.After the human entered the experimental scene, he was free to interact with the robot. The executable intent set under the robot’s preset is shown in Table2. The intent set is divided into 5 categories, and the specific explanation is shown in Table 2. A simple example: someone say “I want this.” He pointed to the apple on the table. The system should understand the intention of “Give him an apple.” When the robot is not activated for a long time, pepper will actively ask “do you need any help?.” At the same time, pepper is always monitoring the whole environment. Once there are changes that meet the conditions of executable intention, pepper will respond immediately.Table 2
Set of experimental executable intents.
Intent numberRobot control levelExample intention1Cancel“Stop,” “cancel” “not”…2Confirm“Go on,” “OK(gesture),” “OK,” “yes”…3Dialogue“What time is it? ,“ “tell a story”…4Move“Come,” “go forward,” “turn around “…5Acts on the scene object“Take the water cup,” “look for a phone”…


## 4.3. Experimental Design and Procedures

In order to verify whether the proposed STMMI algorithm has achieved the purpose of improving interaction efficiency and performance in daily service scenarios, we designed a simulation experiment. The experiment invited 20 volunteers, aged from 20 to 60, and the ratio of male to female was 1 : 1. Each person conducted 5 interaction experiments at intervals. During the process, the interaction instructions and interaction sequence were not preset to maximize the real purpose of the interaction.In the experiment process, the robot acted on the entire scene after executing the instructions, which caused changed the scene information. So, the scene state must be updated after each interaction. The entire system was always in a monitoring and waiting state, and no special wake-up instructions were required during the experiment. All interactions of the system were driven by the robot to cooperate with the experimenter.In order to clearly verify the fusion effect of STMMI, the following requirements were set for the participants:The experimenter should speak and make gestures as naturally as possible during the interactionThe experimenter should not intentionally be too close to the Kinect and the camera, so as not to block too much scene information and cause invalid interactionThe experimenter should put forward regular interaction instructions, and must not put forward instructions that are contrary to common sense, so as not to cause invalid interaction recordsThe experimenter should intersperse the experiment to prevent the experimenter’s interaction behavior from solidifyingDuring the experiment, we randomly select the multimodal fusion algorithm. However, the volunteers who participated in the interaction did not know which fusion algorithm was used. We counted the user ratings and experimental results. The data were used for subsequent analysis.


## 4.4. Experimental Results and Analysis

### 4.4.1. Interaction Results

The experiment verified the feasibility of the algorithm proposed in this paper, and a good interaction effect was achieved. Taking a water cup as an example, we selected several images to show the process of the experiment. Figure3(a) shows that the experimenter was sending instructions, Figures 3(b) and 3(c) show that the robot was executing instructions after understanding the intent. All the images came from the real experimental environment.Figure 3
The experimenter gave instructions and the robot executed the instructions.
(a)(b)(c)Experiments verified the effectiveness of the STMMI algorithm in this paper, and a better interaction effect was obtained. Especially compared with single-modal interaction, multimodal interaction achieved the interaction of information between modalities and obtained more accurate interaction intent. This method could be used for multimodal interactions that can complete collective coding. Our experiments proved the feasibility of the proposed algorithm in current applications.


### 4.4.2. Quantitative Results

The experimental data of each experiment are given in Table3, which contains the number of interactive instructions issued under STMMI, KDI [20] and Bayes-based multimodal fusion algorithm, the number of instructions recognized by the algorithm.Table 3
Experimental data.
CancelConfirmDialogueMoveActs on the scene objectThe intention expressed by the interlocutor (STMMI)463594167114Number of recognized instructions (STMMI)372180149106Number of correctly recognized instructions (STMMI)351971125101The intention expressed by the interlocutor (KDI)272379124105Number of recognized instructions (KDI)2113619790Number of correctly recognized instructions (KDI)1812527981The intention expressed by the interlocutor (Bayes)51398411792Number of recognized instructions (Bayes)2716517567Number of correctly recognized instructions (Bayes)2413455764Further analyze the experimental data and calculate the interactive accuracy and recall of the two algorithms. Among them, the interaction accuracy rate is equal to the number of correctly recognized instructions divided by the number of recognized instructions, and the interaction recall rate is equal to the number of recognized instructions divided by the total number of issued instructions. Figure4 shows a bar chart comparing the interactive accuracy and recall of different algorithms. From the data in the chart, it can be seen that STMMI method is obviously superior to other algorithms in the understanding performance of class 3 and 4 intentions. This is because the third and fourth types of intentions are complex, and the experimenter’s ways of conveying intentions vary. In addition, if the interlocutor pauses for a long time in an intention expression, it may cause the system to stop modal information collection and fusion before the interlocutor fully expresses it. At this time, if the key information in the intention is missing, the system will not be able to get an executable intention or misinterpret it as another intention. However, due to the adaptive ability of STMMI algorithm, the correct intention can still be obtained when noncritical information is missing.Figure 4
Accuracy and Recall of interaction.Through the experimental data in Figure4, it can be found that the first kind of intention-“cancel” STMMI has 14.1% higher accuracy and 35.9% higher recall than the Bayesian fusion method. This is because when expressing the intention of “no,” the interactor often uses body gestures such as waving his hand and shaking his head. Similarly, the STMMI of the second kind of intention “confirmation” has 30% higher accuracy and 40.2% higher recall than the Bayesian fusion method. When the interlocutor uses the “OK” gesture or nods to express the “confirm” intention, the Bayesian fusion method depends on the single-mode recognition results. The third kind of intention-“dialogue,” the accuracy of the two methods is similar, but the recall rate of STMMI is 27.5% higher than that of Bayes. And the recall rate of STMMI is 12% higher than that of KDI. Analyzing the experimental process, it is found that the experimenter often appears the information of the scene object in the dialogue, such as “where is my cup” Such a dialogue. The fourth type of intention-“moving,” when the interactor uses gestures to signal the robot to come, there will be false recognition caused by occlusion, which will affect the result of intention understanding. The fifth kind of intention-“acts on the scene objects.” It can be seen that when the recognition accuracy of the two algorithms is similar, the recall rate of STMMI is 22.7% higher than that of the Bayesian fusion method. And the recall rate of STMMI is 7% higher than that of KDI. This kind of intention needs the most complex information, and three kinds of modal information: voice, gesture, and scene recognition play a role.On the whole, STMMI performs significantly better than KDI method and Bayesian fusion method in recall rate. This shows that STMMI can adapt to the variability of interactive scenes. Although the Bayesian fusion method performs well in accuracy, it cannot understand an intention in many cases, resulting in no response of the robot. This kind of intention is weakly dependent on multimodal information, which depends on the comprehensiveness of the robot knowledge base to a greater extent.STMMI shows that it can get the experimenter’s intention more accurately when getting multimodal information. This is because STMMI adjusts its fusion operator in combination with instruction search in the fusion process in order to find the best result. Therefore, STMMI is more flexible to adapt to different information expression methods and find the best combination mode.


### 4.4.3. Modal Fusion Results

In order to verify the effectiveness of multimodal interaction and the efficiency of the STMMI algorithm in using multimodal information, we recorded the final fusion results of each experiment. Every time the system calculates the interaction intention, there is a set fusion operator. All experimental data are divided according to five kinds of intentions, and the contribution rate of each mode is taken as the average of multiple experiments. This paper summarizes and forms Figure5. The horizontal axis represents the average contribution rate of three modal information in intention fusion, and the vertical axis represents five types of intention. Obviously, due to the different complexity of the five types of intentions, the contribution of voice, gesture, and scene object in the final result is also different. Generally speaking, voice mode is the mode that expresses the most information in interaction. In fact, this is consistent with the characteristics of human interaction. Speaking can convey most of the daily information. Gesture and scene objects play different roles in different interaction intentions. For example, the intent of “confirm” and “dialogue” does not need the modal information of the scene object. Observing the experimental data, we can see that the information sources expressing different intentions are different, so we cannot understand the interaction intention in a fixed paradigm. In intention understanding, patterns cannot be regarded as a fixed, consistent, or complementary relationship. STMMI algorithm can adaptively find the fusion operator according to each expression of the user, and then understand the user tention. Experiments show that STMMI algorithm is a robust multimodal fusion method.Figure 5
The effectiveness of multimodal information.


### 4.4.4. User Evaluation

For users, interaction experience is very important. The experiment used user evaluation to verify the interaction effect of the proposed method and whether the user experience met the initial needs of the design. In order to better illustrate the advantages of the multimodal fusion algorithm proposed in this paper, we investigated the evaluation of the system by the experimenters. User evaluation indicators include mental demand (MD), physical demand (PD), task reachability complexity (TRC), robot operation performance (ROP), and satisfaction (S). Each indicator is a 5-point system, divided into 5 levels. Specifically, the higher the scores of MD, PD, and TRC, the heavier the burden. The higher the ROP and S scores, the better the performance. Figure6 shows the average score of each interaction of 20 experimenters. From the figure, it could be known that in terms of user experience, STMMI’s multimodal fusion method had the advantages of a small operation burden and no special memory, and STMMI could have a higher task completion evaluation.Figure 6
User evaluation.According to the evaluation of the participants in the experiment, the interaction of the two fusion algorithms has a certain mental and physical burden, but the scores are less than 2.5. On the task reachability complexity index, due to the repeated interaction of the Bayesian fusion method, the complexity of users completing a task is high. The adaptability of the STMMI method reduces the number of interaction repetitions and verifies the impact of user personalization on experimental interaction. STMMI performs better than KDI in terms of robot operation performance and satisfaction. STMMI algorithm can understand the different intentions of different users and get a good evaluation in the dimension of task reachability complexity. Finally, in the scoring system with a full score of 5, STMMI is 0.5 higher than the Bayesian fusion method. However, both can get a score of more than 3 points, indicating that the experimenter has a high acceptance of human-computer interaction under multimodal fusion.


## 4.4.1. Interaction Results

The experiment verified the feasibility of the algorithm proposed in this paper, and a good interaction effect was achieved. Taking a water cup as an example, we selected several images to show the process of the experiment. Figure3(a) shows that the experimenter was sending instructions, Figures 3(b) and 3(c) show that the robot was executing instructions after understanding the intent. All the images came from the real experimental environment.Figure 3
The experimenter gave instructions and the robot executed the instructions.
(a)(b)(c)Experiments verified the effectiveness of the STMMI algorithm in this paper, and a better interaction effect was obtained. Especially compared with single-modal interaction, multimodal interaction achieved the interaction of information between modalities and obtained more accurate interaction intent. This method could be used for multimodal interactions that can complete collective coding. Our experiments proved the feasibility of the proposed algorithm in current applications.


## 4.4.2. Quantitative Results

The experimental data of each experiment are given in Table3, which contains the number of interactive instructions issued under STMMI, KDI [20] and Bayes-based multimodal fusion algorithm, the number of instructions recognized by the algorithm.Table 3
Experimental data.
CancelConfirmDialogueMoveActs on the scene objectThe intention expressed by the interlocutor (STMMI)463594167114Number of recognized instructions (STMMI)372180149106Number of correctly recognized instructions (STMMI)351971125101The intention expressed by the interlocutor (KDI)272379124105Number of recognized instructions (KDI)2113619790Number of correctly recognized instructions (KDI)1812527981The intention expressed by the interlocutor (Bayes)51398411792Number of recognized instructions (Bayes)2716517567Number of correctly recognized instructions (Bayes)2413455764Further analyze the experimental data and calculate the interactive accuracy and recall of the two algorithms. Among them, the interaction accuracy rate is equal to the number of correctly recognized instructions divided by the number of recognized instructions, and the interaction recall rate is equal to the number of recognized instructions divided by the total number of issued instructions. Figure4 shows a bar chart comparing the interactive accuracy and recall of different algorithms. From the data in the chart, it can be seen that STMMI method is obviously superior to other algorithms in the understanding performance of class 3 and 4 intentions. This is because the third and fourth types of intentions are complex, and the experimenter’s ways of conveying intentions vary. In addition, if the interlocutor pauses for a long time in an intention expression, it may cause the system to stop modal information collection and fusion before the interlocutor fully expresses it. At this time, if the key information in the intention is missing, the system will not be able to get an executable intention or misinterpret it as another intention. However, due to the adaptive ability of STMMI algorithm, the correct intention can still be obtained when noncritical information is missing.Figure 4
Accuracy and Recall of interaction.Through the experimental data in Figure4, it can be found that the first kind of intention-“cancel” STMMI has 14.1% higher accuracy and 35.9% higher recall than the Bayesian fusion method. This is because when expressing the intention of “no,” the interactor often uses body gestures such as waving his hand and shaking his head. Similarly, the STMMI of the second kind of intention “confirmation” has 30% higher accuracy and 40.2% higher recall than the Bayesian fusion method. When the interlocutor uses the “OK” gesture or nods to express the “confirm” intention, the Bayesian fusion method depends on the single-mode recognition results. The third kind of intention-“dialogue,” the accuracy of the two methods is similar, but the recall rate of STMMI is 27.5% higher than that of Bayes. And the recall rate of STMMI is 12% higher than that of KDI. Analyzing the experimental process, it is found that the experimenter often appears the information of the scene object in the dialogue, such as “where is my cup” Such a dialogue. The fourth type of intention-“moving,” when the interactor uses gestures to signal the robot to come, there will be false recognition caused by occlusion, which will affect the result of intention understanding. The fifth kind of intention-“acts on the scene objects.” It can be seen that when the recognition accuracy of the two algorithms is similar, the recall rate of STMMI is 22.7% higher than that of the Bayesian fusion method. And the recall rate of STMMI is 7% higher than that of KDI. This kind of intention needs the most complex information, and three kinds of modal information: voice, gesture, and scene recognition play a role.On the whole, STMMI performs significantly better than KDI method and Bayesian fusion method in recall rate. This shows that STMMI can adapt to the variability of interactive scenes. Although the Bayesian fusion method performs well in accuracy, it cannot understand an intention in many cases, resulting in no response of the robot. This kind of intention is weakly dependent on multimodal information, which depends on the comprehensiveness of the robot knowledge base to a greater extent.STMMI shows that it can get the experimenter’s intention more accurately when getting multimodal information. This is because STMMI adjusts its fusion operator in combination with instruction search in the fusion process in order to find the best result. Therefore, STMMI is more flexible to adapt to different information expression methods and find the best combination mode.


## 4.4.3. Modal Fusion Results

In order to verify the effectiveness of multimodal interaction and the efficiency of the STMMI algorithm in using multimodal information, we recorded the final fusion results of each experiment. Every time the system calculates the interaction intention, there is a set fusion operator. All experimental data are divided according to five kinds of intentions, and the contribution rate of each mode is taken as the average of multiple experiments. This paper summarizes and forms Figure5. The horizontal axis represents the average contribution rate of three modal information in intention fusion, and the vertical axis represents five types of intention. Obviously, due to the different complexity of the five types of intentions, the contribution of voice, gesture, and scene object in the final result is also different. Generally speaking, voice mode is the mode that expresses the most information in interaction. In fact, this is consistent with the characteristics of human interaction. Speaking can convey most of the daily information. Gesture and scene objects play different roles in different interaction intentions. For example, the intent of “confirm” and “dialogue” does not need the modal information of the scene object. Observing the experimental data, we can see that the information sources expressing different intentions are different, so we cannot understand the interaction intention in a fixed paradigm. In intention understanding, patterns cannot be regarded as a fixed, consistent, or complementary relationship. STMMI algorithm can adaptively find the fusion operator according to each expression of the user, and then understand the user tention. Experiments show that STMMI algorithm is a robust multimodal fusion method.Figure 5
The effectiveness of multimodal information.


## 4.4.4. User Evaluation

For users, interaction experience is very important. The experiment used user evaluation to verify the interaction effect of the proposed method and whether the user experience met the initial needs of the design. In order to better illustrate the advantages of the multimodal fusion algorithm proposed in this paper, we investigated the evaluation of the system by the experimenters. User evaluation indicators include mental demand (MD), physical demand (PD), task reachability complexity (TRC), robot operation performance (ROP), and satisfaction (S). Each indicator is a 5-point system, divided into 5 levels. Specifically, the higher the scores of MD, PD, and TRC, the heavier the burden. The higher the ROP and S scores, the better the performance. Figure6 shows the average score of each interaction of 20 experimenters. From the figure, it could be known that in terms of user experience, STMMI’s multimodal fusion method had the advantages of a small operation burden and no special memory, and STMMI could have a higher task completion evaluation.Figure 6
User evaluation.According to the evaluation of the participants in the experiment, the interaction of the two fusion algorithms has a certain mental and physical burden, but the scores are less than 2.5. On the task reachability complexity index, due to the repeated interaction of the Bayesian fusion method, the complexity of users completing a task is high. The adaptability of the STMMI method reduces the number of interaction repetitions and verifies the impact of user personalization on experimental interaction. STMMI performs better than KDI in terms of robot operation performance and satisfaction. STMMI algorithm can understand the different intentions of different users and get a good evaluation in the dimension of task reachability complexity. Finally, in the scoring system with a full score of 5, STMMI is 0.5 higher than the Bayesian fusion method. However, both can get a score of more than 3 points, indicating that the experimenter has a high acceptance of human-computer interaction under multimodal fusion.


## 5. Conclusions

In this paper, the intent understanding of intelligent service robots under multimodal situation is investigated. In the recognition of single-mode information by the neural network, we have achieved good recognition accuracy. The STMMI algorithm attempts to migrate applications at the neural network structure level, which can make the robot flexibly respond to different interaction scenes. This research focuses on the combination of human-computer interaction and neural network technology.



---

*Source: 3952758-2022-08-29.xml*