# A Parallel Multimodal Integration Framework and Application for Cake Shopping
## 2 Methods
### Parallel Multimodal Integration Framework  
The proposed parallel multimodal integration framework is illustrated in Figure 1. It comprises input modality operation flow (red part in Figure 1), output modality operation flow (green part), the data distribution module (gray part), and the situated logic processor (blue part). The framework is so robust that the unexpected disconnection or damage to a device does not impact the overall system. Each modality in the system operates parallelly and has unrestricted resource allocation, reducing the development barriers. Furthermore, our independent and asynchronous unimodal devices, the efficient and straightforward fusion method, and the flexible and rapid data distribution module collectively contribute to reducing system latency and meeting the practical demands of various applications. To obtain the final decision, we propose a relative weighted fusion method and modality delay strategy for fusing multiple modalities at the decision level. During the fusion phase, each modality's independent decisions \(L_{1},\ldots,L_{N}\) are sent to the fusion module and saved in the corresponding modality slots. For the \(j\)-th modality, the value of the modality slot defaults to the zero feature \(D_{0}\) in the absence of classification results, i.e., \(L_{j}=D_{0}\). When receiving the classification result \(D_{j}\), \(L_{j}\) is updated, i.e., \(L_{j}=D_{j}\). This approach ensures the scalability and robustness of the system. Unlike fusion methods designed for well-aligned datasets [43], our fusion module must handle real-time multimodal data from users. Inevitably, there is a temporal misalignment issue in which the modality input occurs. The misalignment comes from user-triggered time variations (e.g., waving hands after saying \"Goodbye\") and computation delays. Dealing with misalignment--often called multimodal synchronization--is particu  
Figure 1: Overview of the proposed parallel multimodal integration framework. larly important to multimodal systems [40]. Inspired by [40], we propose a modality delay strategy to address this problem. Specifically, we set a fusion delay threshold. When the fusion module receives the first valid classification result, the timestamps are recorded. The fusion module waits for other possible modality results within the delay threshold. If the same modality receives the other valid result, the result will be updated. The valid classification results refer to the data with a straightforward user intent. Regarding expression modality, the neutral expression category is considered invalid, while the happy expression category is considered valid. The modality delay strategy ensures that the system effectively captures intent decisions across various modalities, enhancing the accuracy of intent recognition. It is crucial to set a reasonable fusion delay threshold. If the threshold is too small, the chances of valid user input being disregarded will be high. If the threshold is too large, users will perceive and become aware of system latency, negatively impacting the user experience and satisfaction [44]. We recommend adjusting the threshold based on the specific usage scenarios, device latency, and users' familiarity with the system. For example, controller-based selection in virtual reality takes longer than mouse-based selection on a computer. The threshold should be set larger. Furthermore, the longer it takes for unimodal decision making, and the less familiar users are with the device, the larger the delay threshold should be set. After the waiting phase, The individual modality results are fused using the relative weighted (RW) fusion method. RW can be formulated as:  
\[H=\operatorname{argmax}\sum_{j=1}^{N}w_{j}L_{j} \tag{1}\]  
where \(H\) is the final decision, and \(w_{j}\) is the weight of \(j\)-th modality. For each modality result, the categories with the highest probability are \(q_{1},\dots,q_{N}\).