# A Parallel Multimodal Integration Framework and Application for Cake Shopping
## 2 Methods
### Multimodal Virtual Shopping System Prototype
#### 2.3.3 Situated Multimodal Dialogue  
In our task-oriented dialogue system, the situated dialogue dataset adopts a hierarchical and recursive semantic representation for user dialog states and system dialog acts, similar to [PERSON]'s work [51]. Every meaning is rooted at a user or system node to distinguish between the two classes. Non-terminals of the representation include user verbs, system actions, and slots. User verbs represent the predicate of the user intent, such as Ask, Update, and Book. For example, users can **Update** the recommended cake by specifying an attribute value of cakes (see Turn 2 of Figure 3). System actions represent the system dialog act in response to a user intent. For example, the system could **Prompt** for a slot value, and **Show** the object that matches the slot values in the environment (see Turn 1 of Figure 3). Slots are categorized into attribute and intent slots, with an example in Figure 6. Attribute slots define the object properties and accept either a categorical label (e.g., \"around 500 Yuan\" for cake price) or an open value (e.g., \"450 yuan\"). Intent slots specify a set of required slots and corresponding system actions for each user intent. We employ regular expressions to identify user intent and slot values. Regular expression patterns work well in limited-domain conversations. Through extensive user dialogue experiments, we collect user intent keywords and utterance formats and formulate regular expressions for intent and attribute slots. Figure 7 illustrates an example of the intent recognition process. The regular expression recognizes price keywords and updates attribute values. In summary, our multimodal dialogue dataset consists of 5 attribute slots, 6 intent slots, 6 system actions, 18 regular expressions, and 60 types of system utterances.  
In the cake database, each cake has five attributes: size, price, flavor, theme, and population. Table 2 lists the attribute values for all cakes in the database. Based on the history attribute values and the updated ones, the system matches (system.Find) the cake in the database that best meets the specified attributes. The cake matching follows a hierarchical attribute matching method. For example, as shown in Turn 3 of Figure 3, the attribute values are [\"\(>\)11\", \"around_500\", \"cream\", \"old\", \"old\"]. The attribute importance order is \([1,3,4,5,2]\). During the matching process, the system finds cakes in the database that meet the attribute with the highest importance, denoted as '\(R_{1}-\)size-\(>\)11'. The results are [\"cake018\", \"cake019\",..., \"cake024\"]. Next, the system operates '\(R_{2}-\)population \(-\)old', resulting in [\"cake018\", \"cake022\"]. The result of '\(R_{3}-\)price \(-\)around_500' is [\"cake018\"]. Since there is only one cake in the result, there is no need to match other attributes. \"cake018\" is the final choice. If the results have not been narrowed down to one cake after matching all attributes, the system will randomly select one from the results as the final choice. If the matching results of \(R_{j}\) are empty, the final result will be randomly selected from the results of \(R_{j-1}\). In our hierarchical attribute matching, the attribute importance order is variable. The initial importance order is manually specified based on experience, that is, \([1,2,3,4,5]\). When users update a cake attribute value, the importance order of this attribute is set to one, and other attributes are adjusted accordingly (see Turn 2 and Turn 3 of Figure 3). If users update multiple attribute values simultaneously, the importance order of these attributes is determined according to the initial order. Additionally, any attributes that have not been updated in the conversation will be set to the attribute values of the recommended cake (see Turn 3 of Figure 3).  
Figure 6: Examples of attribute slots and intent slots.  
Figure 7: An example of the intent recognition process. User utterances are marked in red. The regular expression for **Update** price is shown in the box.