## 1 Introduction  
Since the first personal computers were released in the late 1970s, computer graphics, networking, and data processing technologies have evolved to provide high accuracy and powerful performance for rich interactive applications. These developments have enabled various methods of interaction between humans and machines to provide more natural and user-friendly interfaces, thereby contributing to improve user experience and usability [1]. Increased network performance allows for communication to handle interaction modalities in real-time, which is exemplified by smart environments [2, 3] and autonomous vehicles [4]. In this study, we focus on interaction methods that are implemented through one or more modalities. \"Modality\" is defined as a method of transferring data that can be used to interpret the sender's state [5] or intent [6]. A single modality, such as touch on a keyboard, mouse, or screen of a mobile device, is commonly used in many interactive systems. However, by combining modalities, more information can be communicated, leading to an enhanced experience. Thus, it is unsurprising that multiple modalities have started to be considered within a single system [5, 6]. Furthermore, users have different interaction preferences and contextual requirements for different modalities (e.g., visual modality in loud environments). Thus, modality flexibility has become a desired feature of interaction systems that aims to meet the requirements of multiple needs, use cases, and contexts. Expanding the diversity of modalities aims not only to overcome the limitation of available modalities for the end user, but also provide a more natural interaction to enhance usability [5, 6]. A system that employs multiple modalities in the input or output channel is referred to as a multimodal interaction (MI) system. Figure 1 illustrates our interpretation of a general architecture of MI systems. An MI system allows an agent to send input data to it by combining and organizing more than one modality. In addition, the MI system can provide output data in multiple modalities. The agent that interacts with the MI system can be either a human or machine. Advances in technology have increased the possibility of developing new forms of MI systems that can bridge the real world and virtual world. The advancement of network and computing power allows a variety of devices to connect to the Internet with sufficient performance levels, thereby making it possible to achieve real-time rendering of virtual content that is based on the agent's context data. The Internet of Things (IoT) is the technology that realizes this objective, whereby information and communication technologies (ICT), such as sensors, are mounted on physical objects and connected to the Internet [7]. It is expected that physical items will be increasingly equipped with ICT and turned into IoT [8], thus introducing opportunities to apply interaction modalities in new ways. While IoT mainly focuses on enabling ubiquitous connectivity, connecting physical objects to the Internet, and providing contextual data through devices, augmented reality (AR) is a type of technology that can be utilized to provide interactive interfaces in MI systems. AR is used to visualize graphical objects on a view of the real world, where a real-time camera feed on a device or head-mounted display (HMD) typically represents the view. AR objects can not only be used as simple data representations [9; 10], but can also be combined with an input modality, such as touch on a mobile device [11], as part of a user interface (UI) to allow an agent to interact with an MI system [12; 13]. This property distinguishes between applications that only utilize AR as a data visualization tool and those that use AR as part of their UI to interact with the underlying MI system. For efficient interaction with the IoT, we need new means of interaction technologies.