Augmented Things: Enhancing AR Applications leveraging the Internet of Things and Universal 3D Object Tracking Jason Rambach∗1, Alain Pagani†1 and Didier Stricker‡1,2 1German Research Center for Artiﬁcial Intelligence (DFKI), Kaiserslautern, Germany 2TU Kaiserslautern, Kaiserslautern, Germany ABSTRACT With Augmented Reality (AR) reaching its technological maturity, there is a constantly increasing request for AR applications that ad- dress a broadened user base. Current AR applications still tend to be spatially and temporally conﬁned either by applying to a sin- gle use case or by their tracking requirements. At the same time, with the rise of the Internet of Things (IoT), more and more every- day objects are ﬁtted with wireless connection interfaces. In this work, we present the concept of Augmented Things, in which ob- jects carry all the necessary tracking and augmentation information required for AR applications. This allows a user to connect to them, load this information on his personal device and have augmenta- tions such as maintenance instructions or product origin and usage displayed. For this, we also present and evaluate a universally ro- bust 3D object tracking framework based on high quality 3D scans of the objects. Index Terms: H.5.1 [Multimedia Information Systems]: Arti- ﬁcial, augmented, and virtual realities—; I.4.8 [Scene Analysis]: tracking— 1 INTRODUCTION Augmented Reality is an emerging technology with numerous ap- plications in a variety of ﬁelds such as industrial construction and maintanance, medicine and rehabilitation, entertainment, education and many others [4]. Accurate camera pose tracking is one of the key enabling tech- nologies for AR using handheld or wearable devices. Precise track- ing of the camera’s six degree of freedom pose (6DoF) correspond- ing to its orientation and position in every video frame allows for realistic placement of 3D augmentations and integration in the real world [10, 16]. Existing tracking approaches rely either on uncov- ering prediﬁned targets such as markers or natural features (e.g. edges and lines) of objects [6, 7] or employ Simultaneous Localiza- tion and Mapping (SLAM) methods to uncover the structure of the environment. In any case, an AR application needs to contain its own tracking solution, either self-implemented or as an integration of a commercially available tracking framework. In parallel, with the advance of Internet of Things (IoT), origi- nating from the reduced cost and size of sensors and miniature hard- ware components, more and more devices and everyday objects are embedded with microcomputers and wireless network connectivity [12]. This leads to the idea of Augmented Things presented in this paper. We propose to have objects that carry all the information that ∗Jason Raphael.Rambach@dfki.de †Alain.Pagani@dfki.de ‡Didier.Stricker@dfki.de Figure 1: In Augmented Things connected objects carry and share their AR information is required for them to be used as tracking targets and the augmen- tation models that should be displayed for this model. Thus, a user that wishes to display AR augmentations on that object (e.g. for usage instructions, maintenance or simply entertainment) can use a universal object tracking application from his personal device that will connect to the object, receive the object AR package (tracking information and augmentations) and directly track the object and display augmentations. This leads to a separation of tracking and content creation for AR, that greatly simpliﬁes the creation of AR applications. The main contributions of this work are: • The presented concept of Augmented Things and the applica- tion architecture for its realization. • A robust universal 3D object tracking framework suited for the Augmented Things architecture utilizing off-screen ren- dering of the object 3D model and a high degree of paral- lelization. • A fast and robust mulithreaded seamless initialization and reinitialization method for tra