TUIO: A protocol for table-top tangible user interfaces

In this article we present Tuio, a simple yet versatile protocol designed specifically to meet the requirements of table-top tangible user interfaces. Inspired by the idea of interconnecting various existing table interfaces such as the reacTable* , being developed in Barcelona and the tDesk  from Bielefeld, this protocol defines common properties of controller objects on the table surface as well as of finger and hand gestures performed by the user. Currently this protocol has been implemented within a fiducial marker-based computer vision engine developed for the reacTable* project. This fast and robust computer vision engine is based on the original d-touch concept , which is also included as an alternative to the newer fiducial tracking engine. The computer vision framework has been implemented on various standard platforms and can be extended with additional sensor components. We are currently working on the tracking of finger-tips for gestural control within the table interface. The Tuio protocol has been implemented using OpenSound Control [4] and is therefore usable on any platform supporting this protocol. At the moment we have working implementations for Java, C++, PureData, Max/MSP, SuperCollider and Flash. 1 General Observations This protocol definition is an attempt to provide a general and versatile communication interface between tangible table-top controller interfaces and underlying application layers. It was designed to meet the needs of table-top interactive surfaces, where the user is able to manipulate a set of objects. These objects are tracked by a sensor system and can be identified and located in position and orientation on the table surface. Additionally we defined a special cursor object, which doesn’t have a unique ID and doesn’t provide rotation information. The protocol’s flexible design offers methods for selecting which information will be sent. This flexibility is provided without affecting existing interfaces, or requiring re-implementation to maintain compatibility. 2 Implementation Details The Tuio protocol defines two main classes of messages: set messages and alive messages. Set messages are used to communicate information about an object’s state such as position, orientation, and other recognized states. Alive messages indicate the current set of objects present on the surface using a list of unique session IDs. To avoid possible errors evolving out of packet loss, no explicit add or remove messages are included in the Tuio-protocol. The receiver deduces object lifetimes by examining the difference between sequential alive messages. In addition to set and alive messages, fseq messages are defined to uniquely tag each update step with a unique frame sequence ID. To summarize: – object parameters are sent after state change using a set message – on object removal an alive message is sent – the client deduces object addition and removal from set and alive messages – fseq messages associate a unique frame id with a set of set and alive messages 2.1 Efficiency & Reliability In order to provide low latency communication our implementation of the Tuio protocol uses UDP transport. When using UDP the possibility exists that some packets will be lost. Therefore, our implementation of the Tuio protocol includes redundant information to correct possible lost packets, while maintaining an efficient usage of the channel. An alternative TCP connection would assure the secure transport but at the cost of higher latency. For efficiency reasons set messages are packed into a bundle to completely use the space provided by a UDP packet. Each bundle also includes a redundant alive message to allow for the possibility of packet loss. For larger object sets a series of packets, each including an alive message are transmitted. When the surface is quiescent, emphalive messages are sent at a fixed rate dependent on the channel quality, for example once every second, to ensure that the receiver eventually acquires a consistent view of the set of alive objects. The state of each alive but unchanged object is periodically resent with additional set messages. This redundant information is resent at a lower rate, and includes only a subset of the unchanged objects at each update. The subset is continuously cycled so that each object is periodically addressed. Finally, each packet is marked with a frame sequence ID (fseq) message: an increasing number which is the same for all packets containing data acquired at the same time. This allows the client to maintain consistency by identifying and dropping out-of-order packets . To summarize: – set messages are bundled to fully utilize UDP packets – each bundle of set messages includes an alive message containing the session IDs of all currently alive tangible objects – when the surface is quiescent the alive message is resent periodically – the state of a cycling subset of alive but unchanged objects is continuously resent via redundant set messages – each bundle contains a frame sequence (fseq) message It should be noted that the retransmission semantics described here are only one possible interpretation of the protocol. Other possible methods include: (1) weighting the frequency of retransmission according to recency of value changes using a logarithmic back-off scheme and, (2) trimming the set of values to be retransmitted using asynchronous acknowledgments from the client.