FORM: An Extensible, Kinematically-based Gesture Annotation Scheme.

Annotated corpora have played a critical role in speech and natural language research; and, there is an increasing interest in corpora-based research in sign language and gesture as well. As examples, consider the tools Anvil and MediaTagger. These are excellent tools which allow for multi-track annotation of videos of speakers or signers. With tools such as these, researchers can create corpora containing, for example, grammatical information, discourse structure, facial expression, and gesture. The issue, then, is not the ability to create corpora containing gesture and speech information, but the type of information captured when describing gestures. We present a non-semantic, geometrically-based annotation scheme, FORM, which allows an annotator to capture the kinematic information in a gesture just from videos of speakers. In addition, FORM stores this gestural information in Annotation Graph format—allowing for easy integration of gesture information with other types of communication information, e.g., discourse structure, parts of speech, intonation information, etc. 1. Annotated Corpora and Multi-Modal Data Annotated corpora have played a critical role in speech and natural language research; and, there is an increasing interest in corpus-based research in sign language and gesture as well. It was for this purpose that the FORM annotation scheme was developed1. This is a useful tool which allows for multi-tier gesture annotation of videos of speakers or signers. For example, Figure 1’s four stills are from a video sequence of Brian MacWhinney teaching a research methods course at Carnegie Mellon University. These data were chosen because they are part of the TalkBank collection (http://www.talkbank.org). Figure 1: Snapshots of Brian MacWhinney on January 24, 2000 The FORM annotation of the video, from timestamp 1:13.34 (1 minute 13.34 seconds) to timestamp 1:14.01 is The author wishes to sincerely thank Adam Kendon for his input on the FORM project. He has provided not only suggestions as to the direction of the project, but also his unpublished work on a kinematically-based gesture annotation scheme was the FORM project’s starting point (Kendon, 2000). shown in Figure 2. Figure 2: FORM annotation of Jan24.mov, using Anvil as the annotation tool This is the view on the data that a particular tool, Anvil (Kipp, 2001), presents to the annotator. However, FORM uses annotation graphs (Bird and Liberman, 1999) as its logical representation of the data. So regardless of which annotation tool is used, FORM’s internal view is the annotation graph given in Figure 3. Both Annotation Graphs (AGs) and FORM are presented in greater detail, below. However, for now note, first, that an annotation graph is a directed acyclic graph (DAG) such that the nodes represent timestamps of some given signal and the arcs represent some linguistic event that spans the time between the timestamps. Second, note that FORM uses vectors of attribute:value pairs to capture the gestural 1:13.34 1:13.57 1:13.61 1:13.67 1:14.01 UpperArm.Movement ForeArm.Movement UpperArm.Movement ForeArm.Movement HandandWrist.Movement