Patent ID: 8194081

Claim:
A pen-based computing system for capturing and displaying writing and audio, the system comprising: a smart pen device comprising: a microphone configured to record audio and produce audio data based thereon, the audio data including a time component, a gesture capture system configured to capture writing gestures and produce writing gesture data based thereon, the writing gesture data including a capture timing relative to a time component; a processor, a storage medium, instructions contained on the storage medium and capable of being executed by the processor, the instructions for combining the audio data and the writing gesture data according to the time components thereof to generate audio ink data, and an interface for uploading the audio ink data to a computer system; and a computer program product for playing back the audio ink data on the computer system, the computer program product comprising a computer-readable storage medium containing computer program code for: receiving a selection to play the audio ink data in a user interface of the computing system; displaying the captured writing gesture data in a first color; and playing back the audio ink data by animating the captured writing gestures over time and playing the recorded audio in synchronization with the animating of the captured writing gestures, the animating comprising changing the writing gesture data from the first color to a second color using a playback timing that corresponds to the capture timing of the writing gestures when originally captured, wherein the writing gestures captured prior to a current playback time are displayed in the second color and writing gestures captured after the current playback time are displayed in the first color.