A data access system

Abstract: A note-taking system based on a notepad computer with an integrated audio/video-recorder is described. A document is created or retrieved. As the user types on the keyboard or writes with the stylus or similar input instrument, each character or stroke that is input by the user is invisibly time-stamped by the computer. The audio/video stream is also continuously time-stamped during recording. To play a section of recording back, the user selects part of the note (perhaps by circling it with a stylus) and invokes a "playback selection" command. The computer then examines the time-stamp and "winds" the record to the corresponding place in the audio/video recording, where it starts playing - so that the user hears and/or sees what was being recorded at the instant the selected text or strokes were input. With a graphical user interface, the user may input key "topic" words and subsequently place check marks by the appropriate word as the conversation topic veers into that neighborhood.

Description: This invention relates to a data access system, and in particular to one in which the taking of notes usually accompanies the recording of data. The notes themselves are used to gain selective access to the data. At seminars, interviews or meetings, it has long been recognised that the simple act of taking notes helps the writer memorize key facts. On the other hand, to make a note, the listener has to divert attention from the speaker, figure out how to encode the information he has heard or the idea he has had, and then focus on writing it down. It doesn't seem surprising that the listener often loses the speaker's thread! Nowadays many people resort to making audio or video recordings of important events and then spending time transcribing key ideas afterwards Locating the interesting parts of an audio or video recording is often very time-consuming and tedious, which reduces the likelihood that the listener will bother to transcribe the tape, or even bother to make the recording in the first place. In many situations the need to record is not anticipated. Only after the event, sometimes long afterwards, does the need become evident. To overcome this problem it may eventually become technologically possible, from a storage point of view, to record and store every second of a person's working day. The problem of locating key pieces of information from this huge and expanding base of recordings becomes overwhelming. US-A-4,841,387 discloses an arrangement for recording information relating to an event on an audio or video tape, and for indexing handwritten notations (made on an electronic touchpad) concerning the event to the recorded information, including a microprocessor for correlating relative positions, in a correlation table, of the handwritten information (in terms of its x,y coordinates on the touchpad) and the recorded information (in terms of its position on a tape as determined by a tape motion sensor). The correlation table is used in playing back part of the recorded information corresponding to a user-selected notation. US-A-4,425,586 discloses an apparatus and method for storing and iterrelating visual data and computer information. Each of a video tape recorder and a computer automatically record and display where related information is stored in its own mechanism as well as its counterpart mechanism. This allows the user to determine the location of all the corresponding data stored both on video tape and on diskette, by examining only one storage medium. US-A-4,924,387 discloses a computerised court reporting system for providing a simultaneous written and video record. The system includes a system clock and a control system for periodically obtaining the current time and date from the clock and appending it to the recording of the video and audio and a signal representing keystrokes entered via the court reporter's keyboard. Provision is made for flagging, searching and correcting questionable words and for reminding the reporter to do so. The present invention provides a system for providing random access to a data set, preferably an audio and/or video data set, comprising: means, including a device, for capturing displayable, indicia generated by a user interface receiving, in use, manual input from a user, for subsequent selection by the user; and means for indexing into said data set to identify the part thereof that is correlated with a user-selected indicium; characterised by including means for automatically time-stamping each indicium upon its generation, and means for automatically time-stamping said data set, said time-stamped data set and said indicia having a common time base; means for automatically storing, retrieving and displaying said indicia; and means for determining the time-stamp of each user-selected indicium, for use by said indexing means; said device for capturing displayable indicia being adapted for electronically both capturing and displaying said user generated indicia. This invention builds upon these ideas by providing a semi-automatic, fine-grained, audio/video indexing tool. Although the invention focuses on gaining access to audio and video records, it is not limited to this application. It is envisaged that the invention could also be applied to accessing files in a computer system, phone call records, or any other time-stamped data set. The present invention will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 is a diagrammatic view of one form of the invention using a laptop computer; Figure 2 is a diagrammatic view of another form of the invention using a notepad computer with stylus input, and Figure 3 is a diagrammatic view of one form of the architecture of the invention. The example of the invention shown in Figure 1 combines a laptop computer 2 with a video or audio recording device 4. In the example of the invention shown in Figure 2, the portable computer is replaced by a notepad computer 6 with stylus or other graphical input device 8, connected to video or audio recording device 4. The stylus input device may take the place of the keyboard of the portable computer as the user's means of input. Alternatively, a keyboard 10 may be connected to the notepad computer for input. Either form of computer is interfaced to the recorder in such a manner that the computer can monitor the recorder's progress during recording. To enable this, the recorder may emit time-code or make other unique index information continuously available to the computer. Additionally the computer can operate the recorder directly, causing it to record, or play from a specific index point as required. In any situation it is possible for the computer to find out the time-code or index information. In the general system shown in Figure 3, the computer presents a document editor style user interface to the user. This interface may resemble that of a word-processor, an illustrator, or an application specific interface, depending on the user's needs and computer hardware. The user interface provides commands for creating a new document and starting and stopping recording. A new document is created, or an old one recalled, and recording commences. The editor 22 allows the user to draw, type or sketch on the blank document using keyboard 10, stylus 8, or other graphical input device. As each mark (or indicium) 26, is added to the document (an 'indicium' is any indivisible displayable symbol created by the interface, e.g. pen-stroke, character, graphical symbol, etc.), the editor time-stamps and stores it in an indicium-to-timestamp index 14. The video-frame time-stamper 20 also continuously notes the index information 32 arriving from the recorder 4 at that instant, and time-stamps it and stores it in a timestamp-to-timecode index 16. The time-stamps are not visible to the user - they are stored with the computer's internal representation of the indicia. When the session is over, the user stores away the document; the contents of the two indexes, and the recording for subsequent use. Any method that allows time-stamps to be used as an index into the recording may be used instead of the timestamp-to-timecode index, e.g. interpolation of the timecode itself. When the user wants to recall sections of the recording, the appropriate related document is retrieved and recalled to the display 30 by the browser 24. Using the stylus 8, or the keyboard 10, the user selects one or more indicia on the document, perhaps by circling them, positioning a cursor, or typing an identifying name, and instructs the browser 24 to play the associated video section. The browser 24 identifies the selected indicium or indicia, looks up the time-stamp(s) in the indicium-to-timestamp index 14; looks up the timestamp(s) in the timestamp-to-timecode index 16, and plays the section of the recording in the area indicated by the resultant timecode. Thus the user sees what was recorded at the time the marks were made on the video monitor 28. Indicium-to-timestamp and timestamp-to-timecode indices may be combined into a single indicium-to-timecode index. It is expected that users will develop shorthand notations for marking the document to indicate an interesting idea, change of topic or speaker, and so forth. It also is expected that users will type or handwrite key words as the topic arises, and then add additional marks nearby each time the subject veers in the direction of that topic. Thus, to hear all section of a seminar associated with the same topic, it is necessary only to circle all the marks in the location of the topic keyword. Accordingly it will be seen that the present invention provides a simple, inexpensive, lightweight, low-power device for quick access to data records.

Claims: A system for providing random access to a data set, preferably an audio and/or video data set, comprising: means (8,30,22), including a device (30), for capturing displayable indicia (26) generated by a user interface (22) receiving, in use, manual input from a user, for subsequent selection by the user; and means (24,14,16) for indexing into said data set to identify the part thereof that is correlated with a user-selected indicium; characterised by including means for automatically time-stamping (22) each indicium upon its generation, and means (20) for automatically time-stamping said data set, said time-stamped data set and said indicia (26) having a common time base; means for automatically storing (14), retrieving (24) and displaying (30) said indicia (26); and means (24,14) for determining the time-stamp of each user-selected indicium,for use by said indexing means (24,16); said device (30) for capturing displayable indicia (26) being adapted for electronically both capturing and displaying said indicia (26). The system according to claim 1, wherein said indexing means (24,16) includes means (16) for identifying the part of the data set having the same time-stamp as that determined by the determining means (24,14). A system as claimed in claim 1 or 2, wherein said means for automatically retrieving (24) and displaying (30) said displayable indicia (26) includes user means (30) for displaying the captured indicia (26) and user means (30,8) for selecting one of the displayed captured indicia (26) as said user-selected indicium. A system according to any of the preceding claims and adapted to be coupled to a reproduction device (4,28), preferably an audio and/or video reproduction device, further including means (24) for reproducing the identified part of said data set. A system according to claim 4, including means (24) for commanding the reproduction device (4,28) to reproduce said identified part, commencing with the video frame corresponding to the time stamp of the selected indicia (26). A system as claimed in any of claims 1-5, which is implemented (A) by means of a portable computer (2), and/or (B) by means of a portable video recorder (4), preferably a camcorder. A system as claimed in claim 6, in which the portable computer (2) is a notepad computer with stylus input.

Inventors: LAMMING MICHAEL G, LAMMING, MICHAEL G.
Applicants: XEROX CORP, XEROX CORPORATION
Application year: 1992
Main classification: G06F15
IPC codes: G06F17, G06F15, G06F12, H04N7