Method, apparatus and system for applying an annotation to a portion of a video sequence

A method, system and apparatus for applying an annotation to a portion of a video sequence. The method comprises the steps of receiving the video sequence in real-time during capture of the video sequence, monitoring in real-time a plurality of signals associated with the video sequence, and receiving an indication associated with a spatial area of interest of at least one frame during capture of the video sequence. The method further comprises selecting, from the plurality of monitored signals, a temporal portion of one of the plurality of monitored signals for annotation, said selection being based upon at least the spatial area of interest and a temporal variation measure in at least one of the plurality of monitored signals, applying an annotation to a portion of the video sequence corresponding to the selected temporal portion; and storing the annotation in an annotation record associated with the video sequence.

REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit under 35 U.S.C. § 119 of the filing date of Australian Patent Application No. 2015203661, filed 30 Jun. 2015, hereby incorporated by reference in its entirety as if fully set forth herein.

TECHNICAL FIELD

The present invention relates to cinematography and digital cinema. In particular, the present invention relates to a method apparatus and system for applying an annotation to a portion of a video sequence.

BACKGROUND

The advent of digital imaging technology has altered the behaviour of the film industry, in the sense that more and more films are produced digitally. Digital cinematography, the process of capturing video content as digital content items, has become increasingly prevalent for film production.

In addition to simplifying the transition of source materials between production and post-production, digital cinematography has improved the work flow of film production. For example, digital cinematography has enabled on-set monitoring, which means directors, clients, and others on set are able to watch the live video sequences of every shot during film production. While live review can provide useful information to the director, there is presently no easy way for this information to be passed on to the post-production stage. This means that the director must carry over the knowledge of the shot to the post-production process. If the director does not carry over all knowledge of each shot to the post-production stage, information loss can occur.

Given the real-time nature of live video streams, it is desirable that annotations be added to captured video sequences with minimum interaction time so as not to impede the monitoring. However, it is challenging for the director to specify the details of annotations within a short time. Voice annotations cannot be used during capture. Voice annotations have limited ability to capture or be associated with a video sequence.

A need exists to facilitate applying annotations to a video sequence.

SUMMARY

It is an object of the present disclosure to substantially overcome, or at least ameliorate, at least one disadvantage of present arrangements.

A first aspect of the present disclosure provides a processor-implemented method of applying an annotation to a portion of a video sequence, said method comprising the steps of: receiving the video sequence in real-time during capture of the video sequence; monitoring in real-time a plurality of signals associated with the video sequence; receiving an indication associated with a spatial area of interest of at least one frame during capture of the video sequence, selecting, from the plurality of monitored signals, a temporal portion of one of the plurality of monitored signals for annotation, said selection being based upon at least the spatial area of interest and a temporal variation measure in at least one of the plurality of monitored signals; applying an annotation to a portion of the video sequence corresponding to the selected temporal portion; and storing the annotation in an annotation record associated with the video sequence.

In one implementation, the processor-implemented method further comprises determining a region of interest of the at least one frame of the video sequence using the spatial area of interest, wherein the selection of the temporal portion is based upon at least the region of interest and the temporal variation measure.

In another implementation, the region of interest comprises the spatial area of interest and a portion of the at least one video frame having content associated with content of the spatial area of interest.

In another implementation, each the plurality of monitored signals is associated with a spatial region of the at least one video frame.

In another implementation, the spatial region of the at least one video frame is a portion of the video frame.

In one implementation, the spatial region of the at least one video frame is the entire video frame.

In another implementation, the selection indication is a touch gesture received by a touch screen displaying the video sequence.

In another implementation, the temporal variation measure is based upon a monitored signal having a most recent time of change prior to receiving the selection indication.

In another implementation, the temporal variation measure is based upon a monitored signal having a greatest relative transition in a time period prior to receiving the selection indication.

In another implementation, the selected temporal portion starts at a transition time of the selected monitored signal.

In another implementation, the selected temporal portion ends at a further transition of the selected monitored signal.

In another implementation, the processor-implemented further comprises determining a category of the annotation from the selected temporal portion.

In another implementation, a subject of the annotation is identified in the at least one frame by matching a type of the indication to the selected temporal portion.

In another implementation, the plurality of monitored signals include at least one of signals relating to image capture apparatus motion, image capture apparatus zoom, image capture apparatus frame rate, video image lighting, video image colour, video image blur, video image edge density, video image corner density, video image face appearance, video image character motion, video image object motion, video image ambient noise and video image dialog.

In another implementation, an area of the annotation comprises the spatial area of interest.

In another implementation, the area of the annotation further comprises a region of the at least one frame having similar texture content to the spatial area of interest.

In another implementation, the area of the annotation further comprises a region of the at least one frame having a similar motion signature to the spatial area of interest.

A further aspect of the present disclosure provides a computer-readable medium having computer program stored thereon for applying an annotation to a portion of a video sequence, said program comprising: code for receiving the video sequence in real-time during capture of the video sequence; code for monitoring in real-time a plurality of signals associated with the video sequence; code for receiving an indication associated with a spatial area of interest of at least one frame during capture of the video sequence, code for selecting, from the plurality of monitored signals, a temporal portion of one of the plurality of monitored signals for annotation, said selection being based upon at least the spatial area of interest and a temporal variation measure in at least one of the plurality of monitored signals; code for applying an annotation to a portion of the video sequence corresponding to the selected temporal portion; and code for storing the annotation in an annotation record associated with the video sequence.

A further aspect of the present disclosure provides an apparatus for applying an annotation to a portion of a video sequence, the apparatus comprising: means for receiving the video sequence in real-time during capture of the video sequence; means for monitoring in real-time a plurality of signals associated with the video sequence; means for receiving an indication associated with a spatial area of interest of at least one frame during capture of the video sequence, means for selecting, from the plurality of monitored signals, a temporal portion of one of the plurality of monitored signals for annotation, said selection being based upon at least the spatial area of interest and a temporal variation measure in at least one of the plurality of monitored signals; means for applying an annotation to a portion of the video sequence corresponding to the selected temporal portion; and means for storing the annotation in an annotation record associated with the video sequence.

A further aspect of the present disclosure provides system for applying an annotation to a portion of a video sequence, the system comprising: a memory for storing data and a computer program; a processor coupled to the memory for executing said computer program, said computer program comprising instructions for: receiving the video sequence in real-time during capture of the video sequence; monitoring in real-time a plurality of signals associated with the video sequence; receiving an indication associated with a spatial area of interest of at least one frame during capture of the video sequence, selecting, from the plurality of monitored signals, a temporal portion of one of the plurality of monitored signals for annotation, said selection being based upon at least the spatial area of interest and a temporal variation measure in at least one of the plurality of monitored signals; applying an annotation to a portion of the video sequence corresponding to the selected temporal portion; and storing the annotation in an annotation record associated with the video sequence.

DETAILED DESCRIPTION INCLUDING BEST MODE

Narrative films, which are probably the most widely screened films in theatres, are one type of film product that tells a story. The goal of narrative film making is to compose a sequence of events in audio and/or visual form based on a written story. With the advent of digital imaging technology, digital cinematography, being high-quality acquisition of video data using digital cinema cameras during film production, has become increasingly widespread for narrative film making

FIG. 1shows a method100representative of a workflow used in digital cinematography for narrative film making The method100mainly comprises the following stages: a development stage110, a pre-production stage120, a production stage130, and a post-production stage140. The stages110to140are typically executed in sequence to produce a final film.

The pre-production stage120follows the planning stage110. At the development stage110, a film producer selects a story and develops a script with the help of a screenwriter. During the production stage100, key elements such as financing, principal cast members, directors, and cinematographers for the film are confirmed.

At the pre-production stage120, storyboards, which are visual images helping communicating ideas for the script, are developed. Furthermore, during the pre-production stage120, each step of actually making the film is designed and planned.

Following the pre-production stage120is the production stage130. At the production stage130, raw footage for the narrative film is generated. In particular, shots, which are short recorded video sequences, are captured or recorded for different scenes of the film. Shots are recorded using an image capture apparatus, such as a digital video camera. A shot is a basic unit of the production stage130corresponding to a continuous recording of a scene of the narrative film from the time the image capture apparatus starts recording until the imaging apparatus stops recording. It is common for multiple shots for any given scene to be acquired at the production stage130. Acquiring multiple shots for a given scene helps ensure there is footage of sufficient quality for use in the post-production stage140. Each alternative shot captured is referred to as a take. Each shot captured is stored with associated metadata relating to the captured video sequence.

Following the production stage130is the post-production stage140. At the post-production stage140, the captured shots are edited and then exported to various formats such as Digital Versatile Disc (DVD), Blue-ray Disc (BD), Holographic Versatile Disc (HVD), etc. for distribution. The editing process of the post-production stage140consists of reviewing the content and assembling the narrative film. Metadata created in the production stage130is utilized for editing at the post-production stage140. For example, colour grading may be utilized to enhance or alter the colour of a particular scene of the acquired shots, in light of a cinematographer's or director's notes on colours.

Variations of the method100ofFIG. 1are possible in practice. However, film making typically employs pre-production (planning), production (capture) and post-production (editing) stages in some form.

During the production stage130, directors of the film review the captured shots and record notes in report sheets. The notes may include possible issues or general comments on the shots. In digital cinematography, reviewing the acquired shots may be supported by on-set monitoring, enabled by wireless encoders and mobile devices. One example is wireless on-set monitoring to a tablet device such as an iPad™ using Teradek Cube™. The Teradek Cube™ acts as a Wi-Fi (IEEE 802.11) hotspot, which an iPad™ can connect to and communicate via. Once the Teradek Cube™ has been connected to the image capture apparatus via a connection such as a HDMI/HD-SDI connection, the Teradek Cube™ transmits live video (live capture of shots) from the image capture apparatus to the iPad™ with little delay. Such allows monitoring of the take, live on the iPad™.

Annotations that the directors are interested in can be classified into a number of categories. Typical categories of annotations may comprise performance, camera (image capture apparatus) parameters and quality. The performance category includes annotations relating to characters of the narrative film. Example annotation types include script, voice and character positioning. Camera parameter annotations typically include annotation types such as framing and zoom speed. Framing refers to selection of what to include in the scene captured using the camera. Expressive qualities of framing include an angle of the camera to an object of, the scene an aspect ratio of the projected image, and the like. Zooming means a change of focus length of a lens of the camera while the shot is in progress. Different effects may be created by different zooming speed. For example, zooming in creates a feeling of seemingly “approaching” a subject of the shot while zooming out makes audience feel that they are seemingly “distancing” the subject. Quality annotation types relate to issues of quality of the video sequence captured by the camera such as blur and focus. Different quality requirements may affect the camera movements. For example, a smooth camera pan may allow the scene to be sharp enough for the audience to observe, whereas a fast pan may create motion blur to the scene. Such information may be used in adjusting camera movement when making the next shot. The abovementioned annotations may provide some guidance at the production stage130as to how to improve shooting the next shot, or at the post-production stage140to improve editing.

In the pre-production stage130, a type of the production will be specified. Production types include, for example wedding, drama, TV commercial, and company training and the like. Different production types may require different annotations. For example, for shooting wedding videos, special annotations may be required for marking special moments such as cake cutting and exchange of rings. Similarly, TV commercial productions may require designated annotations on marking continuity of an associated story flow. Given the large variety of annotations that a director may wish to record, it would be advantageous to provide a means for capturing different annotations to prevent loss of information between the production stage130and the post-production stage140. It would be useful to have a streamlined means for recording annotations on a live stream of video while a shot is being captured. If the annotations cannot be made in near real-time, completion of the production method100will be unnecessarily extended.

An issue with the captured video sequence may be restricted to a particular spatial region and time. It would therefore be advantageous to allow directors to record an annotation for a portion of a video sequence. An example annotation would specify attributes of the annotation such as annotation type, annotation subject, associated temporal and spatial region. Annotation types refer to the types for annotation described above, while annotation subjects relate to description or content of the annotation. Such attributes are hereafter collectively referred to as ‘semantics of an annotation’.

It is useful to implement a touch-to-annotate operation in the production stage as touch-to-annotate is convenient to the user (e.g., director) for reviewing and editing video sequence.

A range of single and multi-touch gestures are known and widely supported by mobile touch screen based devices like the iPad™ and include such things as a tap, double tap, a pinch, a two finger rotate, stroking out a line, multi-finger swipe and the like. Despite the convenience of the touch-to-annotate operation, some multi-touch gestures, e.g., a tap, have far shorter operative duration than the gesture's underlying semantics. It is therefore desirable to have a method to determine the semantics of annotation intelligently.

A need exists to facilitate creation of annotations during digital film production. The arrangements described determine semantics of annotations by monitoring temporal properties of (i) underlying video signals and (ii) spatial properties of a multi-touch gesture.

The methods described are typically implemented using an electronic device such as a tablet device, a smartphone, or the like, having a display suited to real-time video reproduction.FIGS. 9A and 9Bcollectively form a schematic block diagram of a general purpose electronic device901including embedded components, upon which the methods of annotating a portion of a video stream to be described are desirably practiced. The electronic device901is in the example described hereafter is a tablet device. However, in other implementations the electronic device901may be another type of electronic device in which processing resources are limited, for example a mobile phone, a portable media player, or a smartphone, or an electronic image capture apparatus such as a camera or video camera. Nevertheless, the methods to be described may also be performed on higher-level devices such as desktop computers, server computers, and other such devices with significantly larger processing resources.

As seen inFIG. 9A, the tablet device901comprises an embedded controller902. Accordingly, the tablet device901may be referred to as an “embedded device.” In the present example, the controller902has a processing unit (or processor)905which is bi-directionally coupled to an internal storage module909. The storage module909may be formed from non-volatile semiconductor read only memory (ROM)960and semiconductor random access memory (RAM)970, as seen inFIG. 9B. The RAM970may be volatile, non-volatile or a combination of volatile and non-volatile memory.

The tablet device901includes a display controller907, which is connected to a video display914, such as a liquid crystal display (LCD) panel or the like. The display controller907is configured for displaying bitmap and graphical images on the video display914in accordance with instructions received from the embedded controller902, to which the display controller907is connected.

The tablet device901also includes user input devices913which are typically formed by keys, a keypad or like controls. In the example described herein, the user input devices913includes a touch sensitive panel physically associated with the display914to collectively form a touch screen. For ease of reference, the combination of the display1914and user input devices1913are referred to as a touch screen1914in the arrangements described, consistent with that type of structure as found in traditional tablet devices, such as the Apple iPad™. The touch screen914may thus operate as one form of graphical user interface (GUI) as opposed to a prompt or menu driven GUI typically used with keypad-display combinations. Other forms of user input devices may also be used, such as a microphone (not illustrated) for voice commands or a joystick/thumb wheel (not illustrated) for ease of navigation about menus.

As seen inFIG. 9A, the tablet device901also comprises a portable memory interface906, which is coupled to the processor905via a connection919. The portable memory interface906allows a complementary portable memory device925to be coupled to the tablet device901to act as a source or destination of data or to supplement the internal storage module909. Examples of such interfaces permit coupling with portable memory devices such as Universal Serial Bus (USB) memory devices, Secure Digital (SD) cards, Personal Computer Memory Card International Association (PCMIA) cards, optical disks and magnetic disks.

The tablet device901also has a communications interface908to permit coupling of the device901to a computer or communications network920via a connection921. The connection921may be wired or wireless. For example, the connection921may be radio frequency or optical. An example of a wired connection includes Ethernet. Further, an example of wireless connection includes Bluetooth™ type local interconnection, Wi-Fi (including protocols based on the standards of the IEEE 802.11 family), Infrared Data Association (IrDa) and the like.

Typically, the tablet device901is configured to perform some special function. The embedded controller902, possibly in conjunction with further special function components910, is provided to perform that special function. For example, where the device901is a digital camera, the components910may represent a lens, focus control and image sensor of the camera. The special function component910is connected to the embedded controller902. As another example, the device901may be a mobile telephone handset. In this instance, the components910may represent those components required for communications in a cellular telephone environment. Where the device901is a portable device, the special function components910may represent a number of encoders and decoders of a type including Joint Photographic Experts Group (JPEG), (Moving Picture Experts Group) MPEG, MPEG-1 Audio Layer 3 (MP3), and the like. The special function components910may also relate to operation of the touch screen914.

The methods described hereinafter may be implemented using the embedded controller902, where the processes ofFIGS. 2 to 4 and 7may be implemented as one or more software application programs933executable within the embedded controller902. The tablet device901ofFIG. 9Aimplements the described methods. In particular, with reference toFIG. 9B, the steps of the described methods are effected by instructions in the software933that are carried out within the controller902. The software instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules perform the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user.

The software933of the embedded controller902is typically stored in the non-volatile ROM960of the internal storage module909. The software933stored in the ROM960can be updated when required from a computer readable medium or via communication with a server computer such as a cloud computer. The software933can be loaded into and executed by the processor905. In some instances, the processor905may execute software instructions that are located in RAM970. Software instructions may be loaded into the RAM970by the processor905initiating a copy of one or more code modules from ROM960into RAM970. Alternatively, the software instructions of one or more code modules may be pre-installed in a non-volatile region of RAM970by a manufacturer. After one or more code modules have been located in RAM970, the processor905may execute software instructions of the one or more code modules.

The application program933is typically pre-installed and stored in the ROM960by a manufacturer, prior to distribution of the tablet device901. However, in some instances, the application programs933may be supplied to the user encoded on one or more CD-ROM (not shown) and read via the portable memory interface906ofFIG. 9Aprior to storage in the internal storage module909or in the portable memory925. In another alternative, the software application program933may be read by the processor905from the network920, or loaded into the controller902or the portable storage medium925from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that participates in providing instructions and/or data to the controller902for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, flash memory, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the device901. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the device901include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like. A computer readable medium having such software or computer program recorded on it is a computer program product.

The second part of the application programs933and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display914ofFIG. 9A. Through manipulation of the user input device913(e.g., the keypad), a user of the device901and the application programs933may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via loudspeakers (not illustrated) and user voice commands input via the microphone (not illustrated).

FIG. 9Billustrates in detail the embedded controller902having the processor905for executing the application programs933and the internal storage909. The internal storage909comprises read only memory (ROM)960and random access memory (RAM)970. The processor905is able to execute the application programs933stored in one or both of the connected memories960and970. When the tablet device901is initially powered up, a system program resident in the ROM960is executed. The application program933permanently stored in the ROM960is sometimes referred to as “firmware”. Execution of the firmware by the processor905may fulfill various functions, including processor management, memory management, device management, storage management and user interface.

The processor905typically includes a number of functional modules including a control unit (CU)951, an arithmetic logic unit (ALU)952, a digital signal processor (DSP)953and a local or internal memory comprising a set of registers954which typically contain atomic data elements956,957, along with internal buffer or cache memory955. One or more internal buses959interconnect these functional modules. The processor905typically also has one or more interfaces958for communicating with external devices via system bus981, using a connection961.

The application program933includes a sequence of instructions962through963that may include conditional branch and loop instructions. The program933may also include data, which is used in execution of the program933. This data may be stored as part of the instruction or in a separate location964within the ROM960or RAM970.

In general, the processor905is given a set of instructions, which are executed therein. This set of instructions may be organised into blocks, which perform specific tasks or handle specific events that occur in the tablet device901. Typically, the application program933waits for events and subsequently executes the block of code associated with that event. Events may be triggered in response to input from a user, via the user input devices913ofFIG. 9A, as detected by the processor905. Events may also be triggered in response to other sensors and interfaces in the tablet device901.

The execution of a set of the instructions may require numeric variables to be read and modified. Such numeric variables are stored in the RAM970. The methods described use input variables971that are stored in known locations972,973in the memory970. The input variables971are processed to produce output variables977that are stored in known locations978,979in the memory970. Intermediate variables974may be stored in additional memory locations in locations975,976of the memory970. Alternatively, some intermediate variables may only exist in the registers954of the processor905.

The execution of a sequence of instructions is achieved in the processor905by repeated application of a fetch-execute cycle. The control unit951of the processor905maintains a register called the program counter, which contains the address in ROM960or RAM970of the next instruction to be executed. At the start of the fetch execute cycle, the contents of the memory address indexed by the program counter is loaded into the control unit951. The instruction thus loaded controls the subsequent operation of the processor905, causing for example, data to be loaded from ROM memory960into processor registers954, the contents of a register to be arithmetically combined with the contents of another register, the contents of a register to be written to the location stored in another register and so on. At the end of the fetch execute cycle the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed this may involve incrementing the address contained in the program counter or loading the program counter with a new address in order to achieve a branch operation.

Each step or sub-process in the processes of the methods described below is associated with one or more segments of the application program933, and is performed by repeated execution of a fetch-execute cycle in the processor905or similar programmatic operation of other independent processor blocks in the tablet device901.

The tablet device901is in communication with an image capture apparatus990. The image capture apparatus is a digital video camera in the example described, referred to hereafter as the camera990. In other implementations, the image capture apparatus990may be any other device capable of recording digital video. The tablet device901is in communication with the camera990via a network such as the network920, via a connection991. In an alternative embodiment, the tablet device901is integrally formed with the image capture device990.

A method200of applying an annotation to a portion of a video sequence is shown inFIG. 2. The method200may be implemented by one or more submodules of the application933stored on the memory906, and controlled by execution of the processor905. The method200is executed during live review of a video sequence, that is during capture of the video sequence.

The method200executes as the video sequence is captured (recorded) by the camera990. The method200starts at a step210when the video sequence is received by the tablet device901. The video sequence is received at the step210in real-time from the camera990. The step210is executed on the processor905to analyse and display the video sequence in near real-time to a user of the tablet, in this example the director, on the touch screen914. A method300of receiving, analysing and displaying a video frame of the video sequence, as executed at step210, is described in relation toFIG. 3.

At any time during display of the video sequence by the touch screen914, the director is able to initiate an annotation by executing one of a defined set of gestures to operate the touch screen914. The method200executes on the processor905to progress to a step220upon operation of the touch screen. At the step220, the touch screen914operates to receive the gesture executed by the director in real-time during capture of the video sequence. In receiving the gesture, the tablet device901operates to receive an indication from the director.

The method200executes on the processor905to progress to a step230. At step230, based on the gesture received at step220, and monitored spatial and temporal attributes of the captured video sequence determined in step210, the application933executes to determine semantics for the annotation by analysing the gesture and determining the intended annotation. The semantics for the annotation may include annotation type, annotation subject, and the temporal and spatial extent for the annotation. A method400of analysing the gesture and determining the intended operation, as executed at the step230, is described hereafter with reference toFIG. 4.

The method200executes on the processor905to progress to a step240. The step240executes to determine, based on the how specific the gesture is, whether the intended annotation determined in the step230is ambiguous. If the step240determines that the possible annotation is not ambiguous (‘No’ at step240), no confirmation from the director is required, and the method200executes to progress to step260.

If the indicated annotations are determined to be ambiguous (‘Yes’ at step240), the method200executes to progress to a step250. On execution of the step250, the director is requested to confirm the annotation. The step250may also include presenting alternate possible annotations and/or additional semantic information to the director on the touch screen914for selection. In this event, the alternative annotation options are presented on the touch screen914as menu items. In other implementations, other user interface methods may be used to present alternative annotation options to the director. Once the application933has executed to receive confirmation from the director, the method200executes to progress to the step260. In yet other implementations, a flag may be added as an annotation if the gesture is found to be ambiguous in step250. In some implementations the step240is optional and the method200progresses directly from step230to step260.

The step260executes to apply the determined annotation to a portion of the video sequence by associating the annotation with the portion of the video sequence. The step260executes to embed the determined annotation as metadata by storing the determined annotation and associated semantics as metadata in an output video stream. Alternatively, the determined annotation is stored as an annotation record associated with the video sequence. The output stream, the associated metadata stream, and the annotation record may be stored on the memory909. The application933may in some implementations execute to display the annotation on the display screen914briefly to indicate to the director that the annotation has been stored. Other forms of feedback such as audio feedback may also be used in the step200to indicate to the director that the annotation has been stored. In embedding the determined annotation as metadata, the step260operates to apply the annotation to a portion of a selected one of the signals of step210.

The method300of receiving, analysing and displaying a frame of the video sequence to the director, as executed at step210, is now described in more detail inFIG. 3. The method300may be implemented by one or more submodules of the application933stored on the memory906, and controlled by execution of the processor905.

The method300of receiving, analysing and displaying video frames executes continuously in real-time as the video sequence is captured. The method300is described in relation to a single video frame of the video sequence. The method300starts at a step310. The step310executes to receive the video frame and camera990information in real-time during capture of the video sequence. The camera information may include a state change in one or more parameters of the camera990.

The method300executes to progress to a step320. A set of analytic signals associated with the video frame are determined in real-time in execution of the step320.

The method300executes to progress to the330. Execution of the step330buffers the analytic signals so that there is a history, for a predetermined time window, of the value for each signal. In buffering the analytic signals, the application933executes known buffering techniques, such as storing the analytical signals temporarily on the memory909. The method300executes to progress to step340. In execution of the step340, subsequently, or in parallel to the buffering step330, the video frame is displayed to the director on the touch screen914. In executing the step330, the method300operates to monitor the analytic signals in real-time during capture of the video sequence.

The analytic signals determined in the step320describe specific properties of video data associated with the video frame. The analytic signals can relate to a status of the camera990, such as camera motion, zoom and frame rate. The analytic signals can relate to the captured video stream. Examples of analytic related to the captured video stream include analysis of appearance (such as lighting, colour, blur, edge density, corner density, face appearance), movement (such as character motion such as walking and running, scene motion) and sound (such as ambient noise, dialog). The analytic signals typically comprise signals stored in relation to regions of the frame (such as grids, described in relation toFIG. 5) or may be global signals relating to the entire video frame, e.g., audio signals, light signals, and the like.

The camera990status signals can be derived in at least two ways. In one implementation, the camera device status signals are derived directly from camera990operating parameters and read directly from the camera990using an appropriate application programming interface (API). Alternatively, camera status information can be derived directly or indirectly from the metadata relating to the camera990embedded in the video stream received at the step210. Examples of camera operating signals include the aperture and shutter speed settings and whether or not a zoom or focus pull is being operated.

Methods used to determine the analytic signals by processing the received video stream are known. Some implementations execute to analyse the video stream for motion characteristics (speed and direction), dominant chromaticity, focus (high frequency energy), and face likelihood. For example, motion speed and direction in two dimensions (2D) can be determined using a block matching technique employed in video compression. Such an implementation is useful due to efficiency and existence of hardware support for block matching techniques. However, other similar techniques for estimating optical flow could equally be employed. Chromaticity is determined by considering the distribution of blue-difference (Cb) and red-difference (Cr) chroma components values for an image region. Both motion and colour have signals that are multi-dimensional. While multi-dimensional distance measures are common, the description herein is restricted to discussion of the further analysis of signals to a one dimension (1D) example for simplicity and clarity. Computation and comparisons of 1D signals are also computationally cheaper to achieve for real-time performance Also, the focus quality signal can be generated, as a 1D signal, by measuring the Root mean square (RMS) output of a band pass or high pass filter applied to the relevant image area. Similarly, face detection is typically performed using a statistical process that provides a confidence of and given image area containing a face. Other signals, derived from the camera990, are inherently 1D. Such signals include a status of operation of focus and zoom as well as numeric values of various capture parameters such as aperture, shutter, exposure value, etc. Some signals type can be associated with, and monitored independently for, various spatial regions of the frame in execution of step330. For example, the exposure signal is measured at a grid of points. In such implementations, the step330stores data for spatial regions corresponding to portions of the video frame, which can be used later to match a spatial area of interest indicated by the annotation gesture of step220. Matching a spatial area of interest to a portion of a frame is described with reference toFIG. 6AandFIG. 6B.

Audio content of the video sequence may be monitored under execution of the method300to track beginning and ending of a conversation. Examples of audio signals may include, but are not limited to, silence in speech, voice pitch, and the like. Conventional speech-to-text conversion methods may be used to convert the audio content to text. Natural language processing techniques may be used to identify sentence boundaries to determine the conversation durations.

The analytic signals that are monitored at the step330can be customised based on the production type that is determined at the pre-production stage120. For example, to support the precise timing of TV commercial shots, signals associated with typical actions may be employed. Examples of signals associated with typical actions include, but are not limited to, walking, hand shaking, running, and the like. To support such implementations, the method300allows monitoring of the different signal types to be enabled or disabled by the director using know techniques such as presentation of a selectable menu of signals. The method300also, in some implementations, allows customisation of a region in which the analytic signals are monitored. For example, a director may pay more attention to a character motion in the centre of the video frame and disable monitoring outside the centre region. Resultant execution of the step330to monitor motion only in the centre of the video frame may provide a method of reducing determination of analytic signals while still allowing the director to make annotations on the character motion of importance.

A rate of change of any given signal is also monitored in a similar manner and made available to the annotation semantic determination step230.

Execution of the method300is repeated for each video frame of the video sequence received. Execution of the method300therefore runs in real-time at the same rate as the video frames are captured. Depending on the frame rate and the processing capabilities of the tablet device901, analytic signals may in some implementations be determined on a subset of the received frames, such as every second frame or every third frame. Such reduction in the rate of the analytic signal calculation can be used to ensure the video frames are displayed in the step340at the capture rate and/or with minimum lag.

Referring back toFIG. 2, step210is followed in execution of the method200with step220when the director initiates an annotation by making a gesture. In the arrangements described, the gesture is made on the touch screen914. While use of the touch screen914is particularly suited to the arrangements described, other methods for receiving an indication may nevertheless be appropriate (e.g. a mouse drag, a joystick, hovering of fingers, engaging of buttons, and the like) as means for receiving an indication, provided such can be associated with a particular spatial portion of the video frame. Both single-touch and multi-touch gestures may be used by the director to operate with the touch screen914. Examples of gestures may include, but are not limited to tapping, dragging, flicking, swiping, double tapping, pinching, and shaking.

The method200progresses from step220to step230. The method400of analysing the gesture input and determining the annotation and the annotation semantics, as executed at step230, is now described in detail with reference toFIG. 4. The method400may be implemented by one or more submodules of the application933stored on the memory906, and controlled by execution of the processor905.

Execution of the method400starts with step410. The step410executes to determine the gesture type. The gesture type is used to identify a spatial area of interest associated the gesture. The spatial area of interest relates to a spatial area of the video frame to which the gesture is applied. In alternative implementations, identification of the spatial area of interest maybe completed using rules associated with a particular gesture type.

The method400executes to progress from step410to step420once the gesture type and spatial area of interest are determined The step420executes to select monitored signals related to the spatial area of interest, for use in selecting one of the monitored signals that matches the received gesture.

Once the signal is selected, the method400executes to progress to step430. The step430executes to determine a temporal and spatial range bounding the annotation. The temporal and spatial range of the annotation relate to a temporal portion of the selected signal. A method700of determining the temporal and spatial range of the annotation, as executed at step430, is described in relation toFIG. 7. In an alternative embodiment, only the temporal range bounding the annotation is determined.

Gestures received in the step220are used to determine an annotation type as well as indicating the spatial area of interest.FIGS. 5A to 5Dshow examples500of gestures that might be executed by the director and the corresponding spatial area of interest that each gesture indicates. The gestures are detected when the director touches a surface of the touch screen914.FIG. 5Ashows a representation of a tap gesture510. The tap gesture510is executed when a finger is pressed down to touch the screen914, as indicated by an arrow513and, afterwards, the finger is lifted, as indicated by an arrow515. Execution of the tap gesture510results in a region518where the tap gesture was made being determined as the spatial area of interest. That is, the spatial area of interest is the area of the touch screen914to which the gesture is applied.

FIG. 5Bshows a representation of a drag gesture520having an outline523being specified by a finger drag which returns to a drag start point525. An area528enclosed by the outline523is identified to be the corresponding spatial area of interest.

FIG. 5Cshows a representation of a drag gesture530along a diagonal line533. The drag gesture530identifies a spatial area of interest536along the line536.

FIG. 5Dshows a representation of a multi-touch pinch open gesture540. Two fingers are separated while touching the touch screen914and moved apart as indicated by a line543. End positions of the two fingers define corners of a rectangular space defining a spatial area of interest545.

In some implementations, the spatial area of interest is used to determine a region of interest associated with the gesture. The region of interest includes the spatial area of interest in addition to areas of the video frame which have content associated with a subject of the spatial area of interest. An example of determining a region of interest is discussed in relation toFIG. 6B. A determined region of interest is used in the manner to the spatial area of interest in selecting a temporal portion of one of the monitored signals for annotation.

The spatial area of interest is used to determine annotation area of the annotation at step430.The annotation area is the spatial area of the video frame to which the annotation is applied. A determined region of interest may also be used in a similar manner the spatial area of interest in determining the annotation area.

In some implementations, the spatial area of interest determined from by the gesture will be the same as the annotation area. In other implementations, the spatial area of interest will only identify a part of the annotation area. For example, the drag gestures520and530ofFIGS. 5A and 5Brespectively define spatial areas of interest. The corresponding annotation area can be extended to regions inside the video image frame which share similar texture content. Texture content refers to information about the spatial arrangement of colour or intensities in a region of the video frame. Texture content can be described using different techniques. Such techniques may include but not limited to edge detection which determines the number of edge pixels in a specified region, co-occurrence matrices which captures the spatial relations of similar grey tones, and autocorrelation function which describes the repetitive patterns inside the region.

In yet further implementations, the annotation area can be extended to regions inside the video image frame which share similar motion content with the spatial area of interest indicated by the drag gestures520and530ofFIGS. 5B And 5C. Such motion content is related to movements inside the video frame, which may be quantified by an estimate of a velocity either at each point in the video frame or in 3D scenes or of the camera device990. In another example, the pinch open gesture540ofFIG. 5Dmatches a “zoom in” where the annotation area is a full screen548. However, while spatial area of interest determined from the final finger separation only identifies the area545of the touch screen914. To differentiate the spatial area of interest545and the extended annotation548, the application933may prompt the director for confirmation of the desired annotation area, as at the step240ofFIG. 2. In an alternative implementation, a default option can be set so that method200operates to choose a default annotation area. For example, the method430may select the full video image as the default annotation area for gestures like pinching open, depending on the rules relating to the gesture.

Once the spatial area of interest has been determined, the method400progresses to the step420. Execution of the step420includes selection of a plurality of monitored signals for consideration based on the identified area of interest. As described hereinbefore, a particular signal types can be monitored independently at various spatial regions of the video frame. Such a signal type may be considered as a local signal type. Local signal types contrast to global signal types such as ambient noise, dialog, camera frame rate, and the like which are monitored for the entire video frame.

FIGS. 6A to 6Cshow examples of a selection process executed in step420depending on the identified spatial area of interest.

FIG. 6Ashows a video frame600. The video frame600is divided into equal-sized grids, such as grids602,604,606,608and610. A size of the grids602to610may be defined as a minimum size for which computation of a certain signal type can support. For example, determining exposure value for a grid size of minimum 4-by-4 pixels may be suitable for computation by the tablet device901. For each of the grids602-610a plurality of local signal types are monitored independently of one another. The director executes a gesture612on the touch screen914indicating the identified spatial area of interest. InFIG. 6A, the corresponding spatial area of interest612intersects with four local grids inside the frame600, namely the grids604,606,608and610. The resultant annotation may be associated with signal types defined within one or more of the grids604,606,608and610. Accordingly the step430executes to examine signals monitored inside the grids604,606,608and610.

FIG. 6Bshows another video frame614. Similarly to the video frame600, the video frame614is divided into grids including grids616and618. InFIG. 6B, an identified spatial area of interest620is contained within a single grid620. As a result, only the monitored signals in grid620are associated with the annotation.

The step420selects a number of local signal types by filtering out signals not associated with the identified spatial area of interest. Global signal types, however, cannot be excluded, and are by default taken into consideration.

In some implementations, the spatial area of interest is used to determine a region of interest. For example, the director uses the gesture of620to touch a display of a portion of blue car on an otherwise grey background (not shown) on the touch screen914. Such results in the spatial area of interest being contained within the grid620. However, if the blue car extends across the video frame from the grid620into a grid622, the grid622includes content associated with the content of the spatial area of interest as the grid shares chromatic properties (blue colour) of the remainder of blue car. In such implementations, the application933executes to identify shared content of the grids620and622associated with the gesture. The application933determines the region of interest to relate to the grids620and622. In other implementations, a region of interest may be identified according to rules associated with received gestures—e.g., a double tap on the touch screen indicates to determine the region of interest from the identified spatial area of interest.

A resultant annotation area may be identified by including grid coordinate references, such as for the grids620and622, in applying the annotation at step260.

FIGS. 6C(i) to6C(iv) show examples of how monitored signals are selected from the determined grids. InFIGS. 6C(i) to6C(iv), four types of signals are monitored in the grids identified using the identified spatial region of interest. The monitored signals are a motion signal623shown inFIG. 6C(i), an exposure signal624shown inFIG. 6C(ii), a face likelihood signal626shown inFIG. 6C(iii), and an ambient noise level628shown inFIG. 6C(iv). The signals623,624and626are local signals, while the ambient noise level signal628is a global signal.

InFIGS. 6C(i) to6C(iv) the signals623,624,626and628are monitored over 7 seconds. A gesture is detected on the surface of the touch screen914displaying the video sequence in real-time at 7 seconds, as indicated by a marker630. An examination of temporal variation of signals within a predefined time window (for example, 2 seconds) indicates that changes occur on the motion, exposure, and face likelihood signals623,624and626respectively. The motion signal623varies at between 6 seconds and 7 seconds as indicated by a marker632. The exposure signal624changes around 5.3 seconds as indicated by a marker634, while the face likelihood signal626alters at 6 seconds as indicated by a marker636. The ambient noise signal628fluctuates over 7 seconds but no relatively significant change is observed. Accordingly, the ambient noise signal628is not considered relevant to the annotation. A signal considered relevant to an annotation is also referred to as an active signal.

Out of the three local active signals623,624and626, the change of the motion signal623is the most recent signal change near the time of the gesture630. As a result the motion signal623is most likely the signal that triggered the gesture. Accordingly, the motion signal623is selected as the signal for the annotation. Alternatively, selection of the signal for annotation can occur by choosing a signal which was changing at the time that the gesture was made.

The arrangements described use monitored signals related to the spatial area of interest to determine select a temporal portion of a signal of the video frame to annotate. The selected temporal portion may relate to one of the monitored signals related to the spatial area of interest, or to one of the global signals of the video frame.

Once the gesture type and spatial area of interest have been determined, and some active signals have been selected, the step420execute to match the gesture type, the spatial area of interest and the selected signals with one of the monitored signals to determine the annotation subject.

The particular gesture made by the director can be used in execution of the step430to identify which annotation subjects that director is annotating by matching the type of gesture to the signal. Table 1 shows an example of the signals and the gesture which identifies it.

TABLE 1Possible AnnotationSignalDescriptionSubjectGestureZoomZoom in or zoom out. This can beZoom too quick, tooPinch close fordetermined from the camera attributes.slow, too far etc.zoom out and pinchopen for zoom inPanThis can be determined from objectUneven pan, too slow,Dragtracking or camera motion attributes.too fast, too far etc.TempoThis can be determined from the rate ofToo fast/slow; matchDrag a zigzagchanges in scene motiontempo to other shotDialogThis can be determined from audioDialog volume uneven,Press or dragprocessing or a combination of audiotoo soft, loud, requiresfollowing characterprocessing and face monitoring.specific processingenhancementCharacterThis can be determined from face orInconsistent framingDrag followingMotionbody detection and backgroundwhen the camera ischaractertrackingtracking the motion etc.BlurThis can be determined from the levelInaccurate focus,Press or dragof high frequency detail and contrastcontrast enhancementoutlinewithin the specified regions.required etc.ExposureThis can be determined from theRegion underexposed?,Press or dragintensity histogrammore/less contrast inoutlineregion etc.ColourThis can be determined from the 2DInsufficient colourPress or dragchroma histogramcontrast, needs to beoutlinewarmer/colder etc.FaceThis can be determined by applying aFaces to emphasize,OutlineLikelihoodface detector to the outlined region.blurry faceThe response of the detector is thelikelihood of any face existence in theregion

In some implementations, the annotation subjects contain relative attributes such as “too quick”, “too slow”, “too loud”, etc. This may be realized by comparing the signal values with predetermined thresholds. For example, if the exposure threshold is set to 0.5, then any exposure value smaller than 0.5 will be considered as underexposed.

As indicated in Table 1, the spatial area of interest specified by the gesture might be associated with a number of such signals. If the signals have a consistent state, an annotation can be created for the spatial area of interest. To determine the annotation subjects, the method200may prompt the director with alternate possible annotation subjects associated with those signals, as described in relation to the step250. In the example of exposure an annotation of the region that is underexposed can be generated.

Signal subjects can also be combined to derive more meaningful semantics. For example, a blur signal in conjunction with a face likelihood signal will have a related “face out of focus” annotation. Such a situation occurs when the region of interest includes consistent blur and face likelihood signals. The blur signal indicates that the outline region being blurry and the face likelihood signal shows that the region of interest has high likelihood of being a face. Accordingly the combined subject may be “face out of focus”.

The method700for determining the temporal and spatial range for an annotation, as executed at the step430, is now described with reference toFIG. 7. The method700may be implemented by one or more submodules of the application933stored on the memory906, and controlled by execution of the processor905.

Prior to execution of the method700, the spatial extent of the gesture and the relevant signals has been determined at the step420. The method700starts at step710. The step710executes to consider changes in the relevant monitored signals. Changes in the monitored signals are determined by examining a derivative of each monitored signal. The derivative may be cached as a separate signal history stored in the memory909, or determined as required at step710.

The method700executes on the processor905to progress to a step720. The step720executes to determine a closest time prior to the time at which the gesture was received which exhibited a transition larger than a threshold amount for each relevant monitored signal. A transition larger than the threshold amount identifies a change point in a signal.

The method700executes on the processor905to progress to a step730. The step730executes by prioritising the identified change points to determine an active signal. The step730executes to mark the time of the transition for the active signal as a start time for the annotation.

The method700executes on the processor905to progress to a step740. In execution of the step740the active signal is monitored to determine a subsequent transition time. The step740executes to use the subsequent transition time as an end time for the annotation. The determination of the start and end times at the step740need not be based solely on the value of signals. In some implementations, the start and the end times may be determined, either solely or in part, by an elapsed time. For example, if no suitable signal transition or key value can be identified within a defined interval of the gesture time then a start time may be determined as being a fixed interval of time prior to the gesture. Similarly, if a transition corresponding to an end of annotation is not determined within a predetermined period of time, or if a new annotation gesture is initiated, then the end time for the annotation can be determined to be a fixed time after the start. The start and end times of the annotation define a temporal portion of the selected active signal.

In other implementations, additional information such as the annotation type and the production type may be taken into account when making such determinations. Further implementations may combine using a predetermined time interval with a signal trigger by weighting the size of the signal transition that would be interpreted as the beginning or end of an annotation. In such instances, weighting would be performed so that outside of a pre-determined time interval, even very small transitions would be interpreted as significant. Analysing the transition of the active signals over time represents analysing a temporal variation measure of each of the active signals. In some implementations, the temporal variation measure relate to the greatest relative change in signal over a period of time. In assigning start and end times of the active signal, the step740determines a temporal range of the active signal used for the annotation. In finding the closest change point at step720, marking the active signal at step730and identifying the change in the active signal at step740, the method700operates to select a portion of one of the monitored signals based up the spatial area of interest and a temporal variation measure in at least one of the plurality of monitored signals.

FIGS. 8A to 8Dshow examples of how a temporal range is assigned to the annotation.FIG. 8Ashows a plot of a monitored zoom signal810over 7 seconds. When the camera990starts recording the signal810is at a level811at which the camera990is not zooming After about 1 second (812) the camera990“zooms in” at a steady rate until a 2 second mark (814) at which time the zooming stops (816). A “zoom out” occurs between 3 seconds (818) and 4 seconds (820). Then between 4.5 seconds (822) and 6.7 seconds (832) a steady “zoom in” is shown in the signal810. At 6.3 seconds, shown by a marker830, an annotation gesture is received at the electronic device901. Although the method300has been monitoring the state of zooming and recognising the transitions over the past 6 seconds (812,814,818,820and822), only a record of the latest zoom in transition (822) at 4.5 seconds will be maintained for annotation processing at 6.3 seconds (marker830). The gesture at 6.3 seconds indicated by the marker830is a pinch open gesture (seeFIG. 5D, 540). This gesture is compatible with the “zoom in” and a zoom change annotation type is determined. As the “zoom in” relates to the whole screen the area of interest is determined to be the whole screen. At 6.7 seconds (832) the “zoom in” stops and the method700determines the interval from 4.25 seconds (822) to 6.7 seconds (832) as the temporal range for the annotation.

Alternatively, if the gesture occurs at 7 seconds (marker834) (instead of 6.3 seconds at the marker830), the same annotation will be applied because the time interval 4.25 seconds (832) to 6.7 seconds (834) is within a time threshold. A threshold of 300 milliseconds may for example be used given that the average human reaction time is roughly 250 milliseconds. Other settings for the threshold may be used that may depend on the signal type or a personalised system setting.

Use of a threshold is demonstrated in a plot835ofFIG. 8B. The plot835has a zoom in recorded at 3.9 seconds (836) to 6 seconds (837). If a gesture indicating a zoom in is made at time 6.3 seconds (830), the gesture results in the annotation with temporal range 3.9 seconds (836) to 6 seconds (837) being applies. Any gesture after the time 6.3 seconds (838) will not associate an annotation with the temporal range 3.9 (836) to 6 seconds (837). The step430will execute to examine other signals whose timeout occurs after the time indicated by837.

FIG. 8Cshows a signal840having example transitions for a character motion signal. If more than 1 character is displayed on the touch screen914the method step330ofFIG. 3will monitor multiple motion signals to monitor each of the individual characters. Initially, the character of the signal840is stationary and a zero motion level (844) is recorded. The character moves with steady motion speed between 2 seconds (846) and 6.6 seconds (848). A drag gesture, (e.g.FIG. 5C, 530) is received that follows the path of the character, the drag gesture starting at time 3.6 seconds (850) and ending at time 6.3 seconds (830). Such is determined to relate to the character motion signal, and an annotation generated is applied with a temporal range from 2 seconds (846) to 6.6 seconds (848). Alternatively, a drag gesture that is initiated between time 6.3 seconds (830) and 7 seconds (834) and that follows the path or part of the path taken by that character's movement will result in the same annotation being applied.

In the signals810,835and840ofFIGS. 8A, 8B and 8Crespectively, the monitored signals have discrete values. In some instances, however, the underlying raw signal may be continuous. By processing the raw signal using techniques such as threshold crossing, the continuous signal can be converted to discrete values resulting in step function similar to the ones shown in the signal810ofFIG. 8Aand the character motion signal840ofFIG. 8C. In other implementations, the signal being monitored will not be characterised by discrete states. A signal860ofFIG. 8Dis an example of a signal with non-step transitions. In the example ofFIG. 8D, the average intensity is monitored independently for a matrix of grids on the screen. If an absolute value of the rate of change of the signal exceeds a defined threshold as occurs at 5.5 seconds (864) then this time is a candidate for a start of a temporal range. A tap gesture (FIG. 5A, 510) in the same grid where the transition (864) occurs will result in application of an annotation. The start of the temporal range of the annotation will be time 5.5 seconds (864). The end of the temporal range of the annotation will be when the signal860returns below the threshold at 7 seconds (868). In other words the temporal region of the annotation is the time when the signal860has an absolute rate of change greater than defined threshold.

Once the spatial and temporal range bounding for an annotation have been determined by execution of the step430, information relating to the annotation may be stored as metadata and associated with the video sequence, as described in relation toFIG. 2, step260.

The relevant information of the annotation may be stored as metadata in the form of <annotation semantics, temporal extent, spatial extent >. The metadata can be stored as an XML file as an appendix to the video sequence file. Alternatively, both the metadata and the video sequence data can be stored as container format files, such as the Material eXchange Format (MXF) files. The MXF is a wrapper format which supports a number of different streams of coded video data, together with a metadata wrapper which describes the material contained within the MXF file.

The arrangements described are applicable to the media processing industries and particularly for the digital cinematography industries. In an example use, the director uses the gesture of530in relation toFIG. 5Cto follow motion of a character due to inconsistent framing in tracking that motion (see, e.g., Table 1). The application933executes to apply an annotation to a portion of the video sequence according to the methods described above. For example, the application933executes to select a portion of the character signal840ofFIG. 8Cand applies an annotation to that portion of the video sequence. At the post-production stage140, the annotation is accessible to the director upon accessing the stored video sequence. The methods described allow the director to view video sequences and add annotations during real-time capture of the video sequence. The director does not need to access a report book or other manual means of storing annotations or notes. As the stored annotation information is associated with the video frame, loss of information between the production and pre-productions stages may be reduced.

The described arrangements can be used in reviewing live takes rather than playback of video sequences, in contrast to some methods of annotating video sequences. In contrast to some methods of annotating video sequences, the arrangements described do not require manual input of the temporal portion or area of the annotation. Rather, the temporal region and annotation area may be determined from the spatial region of interest and a temporal variation measure in each of the monitored signals. Unlike some other methods of annotating video sequences, the director does not have to infer start and end times of the annotation temporal region. Rather, the arrangements described use the monitored signals to determine the temporal region of the annotation. Further, the arrangements described consider the spatial area of interest of the received gesture or indication.

In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of”. Variations of the word “comprising”, such as “comprise” and “comprises” have correspondingly varied meanings.