Providing a visual indication of the content of a video by analyzing a likely user intent

A method for providing a visual indication of a content of a video that reflects a likely intent of a user that acquired the video. The present method enables repurposing of a video into a variety of output forms that are adapted to the likely intent of the user and enables more efficient browsing of a stored video.

BACKGROUND

A wide variety of video systems may include the capability of generating, acquiring, or storing a video. Examples of video systems include video cameras, computer systems, home video appliances, PDAs, digital phones, etc. For example, a video camera may include mechanism for acquiring a video and storing the video on an storage media. In another example, a computer system may include the capability of obtaining a video from an attached camera, recording device, or some other video source, and storing the video locally.

A video system may include a mechanism for providing an indication of the content of a stored video. For example, a video camera or a computer system may include a display and rendering hardware for generating a playback of a stored video. In another example, a video appliance may include a display and a mechanism for rendering descriptive information, e.g. titles, file names, length of a video, data/time a video was acquired, etc. pertaining to a stored video. In another example, a video system may include a display and a mechanism for generating thumbnail images that depict portions of a stored video.

Unfortunately, the mechanisms in prior video systems for providing an indication of the content of a stored video may not reflect an intent of a user that acquired the video. For example, descriptive information such as file names, date/time, etc., or thumbnail images randomly generated may not reflect the content in the video that was of interest to a user that acquired a video. As a consequence, a user of a video system may have to engage in a time consuming manual review of the entire content of a stored video in order to obtain useful evaluation on its content.

SUMMARY OF THE INVENTION

A method is disclosed for providing a visual indication of a content of a video that reflects a likely intent of a user that acquired the video. The present method enables repurposing of a video into a variety of output forms that are adapted to the likely intent of the user and enables more efficient browsing of a stored video.

A video system according to the present techniques includes a processor that determines a likely intent of a user that captured a video by detecting at least one motion type in the video. The processor repurposes the video by generating a visual output that is adapted to the motion type. A video system according to the present techniques may also include a video index store for holding indices to video frames in the video.

Other features and advantages of the present invention will be apparent from the detailed description that follows.

DETAILED DESCRIPTION

FIG. 1shows a video system200that incorporates the present techniques. The video system200includes a video camera210, a video store212, a processor214, a display216, a printer218, and a key frame store220. The video camera210generates a video12that includes a series of video frames acquired by the video camera210under control of a user of the video camera210. The video12may be stored in video store212and/or provided directly to the processor214.

The processor214repurposes the video12by generating a visual output that indicates a content of the video12. The visual output reflects a likely intent of a user that acquired the video12. The processor214renders the visual output using the display216and/or the printer218.

The processor214generates a visual output that indicates a content in the video12by first detecting a motion type in the video12that may indicate an intent of a user that captured the video12. Examples of motion types that may indicate an intent of a user include a panning motion type, a zoom in motion type, an object motion type, a still motion type, i.e. no motion, and a complex motion type.

For example, a panning motion type in the video12may indicate that the user of the video camera210that captured the video12had intended to capture a panorama using a panning movement of the video camera210. In response to a detected panning motion type in the video12, the processor214may generate a panoramic output using a portion of the video12that corresponds to the panning motion type. The processor214may render the panoramic output using the display216or the printer218.

In another example, a zoom in motion type in the video12may indicate that the user of the video camera210had intended to capture a close up of an object of interest by zooming the video camera210into the object of interest and then holding the video camera210relatively still on the object of interest. In response to a detected zoom in motion type followed by a still motion type, the processor214may generate a photographic output using a portion of the video12that corresponds to the still motion type. The processor214may render the photographic output that depicts the object of interest using the display216or the printer218.

In yet another example, a still motion type in the video12may indicate that the user of the video camera210had intended to capture an area of interest in a scene by holding the video camera210relatively still on the area. In response to a detected still motion type, the processor214may generate a photographic output using a portion of the video12that corresponds to the still motion type. The processor214may render the photographic output that depicts the area of interest using the display216or the printer218.

In another example, an object motion type in the video12may indicate that the user of the video camera210had intended to capture a moving object of interest. In response to a detected object motion type, the processor214may generate a flipbook output or slideshow output using portions of the video12that correspond to the object motion type. The processor214may render the slideshow output using the display216or render the flipbook output using the printer218.

The processor214detects a motion type in the video12by analyzing a relative motion among a series of video frames of the video12. The processor214may determine a relative motion among the video frames in the video12using a block-based motion analysis such as that associated with MPEG 1 encoding. If the video system200is implemented in a system that already includes MPEG functionality then the processor214may share available MPEG hardware and/or code.

FIG. 2a-2cillustrate one method that may be employed by the processor214to determine a relative motion among a pair of adjacent video frames60-62in the video12. The processor214compares the pixel content of the video frames60and62and determines that a block70of the video frame60is substantially similar to a block72in the video frame62. For example, the processor214may determine a correlation metric between the blocks70and72based on the pixel data values in the blocks70and72to determine the similarity.

The processor214generates a motion vector74that indicates a spatial relationship between the blocks70and72based on the video frame60as a reference frame. The processor214generates a set of motion vectors for the video frames60-62, each motion vector corresponding to a block of the reference video frame60.

The processor214examines an arrangement of the motion vectors for pairs of adjacent video frames in the video12to detect a motion type. For example, an arrangement of the motion vectors for a pair of adjacent video frames may indicate a panning motion type, a zooming motion type (in/out), an object motion type, a still motion type, or a complex motion type. The motion vectors may be calculated over a number of video frames rather than a pair of adjacent video frames.

FIG. 3illustrates an arrangement of motion vectors50for a pair of adjacent video frames in the video12that indicate that the video camera210was panning while the adjacent video frames were being captured. The magnitudes and directions of the motion vectors50depict a relatively consistent and uniform pattern as the video camera210was moved to the right during video capture. The processor214recognizes an arrangement exemplified by the motion vectors50to detect a panning motion type in the video12.

FIG. 4shows an arrangement of motion vectors52for a pair of adjacent video frames in the video12that indicate that the video camera210was zooming out while the adjacent video frames were being captured. The directions of the motion vectors52point to the center of a video frame as the video camera210zooms out. The processor214recognizes an arrangement exemplified by the motion vectors52to detect a zoom out motion type in the video12.

FIG. 5shows an arrangement of motion vectors54for a pair of adjacent video frames in the video12that indicate that the video camera210was zooming in while the adjacent video frames were being captured. The directions of the motion vectors54point away from the center of a video frame as the video camera210zooms in. The processor214recognizes an arrangement exemplified by the motion vectors54to detect a zoom in motion type in the video12.

FIG. 6shows an example arrangement of motion vectors56for a pair of adjacent video frames in the video12that indicate the presence of moving objects. The magnitudes and directions of the motion vectors56vary locally as objects move within the video12. The processor214recognizes an arrangement of the motion vectors that vary locally to detect an object motion type in the video12.

A set of motion vectors for a pair of adjacent video frames in the video12having magnitudes close to zero and the overall vector average of zero indicate that the video camera210was relatively still while the adjacent video frames were being captured. The processor214recognizes a set of near zero magnitude motion vectors to detect a still motion type in the video12.

A set of motion vectors for a pair of adjacent video frames in the video12having a complex arrangement of directions and magnitudes may indicate a complex motion type. One example of a complex motion type is a panning movement of the video camera210while objects move in a captured image scene. Another example of a complex motion type is a movement of one or more objects in a captured image scene when the video camera210is relatively still.

The processor214may determine an intent of a user of the video camera210by analyzing the velocity and/or changes in the velocity of a panning motion type in the video12. The processor214may determine a velocity of a panning motion type in response to the rate of acquired video frames in the video12and the relative motion between successive video frames. For example, a set of motion vectors for a pair of adjacent video frames that indicate a panning motion type and that have a relatively large magnitude indicate a relatively high velocity whereas a set of motion vectors that indicate a panning motion type and that have a relatively small magnitude indicate a relatively low velocity.

If the velocity of a panning motion type is relatively slow and steady then it may be inferred that the user of the video camera210was intending to acquire a panorama. On the other hand, if the velocity of a panning motion type is relatively fast then it may be inferred that the user of the video camera210was intending to quickly move to an area or object of interest and that the user had little or no interest in the intervening areas. A relatively fast panning movement may be regarded as the equivalent of switching off image acquisition, moving the video camera210to the area or object of interest, then switching on image acquisition.

For example, the processor214may determine an intent to acquire a panorama by detecting a panning motion type in the video12for a period of time (Ts) during which the velocity in the panning motion is relatively constant. The period Tsmay be bounded by an initial video frame (Fi) and a final video frame (Ff) in the video12. The processor214may record indices for the video frames Fiand Ffin the key frame store220. The processor214may use the video frames Fiand Ffto generate the a panoramic output. For example, the processor214may use the video frames Fiand Ffas well as the intervening frames between the video frames Fiand Ffto generate a panoramic output.

The processor214may generate a panoramic output by stitching together the video frames Fithrough Ffin a known manner. For example, the processor214may determine motion vectors that indicate the relative spatial offset among adjacent video frames and then determine the overlapping areas of adjacent image frames in response to the motion vectors, and then combine the adjacent video frames so as to exclude the overlapping areas from the combination.

FIG. 7shows an embodiment of a method for determining an intent by a user of the video camera210to acquire an area or object of interest. At step20, the processor214detects a panning motion type in the video12.

At step22, the processor214repeatedly determines a velocity of the panning motion type in response to the rate of acquired video frames and the relative motion between adjacent video frames throughout the duration of the panning motion type. If the velocity of the panning motion type exceeds a predetermined threshold (VTH) then the processor214at step24detects when a velocity of the panning motion type reduces to a relatively stable still motion type. If the still motion type is then maintained for a predetermined time period (TP) the processor214at step26records an index to a video frame in the video12.

The index recorded into the key frame store220at step26corresponds to a video frame of the video12that may have contained an area or object of interest to the user of the video camera210. The index recorded at step26corresponds to a video frame contained in the time period Tp. The processor214may then use the index recorded at step26to obtain a video frame from the video frame store212and then use the obtained video frame to generate a photographic output or a thumbnail output.

The processor214may determine an intent of a user of the video camera210by detecting a zoom in motion type. For example, a user of the video camera210may zoom in to obtain more detail of a particular object of interest. If a zoom in motion type is detected in the video12followed by a still motion type that lasts a predetermined length of time, then it may be inferred that the user of the video camera210intended to zoom in and record an object of interest, e.g. a person's face in a group scene.

FIG. 8shows an embodiment of a method for determining an intent to acquire a close up of an object of interest. At step30, the processor214detects a zoom in motion type in the video12. At step32, the processor214waits until the zoom in motion type stops and then detects a still motion type in the video12. If a still motion type is maintained for a predetermined period of time (Tz) then at step34the processor214records an index to a video frame of the video12into the key frame store220.

The index recorded at step34corresponds to a video frame within the time period TZ. The processor214may then obtain a video frame from the video store212using the index from at step34and then use the obtained video frame to generate a photographic output or a thumbnail output.

The processor214may determine an intent by a user of the video camera210to acquire a moving object by detecting an object motion type in the video12having a duration To. The period Tomay be bounded by an initial video frame (Fio) and a final video frame (Ffo) in the video12. The processor214may record indices into the video12corresponding to the video frames Fioand Ffointo the key frame store220. The processor214may then use the recorded indices for the video frames Fioand Ffoto generate a flipbook output or a slideshow output. For example, the processor214may use the video frames Fioand Ffoas well as one or more of the intervening video frames between the video frames Fioand Ffoto generate photographs for a flipbook. The flipbook photographs may be printed on the printer218. Alternatively, the processor214may use the video frames Fioand Ffoas well as one or more of the intervening video frames between the video frames Fioand Ffoto generate a slideshow output on the display216.

FIG. 9shows a method for repurposing the video12by generating multiple visual outputs that indicate the content of the video12. The processor214performs a motion analysis step40on the video12, followed by a motion type clustering step42on the results of the motion analysis step40, followed by an output generation step44in response to the motion type clustering step42.

In the motion analysis step40, the processor214analyzes the relative motion between adjacent video frames in the video12to detect motion types. The motion analysis step40may detect any combination of panning motion types, zoom in motion types, zoom out motion types, object motion types, still motion types, and complex motion types in the video12.

FIG. 10illustrates the motion type clustering step42. In the motion type clustering step42, the processor214subdivides the video12into a set of segments80-94each segment corresponding to a cluster of video frames having the same motion type. For example, the segment80is a sequence of video frames of the video12having a panning motion type and the segment82is a sequence of video frames of the video12having an object motion type. The segment84has a zoom in motion type, the segment86has an object motion type, the segment88has a still motion type, the segment90has a complex motion type, the segment92has a panning motion type, and the segment94has a zoom in motion type.

FIG. 11shows a set of visual outputs100-114generated at the output generation step44for the example segments80-94of the video12. The processor214generates the visual outputs100-114in response to the motion types assigned to the segments80-94.

The processor214generates a panoramic output100for the segment80at the output generation step44because the segment80has the panning motion type. The processor214also records indices to a set of video frames for the panoramic output100into the key frame store220. Similarly, the processor214generates a panoramic output112for the segment92at the output generation step44because the segment92is the panning motion type and records indices for the panoramic output112into the key frame store220.

The processor214generates a flipbook output102for the segment82at the output generation step44because the segment82is of the object motion type. The flipbook output102includes a set of photographs derived from a set of video frames in the segment82. The photographs for the flipbook output102may be enhanced over and above the resolution of a video frame. The processor214records indices to the video frames used to generate the photographs of the flipbook output102into the key frame store220. Similarly, the processor214generates a flipbook output106for the segment86at the output generation step44because the segment86is of the object motion type and records indices for the flipbook output106into the key frame store220.

The processor214generates a photographic output104for the segment84at the output generation step44because the segment84is of the zoom in motion type. The photographic output104is derived from a video frame in the segment84, e.g. a video frame indexed at step34above. The processor214may perform a resolution enhancement step to increase the resolution of the relevant video frame when generating the photographic output104. The processor214records an index to the video frame used to construct the photographic output104into the key frame store220. Similarly, the processor214generates a photographic output114for the segment94at the output generation step44because the segment94is of the zoom in motion type and records an index for the photographic output114into the key frame store220.

The processor214generates a photographic output108for the segment88at the output generation step44because the segment88is of the still motion type. The photographic output108is based on a video frame in the segment88. The processor214records an index to the video frame used to construct the photographic output108into the key frame store220. An enhancement step may be performed to increase the resolution of the video frame when generating the photographic output108.

FIG. 12illustrates the contents of the key frame store220for the example segments80-94of the video12. The indices (INDEX A through INDEX L) are pointers to corresponding video frames in the video12that are stored in the video store212. The key frame store220associates the indices INDEX A through the INDEX L with the corresponding visual outputs100-112. The processor214may use key frame store220to provide a user interface that enables a user to browse the significant content of the video12.

For example, the processor214may generate a user interface by rendering the visual outputs100-114and generating a selection option that enables a user to individually select the rendered visual outputs100-114. In response to a user selection of the panoramic output100, the processor214obtains the INDEX A and the INDEX B from the key frame store220. The INDEX A points to the frame Fiin the panoramic output100and INDEX B points to the video frame Ffin the panoramic output100. The processor214may use the obtained pointers to replay the video12at the segment that contains the panoramic output100, i.e. the segment80. Similarly, in response to the user selection of the photographic output108, the processor214obtains the INDEX I from the key frame store220and replays the video12at that point in the video12.

The methods disclosed above may be employed in any system that may generate, obtain, or store a video. Examples include computer systems, digital cameras, video appliances, e.g. video recorder/player, home video equipment, etc. For example, alternative mechanisms for acquiring a video that may be repurposed include mechanisms for reading recorded videos and mechanisms for obtaining videos via a communication network, e.g. Internet.

The foregoing detailed description of the present invention is provided for the purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise embodiment disclosed. Accordingly, the scope of the present invention is defined by the appended claims.