Systems and methods for efficient video analysis

Systems (100) and methods (300) for efficient video analysis. The methods involve: automatically identifying features of at least one feature class which are contained in a first video stream; simultaneously generating a plurality of first video chips (904) using first video data defining the first video stream; displaying an array comprising the first video chips within a graphical user interface window; and concurrently playing the first video chips. Each of the first video chips comprises a segment of the first video stream which includes at least one identified feature.

BACKGROUND OF THE INVENTION

1. Statement of the Technical Field

The invention concerns computing systems. More particularly, the invention concerns computing systems and methods for efficient video analysis.

2. Description of the Related Art

The large amount of video surveillance data collected and maintained today requires increasingly efficient methods for analysis. There are many challenges to the analysis of video which are imposed by it's usage as a forensic tool across military applications, law enforcement applications and commercial applications. For example, video analysis is used in unmanned mission applications, critical transportation surveillance applications, energy infrastructure surveillance applications, online geospatial video portal applications, medical applications and industrial production applications. These applications share a common need for efficient analysis of video data which may or may not exist within a geospatial context.

Some traditional approaches for video analysis involve manual video play-back and/or frame-by-frame analysis. One can readily appreciate that techniques employing manual video play-back and frame-by-frame analysis are ad-hoc, time consuming and expensive. Other traditional approaches for video analysis involve comparing the content of two or more video streams. This comparison is achieved by toggling between different video streams or by viewing different video streams that are presented in a side by side manner. Such comparison techniques are time consuming and subject to human error as a result of operator fatigue.

SUMMARY OF THE INVENTION

Embodiments of the present invention concern implementing systems and methods for efficient video analysis. The methods generally involve: automatically identifying features of at least one feature class or visual representations of at least one object which are contained in a first video stream; simultaneously generating a plurality of first video chips using first video data defining the first video stream; displaying an array comprising the first video chips within a Graphical User Interface (“GUI”) window; and concurrently playing the first video chips. The phrases “playing a video chip”, as used herein, means the reproduction of a segment of a video recording after it has been made by sequentially displaying images in an image sequence. Each of the first video chips comprises a segment of the first video stream which includes at least one identified feature.

DETAILED DESCRIPTION

The present invention concerns implementing systems and methods for efficient video analysis. In this regard, the present invention implements a novel technique for simultaneously or concurrently visually inspecting numerous segments of a video stream. The particularities of the novel technique will become more evident as the discussion progresses. Still, it should be understood that the present invention overcomes various drawbacks of conventional video analysis techniques, such as those described above in the background section of this document. For example, the present invention provides more efficient, less time consuming and less costly video analysis processes as compared to those of conventional video analysis techniques. In this regard, it should be understood that the present invention allows an analyst to rapidly review video-based-features at differing temporal resolutions by viewing a video chip grid in a plug-in window. The phrase “temporal resolution”, as used herein, refers to a numerical value representing a total duration of a video stream and/or a numerical value representing a total number of images contained in an image sequence defining a video stream. In this way, the present invention provides an alternative to fast reversing and fast forwarding through an entire video stream or stepping through a video stream frame-by-frame to isolate instances of feature occurrence. Also, the present invention allows fine-scale visualization of a time-of-interest within a video stream with minimal user-software interaction.

The present invention can be used in a variety of applications. Such applications include, but are not limited to, imagery applications, sensor applications, mapping applications, situational awareness applications, natural disaster analysis applications, unmanned vehicle applications, video applications, forensic applications, law enforcement applications, geospatial information based applications, medical applications, military applications, and any other application in which video data needs to be analyzed. Exemplary implementing system embodiments of the present invention will be described below in relation toFIGS. 1-2,5,7,8A and8B. Exemplary method embodiments of the present invention will be described below in relation toFIGS. 3A-30.

Exemplary Systems Implementing the Present Invention

Referring now toFIG. 1, there is provided a block diagram of an exemplary system100that is useful for understanding the present invention. The system100comprises a computing device102, a network104, a server106, a video stream source108, and at least one data store110,112. The system100may include more, less or different components than those illustrated inFIG. 1. However, the components shown are sufficient to disclose an illustrative embodiment implementing the present invention.

The hardware architecture ofFIG. 1represents one embodiment of a representative system configured to facilitate feature data maintenance using video stream source for feature display, quality control, and change detection. As such, system100implements a method for efficient spatial feature data analysis in accordance with embodiments of the present invention. The method will be described in detail below in relation toFIGS. 3A-30. However, it should be understood that the method implements a data driven approach for enabling an efficient evaluation of geospatial data using video data. The phrase “video data”, as used herein, refers to data defining a video stream.

The geospatial data and video data can be stored in the same or different data stores. For example, as shown inFIG. 1, the geospatial data is stored in a feature data store112and the video data is stored in a video data store110. The video data is collected by the video stream source108. The video stream source108comprises an object comprising a video camera. Such objects include, but are not limited to, a satellite, an Unmanned Aerial Vehicle (“UAV”), a plane, a vehicle, a building, a tree or a post. Also, the video data can be communicated to the data store110via network104and server106.

The computing device102facilitates video data analysis. Accordingly, the computing device102has installed thereon a Video Analysis (“VA”) software application and at least one feature analysis plug-in. The VA software application includes, but is not limited to, a Kinovea software application, a MOTIONPRO® software application, a DARTFISH® software application, a Sports Computer Aided Design (“CAD”) software application, or a Full-motion video Asset Management Engine (“FAME™”) software application. Each of the listed VA software applications is well known in the art, and therefore will not be described in detail herein. However, it should be understood that the VA software applications facilitate the display of video streams in an application window. The VA software applications also facilitate the fast forwarding and fast reversing of the displayed video streams.

The feature analysis plug-in is a set of software components that adds specific abilities to the VA software application. For example, the feature analysis plug-in provides the ability to: concurrently and/or simultaneously generate a plurality of video chips using video data defining a video stream; and display all or a portion of the generated video chips in a display area of a plug-in window at the same time. The phrase “video chip”, as used herein, refers to a spatial-temporal segment of video in which at least one feature of a feature class has been identified. The phrase “spatial-temporal segment of video”, as used herein, refers to a segment of a video stream which has timing information (e.g., timestamps indicating when images of the segment were captured) and spatial information associated therewith. The timing information includes, but is not limited to, timestamps indicating when images of a video stream are captured. The spatial information includes, but is not limited to, information indicating locations on the Earth which are visually represented by content of images of a video stream (e.g., Global Positioning System information). The feature of the video chip can be used as a “finger print” for purposes of matching, feature change detection, causality identification, feature maintenance and performing other tasks. The feature changes can include, but are not limited to, the addition/destruction of a road, the addition/destruction of a railroad, the addition/destruction of a transmission line, the addition/destruction of a pipeline, and the expansion/destruction of a building. The destruction of a feature can result from a natural disaster, a public disorder, a military operation, a demolition or other cause.

A video chip may include one or more features of interest. The term “feature”, as used herein, refers to a visual representation of an object. Such objects include, but are not limited to, bridges, water towers, boats, planes, roads, lakes, buildings, gas stations, restaurants, malls, stores, vehicles, people, and cisterns. Notably, the video chips may be displayed in the plug-in window in a grid format or a matrix format. In the grid scenario, each cell of a grid includes one video chip. As a result of such a grid arrangement of video chips, a user can perform feature analysis in a shorter period of time as compared to that needed to perform a feature analysis using the conventional technique employed by the VA software application. This conventional technique generally involves manually fast forwarding and/or fast reversing to each instance of a feature class.

Referring now toFIG. 2, there is provided a block diagram of an exemplary embodiment of the computing device102. The computing device102can include, but is not limited to, a notebook, a desktop computer, a laptop computer, a personal digital assistant, and a tablet PC. The server106ofFIG. 1can be the same as or similar to computing device102. As such, the following discussion of computing device102is sufficient for understanding server106ofFIG. 1. Notably, some or all the components of the computing device102can be implemented as hardware, software and/or a combination of hardware and software. The hardware includes, but is not limited to, one or more electronic circuits.

Notably, the computing device102may include more or less components than those shown inFIG. 2. However, the components shown are sufficient to disclose an illustrative embodiment implementing the present invention. The hardware architecture ofFIG. 2represents one embodiment of a representative computing device configured to facilitate feature data analysis in an efficient manner. As such, the computing device102ofFIG. 2implements improved methods for feature data analysis in accordance with embodiments of the present invention.

As shown inFIG. 2, the computing device102includes a system interface222, a user interface202, a Central Processing Unit (“CPU”)206, a system bus210, a memory212connected to and accessible by other portions of computing device102through system bus210, and hardware entities214connected to system bus210. At least some of the hardware entities214perform actions involving access to and use of memory212, which can be a Random Access Memory (“RAM”), a disk driver and/or a Compact Disc Read Only Memory (“CD-ROM”).

System interface222allows the computing device102to communicate directly or indirectly with external communication devices (e.g., server106ofFIG. 1). If the computing device102is communicating indirectly with the external communication device, then the computing device102is sending and receiving communications through a common network (e.g., the network104shown inFIG. 1).

Hardware entities214can include a disk drive unit216comprising a computer-readable storage medium218on which is stored one or more sets of instructions220(e.g., software code) configured to implement one or more of the methodologies, procedures, or functions described herein. The instructions220can also reside, completely or at least partially, within the memory212and/or within the CPU206during execution thereof by the computing device102. The memory212and the CPU206also can constitute machine-readable media. The term “machine-readable media”, as used here, refers to a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions220. The term “machine-readable media”, as used here, also refers to any medium that is capable of storing, encoding or carrying a set of instructions220for execution by the computing device102and that cause the computing device102to perform any one or more of the methodologies of the present disclosure.

In some embodiments of the present invention, the hardware entities214include an electronic circuit (e.g., a processor) programmed for facilitating efficient feature data analysis. In this regard, it should be understood that the electronic circuit can access and run VA software applications (not shown inFIG. 2), feature analysis plug-ins (not shown inFIG. 2) and other types of applications installed on the computing device102. The VA software applications are generally operative to facilitate the display of video streams in an application window, the fast forwarding of displayed video streams, and the fast reversing of displayed video streams. The listed functions and other functions implemented by the VA software applications are well known in the art, and therefore will not be described in detail herein. A schematic illustration of an exemplary application window504is provided inFIG. 5.

The feature analysis plug-ins are generally operative to display a plug-in window on a display screen of the computing device102. A schematic illustration of an exemplary plug-in window702is provided inFIG. 7. Various types of information can be presented in the plug-in window. Such information includes, but is not limited to, video chips and feature attributes. The feature attributes can include, but are not limited to, attributes of an object which a feature visually represents (e.g., heights, lengths, diameters, longitudes, latitudes, addresses, names, text and sales volumes).

The feature analysis plug-ins are also operative to perform one or more of: automatically and simultaneously generate a plurality of video chips in response to a user-software interaction; generate at least one page of video chips arranged in a grid or matrix format; display pages of video chips in a plug-in window; simultaneously or concurrently play a plurality of video chips of a displayed page of video chips; aggregate two or more video chips in response to a user-software interaction; generate at least one page of aggregated video chips arranged in a grid or matrix format; display pages of aggregated video chips in a plug-in window; simultaneously or concurrently play a plurality of video chips of a displayed page of aggregated video chips; display at least one attribute of a selected video chip image in a plug-in window; automatically fast forward and/or fast reverse a video stream displayed in an application window until the segment of the video stream comprising the selected video chip is displayed in an application window; sort a plurality of video chips based on at least one feature attribute; generate and display at least one page of video chips which are arranged in a sorted order; simultaneously or concurrently playing video chips of a page of sorted video chips; filter video chips based on at least one feature attribute; randomly select and display only a percentage of a plurality of video chips; change a grid size in response to a user-software interaction; change a temporal level of resolution of displayed video chips in response to a user-software interaction; change a spatial zoom level of resolution of displayed video chips in response to a user-software interaction; cycle through pages of video chips that were generated using a plurality of video streams; mark video chips in response to user software-interactions; unmark video chips in response to user-software interactions; and remember various settings that a user sets for each feature class (e.g., bridges, water towers and gas stations) during at least one session. The listed functions and other functions of the feature analysis plug-ins will become more apparent as the discussion progresses. Notably, one or more of the functions of the feature analysis plug-ins can be accessed via a toolbar, menus and other GUI elements of the plug-in window.

A schematic illustration of an exemplary toolbar704of a plug-in window (e.g., plug-in window702ofFIG. 7) is provided inFIG. 8A. As shown inFIG. 8A, the toolbar704comprises a plurality of exemplary GUI widgets802-830. Each of the GUI widgets802-830is shown inFIG. 8Aas a particular type of GUI widget. For example, GUI widget802is shown as a drop down menu. Embodiments of the present invention are not limited in this regard. The GUI widgets802-830can be of any type selected in accordance with a particular application.

GUI widget802is provided to facilitate the display of an array of video chips including features of a user selected feature class (e.g., chimney/smokestack, gas station, restaurant, lake, road, water tower, and building). The array of video chips is displayed in the display area (e.g., display area706ofFIG. 7) of the plug-in window (e.g., plug-in window702ofFIG. 7) in a grid format. In the embodiment shown inFIG. 8A, the GUI widget802includes, but is not limited to, a drop down list that is populated with the feature classes identified in a previously generated feature list. Drop down lists are well known in the art, and therefore will not be described herein.

GUI widget804is provided to facilitate moving through pages of video chips associated with a single feature class. If a selected feature class has more than the maximum number of features that can fit in a grid of the a selected grid size (e.g., three cells by three cells), then the feature analysis plug-in generates a plurality of pages of video chips. Each page of video chips includes a grid with video chips contained in the cells thereof. As shown in the embodiment ofFIG. 8A, the GUI widget804includes, but is not limited to, a text box, a forward arrow button and a backward arrow button. Text boxes and arrow buttons are well known in the art, and therefore will not be described herein. This configuration of the GUI widget804allows a user to move forward and backward through the pages of video chips associated with a single video stream. Paging forward or backward will cause the video chip in an upper left corner grid cell of the new page to be selected. The page context is displayed in the text box as the numerical range of video chips displayed (e.g., video chips one through nine) and the total number of video chips (e.g., twenty) providing visual representations of features of a selected feature class.

GUI widget806is provided to facilitate jumping to a desired page of video chips for review. As shown in the embodiment ofFIG. 8A, GUI widget806includes, but is not limited to, a text box and a search button. The text box is a box in which to enter a page number (e.g., three). Clicking the search button will cause the page of video chips having the entered page number to be displayed in the display area (e.g., display area706ofFIG. 7) of the plug-in window (e.g., plug-in window702ofFIG. 7).

GUI widget808is provided to facilitate a selection of a grid size from a plurality of pre-defined grid sizes. As shown inFIG. 8A, the GUI widget808includes, but is not limited to, a drop down list listing a plurality of pre-defined grid sizes. In some embodiments, the pre-defined grid sizes include one cell by one cell, two cells by two cells, three cells by three cells, four cells by four cells, five cells by five cells, six cells by six cells, seven cells by seven cells, eight cells by eight cells, nine cells by nine cells, and ten cells by ten cells. The grid size of two cells by two cells ensures that a maximum of four video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of three cells by three cells ensures that a maximum of nine video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of four cells by four cells ensures that a maximum of sixteen video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of five cells by five cells ensures that a maximum of twenty-five video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of six cells by six cells ensures that a maximum of thirty-six video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of seven cells by seven cells ensures that a maximum of forty-nine video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of eight cells by eight cells ensures that a maximum of sixty-four video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of nine cells by nine cells ensures that a maximum of eight-one video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. The grid size of ten cells by ten cells ensures that a maximum of one hundred video chips will be simultaneously or concurrently displayed in the display area of the plug-in window. Embodiments of the present invention are not limited to grid having an equal number of cells in the rows and columns thereof. For example, a grid can alternatively have a grid size of four cells by three cells such that each column thereof comprises four cells and each row thereof comprises three cells, or vice versa.

Notably, the display area for each video chip is different for each grid size. For example, the display area for each video chip in a grid having a grid size of two cells by two cells is larger than the display area for each video chip in a grid having a grid size of three cells by three cells. Also, if each video chip has the same spatial zoom level of scale or resolution, then the portion of a video stream contained in a video chip displayed in a two cell by two cell grid is larger than the portion of a video stream contained in a video chip displayed in a three cell by three cell grid. It should also be noted that, in some embodiments, a selected video chip of a first grid will reside in an upper left corner cell of a second grid having an enlarged or reduced grid size.

GUI widget812is provided to facilitate a selection of features for display in the display area (e.g., display area706ofFIG. 7) of the plug-in window (e.g., plug-in window702ofFIG. 7) based on their attributes. As shown inFIG. 8A, the GUI widget812includes a “filter control” button and a “filter setting” drop down button. The “filter control” button facilitates the enablement and disablement of an attribute filter function of the feature analysis plug-in. The “filter setting” drop down button facilitates the display of a drop-down box for assembling a query phrase defining an attribute filter (e.g., [“HEIGHT”=‘100 Feet’], [“HEIGHT”<‘100 Feet’], [“HEIGHT”< >‘100 Feet’], [“HEIGHT” IS NULL], [“HEIGHT” IS NOT NULL], [“HEIGHT”≧‘100 Feet’ AND “DIAMETER”>‘40 Feet’], or [“HEIGHT”≦‘100 Feet’ OR “DIAMETER”>‘40 Feet’]). A schematic illustration of an exemplary drop-down box850is provided inFIG. 8B. When the attribute filter function is enabled, the query phrase takes effect immediately.

Notably, the feature analysis plug-in remembers the filter query phrase that a user sets for each feature class during a session. Accordingly, if the user changes a feature class from a first feature class (e.g., bridges) to a second feature class (e.g., water towers) during a session, then the previously set filter query for the second feature class will be restored. Consequently, only features of the second feature class (e.g., water towers) which have the attribute specified in the previously set filter query (e.g., “HEIGHT”=‘100 Feet’) will be displayed in the plug-in window.

GUI widget814is provided to facilitate the sorting of video chips based on one or more attributes of the features contained therein. For example, a plurality of video chips are sorted into an ascending or descending order based on the heights and/or diameters of the water towers visually represented by the features contained therein. As shown inFIG. 8A, the GUI widget814includes a drop down list. Embodiments of the present invention are not limited in this regard. For example, the GUI widget814can alternatively include a button and a drop down arrow for accessing a drop down box. The button facilitates the enablement and disablement of a sorting function of the feature analysis plug-in. The drop down box allows a user to define settings for sorting video chips based on one or more attributes of an active feature class. As such, the drop down box may include a list from which an attribute can be selected from a plurality of attributes. The drop down box may also include widgets for specifying whether the video chips should be sorted in an ascending order or a descending order.

Notably, the feature analysis plug-in remembers the sort settings that a user defines for each feature class during a session. Accordingly, if the user changes a feature class from a first feature class (e.g., bridges) to a second feature class (e.g., water towers) during a session, then the previously defined sort settings for the second feature class will be restored. Consequently, video chips containing features of the second feature class (e.g., water towers) will be displayed in a sorted order in accordance with the previously defined sort settings.

GUI widget820is provided to facilitate the display of a random sample of video chips of features of a particular feature class for visual inspection and quality control testing. As such, the GUI widget820includes, but is not limited to, a button for enabling/disabling a random sampling function of the feature analysis plug-in and a drop down menu from which a percentage value can be selected.

GUI widget822is provided to facilitate the aggregation of at least one set of video chips to obtain at least one combined video chip with a longer duration than each of its component parts (i.e., the video chips of the set of video chips). For example, a first video chip is combined with an immediately following second video chip so as to form a combined video chip. In this scenario, each of the first and second video chips has a duration of five minutes. Consequently, the combined video chip has a duration of ten minutes. Embodiments of the present invention are not limited in this regard. As shown inFIG. 8A, the GUI widget822includes, but is not limited to, a button for enabling and disabling an aggregation function of the feature analysis plug-in. The GUI widget822may also comprise a means (not shown) for defining how many video chips should be combined to form each combined video chip (e.g., a text box and a drop down list).

GUI widget810is provided to facilitate the selection of a video stream from a plurality of video streams. As shown inFIG. 8A, the GUI widget810includes, but is not limited to, a text box and a drop down list populated with the names of video streams. If a user selects a new item from the drop down list, then the feature analysis plug-in generates and displays at least one page of video chips using the video stream indentified by the newly selected item. The video chips contain features of the same feature class as the immediately preceding displayed video chips. The text box displays information identifying the video stream from which the currently displayed video chips were generated. The contents of the text box can be updated in response to a user selection of a new item from the drop down list. The contents of the text box can also be updated by the feature analysis plug-in during the performance of video stream cycling operations, which will be described below in relation to GUI widget824. Accordingly, the information contained in the text box always identifies the video stream from which the currently displayed video chips were generated.

GUI widget824is provided to facilitate the cycling through video chip pages for a plurality of video streams. A user may want to cycle through such video chip pages for change detection purposes. The GUI widget824is configured to allow manual cycling and/or automatic cycling between video chip pages for a plurality of video streams. As such, the GUI widget824includes, but is not limited to, a check box for enabling and disabling video stream cycling operations of the feature analysis plug-in, a slider for setting the rate at which the video streams automatically cycle, and/or a button for manually commanding when to change the video stream.

GUI widget826is provided to facilitate the performance of manual-spatial scale operations by the feature analysis plug-in. The manual-spatial scale operations are operative to adjust the spatial zoom level of scale of images of all of the displayed video chips from a first spatial zoom level of scale to a second spatial zoom level of scale in response to a user-software interaction. The first spatial zoom level of scale is a default spatial zoom level of scale (e.g., 100%) or a previously user-selected spatial zoom level of scale (e.g., 50%). The second spatial zoom level of scale is a new user-selected spatial zoom level of scale (e.g., 75%). As such, the GUI widget826includes, but is not limited to, a drop down list populated with a plurality of whole number percentage values. The percentage values include, but are not limited to, whole number values between zero and one hundred.

GUI widget828is provided to facilitate the viewing of each displayed feature at its best-fit spatial zoom level of scale or its pre-defined maximum spatial zoom level of scale. As such, the GUI widget828includes, but is not limited to, a button for enabling and disabling auto-spatial scale operations of the feature analysis plug-in. When the auto-spatial scale operations are enabled, the manual-spatial scale operations are disabled. Similarly, when the auto-spatial scale operations are disabled, the manual-spatial scale operations are enabled.

GUI widget830is provided to facilitate the viewing of a plurality of “temporally zoomed” video chips using a feature analysis plug-in. In this regard, it should be understood that temporal resolution operations are initiated via GUI widget830. The temporal resolution operations involve modifying a temporal level of resolution of at least one video chip. The temporal resolution is modified by (a) altering the temporal resolutions of video chips which precede a selected video chip in a temporal order, (b) altering the temporal resolutions of video chips which succeed the selected video chip in the temporal order, or (c) altering the temporal resolution of only the selected video chip. For example, in scenario (c), a selected video chip has an original temporal level of resolution of one minute. In response to a user-software interaction, the temporal level of resolution of the selected video chip is changed to ten seconds. Consequently, the content of the selected video chip is re-displayed in a plug-in window as six, ten second video chips, rather than one sixty second video chip. Embodiments of the present invention are not limited to the particularities of the above-provided examples. The value of the temporal level of scale can be increased and/or decreased via GUI widget830. As shown inFIG. 8A, the GUI widget830includes, but is not limited to, a button for enabling and disabling temporal resolution operations of the feature analysis plug-in. The GUI widget830may also comprise a means (not shown) for selecting a new value (e.g., ten seconds) for the temporal level of resolution (e.g., a text box and a drop down list).

GUI widget816is provided to facilitate the writing of all “flagged” video chips to an output file stored in a specified data store (e.g., feature data store112ofFIG. 1). GUI widget818is provided to facilitate the saving of all video chips which have been “flagged” during a session to a user-named file. The user-named file can include, but is not limited to, a shapefile. Shapefiles are well known in the art, and therefore will not be described herein. In some embodiments of the present invention, a video chip is “flagged” by right clicking on the video chip to obtain access to a “chip context” GUI and selecting a “flag” item from the “chip context” GUI.

As evident from the above discussion, the system100implements one or more method embodiments of the present invention. The method embodiments of the present invention provide implementing systems with certain advantages over conventional video analysis systems. For example, the present invention provides a system in which an analysis of video data can be performed in a shorter period of time as compared to the amount of time needed to analyze video data using conventional fast forward/fast reverse techniques. The present invention also provides a system in which video data is analyzed much more efficiently than in conventional video data analysis systems. The manner in which the above listed advantages of the present invention are achieved will become more evident as the discussion progresses.

Exemplary Methods of the Present Invention

Referring now toFIGS. 3A-3G, there is provided a flow diagram of an exemplary method300for efficient video data analysis that is useful for understanding the present invention. As shown inFIG. 3A, the method300begins with step301and continues with step302. In step302, video data and spatial feature data is collected. The spatial feature data can include, but is not limited to, data defining feature identifiers, geographic locations of objects visible in a video stream, and spatial relationships between the objects.

After the video data is collected, it is stored in a first data store (e.g., video data store110ofFIG. 1) that is accessible by a computing device (e.g., computing device102ofFIG. 1), as shown by step303. In a next step304, video metadata is obtained from the first data store. The video metadata can include, but is not limited to, data defining video stream identifiers, video stream sizes, video stream collection dates and times, video stream content, and the correspondence of video stream content to ground. The video metadata is then used in step305to generate a list of video stream identifiers. This list of video stream identifiers will be subsequently used by a feature analysis plug-in to allow a user to select at least one of a plurality of video streams to analyze at any given time.

The video data, video metadata and/or spatial feature data is used in steps306and307for temporal registration and geospatial registration of the video streams. Methods for temporal registration and geospatial registration are well known in the art, and therefore will not be described herein. However, it should be understood that temporal registration is generally performed to establish correspondences between temporal frames of video sequences. Geospatial registration is generally performed to accurately map between video stream coordinates and geo-coordinates. Any known method for temporal registration and geospatial registration of video streams can be used with the present invention without limitation. Notably, such known techniques may employ place/name databases, GOOGLE® maps, and/or Global Positioning System (“GPS”) information. Step307also involves performing operations by the feature analysis plug-in to identify features of at least one feature class within the video streams.

Similar to the video data, the spatial feature data is stored in a data store (e.g., feature data store112ofFIG. 1) after it is collected, as shown by step308. The spatial feature data is used in step310to generate a feature list including a series of items identifying feature classes (e.g., bridges, water towers, buildings, vehicles and gas stations). Subsequently, the feature list is stored in the data store (e.g., data store112ofFIG. 1).

Upon completing step310, the method continues with step312where a VA software application is launched. The VA software application can be launched in response to a user software interaction. For example, as shown inFIG. 4, a VA software application can be launched by accessing and selecting a “Video Analysis Software Program” entry450on a start menu454of a desktop window452.

In a next step314, an application window is displayed on top of the desktop window. A schematic illustration of an exemplary application window is provided inFIG. 5. As shown inFIG. 5, the application window504includes a toolbar510including GUI widgets for at least fast reversing a video stream, playing a video stream, pausing a video stream, fast forwarding a video stream, and launching a plug-in. The application window504also includes a video display area506in which a video stream can be presented to a user of the computing device (e.g., computing device102ofFIG. 1).

Referring again toFIG. 3A, a first video stream is displayed in the application window, as shown in step316. A schematic illustration showing an exemplary first video stream508displayed in an application window504is provided inFIG. 5. The first video stream508contains video data relating to the destruction of a feature of interest (e.g., a visual representation of a bridge) during a flood. Embodiments of the present invention are not limited in this regard.

After the first video stream is presented to a user of the computing device (e.g., computing device102ofFIG. 1), a feature analysis plug-in is launched, as shown by step318. The feature analysis plug-in can be launched in response to a user-software interaction. For example, as shown inFIG. 6, a feature analysis plug-in is launched by selecting an item602of a drop down menu of a toolbar510. Once the feature analysis plug-in is launched, the method300continues with step319ofFIG. 3B.

Referring now toFIG. 3B, step319involves displaying a plug-in window on top of the desktop window and/or application window. A schematic illustration of an exemplary plug-in window702is provided inFIG. 7. As shown inFIG. 7, the plug-in window702comprises a toolbar704, a display area706, an attribute pane708, and a scrollbar710. A schematic illustration of the toolbar704is provided inFIG. 8A. As shown inFIG. 8A, the toolbar704comprises a plurality of exemplary GUI widgets802-830. Each of the GUI widgets802-830is described above in detail.

Referring again toFIG. 3B, a next step320involves receiving a user input for viewing a paralyzed view of multiple animated video segments of the first video stream comprising “features” of a particular type. In response to the user-software interaction of step320, the method300continues with step322where the feature analysis plug-in automatically and concurrently generates a plurality of first video chips. The first video chips are generated using the feature list generated in previous step310, the spatial feature data for the features which are of the particular type, the video metadata and/or the video data for the first video stream. Each of the first video chips includes a spatial-temporal segment of the first video stream which comprises at least one previously identified feature. The feature may be used as a “finger print” for purposes of matching, change detection, causality identification, feature maintenance and performing other tasks. The first video chips can have a default temporal level of resolution (e.g., thirty seconds, one minute, or two plus minutes) and/or a default spatial zoom level of scale or resolution.

Upon completing step322, step324is performed where at least one page of first video chips is created by the feature analysis plug-in. The first video chips are arranged on the page in a grid or matrix format. The grid or matrix of the video chips has a default size (e.g., ten cells by ten cells) or a user-specified size (e.g., three cells by three cells). In a next step325, a first page of video chips is displayed in a plug-in window. A schematic illustration of a displayed page of video chips902is provided inFIG. 9. As shown inFIG. 9, the page902comprises a grid906defined by a plurality of grid cells908. A different video chip904is presented within each grid cell908of the grid906. Upon completing step325, the method300continues with step326. In step326, the displayed video chips are simultaneously or concurrently played.

In a next step327, a user input is received by the computing device for viewing a page of aggregated video chips. Each of the aggregated video chips has a duration (e.g., one minute) which is greater than the default duration (e.g., thirty seconds). The user input is facilitated by a GUI widget (e.g., GUI widget822ofFIG. 8A) of the plug-in window (e.g., the plug-in window702ofFIG. 7). The GUI widget may be configured to allow a user to enable an aggregation function of the feature analysis plug-in, specify which video chips should be aggregated, and/or define how many video chips should be combined to form each of a plurality of combined video chips. As such, the GUI widget822may include, but is not limited to, a button for enabling and disabling the aggregation function of the feature analysis plug-in, a means for specifying which video chips should be aggregated, and/or a means for defining how many video chips should be combined to form each combined video chip (e.g., a text box and a drop down list).

Referring again toFIG. 3B, the method300continues with step328. In step328, the feature analysis plug-in automatically and simultaneously generates a plurality of aggregated video chips. Each aggregated video chip comprises a spatial-temporal segment of the first video stream including two or more of the first video chips arranged in an ascending temporal order. Also, each of the aggregated video chips has a duration that is greater than the duration of each of its counterparts (i.e., the first video chips which were aggregated). Thereafter, in step329, at least one page of aggregated video chips is created by the feature analysis plug-in. The aggregated video chips are arranged within the page in a grid or matrix format. A schematic illustration of an exemplary page of aggregated video chips1002is provided inFIG. 10. Each of the aggregated video chips is an aggregation of two first video chips904ofFIG. 9. For example, aggregated video chip1004is an aggregation of video chips910ofFIG. 9. Aggregated video chip1006is an aggregation of video chips912ofFIG. 9. Aggregated video chip1008is an aggregation of video chips914ofFIG. 9, and so on. Embodiments of the present invention are not limited in this regard. The page of aggregated video chips is then displayed in the plug-in window, as shown by step330. Upon completing step330, step331is performed where the aggregated video chips are simultaneously or concurrently played.

Subsequently, the method300continues with step332ofFIG. 3C. Step332involves receiving a user input selecting a video chip of the first page of aggregated video chips. The video chip can be selected by moving a mouse cursor over the video chip and clicking a mouse button. A schematic illustration of a selected video chip1004is provided inFIG. 11. As shown inFIG. 11, the selected video chip1104is annotated with a relatively thick and distinctly colored border. Embodiments of the present invention are not limited in this regard. Any type of mark or annotation can be used to illustrate that a particular video chip has been selected.

In response to this user input, step333is performed where attribute information for the feature contained in the selected video chip is displayed in an attribute pane (e.g., attribute pane708ofFIG. 7) of the plug-in window (e.g., plug-in window702ofFIG. 7). A schematic illustration of an exemplary plug-in window702is provided inFIG. 11which has attribute information a1, a2displayed therein. Step333may also involve automatically fast forwarding and/or fast reversing the first video stream displayed in the application window until the segment thereof comprising the selected video chip is displayed in the application window.

In a next step334, a user input is received by the computing device for sorting all or a portion of the first video chips based on at least one attribute of the features contained therein. The user input is facilitated by a GUI widget (e.g., GUI widget814ofFIG. 8A) of the plug-in window (e.g., the plug-in window702ofFIG. 7). The GUI widget may be configured to allow a user to specify the attribute(s) that the sorting should be based on, and/or specify whether the video chips should be sorted in an ascending order or a descending order.

In response to the user input of step334, all or a portion of the first video chips are sorted in an ascending order or a descending order based on the user-specified feature attribute(s), as shown by step335. Thereafter in step336, at least one page of sorted video chips is created by the feature analysis plug-in. The sorted video chips are arranged on the page in a pre-defined grid format or a matrix format. A first page of sorted video chips is then displayed in the plug-in window, as shown by step337. The first page of sorted video chips may or may not include the same segments of the first video stream as the first page of first video chips. For example, if the first grid has a grid size of three cells by three cells, then video chips one through nine of one hundred video chips are presented therein. Thereafter, an ordered list is generated by sorting the one hundred video chips by at least one attribute (e.g., height). In this scenario, the first grid is updated to include the first nine video chips identified in the ordered list. These first nine video chips of the ordered list may include one or more of the original video chips (e.g., views 1-9), as well as one or more video chips (e.g., views 10-100) different than the original video chips. A schematic illustration of an exemplary first page of sorted video chips1202is provided inFIG. 12. As shown inFIG. 12, the first page of sorted video chips1202includes the video chips A1, A67, A21, A88, A5, A45, A47, A33, A72. Notably, none of the video chips A1, A67, A21, A88, A5, A45, A47, A33, A72is contained in the first page of video chips902ofFIG. 9. Embodiments of the present invention are not limited in this regard. For example, the first page of sorted video chips1202can alternatively include one or more of the video chips contained in page902ofFIG. 9. After the first page of sorted video chips is displayed in the plug-in window, step338is performed where the video chips are simultaneously or concurrently played.

In a next step339, a user input is received by the computing device for viewing a second page of sorted video chips in the plug-in window. The user input is facilitated by a GUI widget (e.g., GUI widget804or806ofFIG. 8A) of the plug-in window (e.g., the plug-in window702ofFIG. 7). The GUI widget may be configured to facilitate moving through pages of unsorted and/or sorted video chips. In this regard, the GUI widget includes arrow buttons that allow a user to move forward and backward through the pages of unsorted and/or sorted video chips. Alternatively or additionally, the GUI widget may be configured to facilitate jumping to a desired page of unsorted and/or sorted video chips for review. In this regard, the GUI widget includes a text box for entering a page number and a search button for causing the page of unsorted and/or sorted video chips having the entered page number to be displayed in the display area (e.g., display area706ofFIG. 7) of the plug-in window (e.g., plug-in window702ofFIG. 7).

After the user input is received in step339, step340where a second page of the sorted video chips is displayed in the plug-in window. A schematic illustration of an exemplary second page of sorted video chips1302is provided inFIG. 13. As shown inFIG. 13, the second page of sorted video chips1302includes the video chips A100, A25, A14, A63, A51, A32, A7, A99, A71. Thereafter, the sorted video chips are simultaneously or concurrently played, as shown by step341. Upon completing step341, the method300continues with step342ofFIG. 3D.

As shown inFIG. 3D, step342involves receiving, by a computing device, a user input for filtering the video chips of the second page of sorted video chips by one or more feature attributes thereof. The user input is facilitated by a GUI widget (e.g., GUI widget812ofFIG. 8A) of the plug-in window (e.g., the plug-in window702ofFIG. 7). The GUI widget is configured to facilitate a selection of features for display in the display area (e.g., display area706ofFIG. 7) of the plug-in window (e.g., plug-in window702ofFIG. 7) based on their attributes. In this regard, the GUI widget may include, but is not limited to, a “filter control” button and a “filter setting” drop down button. The “filter control” button facilitates the enablement and disablement of an attribute filter function. The “filter setting” drop down button facilitates the display of a drop-down box for assembling a query phrase defining an attribute filter (e.g., [“HEIGHT”=‘100 Feet’], [“HEIGHT”<‘100 Feet’], [“HEIGHT”< >‘100 Feet’], [“HEIGHT” IS NULL], [“HEIGHT” IS NOT NULL], [“HEIGHT”≧‘100 Feet’ AND “DIAMETER”>‘40 Feet’], or [“HEIGHT”≦‘100 Feet’ OR “DIAMETER”>‘40 Feet’]). A schematic illustration of an exemplary drop-down box850is provided inFIG. 8B.

Upon receipt of the user input in step342, the feature analysis plug-in performs operations to filter the video chips of the displayed second page of sorted video chips, as shown by step343. In a next step344, a page of filtered video chips is created by the feature analysis plug-in. The page of filtered video chips is created by removing at least one video chip from the displayed second page of sorted video chips in accordance with the results of the filtering operations performed in previous step353. Thereafter, the page of filtered video chips is displayed in the display area (e.g., display area706ofFIG. 7) of the plug-in window (e.g., plug-in window702ofFIG. 7), as shown by step345. The displayed filtered video chips are then simultaneously or concurrently played in step346. A schematic illustration of an exemplary page of filtered video chips1402is provided inFIG. 14. As shown inFIG. 14, the page of filtered video chips1402includes the video chips A25, A14, A63, A51, A32, A7, A71contained in the second page of sorted video chips1302ofFIG. 13. However, the page of filtered video chips1402does not include video chips A100and A99in grid cells thereof. In this regard, it should be understood that video chips A100and A99have been removed from the second page of sorted video chips1302ofFIG. 13to obtain the page of filtered video chips1402. Embodiments of the present invention are not limited in this regard.

Referring again toFIG. 3D, the method300continues with step347where the computer device receives a user input for viewing only a portion (e.g., a percentage) of the video chips generated in previous step322. The user input is facilitated by a GUI widget (e.g., GUI widget820ofFIG. 8A) of the plug-in window (e.g., the plug-in window702ofFIG. 7). The GUI widget is configured to facilitate the display of a random sample of video chips of features of a particular feature class for visual inspection. As such, the GUI widget may include, but is not limited to, a button for enabling/disabling a random sampling function of the feature analysis plug-in and a drop down menu from which a percentage value can be selected.

In response to the reception of the user input in step347, step348is performed where “N” video chips of the first video chips generated in previous step322are randomly selected. The value of “N” is determined based on the percentage value selected in previous step347. For example, if one hundred first video chips were generated in step322and the percentage value of twenty was selected in step347, then twenty video chips would be randomly selected from the one hundred first video chips. Embodiments of the present invention are not limited in this regard.

Upon completing step348, step349is performed where the feature analysis plug-in creates at least one page of sampled video chips including all or a portion of the “N” video chips arranged in a grid or matrix format. Notably, the pages of sampled video chips can have a default grid size or a user-specified grid size. For example, if a grid size is four cells by four cells and “N” equals twenty, then two pages of sampled video chips would be created in step349since each page can contain a maximum of sixteen video chips. In contrast, if the grid size is five cells by five cells and “N” equals twenty, then only one page of sampled video chips would be created in step349since the page can contain a maximum of twenty-five video chips. Embodiments of the present invention are not limited in this regard. In a next step350, the page of sampled video chips is displayed in the plug-in window. Thereafter, the sampled video chips are simultaneously or concurrently played, as shown by step351.

A schematic illustration of an exemplary page of sampled video chips1502is provided inFIG. 15. As shown inFIG. 15, the page of sampled video chips1502includes only three video chips A10, A41, A62. In this regard, the total number of first video chips is nine and the percentage selected in step347is thirty-three percent. Accordingly, the value of “N” is three. The three video chips contained in the page1502were randomly selected from the nine first video chips. Embodiments of the present invention are not limited to the particularities of this example.

Referring again toFIG. 3D, the method300continues with step352where the computing device receives a user input for changing a grid size of the page(s) of sampled video chips from a first grid size (e.g., three cells by three cells) to a second grid size (e.g., two cells by two cells). The user input is facilitated by a GUI widget (e.g., GUI widget808ofFIG. 8A) of the plug-in window (e.g., the plug-in window702ofFIG. 7). The GUI widget is configured to facilitate a selection of a grid size from a plurality of pre-defined grid sizes. As such, the GUI widget may include, but is not limited to, a drop down list listing a plurality of pre-defined grid sizes.

In response to the reception of the user input in step352, step353ofFIG. 3Eis performed. As shown inFIG. 3E, step353involves creating at least one modified page including a grid of sampled video chips having the second grid size. The modified page is then displayed in the plug-in window, as shown by step354. In a next step355, the video chips of the modified page of sampled video chips are simultaneously or concurrently played.

A schematic illustration of an exemplary modified page of sampled video chips1602is provided inFIG. 16. As shown inFIG. 16, the modified page1602comprises a grid1606with a grid size of two cells by two cells. Each of the four grid cells1608includes a video chip A10, A41, A62. The video chips are the same as the video chips in the page of sampled video chips1502ofFIG. 15. Embodiments of the present invention are not limited in this regard. For example, the modified page1602may include less or more than the total number of video chips contained in page1502. In the “less than” scenario, the absence of certain video chips from page1602would be a result of the reduction in grid size from three cells by three cells to two cells by two cells. In the “more than” scenario, the inclusion of certain video chips in page1602would be a results from an increase in grid size. Notably, the display area for each video chip of page1602is larger than the display area for each video chip of page1502. This increase of display area for the video chips is also a result of the reduction in grid size from three cells by three cells to two cells by two cells. It should also be noted that larger portions of the images of the first video stream surrounding features thereof are displayed in the grid cells1608as compared to that displayed in corresponding grid cells1508ofFIG. 15. This increase in surrounding portions of the video stream images is at least partially due to the reduction in grid size as well as the fact that the video chips of grid cells1608have the same spatial zoom level of scale or resolution. Embodiments of the present invention are not limited in the particularities ofFIG. 16.

Referring again toFIG. 3E, the method300continues with step356. In step356, the computing device receives a user input for viewing a plurality of “temporally zoomed” video chips comprising at least one selected video chip. Each of the “temporally zoomed” video chips has a temporal resolution (or duration) which is lower than the temporal resolution of the selected video chip(s). The video chip can be selected by moving a mouse cursor over the video chip and clicking a mouse button. A schematic illustration of an exemplary selected video chip1704is provided inFIG. 17. As shown inFIG. 17, the selected video chip1704is annotated with a relatively thick and distinctly colored border. Embodiments of the present invention are not limited in this regard. Any type of mark or annotation can be used to illustrate that a particular video chip has been selected.

The user input of step356is facilitated by a GUI widget (e.g., GUI widget830ofFIG. 8A) of the plug-in window (e.g., plug-in window702ofFIG. 4). The GUI widget allows a user to modify a temporal level of resolution of at least one selected video chip. For example, a selected video chip has an original temporal level of resolution of one minute. In response to a user-software interaction, the temporal level of resolution of the selected video chip is changed to ten seconds. Embodiments of the present invention are not limited in this regard. The value of the temporal level of resolution can be increased and/or decreased via GUI widget. In this regard, the GUI widget includes, but is not limited to, a button for enabling and disabling temporal resolution operations of the feature analysis plug-in. The GUI widget may also comprise a means for selecting a new value (e.g., ten seconds) for the temporal level of scale (e.g., a text box and a drop down list).

After the reception of the user input in step356, the feature analysis plug-in performs operations for automatically creating a page of “temporally zoomed” video chips having the lower temporal resolution. Thereafter, in step358, the page of “temporally zoomed” video chips is displayed in the plug-in window. In a next step359, the “temporally zoomed” video chips are simultaneously or concurrently played.

A schematic illustration of an exemplary page of “temporally zoomed” video chips1802is provided inFIG. 18. As shown inFIG. 18, the selected video chip1704ofFIG. 17has been broken into a plurality of sub-segments1804-1818. The sub-segments1804-1818have the same temporal level of resolution (or duration) which is less than that of the selected video chip1704.

Referring again toFIG. 3E, the method300continues with step360. In step360, the computing device receives a user input for viewing images of all of the currently video chips at a user-specified spatial zoom level of scale. The user input is facilitated by a GUI widget (e.g., GUI widget826ofFIG. 8A) of the plug-in window (e.g., plug-in window702ofFIG. 7). The GUI widget is configured to facilitate the performance of manual-spatial scale operations by the feature analysis plug-in. The manual-spatial scale operations are operative to adjust the spatial zoom level of scale of images of all of the displayed video chips from a first spatial zoom level of scale to a second spatial zoom level of scale in response to a user-software interaction. The first spatial zoom level of scale is a default spatial zoom level of scale (e.g., 100%) or a previously user-selected spatial zoom level of scale (e.g., 50%). The second spatial zoom level of scale is a new user-selected spatial zoom level of scale (e.g., 75%). As such, the GUI widget may include, but is not limited to, a drop down list populated with a plurality of whole number percentage values.

After the reception of the user input in step360, the feature analysis plug-in performs operations for automatically and concurrently generating a plurality of “fixed spatially zoomed” video chips at the user-specified spatial zoom level of scale, as shown by step361. In a next step362, the feature analysis plug-in performs operations to create a page of “fixed spatially zoomed” video chips. Thereafter in step363, the page of “fixed spatially zoomed” video chips is displayed in the plug-in window. In a next step364ofFIG. 3F, the “fixed spatially zoomed” video chips are simultaneously or concurrently played.

A schematic illustration of an exemplary page of “fixed spatially zoomed” video chip1902is provided inFIG. 19. As shown inFIG. 19, all of the video chips1904have the same spatial zoom level of scale. As such, the smallest feature B1appears smaller than the larger features B2, B3, B4. Similarly, the largest feature B3appears larger than the smaller features B1, B2, B4. Embodiments of the present invention are not limited to the particularities ofFIG. 19.

Referring again toFIG. 3F, the method300continues with step365. In step365, the computing device receives a user input for viewing all of the currently displayed features (e.g., B1, B2, B3, B4ofFIG. 19) at a best-fit spatial zoom level of scale. The user input is facilitated by a GUI widget (e.g., GUI widget828ofFIG. 8A) of the plug-in window (e.g., plug-in window702ofFIG. 7). The GUI widget is configured to facilitate the viewing of each displayed feature at its best-fit spatial zoom level of scale or its pre-defined maximum spatial zoom level of scale. As such, the GUI widget may include, but is not limited to, a button for at least enabling auto-scale operations of the feature analysis plug-in and disabling the manual-spatial scale operations of the feature analysis plug-in.

In response to the reception of the user input in step365, the feature analysis plug-in performs operations to automatically and concurrently generate a plurality of “auto spatially zoomed” video chips comprising the currently displayed features at the best-fit spatial zoom level of scale, as shown by step366. In a next step367, the feature analysis plug-in performs operations to create a page of “auto spatially zoomed” video chips. Thereafter in step368, the page of “auto spatially zoomed” video chips is displayed in the plug-in window. The “auto spatially zoomed” video chips are then simultaneously or concurrently played, as shown by step369.

A schematic illustration of an exemplary page of “auto spatially zoomed” video chips2002is provided inFIG. 20. As shown inFIG. 20, each of the video chips2004has a different spatial zoom level of scale. As such, all of the features B1, B2, B3, B4appear to be of the same size regardless of their actual relative physical sizes. Embodiments of the present invention are not limited to the particularities ofFIG. 20.

Referring again toFIG. 3F, the method300continues with step370where the first page of first video chips is re-displayed in the plug-in window. In a next step372, the computing device receives a user input for cycling through video chip pages for a plurality of video streams. The user input is facilitated by a GUI widget (e.g., GUI widget824ofFIG. 8A) of the plug-in window. The GUI widget is configured to allow manual cycling and/or automatic cycling between video chip pages for a plurality of video streams. As such, the GUI widget may include, but is not limited to, a check box for enabling and disabling automatic video stream cycling operations of the feature analysis plug-in (e.g., check box2106ofFIG. 21), a slider for setting the rate at which the video streams automatically cycle (e.g., slider2104ofFIG. 21), and/or a button for manually commanding when to change the video stream (e.g., button2102ofFIG. 21).

In response to the reception of the user input in step372, the feature analysis plug-in performs operations for cycling through the pages of one or more video streams, as shown by step374. A schematic illustration of an exemplary video stream cycling process performed in step374is provided inFIG. 22. As shown inFIG. 22, a first iteration of the video stream cycling process begins with the display of a first page of video chips2202generated using a first video stream. Upon the expiration of a pre-defined amount of time or in response to a first user input, a second page of video chips2204is displayed in the plug-in window. The second page of video chips was generated using a second video stream different from the first video stream. The first iteration of the video stream cycling process ends with the display of a third page of video chips2206in response to a second user input or upon the expiration of the pre-defined amount of time. The third page of video chips2206was generated using a third video stream that is different than the first video stream and/or the second video stream. Thereafter, a second iteration of the video stream cycling process can begin such that the pages of video chips2202,2204,2206are cycled through again. Embodiments of the present invention are not limited to the particularities ofFIG. 22.

Referring again toFIG. 3F, the method300continues with step375. In step375, the computing device receives a user input for “checking” or “flagging” a video chip of the displayed page of first video chips. A user may desire to mark a video chip which contains an occurrence of information contained in a video stream that is relevant to a particular application. The application can include, but is not limited to, a situational awareness application, a natural disaster application, an unmanned vehicle application, a forensic application, a law enforcement application, a medication application, and a military application. Step375can involve selecting a video chip. The video chip can be selected by moving a mouse cursor over the video chip and clicking a mouse button. In response to the click of the mouse button, a menu is presented to the user of the computing device. The menu includes a list of commands, such as a command for enabling “check/uncheck” operations of the feature analysis plug-in.

Schematic illustrations of exemplary selected video chips2304,2504and exemplary menus2306are provided inFIGS. 23 and 25. As shown inFIGS. 23 and 25, the selected video chip2304,2504is annotated with a relatively thick and distinctly colored border. Also, a selected command “Check/Uncheck” or “Flag/Unflag” of the menu2306is annotated by bolding the text thereof. Embodiments of the present invention are not limited in this regard. Any type of mark or annotation can be used to illustrate that a particular chip image has been selected and/or that a particular command of a menu has been selected.

In response to the reception of the user input in step375ofFIG. 3F, the feature analysis plug-in performs step376. In step376, the selected video chip is automatically marked with a pre-defined mark. A schematic illustration of a video chip2304marked with a check2404is provided inFIG. 24. A schematic illustration of a video chip2504marked with a flag2604is provided inFIG. 26. Embodiments of the present invention are not limited to the particularities ofFIGS. 24 and 26. Any type of mark or annotation can be employed to illustrate that a chip image has been checked or flagged.

After completion of step376, the method300continues with step377ofFIG. 3G. Referring now toFIG. 3G, step377involves receiving, by the computing device, a user input for “unchecking” or “unflagging” the marked video chip (e.g., video chip2304ofFIG. 24or video chip2504ofFIG. 26) of the displayed page of video chips. In response to the user input of step377, the mark (e.g., mark2404ofFIG. 24or mark2604ofFIG. 26) is automatically removed from the marked video chip (e.g., video chip2304ofFIG. 24or video chip2504ofFIG. 26), as shown by step378.

In a next step379, the computing device receives a user input for “flagging” all of the video chips which precede or succeed a selected one of the displayed video chips in a temporal order or a sorted order. The video chip is selected by moving a mouse cursor over the video chip and clicking a mouse button. In response to the click of the mouse button, a menu is presented to the user of the computing device. The menu includes a list of commands, such as a command for enabling “Flag/Unflag Backward” or “Flag/Unflag Forward” operations.

Schematic illustrations of exemplary selected video chips2704,2904and exemplary menus2306are provided inFIGS. 27 and 29. As shown inFIGS. 27 and 29, the selected video chip2704,2904is annotated with a relatively thick and distinctly colored border. Also, the selected command “Flag/Unflag Backward” or “Flag/Unflag Forward” of the menu2306is annotated by bolding the text thereof. Embodiments of the present invention are not limited in this regard. Any type of mark or annotation can be used to illustrate that a particular chip image has been selected and/or that a particular command of a menu has been selected.

Referring again toFIG. 3G, the method300continues with step380. In step380, the feature analysis plug-in performs operations to add a mark or annotation to the selected video chip (e.g., video chip2704ofFIG. 27or video chip2904ofFIG. 29) and to all of the video chips which precede or succeed the selected video chip in a temporal order or a sorted order (e.g., video chips2802-2808ofFIG. 28and chip images3002-3006ofFIG. 30). Upon completing step380, step381is performed where all of the “flagged” chip images are exported to a table or file. The exportation can be initiated by a user of the computing device using a GUI widget (e.g., GUI widget816or818ofFIG. 8A) of the plug-in window.

In a next step382, the computing device receives a user input for unflagging all of the “flagged” video chips. In response to the user input of step382, step383is performed where the marks or annotations are removed from the “flagged” video chips. Subsequently, step384is performed where the method ends or other processing is performed.