Patent ID: 12211278

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the disclosed invention, its various features and the advantageous details thereof, are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted to not unnecessarily obscure what is being disclosed. Examples may be provided and when so provided are intended merely to facilitate an understanding of the ways in which the invention may be practiced and to further enable those of skill in the art to practice its various embodiments. Accordingly, examples should not be construed as limiting the scope of what is disclosed and otherwise claimed.

The embodiments herein provide a deep learning based video information extraction system that is based on ground truth captions for a specific target domain. Video information extraction refers to extracting and assigning types to the terms, entities, relations, and events of interest in the video. This capability can play an important role in video analytics for surveillance cameras in various target domains. For example, it is relevant in homeland security applications such as border protection, as well as related law enforcement, intelligence, security and defense applications. For video entity, relation, and event extraction three embodiments are described. The first embodiment is a pre-IE (pre-information extraction) approach that leverages information extraction capabilities to extract terms, entities, relations, and events from ground truth captions, followed by training a video captioning framework with video features as well as derivatives of the extracted information. The second embodiment is a joint embedding approach that embeds both video features and term/entity/relation/event vectors in a common space. The third embodiment, a post-IE approach, entails training a video captioning framework directly with video features and the ground truth captions such that a descriptive sentence is generated as output. In a post-processing step, information extraction is applied on the output to obtain terms, entities, relations, and events. Referring now to the drawings, and more particularly toFIGS.1A through21, where similar reference characters denote corresponding features consistently throughout, there are shown exemplary embodiments. In the drawings, the size and relative sizes of components, layers, and regions, etc. may be exaggerated for clarity.

For video entity, relation and event extraction, three embodiments are described herein. The first is a pre-IE (pre-information extraction) approach that leverages information extraction capabilities to extract terms, entities, relations, and events from ground truth captions, followed by training a video captioning framework with the extracted information. The second example is a joint embedding approach that embeds both video features and term/entity/relation/event vectors in a common space. The third example entails training a video captioning framework directly with the ground truth captions such that a descriptive sentence is generated as output. In a post-processing step, information extraction is applied on the output to obtain terms, entities, relations, and events. This example may be referred to as a post-IE approach. These examples are especially useful in a situation where (a) a small amount of caption data is available as a byproduct of workflows in the target domain, and (b) object class label and localization datasets necessary for object detection for the target domain are not available and expensive to obtain.

The various modules and corresponding components described herein and/or illustrated in the figures may be embodied as hardware-enabled modules and may be a plurality of overlapping or independent electronic circuits, devices, and discrete elements packaged onto a circuit board to provide data and signal processing functionality within a computer. An example might be a comparator, inverter, or flip-flop, which could include a plurality of transistors and other supporting devices and circuit elements. The modules that include electronic circuits process computer logic instructions capable of providing digital and/or analog signals for performing various functions as described herein. The various functions can further be embodied and physically saved as any of data structures, data paths, data objects, data object models, object files, database components. For example, the data objects could include a digital packet of structured data. Example data structures may include any of an array, tuple, map, union, variant, set, graph, tree, node, and an object, which may be stored and retrieved by computer memory and may be managed by processors, compilers, and other computer hardware components. The data paths can be part of a computer CPU or GPU that performs operations and calculations as instructed by the computer logic instructions. The data paths could include digital electronic circuits, multipliers, registers, and buses capable of performing data processing operations and arithmetic operations (e.g., Add, Subtract, etc.), bitwise logical operations (AND, OR, XOR, etc.), bit shift operations (e.g., arithmetic, logical, rotate, etc.), complex operations (e.g., using single clock calculations, sequential calculations, iterative calculations, etc.). The data objects may be physical locations in computer memory and can be a variable, a data structure, or a function. Some examples of the modules include relational databases (e.g., such as Oracle® relational databases), and the data objects can be a table or column, for example. Other examples include specialized objects, distributed objects, object-oriented programming objects, and semantic web objects. The data object models can be an application programming interface for creating HyperText Markup Language (HTML) and Extensible Markup Language (XML) electronic documents. The models can be any of a tree, graph, container, list, map, queue, set, stack, and variations thereof, according to some examples. The data object files can be created by compilers and assemblers and contain generated binary code and data for a source file. The database components can include any of tables, indexes, views, stored procedures, and triggers.

In other examples, the modules described herein may be programmable modules and may be configured as a computer program product that includes a pre-configured set of instructions, which when performed, can result in actions as stated in conjunction with the methods and techniques described herein. In an example, the pre-configured set of instructions may be stored on a tangible non-transitory computer readable medium or a program storage device. In an example, the tangible non-transitory computer readable medium may be configured to include the set of instructions, which when performed by a device, can cause the device to perform acts similar to the ones described here. Embodiments herein may also include tangible and/or non-transitory computer-readable storage media for carrying or having computer executable instructions or data structures stored thereon.

Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform particular tasks or implement particular abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps. The embodiments herein can include both hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc.

FIG.1Ais a block diagram illustrating a video information extraction system10comprising a memory15to store a video20. In some examples, the memory15may be Random Access Memory, Read-Only Memory, a cache memory, hard drive storage, flash memory, the cloud, or other type of storage mechanism. Furthermore, the memory15may be part of a server computer system or electronic device (not shown) that is remotely linked to the video information extraction system10through any of wired and wireless communication, according to an example. The video20may be stored as any suitable type of video files such as MPG, MP2, MPEG, MPE, MPV, AVI, WMV, MOV, MKV, VOB, or FLV, among other types of video files and may be presented as video frames. Furthermore, the video20may contained embedded images, text, or other graphics.

The video information extraction system10comprises a textual information extraction module25to obtain information about terms, entities, relations, and events30from ground truth captions35corresponding to the video20. The video information extraction system10further comprises a video captioning module40. In an example, the terms, entities, relations, and events30may be a unique object or set of objects in the video20; for example, a specific person(s), place(s), or item(s) in the video20. The textual information extraction module25is configured to automatically extract structured information in the form of terms, entities, relations and events from the unstructured and/or semi-structured machine-readable ground truth captions35corresponding to the video20through pre-programmed instructions or real-time instructions provided by a user. In an example, the textual information extraction module25is a deep learning based textual information extraction module that is executed to obtain the information about the terms, entities, relations, and events30from each caption in the ground truth captions35corresponding to the video20. The set of extracted elements are then used as the target to train the video captioning module40to extract information from the video20. In an example, the ground truth captions35may be textual captions that are pre-programmed in the textual information extraction module25and assigned or linked to the video20based on deep learning techniques, or they may be assigned or linked to the video20based on user input, which may occur in real time. The information from the textual captions; i.e., the ground truth captions35, may be extracted using a nominal or named entity recognition (NER) information extraction technique, for example; although other information extraction techniques may be used. For example, the information extraction technique may locate and extract named entity mentions in the ground truth captions35of video20into pre-defined categories such as person names, organizations, locations, time expressions, quantities, values, percentages, qualifiers, titles, etc. Some example NER techniques which could be used include, without limitations, GATE™, OpenNLP™, and SpaCy™. Moreover, the information extraction technique may be a linguistic grammar-based technique or a mathematical statistical model such as machine learning, which may be trained using training data.

The video captioning module40comprises an encoder45(i) to receive the information about the terms, entities, relations, and events30from the textual information extraction module25, and (ii) to extract video features50from the video20. In an example, the encoder45may comprise a hardware device such as an electronic circuit, integrated circuit chip, or transducer or may comprise a software program to combine the information about the terms, entities, relations, and events30with video features50. For example, the encoder45may comprise a video encoder. In an example, the video features50may be portions of the video20and/or descriptions of the video20that are related to the terms, entities, relations, and events30corresponding to the video20. The video information extraction system10further comprises a decoder55to generate a text caption60based on the extracted video features50. In an example, the decoder55may comprise a hardware device such as an electronic circuit, integrated circuit chip, or transducer or may comprise a software program to generate the text caption60based on the extracted video features50. For example, the text caption60may be a short summary, synopsis, or title associated with the extracted video features50comprised of terms, entities, relations and/or events. In an example, the decoder55may comprise a video decoder.

FIG.1B, with reference toFIG.1A, is a block diagram illustrating an example in which the information about terms, entities, relations, and events30from a ground truth caption35is extracted using an information extraction (IE) technique. In this approach, an information extraction capability is first employed to extract terms, entities, relations, and events30from the ground truth captions35corresponding to the video20. The extracted information and the video20are input for video captioning to extract the video features50and to generate the text caption60comprised of terms, entities, relations or events composed of type:value pairs.

FIG.2, with reference toFIGS.1A and1B, is a block diagram illustrating that the video information extraction system10may further comprise an object detection module65(i) to obtain regional features67corresponding to objects70in the video20, and (ii) to input the regional features67into the decoder55by way of the encoder45of the video captioning module40to generate the text caption60. In an example, the object detection module65may be a pre-trained object detection module such as a one-stage YOLO™ RetinaNet™ or two stage Faster R-CNN, Mask R-CNN, etc., can be used to obtain the regional features67corresponding to the objects70in the video20. According to an example, the regional features67comprise a partial region of the video20, and may be defined by a specified pre-programmed region of the video20or may be identified by a user in real time. The video20may be decomposed into smaller parts to identify the regional features67. For example, the video20may be decomposed based on the duration corresponding to the presence of the specific objects70in the video20or the location of the objects70in the video20, etc. Furthermore, other decomposition techniques may be utilized to parse the video20to identify regional features67corresponding to the objects70in the video20. Moreover, the objects70may correspond to any person(s), place(s), or item(s) or other type of identifier in the video20.

FIG.3, with reference toFIGS.1A through2, is a block diagram illustrating that the encoder45may receive the information about the terms, entities, relations, and events30in the form of vectors75of higher-order co-occurrence information. In an example, the vectors75may be a graphical representation of the terms, entities, relations, and events30from ground truth captions35corresponding to the video features50. As described above, in the video captioning module40, in an example the information from extracted elements such as terms, entities, relations and/or events composed of type:value pairs is used as input to the decoder55. This is achieved using a vector approach leveraging higher-order relations between items. Such higher-order relations reflect semantic regularities well, especially when the event space of the items is sparse, and as such improve on existing approaches that leverage first-order co-occurrence relations such as GloVe, Word2Vec, etc. The encoder45may perform entity resolution on the terms, entities, relations, and events30. For example, entity resolution may involve the processes of deduplication, record linkage, and canonicalization. In an example, the Dedupe™ library may be utilized for the entity resolution process.

FIG.4, with reference toFIGS.1A through3, is a block diagram illustrating that the encoder45may execute a convolutional neural network (CNN)80to extract video features from the video20. The CNN80comprises a deep learning algorithm, which receives an input (such as information about the video regions associated with terms, entities, relations, and events30from the textual information extraction module25), and assigns a rank to various aspects/objects70in the video20in order to differentiate one object from another. Some example CNNs80, which may be executed by the encoder45, include Caffe™, Deeplearning4j™, Dlib™, Microsoft Cognitive Toolkit™, TensorFlow™, Theano™, and Torch™.

FIG.5, with reference toFIGS.1A through4, is a block diagram illustrating that the encoder45may transform input from the video20in the form of video features50through either a pre-trained 2D convolutional neural network85or a pre-trained 3D convolutional neural network90.FIG.6, with reference toFIGS.1A through5, is a block diagram illustrating that the extracted video features50may comprise spatio-temporal features95(i.e., occupying space and time features).FIG.7, with reference toFIGS.1A through6, is a block diagram illustrating that the spatio-temporal features95may be derived from the video20based on a transfer learning process98. In this regard, the video20is converted to the spatio-temporal features95through either a pre-trained 2D convolutional neural network (CNN)85such as a residual network (ResNet) or VGG, etc., or a pre-trained 3D CNNs90such as C3D or I3D, etc. based on the transfer learning process98. These spatio-temporal features95and object level regional features are attended through attention mechanisms during the caption generation process by the decoder55. Here, the 3D CNNs90necessary for modeling temporal features95may not be crucial for prediction of entities but may be useful in predicting relations, events/actions, etc., beyond entities. The transfer learning process98may permit a machine learning algorithm to improve learning capacities for the video20through a previous exposure to a different video and reusing the model developed for the previous video for extracting the video features50from the video20. Some example models used in the transfer learning process98may include Oxford™ VGG Model, Google™ Inception Model, and Microsoft™ ResNet Model, etc.

FIG.8, with reference toFIGS.1A through7, is a block diagram illustrating that the decoder55may execute a recurrent neural network (RNN)99. In some examples, the RNN99may be long short-term memory (LSTM), gated recurrent unit (GRU), bi-directional, or continuous-time networks, etc. to generate the text caption60. The RNN99may be a type of artificial neural network where connections between nodes form a directed graph along a temporal sequence. The RNN99utilizes its memory to process inputs to assist in the modeling of video features50for generating the text caption60.

FIG.9Ais a block diagram illustrating a video information extraction system100comprising a memory115to store a video120. In some examples, the memory115may be Random Access Memory, Read-Only Memory, a cache memory, hard drive storage, flash memory, the cloud, or other type of storage mechanism. Furthermore, the memory115may be part of a server computer system or electronic device (not shown) that is remotely linked to the video information extraction system100through any of wired and wireless communication, according to an example. The video120may be stored as any suitable type of video files such as MPG, MP2, MPEG, MPE, MPV, AVI, WMV, MOV, MKV, VOB, or FLV, among other types of video files and may be presented as video frames. Furthermore, the video120may contained embedded images, text, or other graphics.

The video information extraction system100further comprises a textual information extraction module125to obtain information about terms, entities, relations, and events130from ground truth captions135corresponding to the video120. The video information extraction system100further comprises a common embedding module152. In an example, the terms, entities, relations, and events130may be a unique object or set of objects in the video120; for example, a specific person(s), place(s), or item(s) in the video120. The textual information extraction module125is configured to automatically extract structured information in the form of terms, entities, relations and events from the unstructured and/or semi-structured machine-readable ground truth captions135in video120through pre-programmed instructions or real-time instructions provided by a user. In an example, the textual information extraction module125is a deep learning based textual information extraction module that is executed to obtain the information about the terms, entities, relations, and events130from each caption in the ground truth captions135corresponding to the video120. The information about the set of extracted elements is then used as the target to train the common embedding module152to link information from the video120. In an example, the ground truth captions135may be textual captions that are pre-programmed in the textual information extraction module125and assigned or linked to the video120based on deep learning techniques, or they may be assigned or linked to the video120based on user input, which may occur in real time. The information from the textual captions; i.e., from the ground truth captions135, may be extracted using a nominal or named entity recognition (NER) information extraction technique, for example; although other information extraction techniques may be used. For example, an information extraction technique may locate and classify named entity mentions in the ground truth captions135of the video120into pre-defined categories such as person names, organizations, locations, time expressions, quantities, values, percentages, qualifiers, titles, etc. Some example NER techniques which could be used include, without limitations, GATE™, OpenNLP™, and SpaCy™. Moreover, the information extraction technique may be a linguistic grammar-based technique or a mathematical statistical model such as machine learning, which may be trained using training data.

The video information extraction system100further comprises a first encoder145to receive the information about the terms, entities, relations, and events130from the textual information extraction module125. A second encoder147is provided to extract video features150from the video120. In an example, the first encoder145and the second encoder147may each comprise a hardware device such as an electronic circuit, integrated circuit chip, or transducer or may comprise a software program to combine the information about the terms, entities, relations, and events130with video features150. For example, the first encoder145and the second encoder147may each comprise an encoder that operates on textual and video data respectively. In an example, the video features150may be portions of the video120and/or descriptions of the video120that are related to the terms, entities, relations, and events130corresponding to objects in the video120.

The video information extraction system100further comprises a common embedding module152to encode the information about the terms, entities, relations, and events130and the extracted video features150into vectors154. In an example, the vectors154may include a graphical representation of the video features150. The video information extraction system100further comprises a decoder155to generate a text caption160based on the vectors154. In an example, the decoder155may comprise a hardware device such as an electronic circuit, integrated circuit chip, or transducer or may comprise a software program to generate the text caption160based on the vectors154. For example, the text caption160may be a short summary, synopsis, or title associated with the extracted video features150comprised of terms, entities, relations and/or events. In an example, the decoder155may comprise a video decoder.

FIG.9B, with reference toFIG.9A, illustrates an encoder-decoder framework with two different modalities of data; namely, videos and text, that are mapped to a common space for joint embedding. Without loss of generalization,FIG.9B. depicts the extraction of entities. The video features150of a video120and the vectors154of the terms, entities, relations, and events130are similarly obtained as in the pre-IE approach described above with reference toFIG.1B. Then, the video features150of the video120are encoded into vectors in a common space through respective encoders (e.g., first encoder145and second encoder147), and a vector in the common space is reconstructed into entities through the decoder155. The encoder-decoder system is trained in such a way that paired video and entities are closely located in the common space through respective encoders (e.g., first encoder145and second encoder147), while enabling reconstruction errors, cyclic errors, etc., to be minimized. Generative adversarial learning may be used to leverage unpaired data. During use of the resulting deep learning model, entities can be generated by a cascaded use of a video feature encoder followed by an entity decoder for a given input video120.

FIG.10, with reference toFIGS.9A and9B, is a block diagram illustrating that the video information extraction system100may further comprise an object detection module165(i) to obtain regional features167corresponding to objects170in the video120, and (ii) to input the regional features167into the second encoder147. In an example, the object detection module165may be a pre-trained object detection module such as a one-stage YOLO™, RetinaNet™ or two stage Faster R-CNN, Mask R-CNN, etc., can be used to obtain the regional features167corresponding to the objects170in the video120. According to an example, the regional features167comprise a partial region of the video120, and may be defined by a specified pre-programmed region of the video120or may be identified by a user in real time. The video120may be decomposed into smaller parts to identify the regional features167. For example, the video120may be decomposed based on the duration corresponding to the presence of the specific objects170in the video120or the location of the objects170in the video120, etc. Furthermore, other decomposition techniques may be utilized to parse the video120to identify regional features167corresponding to the objects170in the video120. Moreover, the objects170may correspond to any person(s), place(s), or item(s) or other type of identifier in the video120.

FIG.11, with reference toFIGS.9A through10, is a block diagram illustrating that the first encoder145may receive the information about the terms, entities, relations, and events130in the form of vectors175of higher-order co-occurrence information. The vectors175may be a graphical representation of the information about the terms, entities, relations, and events130. In the common embedding module152, information about extracted elements such as items composed of type:value pairs are mapped to vectors154and used as input to the decoder155. This mapping is achieved using an embedding vector approach leveraging higher-order relations between items in vectors175. Such higher-order relations reflect semantic regularities well, especially when the event space of the items is sparse, and as such improve on existing approaches that leverage first-order co-occurrence relations such as GloVe, Word2Vec, etc. The first encoder145may perform entity resolution on the terms, entities, relations, and events130. For example, entity resolution may involve the processes of deduplication, record linkage, and canonicalization. In an example, the Dedupe™ library may be utilized for the entity resolution process.

FIG.12, with reference toFIGS.9A through11, is a block diagram illustrating that the first encoder145, the second encoder147, and the decoder155may be trained such that paired video120and information about terms, entities, relations, and events130are proximately located in the common embedding module152. This configuration permits reduction in processing time and memory requirements for performing the training.FIG.13, with reference toFIGS.9A through12, is a block diagram illustrating that the common embedding module152may reduce reconstruction errors183of the vectors154. For example, the reconstruction errors183may be errors between the original data point and its low dimensional reconstruction, and may be used as an anomaly score to detect anomalies in the vectors154. Moreover, the reconstruction errors183may be a mismatch between the values of the vectors154.

FIG.14, with reference toFIGS.9A through13, is a block diagram illustrating that the common embedding module152may reduce cyclic errors184of the vectors154.FIG.15, with reference toFIGS.9A through14, is a block diagram illustrating that a generative adversarial learning process187may be applied to unpaired data188in the common embedding module152. For example, the generative adversarial learning process187may be used to generate new videos based on the unpaired data188such that the new videos appear to be authentic to human observers.FIG.16, with reference toFIGS.9A through15, is a block diagram illustrating that the video information extraction system100may further comprise a cascaded arrangement189a,189bof the second encoder147and the decoder155applied to the video120. In an example, the cascaded arrangement189a,189bmay provide for efficiencies in training the unpaired data188.

FIG.17Ais a block diagram illustrating a video information extraction system200comprising a memory215to store a video220. In some examples, the memory215may be Random Access Memory, Read-Only Memory, a cache memory, hard drive storage, flash memory, the cloud, or other type of storage mechanism. Furthermore, the memory15may be part of a server computer system or electronic device (not shown) that is remotely linked to the video information extraction system200through any of wired and wireless communication, according to an example. The video220may be stored as any suitable type of video files such as MPG, MP2, MPEG, MPE, MPV, AVI, WMV, MOV, MKV, VOB, or FLV, among other types of video files and may be presented as video frames. Furthermore, the video220may contained embedded images, text, or other graphics.

The video information extraction system200further comprises a video captioning module240comprising an encoder245to receive the video220and ground truth captions235corresponding to the video220. The video information extraction system200further comprises a decoder255to generate full sentence captions256from the video220based on the ground truth captions235. The video information extraction system200further comprises a textual information extraction module225to obtain terms, entities, relations, and events230from the full sentence captions256corresponding to the video220.

In an example, terms, entities, relations, and events230may be a unique object or set of objects in the video220; for example, a specific person(s), place(s), or item(s) in the video220. A textual information extraction module225is configured to automatically extract structured information from the unstructured and/or semi-structured machine-readable full sentence captions256corresponding to the video220through pre-programmed instructions or real-time instructions provided by a user. In an example, the textual information extraction module225is a deep learning based textual information extraction module that is executed to extract the terms, entities, relations, and events230from the full sentence captions256corresponding to the video220. In an example, the ground truth captions235may be full sentence captions that are assigned or linked to the video220based on deep learning techniques, or they may be assigned or linked to the video220based on user input, which may occur in real time. The terms, entities, relations and/or events230present in the full sentence captions256may be extracted using a nominal or named entity recognition (NER) information extraction technique, for example; although other extraction techniques may be used. For example, an information extraction technique may locate and classify named entity mentions in the full sentence captions256that correspond to objects in the video220into pre-defined categories such as person names, organizations, locations, time expressions, quantities, values, percentages, qualifiers, titles, etc. Some example NER techniques which could be used include, without limitations, GATE™, OpenNLP™, and SpaCy™. Moreover, the information extraction technique may be a linguistic grammar-based technique or a mathematical statistical model such as machine learning, which may be trained using training data.

In an example, the encoder245may comprise a hardware device such as an electronic circuit, integrated circuit chip, or transducer or may comprise a software program to encode the full sentence captions256. For example, the encoder245may comprise a video encoder. In an example, the decoder255may comprise a hardware device such as an electronic circuit, integrated circuit chip, or transducer or may comprise a software program to generate the full sentence captions256. For example, the full sentence captions256may be a full sentence description associated with objects and/or events in the video220. In an example, the decoder255may comprise a video decoder.

FIG.17B, with reference toFIG.17A, is a block diagram illustrating a post-information extraction (post-IE) technique. In this approach, the video captioning module240is trained with ground truth captions235to generate full sentence captions256and information extraction is applied to the generated full sentence captions256. The video captioning module240comprises a CNN encoder and a RNN decoder similar to the pre-IE approach except that it is trained with ground truth captions235to generate full sentence captions256describing the content of the video220(as opposed to generating terms, entities, relations, and events230). As noted, information extraction is applied to the generated full sentence captions256as a downstream task, and the resulting terms, entities, relations, and events230are output.

FIG.18, with reference toFIGS.17A and17B, is a block diagram illustrating that the video information extraction system200may further comprise an object detection module265(i) to obtain regional features267corresponding to objects270in the video220, and (ii) to input the regional features267into the encoder245of the video captioning module240. In an example, the object detection module265may be a pre-trained object detection module such as a one-stage YOLO™, RetinaNet™ or two stage Faster R-CNN, Mask R-CNN, etc., can be used to obtain the regional features267corresponding to objects in the video220. According to an example, the regional features267comprise a partial region of the video220, and may be defined by a specified pre-programmed region of the video220or may be identified by a user in real time. The video220may be decomposed into smaller parts to identify the regional features267. For example, the video220may be decomposed based on the duration corresponding to the presence of the specific objects in the video220or the location of the objects270in the video220, etc. Furthermore, other decomposition techniques may be utilized to parse the video220to identify regional features267corresponding to the objects270in the video220. Moreover, the objects270may correspond to any person(s), place(s), or item(s) or other type of identifier in the video220.

FIG.19, with reference toFIGS.17A through18, is a block diagram illustrating that the encoder245may execute a convolutional neural network (CNN)280, and wherein the decoder255may execute a recurrent neural network (RNN)299. The CNN280comprises a deep learning algorithm, which receives an input and assigns a rank to various aspects/objects in the video220in order to differentiate one object from another. Some example CNNs280, which may be executed by the encoder245, include Caffe™ Deeplearning4j™, Dlib™, Microsoft Cognitive Toolkit™, TensorFlow™, Theano™, and Torch™. In some examples, the RNN299may be long short-term memory (LSTM), gated recurrent unit (GRU), bi-directional, or continuous-time networks, etc. to generate the full sentence captions256. The RNN299may be a type of artificial neural network where connections between nodes from a directed graph along a temporal sequence. The RNN299utilizes its memory to process inputs to assist in generating the full sentence captions256.

FIG.20, with reference toFIGS.17A through19, is a block diagram illustrating that the encoder245may receive input from the ground truth captions235in the form of vectors275of higher-order co-occurrence information. The vectors275may be graphical representations of the regional features267, according to an example. In the video captioning module240, elements of the ground truth captions are mapped to vectors in a continuous vector space and used as input to the decoder255. This mapping is achieved using an approach leveraging higher-order relations between elements. Such higher-order relations reflect semantic regularities well, especially when the event space of the items is sparse, and as such improve on existing approaches that leverage first-order co-occurrence relations such as GloVe, Word2Vec, etc. The encoder245may perform entity resolution on the ground truth captions235. For example, entity resolution may involve the processes of deduplication, record linkage, and canonicalization. In an example, the Dedupe™ library may be utilized for the entity resolution process.

In an example, the embodiments herein can provide a computer program product configured to include a pre-configured set of instructions, which when performed, can result in actions as stated in conjunction with various figures herein. In an example, the pre-configured set of instructions can be stored on a tangible non-transitory computer readable medium. In an example, the tangible non-transitory computer readable medium can be configured to include the set of instructions, which when performed by a device, can cause the device to perform acts similar to the ones described here.

The embodiments herein may also include tangible and/or non-transitory computer-readable storage media for carrying or having computer-executable instructions or data structures stored thereon. Such non-transitory computer readable storage media can be any available media that can be accessed by a general purpose or special purpose computer, including the functional design of any special purpose processor as discussed above. By way of example, and not limitation, such non-transitory computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions, data structures, or processor chip design. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media.

Computer-executable instructions include, for example, instructions and data which cause a special purpose computer or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

The modules provided by the software-enabled embodiments herein may be implemented on an integrated circuit chip (not shown). The chip design is created in a graphical computer programming language, and stored in a computer storage medium (such as a disk, tape, physical hard drive, or virtual hard drive such as in a storage access network). If the designer does not fabricate chips or the photolithographic masks used to fabricate chips, the designer transmits the resulting design by physical means (e.g., by providing a copy of the storage medium storing the design) or electronically (e.g., through the Internet) to such entities, directly or indirectly. The stored design is then converted into the appropriate format (e.g., GDSII) for the fabrication of photolithographic masks, which typically include multiple copies of the chip design in question that are to be formed on a wafer. The photolithographic masks are utilized to define areas of the wafer (and/or the layers thereon) to be etched or otherwise processed.

The resulting integrated circuit chip may be distributed by the fabricator in raw wafer form (that is, as a single wafer that has multiple unpackaged chips), as a bare die, or in a packaged form. In the latter case the chip is mounted in a single chip package (such as a plastic carrier, with leads that are affixed to a motherboard or other higher level carrier) or in a multichip package (such as a ceramic carrier that has either or both surface interconnections or buried interconnections). In any case the chip is then integrated with other chips, discrete circuit elements, and/or other signal processing devices as part of either (a) an intermediate product, such as a motherboard, or (b) an end product. The end product may be any product that includes integrated circuit chips, ranging from toys and other low-end applications to advanced computer products having a display, a keyboard or other input device, and a central processor.

A data processing system suitable for storing and/or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

Input/output (I/O) devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.

A representative hardware environment for practicing the embodiments herein is depicted inFIG.21, with reference toFIGS.1A through20. This schematic drawing illustrates a hardware configuration of an information handling/computer system300in accordance with the embodiments herein. The system300comprises at least one processor or central processing unit (CPU)310. The CPUs310are interconnected via system bus312to various devices such as a random access memory (RAM)314, read-only memory (ROM)316, and an input/output (I/O) adapter318. The I/O adapter318can connect to peripheral devices, such as disk units311and tape drives313, or other program storage devices that are readable by the system. The system300can read the inventive instructions on the program storage devices and follow these instructions to execute the methodology of the embodiments herein. The system300further includes a user interface adapter319that connects a keyboard315, mouse317, speaker324, microphone322, and/or other user interface devices such as a touch screen device (not shown) to the bus312to gather user input. Additionally, a communication adapter320connects the bus312to a data processing network, and a display adapter321connects the bus312to a display device323which may be embodied as an output device such as a monitor, printer, or transmitter, for example. Further, a transceiver326, a signal comparator327, and a signal converter328may be connected with the bus312for processing, transmission, receipt, comparison, and conversion of electric or electronic signals.

The embodiments herein extend the concept of information extraction to videos by providing a deep learning based video information extraction system that is based on ground truth captions generated by humans for a specific target domain. This capability can play an important role in video analytics for surveillance cameras in various target domains. For example, it is relevant in homeland security applications such as border protection, as well as related law enforcement, intelligence, security and defense applications.

The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the appended claims.