Patent Description:
At any time instant, countless events that happen in the real world are captured by sensors such as cameras, microphoens, LiDAR, mmWave and stored as massive sensor data resources. To effectively retrieve such recordings, whether in offline or online settings, scene captioning is an essential technology thanks to its ability to understand scenes and describe events in natural language. Furthermore, robots can interact with humans or other robots to decide next actions based on scene understanding through online scene captioning.

Scene captioing technology has been actively researched in the field of computer vision for video captioning. A deep recurrent neural network (RNN) has been applied to video captioning, where an RNN is trained to convert a sequence of image features extracted from a video clip to a sequence of words that describes the video content. Its goal is to generate a video description (caption) about objects and events in any video clip. Recently, a Transformer model is getting more popular than RNNs as a model for video captioning, since it improves the captioning performance.

Moreover, not only image features but also audio features have been extracted from video clips and utilized to improve the caption quality, where an attention-based multimodal fusion technique has been introduced to effectively fuse the image and audio features according to the video content [<CIT>].

Document <CIT> discloses a vision-based method for forecasting human actions and poses.

Document <CIT> discloses an imaging device including: an imaging element that take an image of a subject; and a plurality of light sources that radiate a light to the subject, wherein optical axes of the plurality of the light sources are inclined outward with respect to an optical axis of the imaging element.

Prior methods for video captioning are basically assumed to work in an offline manner, where each video clip is given before captioning, and therefore the system can access all frames of the video clip to generate the caption. However, such prior methods are not practical in real-time monitoring, surveillance systems, scene-understanding based interaction systems for car navigation and robots, in which it is essential not only to describe events accurately but also to produce captions as soon as possible to find and report the events quickly to take next actions. Thus, low-latency captioning for online systems is required to realize such functionality, where the system needs to decide an appropriate timing to generate a correct caption using only the limited number of frames the system has received so far. Additionally, such a low-latency captioning function allow robots to decide next actions as soon as possible to interact with humans and other robots.

The present invention is based on recognition that scene captioning is an essential technology to understand scenes and describe events in natural language. To apply it to real-time monitoring systems, the systems require not only to describe events accurately but also to produce the captions as soon as possible. Low-latency captioning is needed to realize such functionality, but this research area for online scene captioning has not been pursued yet. This paper proposes a novel approach to optimize each caption's output timing based on a trade-off between latency and caption quality. An audio-visual Transformer is trained to generate ground-truth captions using only a small portion of all scenes, and to mimic outputs of a pre-trained Transformer to which all the scenes are given. A CNN-based timing detector is also trained to detect a proper output timing, where the captions generated by the two Transformers become sufficiently close to each other. With the jointly trained Transformer and timing detector, a caption can be generated in the early stages of an event, as soon as an event happens or when it can be forecasted.

An object of some embodiments of the invention is to provide a system and a method for end-to-end scene captioning capable for online/offline surveillance system and scene-aware interaction to understand the events using natural language as soon the system recognizes the events as possible.

The above problems are solved by the subject-matter according to the independent claims. This disclosure includes a low-latency scene captioning system, which is trained to optimize the output timing for each caption based on a trade-off between latency and caption quality. The present disclosure may train a low-latency caption generator according to the following strategy: (<NUM>) generate groundtruth captions using a low-latency caption generator that only sees a small portion of all scenes acquired by sensors or a small portion of all signals; (<NUM>) mimic outputs of a pre-trained caption generator, where the outputs are generated using the entire scene; and (<NUM>) train a timing detector that finds the best timing to output a caption, such that the caption ultimately generated by the low-latency caption generator becomes sufficiently close to its groundtruth caption or the caption generated by the pre-trained caption generator using the entire scene. The low-latency caption generator based on (<NUM>) and (<NUM>) and the timing detector of (<NUM>) are jointly trained.

The jointly trained low-latency caption generator and timing detector can generate captions in an eraly stage of scene acquired by sensors, as soon as an event happens. Additionally, this framework can be applied to forecast future events in low-latency captioning. Furthermore, by combining multimodal sensing information, an event can be recognized at an earlier timing triggered by the earliest cue in one of modalities without waiting for other cues in other modalities. In particular, an audio-visual Transformer built as a low-latency caption generator according to embodiments of the present invention can be used to generate captions earlier than a visual cue's timing based on the timing of an audio cue. Such a low-latency video captioning system using multimodal sensing information can contribute not only to retrieving events quickly but also to respond scenes earlier.

Some embodiments are based on recognition that experiments with the ActivityNet Captions dataset show that a system based on the present invention achieves <NUM>% of the caption quality of the upper bound given by the pre-trained Transformer using the entire video clips, using only <NUM>% of frames from the beginning.

While the above-identified drawings set forth presently disclosed embodiments, other embodiments are also contemplated, as noted in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation.

The following description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the scope of the subject matter disclosed as set forth in the appended claims.

Specific details are given in the following description to provide a thorough understanding of the embodiments. However, understood by one of ordinary skill in the art can be that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the subject matter disclosed may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Further, like reference numbers and designations in the various drawings indicated like elements.

Also, individual embodiments may be described as a process, which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in a figure. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the function's termination can correspond to a return of the function to the calling function or the main function.

When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium.

Modules and networks exemplified in the present disclosure may be computer programs, software or instruction codes, which can execute instructions using one or more processors. Modules and networks may be stored in one or more storage devices or otherwise stored into computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape, in which the computer readable media are accessible from the one or more processors to execute the instructions.

Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media may be RAM, ROM, EEPROM or flash memory, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both using one or more processors. Any such computer storage media may be part of the device or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media.

According to some embodiments of the present disclosure, a scene captioning system can provide low-latency scene captioning to understand scenes and describe events in natural language from real-time monitoring of scenes or online/offline scene streaming.

In some cases, the scene captioning system can be configured as a video captioning system, and the video captioning system may be referred to as a low-latency video captioning method/system.

In the following, a low-latency video captioning method/system is provided as an example descriptons of the low-latency scene captioning method/system.

The low-latency scene captioning method/system is applied to realtime or offline video streams, multimodal sensor signals are a stream of audio-video signal, a scene encoder is an audio-visual encoder, unimodal/multimoda sensor features are and visual features.

<FIG> is a schematic diagram illustrating a process of low-latency video captioning, according to embodiments of the present invention. Suppose a video stream <NUM> is given as X, in which an event has occurred between time s and time T, where a groundtruth caption <NUM> as Y may be "One of workers is hit by a bulldozer" for the event. The caption is typically a sentence that explains the event in natural language. Video captioning systems based on prior art generate such a caption for a video clip using the entire frames Xs:T of the video stream X, where Xs:T denotes a sequence of video frames Xs,Xs+<NUM>,. ,XT-<NUM>, XT, each of which corresponds to a video frame for each time index. Therefore, such prior video captioning systems are assumed to work only in an offline manner.

However, such prior methods are not practical in real-time monitoring or surveillance systems, in which it is essential not only to describe events accurately but also to produce captions as soon as possible. To this end, a low-latency video captioning module <NUM> needs to have a timing detector <NUM> that decides if the current time t is an appropriate timing to output a caption, using partial video frames <NUM>, i.e., Xs:t the video frames from the beginning time s to the current time t. Only if the timing detector <NUM> detects the appropriate timing, a low-latency caption generator <NUM> generates a caption <NUM> based on the partial video frames <NUM>. The beginning time s may be decided based on a timing the system generated the previous caption or a timing a certain change on pixel intensities in the video is detected (not shown in <FIG>).

One remaining problem is that there is no sufficient dataset to train the timing detector <NUM> and the low-latency caption generator <NUM>, where the dataset should be annotated with appropriately early timings and their captions for various videos. There exist datasets only for offline video captioning, i.e., the datasets were annotated with timings where the events have already finished in the video. In general, correcting a large amount of new dataset with such early timing annotations is very expensive.

To solve this problem, embodiments of the present invention train the timing detector <NUM> and the low-latency caption generator <NUM> using only datasets for offline video captioning, where the detector and the generator are optimized not only to describe events accurately but also to produce captions as soon as possible. Thus, low-latency video captioning is realized by detecting an appropriate timing to generate a correct caption using only the partial video frames <NUM> until the current time t. The system may generate a caption <NUM>, which is equal to the groundtruth <NUM> "One of workers is hit by a bulldozer" even though it uses only partial video frames <NUM>. On the other hand, prior methods that train the generator for offline video captioning may generate an incorrect caption <NUM> as "Workers are walking" for the partial video frames <NUM>, because the generator is trained to generate a caption about an event occurred in each video clip. In the example of low-latency video captioning <NUM>, the event "One of workers is hit by a bulldozer" has not yet occurred at time t. Therefore, it is difficult for systems based on prior art to generate the correct caption, while systems based on the present invention potentially generate the correct caption since the generator is trained to utilize some signs on future events in the partial video frames to generate correct captions.

<FIG> is a block diagram illustrating a low-latency video captioning system <NUM> including a low-latency caption generation training module <NUM> consisting of a traing data set <NUM>, a feature extractor <NUM>, a future frame eliminator <NUM>, a pre-trained caption generator <NUM>, a caption generation loss calculator <NUM>, a caption similarity checker <NUM> and a timng detection loss calculator <NUM>, a low-latency captioning module <NUM> consisting of a timing detector module <NUM> and a low latency caption generator <NUM>, interaction module <NUM> consisting of a caption understnading module <NUM> and a response generator <NUM> and, according to embodiments of the present invention.

The low-latency video captioning system <NUM> includes a human machine interface (HMI) <NUM> connectable with a keyboard <NUM> and a pointing device/medium <NUM>, one or more processor <NUM>, a memory <NUM>, a network interface controller <NUM> (NIC) connectable with a network <NUM> including local area networks and internet network, a display and/or speaker interface <NUM> connectable with a display and/or speaker device <NUM>, a machine interface <NUM> connectable with a machine actuator <NUM>, an multimodal sensor interface <NUM> such as an audio interface <NUM> and a visual interface <NUM> connectable with input devices including multimodal sensors <NUM> such as a microphone device <NUM> and a camera device <NUM> respectively, a printer interface <NUM> connectable with a printing device <NUM>. The memory <NUM> may be one or more memory units. The low-latency video captioning-based interaction system <NUM> can receive multimodal sensing data <NUM> via the network <NUM> connected to the NIC <NUM>. The storage device <NUM> includes a feature extractor <NUM>, a low-latency caption generation training module <NUM>, a low latency captioning module <NUM> and an interaction module <NUM>. The feature extractor <NUM> is configured as an audio-visual feature extractor <NUM> and the system is configured as a video captioning system.

For performing the low-latency captioning, instructions may be transmitted to the low-latency video captioning system <NUM> using the keyboard <NUM>, the pointing device/medium <NUM> or via the network <NUM> connected to other computers (not shown in the figure). The system <NUM> receives instructions via the HMI <NUM> and executes the instructions for performing low-latency video captioning using the processor <NUM> in connection with the memory <NUM> by loading the low-latency captioning module <NUM>.

The low-latency video captioning module <NUM> outputs a token sequence as captioning result for a given multimodal sensing feature sequence obtained by feature extractor <NUM>, and sends the token sequence to the display/speaker device <NUM> via the display/speaker interface <NUM>, the printer device <NUM> via the printer interface <NUM>, or other computers (not shown in the figure) via the network <NUM>. Each token in the token sequence may be a single word, a single letter, a single character, or a word piece in a text form.

<FIG> is a block diagram illustrating a training module <NUM> included in a low-latency video captioning system <NUM> including a low latency captioing module <NUM>, which exploits a pre-trained caption generator <NUM> to jointly optimize the low-latency caption generator <NUM> and the timing detector <NUM>, according to embodiments of the present invention.

Given a training dataset <NUM> including a set of video clips X and a set of groundtruth captions Y, wherein each video clip X in X is associated with its groundtruth caption Y in Y, first a feature extractor <NUM> converts the video clip X to a feature vector sequence Z for the video clip X. Then, a future frame eliminator <NUM> eliminates future frames from Z to simulate a situation of low-latency video captioning, where the future frames are not available. The future frame eliminator <NUM> takes the first K feature vectors of Z, removes the rest, and generates a future-eliminated feature vector sequence Z', where K is determined randomly from <NUM> ≤ K ≤ |Z| (|Z| denotes the length of Z).

The low-latency caption generator <NUM> includes an encoder <NUM>, a decoder <NUM>, and a search module <NUM>, wherein the encoder <NUM> encodes the future-eliminated feature vector sequence Z' to a hidden activation vector sequence H, the decoder <NUM> estimates a posterior probability distribution P(Y|H) over a random variable Y of captions for H, and the search module <NUM> searches for the best caption Ŷ such that <MAT>, where <IMG> indicates a set of all possible word sequences based on a predefined vocabulary <IMG>.

The timing detector <NUM> may include an encoder <NUM> and a timing detection module <NUM>, wherein the encoder <NUM> encodes the future-eliminated feature vector sequence Z' to a hidden activation vector sequence H'. The timing detector <NUM> may use the encoder <NUM> of the low-latency caption generator <NUM> instead of the encoder <NUM>. In this case, the timing detector <NUM> may not have the encoder <NUM>, and may receive the output of the encoder <NUM>, i.e., H' may be obtained as H' = H.

The timing detection module <NUM> estimates a detection probability distribution P(<IMG>|H') over a random variable <IMG> indicating whether the current timing is appropriate or not based on H'. The random variable D may take <NUM> or <NUM>, where <NUM> indicates that the current timing is inappropriate while <NUM> indicates that the current timing is appropriate. In low-latency video captioning <NUM>, the timing detector <NUM> may detect the appropriate timing when P(<IMG> = <NUM>|H') > F, where F is a pre-determined threshold such that <NUM> ≤ F < <NUM>.

To train the low-latency caption generator <NUM>, a caption generation loss calculator <NUM> may compute a loss value based on the posterior probability distribution P(Y|H) and the groundtruth caption Y. The loss value may be computed as a cross-entropy loss; <MAT>.

If the encoder <NUM> and the decoder <NUM> are designed as differentiable functions such as neural networks, the parameters of the neural networks can be trained to minimize the cross-entropy loss using the back-propagation algorithm. Minimizing the cross-entropy loss means letting the low-latency caption generator <NUM> generate appropriate captions as close as possible to the groundtruth captions. The system is configured as a scene captioning system, the encoder <NUM> is configured as an audio-visual feature encoder <NUM>.

To train the timing detector <NUM>, it needs a supervision signal on whether the current timing is appropriate or not. However, such kinds of signals are not included in the training dataset <NUM>. Some embodiments of the present invention may have a caption similarity checker <NUM> to determine the appropriate timing based on the caption similarity between the generated caption Ŷ and the groundtruth caption Y. This is based on an idea that the appropriate timing is a timing when the low-latency caption generator can generate a caption Y for the future-eliminated feature vector sequence Z', which is sufficiently close to the groundtruth caption Y. The caption similarity may be computed based on any sequence similarity measures such as Word accuracy, BLEU score, and METEOR score defined between the word sequences of the two captions.

The supervision signal d may be computed as <MAT> where Sim(Ŷ,Y) is a similarity measure between Y and Y, and S is a predefined threshold such that <NUM> ≤ S < <NUM>.

Based on the supervison d and the detection probability distribution P(<IMG>|H') estimated by the timing detection module <NUM>, a binary cross-entropy loss can be computed as <MAT>.

As well as the low-latency caption generator <NUM>, the timing detector <NUM> can be trained using the back-propagation algorithm if the encoder <NUM> and the timing detection module <NUM> are designed as differentiable functions such as neural networks.

In some embodiments of the present invention, the training procedure <NUM> for a low-latency video captioning system <NUM> may also include a pre-trained caption generator <NUM> to improve the performance of the low-latency video captioning system <NUM>. The pre-trained caption generator <NUM> may include an encoder <NUM>, a decoder <NUM>, and a search module <NUM>, which have already been trained using the training dataset <NUM> to minimize the loss function <MAT> where H" may be generated by the encoder <NUM> from the feature vector sequence Z, which are not eliminated. The generated caption Y' obtained as <MAT> is expected to be closer to the groundtruth caption Y than the low-latency caption Ŷ because the pre-trained caption generator <NUM> can use sufficient video frames (without future frame elimination) to generate captions unlike the low-latency caption generator <NUM>.

Similary, the estimated posterior probability distribution P(Y|H") is expected to be better than P(Y|H) to generate correct captions.

With the pre-trained caption generator <NUM>, the caption generation loss calculator <NUM> may use a Kullback-Leibler (KL) divergence loss as <MAT> where the loss value <IMG> is computed as a KL divergence that indicates the similarity between two probability distributions. Minimizing the KL divergence means training the low-latency caption generator <NUM> to mimic the pre-trained caption generator <NUM>, where the low-latency caption generator <NUM> can use only the future-eliminated feature vector sequence Z' while the pre-trained caption generator <NUM> can use the entire feature vector sequence Z. More specifically, the training is performed to let the low-latency caption generator <NUM> estimate the posterior probability distribution P(Y|H) as close as possible to the posterior probability distribution P(Y|H") estimated by the pre-trained caption generator <NUM> using the entire video frames. This framework potentially avoids overfitting the parameters of the low-latency caption generator <NUM> to the groundtruth caption Y.

A linear combination of the cross-entropy loss LCE and the KL divergence loss LKL may also be used to train the low-latency caption generator <NUM> as <MAT> where λ is a pre-defined scaling factor to balance the two losses.

With the pre-trained caption generator <NUM>, the caption similarity checker <NUM> may compute the supervision signal d as <MAT>.

This mechanism encourages the timing detector <NUM> to detect the timing when the low-latency caption generator <NUM> can generate a caption not only close to the groundtruth caption Y but also to the caption Y' generated by the pre-trained caption generator <NUM> for the entire feature vector sequence Z. This also avoids the supervison for the timing detector <NUM> relys only on the similarity to the groundtruch caption Y, which hopefully improves the robustness of the timing detector <NUM>.

<FIG> is a schematic diagram illustrating an audio-visual Transformer <NUM> built as a low-latency caption generator combined with a timing detector for low-latency video captiong, according to embodiments of the present invention. The audio-visual Transformer <NUM> for low-latency video captioning includes a feature extractor <NUM>, an encoder <NUM>, a decoder <NUM>, and a timing detection module <NUM>.

Given a video stream <NUM>, the feature extractor <NUM> extracts VGGish features <NUM> and I3D features <NUM> from the audio signal <NUM> and the visual signal <NUM> from the audio and visual tracks of the video stream <NUM>, respectively, where the audio signal <NUM> may be an audio waveform and the visual signal <NUM> may be a sequence of images. The frame rate for feature extraction may be different on each track. The encoder <NUM> encodes the VGGish <NUM> and I3D <NUM> feature vectors, where the sequences of audio and visual features from a starting point to the current time are fed to the encoder <NUM> and converted to hidden activation vector sequences through self-attention layers <NUM>, bi-modal attention layers <NUM>, and feed-forward layers <NUM>. Typically, this encoder block <NUM> is repeated N times, e.g., N = <NUM> or greater. The final hidden activation vector sequences are obtained via the N-th encoder block.

Let XA and XV be audio signal <NUM> and visual signal <NUM>. First, the feature extractor <NUM> is applied to the input signals as <MAT> to obtain feature vector sequences corresponding to the VGGish <NUM> and I3D <NUM> features, respectively. Each encoder block <NUM> computes hidden vector sequences as <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> where MHA() and FFN() denote a multi-head attention and a feed-forward network, respectively.

MHA() takes three matrices query Q, key K, and value V, each of which has the length times feature dimensions as <MAT>, K ∈ <MAT>, and returns a matrix as <MAT> <MAT> <MAT> where <MAT> and <MAT> are parameter matrices of each multi-head attention layer, h is the number of heads, and dmodel is the model size. Concat() concatenates matrices in the feature dimension.

The feed-forward network FFN() is computed as FFN(x) = max(<NUM>,xW<NUM> + b<NUM>)W<NUM> + b<NUM>, where <MAT> and <MAT> are parameter matrices and b<NUM> ∈ <MAT> and <MAT> are parameter vectors of each feed-forward layer.

The self-attention layer <NUM> extracts temporal dependency within each modality, where the arguments for MHA() are all same, i.e., An-<NUM> or Vn-<NUM>. The bi-modal attention layers <NUM> further extract cross-modal dependency between audio and visual features, where they take the key and the value from the other modality. After that, the feed-forward layers <NUM> are applied in a point-wise manner. The encoded representations for audio and visual features are obtained as AN and VN.

The timing-detection module <NUM> receives the encoded hidden vector sequences based on the audio-visual information available at the moment. The role of the timing-detection module <NUM> is to estimate whether the system should generate a caption or not for the given encoded features. The timing-detection module <NUM> first processes the encoded vector sequence from each modality with stacked 1D-convolution layers <NUM> as <MAT>.

Each time-convoluted sequences are then summarized into a single vector through pooling <NUM> and concatenation <NUM> operations.

Feed-forward layer FFN() <NUM> and sigmoid function σ() <NUM> convert the summary vector to the probability of <IMG>, where <IMG> ∈ {<NUM>, <NUM>} is a random variable indicating whether a relevant caption can be generated or not. <MAT> <MAT>.

Once the timing-detection module <NUM> provides a higher probability than a threshold, e.g., P(d = <NUM>|XA,XV) > <NUM>, the decoder generates a caption based on the encoded hidden vector sequences (AN, VN).

The decoder <NUM> iteratively predicts the next word from a starting word (<sos>). At each iteration step, the decoder receives the previously generated words <NUM>, and estimate the posterior probability distribution of the next word <NUM> by applying word embedding <NUM>, M decoder blocks <NUM>, a linear layer <NUM>, and a softmax operation <NUM>.

Let <MAT> be partial caption < sos >, y<NUM>,. , yi after i iterations. Each decoder block <NUM> has self-attention <NUM>, bi-modal source attention <NUM>, concatenation <NUM>, and feed-forward layer <NUM> as <MAT> <MAT> <MAT> <MAT> <MAT>.

The self-attention layer <NUM> converts the word embedding vectors to high-level representations considering their temporal dependency. The bi-modal source attention layers <NUM> update the word representations based on the relevance to the encoded multi-modal hidden vector sequences. The feed-forward layer <NUM> is then applied to the outputs of the bi-modal attention layers after concatenation <NUM>. Finally, the linear transform <NUM> and the softmax operation <NUM> are applied to the output of the M-th decoder block to obtain the posterior probability distribution of the next word <NUM> as <MAT> <MAT> where <IMG> denotes the vocabulary.

After picking the one-best word Ŷi+<NUM> the partial caption is extended by adding the selected word to the previous partial caption as Ŷi+ <NUM> = Yi,Ŷi+<NUM>. This is a greedy search process that ends if Ŷi+<NUM> =< eos >, which represents an end word. The posterior probability distribution of caption Y can be computed as <MAT> where H = (AN, VN).

It is also possible to pick multiple words with highest probabilities and consider multiple candidates of captions according to the beam search technique in the search module <NUM>.

The multi-modal encoder, the timing detector, and the caption decoder are jointly trained, so that the model achieves a caption quality comparable to that for a complete video, even if the given video is shorter than the original one by truncating the later part.

The training process for model θ = (θc, θD) repeats the following steps: <MAT> <MAT>.

<FIG> is an evaluation result obtained by performing a video captiong bench marks, according to embodiments of the present invention. The proposed low-latency caption generation was tested using the ActivityNet Captions dataset Krishna et al. (<NUM>), which consists of <NUM> caption sentences associated with temporal localization information based on <NUM> YouTube videos. The dataset is split into <NUM>%, <NUM>%, and <NUM>% for training, validation, and testing. The validation set was split into two subsets on which the performance was reported. The average duration of a video clip is <NUM>, <NUM>, and <NUM> seconds for the training set and the validation subsets <NUM> and <NUM>, respectively. VGGish and I3D features were used. The VGGish features were configured to form a <NUM>-dimensional vector sequence for the audio track of each video, where each audio frame corresponds to a <NUM> segment without overlap. The I3D features were configured to form a <NUM>-dimensional vector sequence for the video track, where each visual frame corresponds to a <NUM> segment without overlap.

A multi-modal Transformer was first trained with entire video clips and their ground-truth captions. This model was used as a baseline and teacher model. N = <NUM> was used for encoder blocks and M = <NUM> for decoder blocks, and the number of attention heads was <NUM>. The vocabulary size was <NUM>,<NUM>, and the dimension of word embedding vectors was <NUM>.

The proposed model for online captioning was trained with incomplete video clips according to the steps in the invention. In the training process, α = β = γ = <NUM>/<NUM> was used for the loss function. The dimensions of hidden activations in audio and visual attention layers were <NUM> and <NUM>, respectively. The timing detector had <NUM> stacked 1D-convolution layers, with a ReLU non-linearity in between. The performance was measured by BLEU3, BLEU4, and METEOR scores.

The latency ratio indicates the ratio of the video duration used for captioning to the duration of the original video clip. With the baseline model, the latency ratio is always <NUM>, which means all frames are used to generate captions.

<FIG> compares captioning methods in METEOR scores on validation subset <NUM>. The model selected for evaluation was trained with S = <NUM> (empirically decided) and had the best METEOR score on validation subset <NUM>. The latency with the detection threshold F was controlled. As shown in <FIG>, the proposed method at a <NUM>% latency achieves <NUM> METEOR score with only a small degradation, which corresponds to <NUM>% of the baseline score <NUM>. It also achieves <NUM> METEOR score at a <NUM>% latency, which corresponds to <NUM>% of the baseline. A naive method was tested which takes video frames from the beginning with a fixed ratio to the original video length and runs the baseline captioning on the truncated video clip. The results show that the proposed approach clearly outperforms the naive method at an equivalent latency.

The table also includes the results using a unimodal Transformer that receives only the visual feature. The results show that the proposed method works for the visual feature only, but the performance is degraded due to the lack of the audio feature. This result indicates that the audio feature is essential even in the proposed low-latency method.

Claim 1:
A computer-executable training method for training a low-latency scene caption generator (<NUM>) and a timing detector (<NUM>), wherein the low-latency scene caption generator (<NUM>) comprises an audio-visual sensor feature encoder (<NUM>) and a scene caption decoder (<NUM>), the method comprising:
providing a training dataset (<NUM>) that includes a set of audio-visual sensor signals and a set of groundtruth scene captions, wherein the set of audio-visual sensor signals is a set of video clips;
converting a video clip in the set of video clips to a feature vector sequence for the video clip using a feature extractor;
eliminating future frames from the feature vector sequence using a future frame eliminator, wherein the future frame eliminator takes the first K feature vectors from the feature vector sequence and removes other feature vectors to generate a future-eliminated feature vector sequence, wherein K is determined randomly from <NUM> ≤ K ≤ |Z| and |Z| denotes a length of the feature vector sequence;
encoding the future-eliminated feature vector sequence to a hidden activation vector sequence;
training the low-latency scene caption generator (<NUM>) by computing a loss value;
computing a loss value based on a posterior probability distribution and the groundtruth captions;
training the timing detector (<NUM>) by computing a supervision signal based on a scene caption similarity between a scene caption for the future-eliminated feature vector sequence and the groundtruth caption;
wherein the timing detector (<NUM>) is trained to detect an output timing to generate a caption as soon as events happen or when the events are forecasted.