WEAKLY SUPERVISED ACTION SEGMENTATION

According to one aspect, weakly-supervised action segmentation may include performing feature extraction to extract one or more features associated with a current frame of a video including a series of one or more actions, feeding one or more of the features to a recognition network to generate a predicted action score for the current frame of the video, feeding one or more of the features and the predicted action score to an action transition model to generate a potential subsequent action, feeding the potential subsequent action and the predicted action score to a hybrid segmentation model to generate a predicted sequence of actions from a first frame of the video to the current frame of the video, and segmenting or labeling one or more frames of the video based on the predicted sequence of actions from the first frame of the video to the current frame of the video.

BACKGROUND

Supervised and unsupervised learning models work in unique ways. The supervised learning approach in machine learning (ML) uses labeled datasets that train algorithms to classify data or predict outputs precisely. The model uses the labeled data, which is typically human labeled, to measure the relevance of different features to gradually improve model fit to the known outcome. With unsupervised learning, ML algorithms are used to examine and group unlabeled datasets. Such algorithms may uncover unknown patterns in data without human supervision. However, unsupervised learning sometimes produces erroneous results. On the other hand, supervised learning may be costly, time consuming, and may require human expertise for label validation. Generally, action segmentation aims to segment a temporally untrimmed video by time, and label each segmented part with a pre-defined action label.

BRIEF DESCRIPTION

According to one aspect, a system for weakly-supervised action segmentation may include a memory and a processor. The memory may store one or more instructions. The processor may execute one or more of the instructions stored on the memory to perform one or more acts, actions, or steps. For example, the processor may perform feature extraction to extract one or more features associated with a current frame of a video including a series of one or more actions, feeding one or more of the features to a recognition network to generate a predicted action score for the current frame of the video, feeding one or more of the features and the predicted action score to an action transition model to generate a potential subsequent action, and feeding the potential subsequent action and the predicted action score to a hybrid segmentation model to generate a predicted sequence of actions from a first frame of the video to the current frame of the video.

The hybrid segmentation model may generate the predicted sequence of actions based on a predicted action length for a predicted action associated with the predicted action score. The action transition model may generate the potential subsequent action based on a transcript of one or more known sequences of actions, one or more of the features, and the predicted action score. The hybrid segmentation model may generate a predicted sequence of action lengths corresponding to the predicted sequence of actions. The processor may detect one or more errors associated with the predicted sequence of action length and the predicted sequence of actions based on an error function. The hybrid segmentation model may be based on an unconstrained Viterbi algorithm. The action transition model may generate the potential subsequent action based on feeding one or more of the features to an anticipation network to generate an expected action for the current frame of the video and based on a comparison between the expected action for the current frame and the predicted action for the current frame. If the comparison is greater than a similarity threshold, generating the potential subsequent action based on a transcript of one or more known sequences of actions. If the comparison is less than a similarity threshold, generating the potential subsequent action based on exploring a universe of possible actions. The anticipation network of the action transition model may be trained during a training phase associated with a constrained version of the hybrid segmentation model.

According to one aspect, a computer-implemented method for weakly-supervised action segmentation may include performing feature extraction to extract one or more features associated with a current frame of a video including a series of one or more actions, feeding one or more of the features to a recognition network to generate a predicted action score for the current frame of the video, feeding one or more of the features and the predicted action score to an action transition model to generate a potential subsequent action, and feeding the potential subsequent action and the predicted action score to a hybrid segmentation model to generate a predicted sequence of actions from a first frame of the video to the current frame of the video.

The hybrid segmentation model may generate the predicted sequence of actions based on a predicted action length for a predicted action associated with the predicted action score. The action transition model may generate the potential subsequent action based on a transcript of one or more known sequences of actions, one or more of the features, and the predicted action score. The hybrid segmentation model may generate a predicted sequence of action lengths corresponding to the predicted sequence of actions. The computer-implemented method for weakly-supervised action segmentation may include detecting one or more errors associated with the predicted sequence of action length and the predicted sequence of actions based on an error function. The hybrid segmentation model may be based on an unconstrained Viterbi algorithm.

According to one aspect, a system for weakly-supervised action segmentation may include a memory and a processor. The memory may store one or more instructions. The processor may execute one or more of the instructions stored on the memory to perform one or more acts, actions, or steps. For example, the processor may perform feature extraction to extract one or more features associated with a current frame of a video including a series of one or more actions, feeding one or more of the features to a recognition network to generate a predicted action score for the current frame of the video, feeding one or more of the features and the predicted action score to an action transition model to generate a potential subsequent action, feeding the potential subsequent action and the predicted action score to a hybrid segmentation model to generate a predicted sequence of actions from a first frame of the video to the current frame of the video, and segmenting or labeling one or more frames of the video based on the predicted sequence of actions from the first frame of the video to the current frame of the video.

The hybrid segmentation model may generate the predicted sequence of actions based on a predicted action length for a predicted action associated with the predicted action score. The action transition model may generate the potential subsequent action based on a transcript of one or more known sequences of actions, one or more of the features, and the predicted action score. The hybrid segmentation model may generate a predicted sequence of action lengths corresponding to the predicted sequence of actions.

DETAILED DESCRIPTION

The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. These examples are not intended to be limiting. Further, one having ordinary skill in the art will appreciate that the components discussed herein, may be combined, omitted, or organized with other components or organized into different architectures.

A “processor”, as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that may be received, transmitted, and/or detected. Generally, the processor may be a variety of various processors including multiple single and multicore processors and co-processors and other multiple single and multicore processor and co-processor architectures. The processor may include various modules to execute various functions.

A “memory”, as used herein, may include volatile memory and/or non-volatile memory. Non-volatile memory may include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory may include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), and direct RAM bus RAM (DRRAM). The memory may store an operating system that controls or allocates resources of a computing device.

A “disk” or “drive”, as used herein, may be a magnetic disk drive, a solid state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, and/or a memory stick. Furthermore, the disk may be a CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and/or a digital video ROM drive (DVD-ROM). The disk may store an operating system that controls or allocates resources of a computing device.

A “bus”, as used herein, refers to an interconnected architecture that is operably connected to other computer components inside a computer or between computers. The bus may transfer data between the computer components. The bus may be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus may also be a vehicle bus that interconnects components inside a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area network (CAN), Local Interconnect Network (LIN), among others.

A “database”, as used herein, may refer to a table, a set of tables, and a set of data stores (e.g., disks) and/or methods for accessing and/or manipulating those data stores.

An “operable connection”, or a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a wireless interface, a physical interface, a data interface, and/or an electrical interface.

A “mobile device”, as used herein, may be a computing device typically having a display screen with a user input (e.g., touch, keyboard) and a processor for computing. Mobile devices include handheld devices, portable electronic devices, smart phones, laptops, tablets, and e-readers.

One of the challenges in human machine interaction may be in automatic vision-based understanding of human actions in instructional videos. These videos often depict a series of low-level actions that collectively accomplish a high-level task, such as preparing a meal or assembling an object or an item. However, labeling each frame of these videos may be arduous and may utilize a significant amount of manual effort to note the start and end times of each action segment. Consequently, there is interest in developing weakly supervised methods to learn the actions. In particular, such methods aim to overcome the challenge of weakly labeled instructional videos, where merely the ordered sequence of action labels (e.g., transcript) may be provided without any information on the duration of each action.

Detection of fine errors and anomalies in tasks performed by human operators may be useful for enhancing quality of work and efficiency. Anomalies may take different forms. According to one aspect, an anomaly may include a fine-grained sequential anomaly in an instructional video. Sequential anomalies may be defined as unseen action sequences that arise due to unexpected permutations (e.g., changes to the order of actions, addition of one or more actions, omission of one or more actions, etc.) in the action sequences seen in the training set. Explained yet again, such permutations may include unexpected changes in the order of actions, or the omission or addition of one or multiple actions at any point in the video. An error may be defined as a sequential anomaly that leads to an undesired outcome and may include an inaction, according to one aspect. This means that not all sequential anomalies are indicative of faulty procedures. The anomalous sequences may or may not entail assembly errors. Although the system for weakly-supervised action segmentation is described in the context of object or item assembly, other aspects are contemplated (e.g., any order based operation, such as cooking, etc.).

Examples of these unseen variations or anomalies at test time may include scenarios where an assembly worker skips fastening a screw or spends too much time idling between actions. It may be desirable to have artificial intelligence systems trained on limited or smaller sets of data, but these systems should be capable of detecting out-of-sequence actions (e.g., anomalies) or interruptions in situations or scenarios where inexperienced workers make mistakes (e.g., unintended actions) or follow sub-optimal sequences. According to one aspect, the system for weakly-supervised action segmentation may implement one or more actions via a controller, actuators, displays, speakers, etc. by notifying a worker when an anomaly has occurred at a test time or during an execution phase.

FIG.1is an exemplary component diagram of a system100for weakly-supervised action segmentation, according to one aspect. The system100for weakly-supervised action segmentation may include a processor102, a memory104, a storage drive106storing one or more neural networks108, and a communication interface110. The memory104may store one or more instructions. The processor102may execute one or more of the instructions stored on the memory104to perform one or more acts, actions, or steps. During a training phase, the communication interface110may receive a video (e.g., training video) and a video transcript (e.g., training transcript) and the system100for weakly-supervised action segmentation may build or train a weakly-supervised action segmentation model based thereon. Videos presented to the system100for weakly-supervised action segmentation may have their features extracted. For example, the processor102may perform feature extraction to extract one or more features associated with a current frame of a video including a series of one or more actions.

A transcript or training transcript may be a known (e.g., correct) ordered sequence of action labels or a known sequence of actions. For example, with reference to assembly of an object or item, examples of actions or action classes may include fasten screw, insert screw, take block, take part, fasten nut, take ring, take plate, tighten screw, spin block, insert pin, balance part, drop, pickup, hammer pin, etc. It will be appreciated that other actions may be classified according to other aspects. According to one aspect, the transcript may include cross-task variations rather than intra-task anomalies.

During an execution phase or a test phase, the system100for weakly-supervised action segmentation may receive a video (e.g., test video) and generate a predicted sequence of actions from a first frame of the video to the current frame of the video based on the received video (e.g., test video).

According to one aspect, the training video may include participants who assemble one or more toys in an expected and consistent manner. On the other hand, the test video and validation sets may include participants who display anomalies in their assembly of toys, including sequence variations, defects, or redundancies. Although the test video and training videos may include the same set of actions, the action sequences of the test video may be distinct and previously unseen relative to the training video.

The training video and/or the transcript may include frame-wise spatial annotations of human errors, atomic actions, human poses, segmentation annotations by frame for non-anomalous, anomalous without error, and anomalous with error videos, and bounding boxes of interactive objects for one or more videos of different tasks, such as assembly tasks. Additionally, temporal action labels, video-level error labels, object bounding boxes, and human poses may be provided. Temporal action labels may be annotated the start and end frames of action segments in each video in addition to the action transcripts.

Video-level error labels may be indicative of sequential anomalies which may, despite the error, still demonstrate a valid and complete assembly sequence. Unseen errors that occurred during the assembly of each object in the test video may be identified and labeled. Each test video may be annotated with one or more video-level error labels that indicate error classes present in the video. Object bounding boxes may be indicative of objects that each participant touches for the frames. Human pose may be modeled based on a number of joints for human participants.

The system100for weakly-supervised action segmentation may or may not include a device150. The device150may be utilized for capturing the video (e.g., either the training video or the test video) or be utilized for implementing an action based on the predicted sequence of actions. The device150may include a processor152, a memory154, a storage drive156, a communication interface158(e.g., which may be in computer communication or communicatively coupled with the communication interface110of the system100for weakly-supervised action segmentation), a controller160, one or more actuators162, one or more sensors170(e.g., video sensor, image capture sensor, etc.), a display172, a speaker174.

Weakly-supervised action segmentation and unseen error detection may be provided for anomalous instructional videos. Examples of instructional videos or ‘videos’ herein may include untrimmed videos of participants assembling different items, for example. These videos may be recorded or presented from various viewpoints.

During the training phase, videos presented to the system100for weakly-supervised action segmentation may not include anomalies. Stated another way, given a fixed set of non-anomalous training transcripts, the system100for weakly-supervised action segmentation may explore and infer unseen anomalous sequences at test time. Again, an anomaly may include one or more previously unseen (e.g., not from a training transcript) action sequences that arise due to unexpected permutations of the action sequences seen in the training set. Additionally, during the training phase, a weakly labeled segmentation algorithm may be introduced. For example, the segmentation algorithm may be implemented in a segmentation model which may be a generalization of a constrained version of the Viterbi algorithm and may identify potential anomalous moments based on the difference between future anticipation and current recognition results, as will be discussed in greater detail with reference toFIG.2.

During the test phase, validation phase, or execution phase, videos presented to the system100for weakly-supervised action segmentation may include anomalies, such as sequential anomalies, defined as unexpected permutations of the training transcripts, such as redundant actions (e.g., inserting an extra screw or unexpected background segments between actions), skipped actions (e.g., not tightening a screw), and major changes in the order of training action subsequences (e.g., when the last phase of an assembly is performed at the beginning of the video).

The segmentation algorithm may be implemented in a segmentation model which may be a generalization of an unconstrained version of the Viterbi algorithm, which may act as an alignment mechanism to align unseen transcripts to the video and identify potential anomalous moments based on the difference between future anticipation and current recognition results. The unconstrained version of the segmentation model may be a hybrid segmentation model and may be an unconstrained Viterbi algorithm that enables real-time segmentation of videos into unseen action sequences. In this way, inference of anomalous action sequences while maintaining real-time performance may be possible.

The majority of weakly-supervised action segmentation methods may be limited by the training transcripts. Specifically, most cannot predict unseen transcripts because they iterate through training transcripts to find the best alignment with the test video. An RNN may be trained to predict the video transcript offline, but the RNN remains biased by the training transcripts and unable to generalize well to unexpected transcript variations.

One advantage or benefit of the weakly-supervised action segmentation described herein may be that it is not restricted by the training transcripts during testing, thereby allowing for the inference of anomalous action sequences while maintaining real-time performance. Well-being of individuals may be enhanced by movement monitoring or action monitoring using the weakly-supervised action segmentation. Additionally, quality of work may be improved in the technical areas of manufacturing and task assembly. Based on these segmentation results, pre-defined human errors that occur during assembly may be detected. Examples of errors which may be detected with reference to assembly of an object or item may include idle time, unfastened leg, dropped item without picking up item, missing leg, missing ring, unfastened screw, extra screw, no balancing, missing screw, missing part, extra ring, etc. It will be appreciated that other errors may be classified according to other aspects.

Weakly-Supervised Action Segmentation Formulation

The processor102of the system may perform weakly-supervised action segmentation tasks discussed herein. Given a set of training videosand corresponding transcripts, a goal may be to partition a test video into sequences of n actions a1n∈and their duration l1n.may be the set of unseen test transcripts, and may be based on the anomaly assumption,∩=Ø.may be defined as the set of || unique actions labels and x1tthe sequence of frame-level features from the beginning of video until time t.

Weakly-Supervised Action Segmentation Model

Given trained parameters and extracted features x1tof a test video, Equation (1) below, may be utilized to approximate the likelihood of n action segments with labels a1nand durations l1nuntil time t. In Equation (1), p(xt|ant) may be the visual model, and may be derived using the Bayes rule on top of the probability output of a recognition network. ntmay be the segment number at time t. Thus, the processor102may perform feeding one or more of the features to a recognition network to generate a predicted action score for the current frame of the video.

The probability of transitioning into action segment ń>1 at time tń=Σ1ń-1l{circumflex over (n)}may be modeled by p(ań|a1ń-1,x1tń). The action transition model described herein may depend on the context of video at the transition point tńin addition to previous action labels a1ń-1. This may facilitate detection of anomalous segments unlike previous works, where action transition occurs merely according to the set of seen training transcripts. Finally, pmodeń(l|a) may be the length model of segment ń, and may estimate the probability of action a lasting l frames.

Equation (1) addresses segmentation for both modes of offline (e.g., off) and online (e.g., on). In offline segmentation, t marks the end of an. Meanwhile, in the online mode, the last segment n may be ongoing, so t may not mark the end of the current action. Hence, the difference between both modes may be the choice of the length model pmoden(l|a) for the last segment n. While poffń(l|a) may be a Poisson function in offline segmentation for ponn(l|a) for merely the last segment n. Poissons may be parameterized by the estimated average length of actions.

Action Transition Model

Let {right arrow over (A)}t=[p(ct|xt-ω;θa), ∀c∈] be the “anticipated” or future action anticipation probability vector for time t given past features at time t-ω. Also, {right arrow over (R)}t=[p(ct|xt;θr), ∀c∈] may denote the “current” action recognition probability vector for time t. {right arrow over (A)}tand {right arrow over (R)}t∈may be outputs of anticipation and recognition networks, parameterized by θaand θrrespectively. Thus, the processor102may perform feeding one or more of the features and the predicted action score to an action transition model to generate a potential subsequent action. The action transition model may generate the potential subsequent action based on a transcript of one or more known sequences of actions, one or more of the features, and the predicted action score.

Anomalous behavior may be detected by the discrepancy between the expected and current action representations {right arrow over (A)}tand {right arrow over (R)}trespectively. Actions that typically occur temporally close to each other may be also more similar in their visual representations. Equation (2) may connect similarity between action representations to their temporal positions.τ(a1ń-1,tń) may be the set of possible action labels for segment ń at transition point tńgiven the previous labels a1ń-1. In Equation (2),(a,b) may be the cosine similarity of two vectors, and{aij} may return the set of all actions that succeed sequence aijaccording to the training transcriptsor all successors for action sequence aij.

Dissimilar anticipated and current action probability vectors may indicate an anomalous transition, so deviation from the transcripts may be allowed by exploring the set of all possible actions. Otherwise, action transitions may follow the sequences in the training transcripts. In this case, the set of all possible actions may be equal to the set of all actions that succeed the Longest Common Subsequence (LCS) aiń-1between the previous sequence a1ń-1and the training transcripts. The transition model may be a special case when τ=0, because in this case deviation from training transcripts never occurs and the LCS may be always a1ń-1. Ultimately, in Equation (1), p(ań|a1ń-1,x1tń)=1 if ań∈τ(a1ń-1,tń), and may be 0 otherwise.

In this way, a constrained dynamic approach and a greedy approach are implemented by Equation (2). The greedy approach may be represented by the set of all possible actions, which may be utilized when an anomaly is detected. The constrained dynamic approach may be utilized when no anomaly is detected, an inference may be made based on the most likely sequence of actions from the set of training sequences or training transcript. The constrained algorithm may return a transcript sequence of actions and durations, and thus, exist in the training or transcript universe.

Algorithm 1, provided herein, may efficiently solve both online and offline segmentation of Equation (1) at each time step t. At each time step t, dynamic programming and the results from the previous time step may be used to generate new segmentation results. Each new sequence may be the result of either continuing the last action or transitioning into a new one. The hybrid segmentation model may be based on an unconstrained Viterbi algorithm. Different than the constrained Viterbi algorithm from the training phase, the execution phase Viterbi algorithm may be unconstrained, because it may be not limited to the training transcripts. This may be useful for inferring unseen and anomalous action sequences. Specifically, Pt[ln,a1n] may be defined as the probability of the most likely alignment of sequence a1nwith video frames until time t, so that anmay be incomplete and have a duration of ln. The mostly likely segmentation result (ā1n,l1n)mode=argmax{Pt[ĺń,á1ń]·pmodeń(ĺń|áń)} may be derived once at the end of video for offline segmentation, and at every time step during online inference. However, merely the current action at=ānmay be selected as the online inference output of time t.

Thus, the processor102may perform feeding the potential subsequent action and the predicted action score to a hybrid segmentation model to generate a predicted sequence of actions from a first frame of the video to the current frame of the video. The hybrid segmentation model may generate the predicted sequence of actions based on a predicted action length for a predicted action associated with the predicted action score. The hybrid segmentation model may generate a predicted sequence of action lengths corresponding to the predicted sequence of actions.

In order to achieve real time performance, separately for each action a∈at time t, merely the set of top B likely segmentation resultstB(a) ending with action a may be kept (e.g., by pruning excess options). Such an action-wise pruning gives the online segmentation method the advantage to infer any possible action, which might have been pruned out otherwise. The overall complexity of Algorithm 1 at each time step may be O(B||(log B+||)). This complexity may be the result of enumerations in addition to the sorting complexity of topB{ } with beam size B.

A weakly-supervised framework may be used to train the anticipation and recognition networks in an iterative fashion. Given a video of length T and its transcript per iteration, training may be done following two steps. First, frame-level pseudo labels ā1Tmay be estimated through offline segmentation in Equation (1) during the training phase. Second, the pseudo labels may be used in a loss functionto update the parameters θaand θrof the anticipation, and recognition networks respectively. The Constrained Discriminative Forward LossCDFLmay be employed, which effectively maximizes the decision margin between valid and hard invalid pseudo labels. In Equation (3),CDFLmay be applied to the recognition outputs {right arrow over (R)}1Tof all frames and to the anticipation output {right arrow over (A)}ωT, weighted by λa, for frames from ω to T, where ω may be the future anticipation range.

Error Detection

Problem Definition: a goal in error detection may be to identify if and the number of times nean error e∈ε has occurred in a test video. ε may be the set of unseen error categories that may be only present in the test video. The dataset provides detailed instructions I of what each error may be when performing a task, e.g., the error label “missed leg” may mean using less than 4 legs to assemble a table. It may be not clear how to temporally locate all errors because certain errors correspond to inaction. Also, some errors may be inferred when the video has ended, e.g. not picking up an item that may be dropped in the process. As a result, detect errors at the end of the video after the task may be fully observed.

Overview: a simple error detection method may be provided as a set of error functions {e}, so that each function

maps frequency f of inferred actions in the test video to the number of instances nethat error e has occurred. Here, f={fa}, and famay be the number of predicted video segments labeled by action a. For example, the function for the error label “Loose Screw” may be defined asLoose Screw:=max(finsert screw−ffasten nut, 0). Thus, the processor102may detect one or more errors associated with the predicted sequence of action length and the predicted sequence of actions based on an error function.

For each test video, two different segmentation results S0and Sτmay be generated for τ=0 and τ>0 respectively. The S0may represent the constrained offline segmentation as a reference, where the estimated transcript may be one of the training transcripts. Then, the respective set of action frequencies f0and fτmay be calculated from the segmentation results S0and Sτ. Finally, f0and fτmay be incorporated in Equation (4) to produce if and how many times each error e has happened:

Error functions may operate based on action frequencies and may not consider the semantics of the video. In other words, error functions may not consider the assembly type. Stated yet another way, the same behavior may be erroneous in task A and may be error free in task B. As a result, the reference action frequency f0may be used to focus on relevant errors and alleviate false positives. Therefore, to remove false positives, a behavior is considered erroneous only if it is error free in its corresponding training transcript. Specifically, term b in Equation (4) may condition the result based on the action frequency discrepancy between the predicted anomalous transcript and its corresponding non-anomalous training transcript. In other words, an erroneous behavior may be detected in the anomalous segmentation result if the same behavior is error-free in the estimated non-anomalous transcript of the video, e.g., skipping action a in a test video may be considered an error if action a has occurred in its non-anomalous reference S0.

According to one aspect, the processor102may perform segmenting or labeling one or more frames of the video based on the predicted sequence of actions from the first frame of the video to the current frame of the video. According to one aspect, the processor102may notify a worker when an anomaly has occurred at a test time or during an execution phase utilizing the display172or the speaker174, for example.

FIG.2is an exemplary component diagram of a system200for weakly-supervised action segmentation which may be a system for training a weakly-supervised action segmentation model, according to one aspect. As seen inFIG.2, video202may have features extracted204. These extracted features may be fed to a recognition network220and an action transition model230with a similarity threshold set to zero so as to stick to the video transcript232for training. The length model210may provide an estimated length for the corresponding estimated action from the recognition network220. The outputs from the length model210, the recognition network220, and the action transition model230may be fed to the segmentation model240, which may be a constrained Viterbi algorithm. The segmentation model240may output actions labels and durations for the actions242, which may be utilized to generate pseudo frame labels for the video202, and the parameters of the length model210, the recognition network, and the action transition model230may be updated246.

FIG.3is an exemplary component diagram of a system300for weakly-supervised action segmentation, according to one aspect. As seen inFIG.3, video302may have features extracted304. These extracted features may be fed to a recognition network320and an action transition model330which may or may not utilize the training transcripts332. The length model310may provide an estimated length for the corresponding estimated action from the recognition network320. The outputs from the length model310, the recognition network320, and the action transition model330may be fed to the hybrid segmentation model340, which may be an unconstrained Viterbi algorithm. The hybrid segmentation model340may output actions labels and durations for the actions342, which may be utilized to action frequency344, and errors may be detected at346.

FIG.4is an exemplary component diagram of an architecture for an action transition model for the system for weakly-supervised action segmentation ofFIG.3, according to one aspect. An anticipation network402of the action transition model330may be trained during a training phase associated with a constrained version of the hybrid segmentation model. The action transition model may generate the potential subsequent action based on feeding one or more of the features to an anticipation network to generate an expected action for the current frame of the video and based on a comparison404between the expected action for the current frame and the predicted action score for the current frame. If the comparison404is greater than a similarity threshold, generating406the potential subsequent action based on a transcript of one or more known sequences of actions. If the comparison is less than a similarity threshold, generating408the potential subsequent action based on exploring a universe of possible actions.

FIG.5is an exemplary flow diagram of a computer-implemented method500for training a weakly-supervised action segmentation model, according to one aspect. The computer-implemented method for training a weakly-supervised action segmentation model may include performing502feature extraction to extract features associated with current frame of video including series of actions and no anomalies, feeding504features to recognition network to generate current action scores for current frame of video, feeding506features and current action scores to action transition model set to follow transcript to generate potential subsequent action, feeding508potential subsequent action and current action scores to constrained version of segmentation model to generate predicted sequence of actions from first frame of video to current frame of video, generating510action labels for frames, and updating512model parameters.

FIG.6is an exemplary flow diagram of a computer-implemented method600for weakly-supervised action segmentation, according to one aspect. The computer-implemented method600for weakly-supervised action segmentation may include performing602feature extraction to extract features associated with current frame of video including series of actions, feeding604features to recognition network to generate current action scores for a current frame of video, feeding606features and current action scores to action transition model to generate potential subsequent action, feeding608potential subsequent action and current action scores to unconstrained hybrid segmentation model to generate predicted sequence of actions from first frame of video to current frame of video, and segmenting or labeling610frames of video based on predicted sequence of actions from first frame of video to current frame of video.

FIG.7is an exemplary illustration of video in association with the system for weakly-supervised action segmentation ofFIGS.1-3, according to one aspect. For example, different tasks T1, T2, T3are shown in connection with assembly of an object. The legend on the bottom ofFIG.7is an exemplary illustration of a legend for the classifications of different actions which are utilized in connection with the assembly of the objects in chronological order, for example.

FIG.8is an exemplary illustration of segmentation in association with the system for weakly-supervised action segmentation ofFIGS.1-3, according to one aspect. According to one aspect, the unconstrained segmentation ofFIG.8may not necessarily be limited to the training transcripts, shown on the right. For example, if a comparison between an expected action for a current frame and a predicted action score for the current frame is greater than a similarity threshold, then the training transcripts may be utilized. On the other hand, if the comparison between the expected action for the current frame and the predicted action score for the current frame is less than the similarity threshold, then a set of all possible actions may be explored.

Still another aspect involves a computer-readable medium including processor-executable instructions configured to implement one aspect of the techniques presented herein. An aspect of a computer-readable medium or a computer-readable device devised in these ways is illustrated inFIG.9, wherein an implementation900includes a computer-readable medium908, such as a CD-R, DVD-R, flash drive, a platter of a hard disk drive, etc., on which is encoded computer-readable data906. This encoded computer-readable data906, such as binary data including a plurality of zero's and one's as shown in906, in turn includes a set of processor-executable computer instructions904configured to operate according to one or more of the principles set forth herein. In this implementation900, the processor-executable computer instructions904may be configured to perform a method902, such as the computer-implemented method500ofFIG.5and the computer-implemented method600ofFIG.6. In another aspect, the processor-executable computer instructions904may be configured to implement a system, such as the system100for weakly-supervised action segmentation ofFIGS.1-3. Many such computer-readable media may be devised by those of ordinary skill in the art that are configured to operate in accordance with the techniques presented herein.

Generally, aspects are described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions may be distributed via computer readable media as will be discussed below. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform one or more tasks or implement one or more abstract data types. Typically, the functionality of the computer readable instructions are combined or distributed as desired in various environments.

FIG.10illustrates a system1000including a computing device1012configured to implement one aspect provided herein. In one configuration, the computing device1012includes at least one processing unit1016and memory1018. Depending on the exact configuration and type of computing device, memory1018may be volatile, such as RAM, non-volatile, such as ROM, flash memory, etc., or a combination of the two. This configuration is illustrated inFIG.10by dashed line1014.

In other aspects, the computing device1012includes additional features or functionality. For example, the computing device1012may include additional storage such as removable storage or non-removable storage, including, but not limited to, magnetic storage, optical storage, etc. Such additional storage is illustrated inFIG.10by storage1020. In one aspect, computer readable instructions to implement one aspect provided herein are in storage1020. Storage1020may store other computer readable instructions to implement an operating system, an application program, etc. Computer readable instructions may be loaded in memory1018for execution by the at least one processing unit1016, for example.

The computing device1012includes input device(s)1024such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, or any other input device. Output device(s)1022such as one or more displays, speakers, printers, or any other output device may be included with the computing device1012. Input device(s)1024and output device(s)1022may be connected to the computing device1012via a wired connection, wireless connection, or any combination thereof. In one aspect, an input device or an output device from another computing device may be used as input device(s)1024or output device(s)1022for the computing device1012. The computing device1012may include communication connection(s)1026to facilitate communications with one or more other devices1030, such as through network1028, for example.

Various operations of aspects are provided herein. The order in which one or more or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated based on this description. Further, not all operations may necessarily be present in each aspect provided herein.