Patent Description:
Machine learning models, among other systems, may be trained to segment or localize patterns in a sequence of frames, i.e. to analyze the sequence of frames in order to detect which sub-ranges of the sequence include some learnt patterns. Generally, the machine learning model is designed to output pattern estimations for each frame, i.e. the probabilities that every one of the frames includes this or that pattern.

Recently, fully supervised approaches for action segmentation in untrimmed videos have achieved encouraging results. However, full supervision is a labor-intensive endeavor as it consists in annotating every instance of the actions of interest with their exact temporal boundaries. Furthermore, in many cases, the exact temporal boundary of actions is subjective and can lead to annotation inconsistencies.

To address the issues of full supervision, a lot of effort has been allocated to weakly supervised approaches. In this perspective, Li et al. (<NPL>, hereinafter "Li") introduced timestamp supervision for action segmentation. Timestamp supervision is a form of weak supervision where the annotator is required to annotate a single frame and its class for each action in the video. This provides a form of temporal information that is crucial for action segmentation and missing from other weak supervision approaches such as transcript supervision, in which transcripts are the ordered set of actions that occur in a video without any temporal information about the start and end of each action instance. While the cost of obtaining timestamp supervision is only marginally higher than transcript supervision, Li showed that timestamp supervision results in action segmentation performance close to the fully supervised setting.

Despite promising results, timestamp supervision as proposed by Li assumes that every action (or more generally every pattern) instance in the sequence is annotated. Nonetheless, an annotator could miss some patterns and as a result, not all pattern segments are necessarily annotated. Generating dense labels in this setting would produce incorrect pseudo-ground-truth, which has a severe negative impact on the performance.

<NPL>, discloses timestamp supervision for surgical phase recognition. Specifically, the annotated frames are treated as anchors and pseudo labels are diffused to both sides, starting from anchors and stopping at the high-uncertainty frames.

In order to at least substantially remedy the above-mentioned drawbacks, the present disclosure relates to a computer-implemented method for generating pseudo-ground-truth data for training a machine learning model to segment or localize patterns in a sequence of frames over a succession dimension, the method comprising:.

For the sake of conciseness, this method is referred to hereinafter as the generating method.

A sequence of frames is an ordered series of frames along one dimension, known as the succession dimension, which can be e.g. time or space.

The contents of the frames are such that a plurality of frames forms a pattern. The segmentation aims to determine which pattern is included in which (desirably continuous) series of the frames. Thus, in the present disclosure, unless otherwise mentioned, the segmentation includes the task sometimes referred to as alignment and corresponding to aligning, in the succession dimension, a known output pattern with an input sequence of frames. Localization aims to associate a given pattern to each of the frames which belong to that pattern. Therefore, as opposed to segmentation, localization may result in some overlapping patterns, and/or in frames associated to no pattern at all (e.g. background).

A machine learning model may comprise one or more mathematical expressions configured to process the frames in order to output information representative of the patterns. Weights of the machine learning model may be iteratively updated by the training method. The machine learning model may comprise one or more artificial neural network. Each artificial neural network (ANN) is a computational model with one or more neurons, each neuron having a transfer function. The ANN therefore has a global transfer function that allows the calculation of at least one output as a function of at least one input. The neurons may be structured in one or more layers. The transfer functions of each neuron or the relative weights of the neurons in the network can be weighted by coefficients called synaptic weights (or simply weights) and bias. The weights can be modulated according to the training of the ANN. In principle, training comprises providing the ANN with a set of situations in which inputs and outputs are known. During the training process, the ANN adapts its synaptic weights and biases to conform to the learned situations, possibly with some tolerance.

The machine learning model is assumed to be able to learn from the pseudo-ground-truth data.

As mentioned above, a timestamp includes identification of a respective one of the frames and a pattern annotation of said frame. The timestamp may be defined as in Li. Basically, the timestamp provides punctual information about what the pattern of a frame is. Annotation may be performed manually or otherwise. Identification of a frame may be any information enabling to identify a frame among the other frames of the sequence, including a position of the frame or the frame itself. Therefore, for conciseness, a timestamp may be used to identify a position on the succession dimension, by referring to the position of the frame identified by that timestamp.

The defining at least one subset may be carried out explicitly, e.g. with a subset having an identified name and content, or implicitly, e.g. when the method works in a certain sub-range of the sequence even without giving it proper identification.

Hereinafter, unless otherwise stated, "the subset" refers to the at least one subset. In an embodiment, "the at least one subset" comprises all the subsets of the sequence, respectively defined between all pairs of consecutive timestamps of the plurality. More generally, herein, the article "the" may refer to "the at least one".

The subset comprises all the frames located between two consecutive timestamps, namely the first timestamp and the second timestamp. That is, the plurality does not comprise any other timestamp between the first timestamp and the second timestamp. Thus, the subset is a continuous portion of the sequence in the succession dimension. The respective frames of the first timestamp and the second timestamp, i.e. the boundaries of the subset, may be included or not in the subset. For conciseness, it will be assumed hereinafter that the first timestamp precedes the second timestamp, i.e. the frame of the first timestamp comes before the frame of the second timestamp in the succession dimension, but the opposite situation is also contemplated.

The subset is divided into three regions. The first region is on the side of the first timestamp, the second region is on the side of the second timestamp, and the neutral region joins the first region to the second region. The regions are adjoining or contiguous, i.e. no frame of the subset is left out of the regions and the regions do not overlap one another. In addition, each region comprises a set of frames which is continuous over the succession dimension.

As with the subset, the three regions may be defined explicitly or implicitly. For instance, instead of focusing on a subset bounded by two consecutive timestamps as detailed before, one may focus on a sub-range of the sequence comprising one timestamp and frames on either side(s) thereof, the sub-range being possibly centered on that timestamp. A first portion of the sub-range, corresponding to the first region, would start from that timestamp and extend towards an end of the sub-range. As applicable, a second portion of the sub-range would start from that timestamp and extend towards the other end of the sub-range. The second portion would correspond to the previously-defined second region of a neighboring subset. End portions, corresponding to parts of the previously-defined neutral region, would extend between the first and second portions and respective ends of the sub-range. Although the portions defined in this way may look formally different from the above-defined regions, they amount to implicitly defining the above-mentioned three regions, as long as the initial assigning provides a neutral region or portion to account for possibly missing timestamps, as will be detailed below.

After initializing, the division in three regions is then optimized based on pattern estimations for the frames of the subset. Pattern estimations for each frame, also known as frame-wise pattern probability estimates, are predicted probabilities that a given frame corresponds to a given pattern. For instance, the frame-wise pattern probability estimates may comprise a N-component vector for each frame, with each i-th component, i from <NUM> to N, being the probability that the given frame corresponds to the i-th pattern learnt by the machine learning model. As mentioned above, the frame-wise pattern probability estimate may be obtained by methods known per se, e.g. machine learning models.

The compromise used to optimize the boundaries between the first region, the second region and the neutral region, balances the size of the neutral region, which is desirably as low as possible because it is not known to which pattern the frames assigned thereto correspond, and a match between the pattern estimations of the frames and the regions to which the frames are assigned, i.e. the confidence with which frames are assigned to the first region and the second region. In other words, this compromise leverages the information included in the first timestamp and the second timestamp, by assuming that the neighboring frames will belong to the same pattern, while leaving an uncertainty area in-between, namely the neutral region, if pattern estimates show that it is not quantitatively reasonable to assign some frames either to the pattern of the first timestamp or to the pattern or the second timestamp. Yet in other words, the compromise corresponds to a trade-off between the confidence in the frames with assigned labels, namely the frames of the first and second region, and the size of the neutral region.

The optimizing may be carried out for each subset separately, or jointly for all the subsets.

Finally, the method outputs the regions after optimization, namely pattern annotation of the first timestamp for the frames assigned to the first region, and pattern annotation of the second timestamp for the frames assigned to the second region. This output may be presented in a variety of more or less explicit formats, which enable to derive this information. For instance, the output may merely comprise a list of frames identifications which correspond to the successive boundaries between the different regions; then, considering the timestamps, it is possible to derive the patterns associated to the frames which were not left in one of the neutral regions after optimization.

Thanks to these provisions, the present generating method relax the assumption that all pattern should be annotated (i.e. that a corresponding timestamp should be provided) and proposes an approach that is designed to deal with potential missing segments in the timestamp supervision framework. In contrast to Li, this approach can skip frames belonging to a missing segment during training while trying to generate labels for all frames belonging to annotated segments. As explained before, the neutral region provides room for possibly unannotated patterns, and may be ignored for subsequent training.

This approach is therefore more robust to missing segments in the timestamps and thus provides improved pseudo-ground-truth data which enhance training.

Optionally, the pattern estimations are obtained by the machine learning model. In particular, the machine learning model may be configured to make probabilistic predictions, e.g. at test time: for instance, a neural network is able to make probabilistic predictions, whereas a support vector machine only makes binary predictions and is not regarded as configured to make probabilistic predictions. In these embodiments, the machine learning model may be the same as the machine learning model which is intended to be trained with the pseudo-ground-truth data. In order for the machine learning model to provide reasonable pattern estimations, the machine learning model may be pre-trained with the sequence of frames and the plurality of timestamps.

Optionally, the neutral region comprises the frames to be associated with an unknown pattern, and optionally wherein the pseudo-ground-truth data comprise, for each frame of the neutral region after the optimizing, identification of said frame and, as a corresponding pattern annotation, the unknown pattern. The frames annotated with the unknown pattern may be ignored during subsequent training. Alternatively, the pseudo-ground-truth data may omit the frames of the neutral region.

Optionally, the sequence of frames is selected from a group comprising a plurality of images, a video, an audio track, an electrical input. More specifically, a few non-limiting examples are set out below:.

The sequence of frames may be obtained from a sensor, e.g. a microphone, an imaging device such as a camera, electrodes, etc., or from a database, e.g. a local or distant server or the like.

Optionally, the pattern is an action. The action may be carried out by a human being, an animal or a machine.

Optionally, the generating method further comprises:.

These steps aim to replicate the principles of the previously-described generating method at the beginning and the end of the sequence of frames. The third region has the same function as the first region or the second region, and the end region has the same function as the neutral region. The explanations given before thus apply mutatis mutandis to these optional features.

Optionally, a match between the pattern estimations of the frames and the regions to which the frames are assigned is quantified using a sum of log probabilities of pattern estimations of the frames assigned to the respective regions. Thus, the match is simple to compute and may be used for real-time applications such as autonomous or assisted driving, or manufacturing monitoring.

The optimizing comprises solving the constrained minimization problem <MAT> where ri is the number of frames of the first region, gi is the number of frames of the neutral region, li+<NUM> is the number of frames of the second region, N is the number of timestamps, pi is the position of the i-th timestamp in the sequence, ypi is the pattern annotation of the i-th timestamp, ỹt is the pattern estimation for frame t, and β is a hyperparameter.

Optionally, the compromise is represented by a non-differentiable model and the optimizing comprises solving a differentiable approximation of the non-differentiable model. By replacing the non-differentiable model by a differentiable approximation thereof, it is possible to use a broader range of iterative optimizations and to make the optimizing step yield faster results for a comparable, if not better performance. As a consequence, the generating method can be used in real-time applications such as autonomous or assisted driving, or manufacturing monitoring.

The present disclosure is further directed to a training method for a machine learning model configured to segment or localize patterns in a sequence of frames over a succession dimension, the training method comprising:.

As mentioned above, the frames assigned to the respective neutral regions may be ignored or otherwise specially handled in order to improve the training. The machine learning model may comprise models of any type, e.g. a neural network such as a convolutional neural network (CNN).

The present disclosure is further directed to a system configured to generate pseudo-ground-truth data for training a machine learning model to segment or localize patterns in a sequence of frames over a succession dimension, the system being configured to:.

The system may be configured to carry out the above-mentioned generating method, and may have part or all of the above-described features. The system may have the hardware structure of a computer.

The present disclosure is further related to a computer program including instructions for executing the steps of the above described method for generating pseudo-ground-truth data when said program is executed by a computer.

The present disclosure is further related to a recording medium readable by a computer and having recorded thereon a computer program including instructions for executing the steps of the above described method for generating pseudo-ground-truth data. The recording medium can be any entity or device capable of storing the program. For example, the medium can include storage means such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or magnetic storage means, for example a diskette (floppy disk) or a hard disk.

Alternatively, the recording medium can be an integrated circuit in which the program is incorporated, the circuit being adapted to execute the method in question or to be used in its execution.

The invention and advantages thereof will be better understood upon reading the detailed description which follows, of embodiments given as non-limiting examples. This description refers to the appended drawings, wherein:.

<FIG> shows a diagram of a computer-implemented method for generating pseudo-ground-truth data (hereinafter the "generating method <NUM>") for training a machine learning model to segment or localize patterns in a sequence of frames over a succession dimension, according to embodiments of the present disclosure. The generating method <NUM> comprises a providing step <NUM> of providing a sequence of frames <NUM> and a plurality of timestamps <NUM>, a defining step <NUM> of defining at least one subset <NUM> comprising the frames located between a first timestamp and a consecutive second timestamp, an assigning step <NUM> of initially assigning the frames of the at least one subset to one of three non-empty regions, and obtaining step <NUM> of obtaining pattern estimations for each frame of the at least one subset <NUM>, an optimizing step <NUM> of optimizing respective boundaries between the regions and an outputting step <NUM> of outputting pseudo-ground-truth data <NUM> comprising pattern annotations of the frames based on the regions to which they are assigned after the optimizing step <NUM>.

The generating method <NUM> may be used in the context of training a machine learning model using the generated pseudo-ground-truth data. In this respect, a training method for a machine learning model configured to segment or localize patterns in a sequence of frames may comprise generating pseudo-ground-truth data <NUM> by the generating method <NUM>, and a training step <NUM> of training the machine learning model based on the pseudo-ground-truth data <NUM>. The training step <NUM> is known per se in the art.

<FIG> illustrates a system configured to generate pseudo-ground-truth data (hereinafter the "generating system <NUM>") for training a machine learning model to segment or localize patterns in a sequence of frames over a succession dimension.

The generating system <NUM> may comprise an electronic circuit, a processor <NUM> (shared, dedicated, or group), a combinational logic circuit, a memory <NUM> that executes one or more software programs, and/or other suitable components that provide the described functionality. In other words, the generating system <NUM> may be a computer device. The memory <NUM>, or an external memory to which the generating system <NUM> may be connected, may store data, e.g. a computer program which when executed, carries out the generating method <NUM> or the training method according to the present disclosure; thus, the memory <NUM> or the external memory may form a recording medium readable by a computer and having recorded thereon a computer program including instructions for executing the steps of the method. In particular, the generating system <NUM> or the external memory may store software which comprises a computer program including instructions for executing the steps of the generating method <NUM> or the training method. More generally, the generating system <NUM> may be configured to implement the generating method <NUM> or the training method.

In the following, an embodiment of the generating method <NUM> is detailed with reference to <FIG>, focusing on an application to segmenting actions in a video, i.e. segmenting patterns in a sequence of frames over a succession dimension, wherein each pattern correspond to an action performed in the video, each frame comprises an image and the succession dimension is time. However, the principles explained hereinafter can be generalized by the skilled person to any pattern other than actions, any succession dimension other than time, and/or any frame other than images. For instance, the sequence may be, as detailed above, a plurality of images, a video, an audio track, an electrical input, etc. In addition, in the following embodiment, the machine learning model is a neural network model (shortened as "network"), but as detailed above, other models can be envisaged.

Given an untrimmed video, action segmentation is the task of predicting the action label for each frame. The video is represented as a set of D -dimensional pre-extracted framewise features X = (x<NUM>, ··· , xT) with length T (i.e. the video comprises T frames). A fixed set of Q action classes is assumed, and the ground truth label is represented by Y = (y<NUM>, ··· , yT). Action segmentation models are required to predict framewise probability estimates Ỹ = (ỹ<NUM>, ··· , ỹT) where ỹt ∈ [<NUM>,<NUM>]Q represents the predicted probability for each class. If full supervision is available during training (Y is provided), a per-frame classification loss can be used to train the network.

Timestamps are a set of N positions in time P = (p<NUM>, ···, pN) and their corresponding labels yp<NUM>, ···, ypN. It is assumed that the timestamps are ordered, <MAT>.

That is, each timestamp including identification of a respective one of the frames, e.g. a position thereof, and a pattern annotation of said frame, e.g. a label.

A naive approach for action segmentation using timestamps is to only use the timestamps for training the neural network model. The neural network model estimates the frame-wise probability for the entire video and the classification loss is calculated only using the frames with the timestamps.

Li assumes that every action segment is annotated with one timestamp. Given this assumption, to uniquely produce the ground truth from the timestamps, one needs to only find the time location between consecutive timestamps where the action label changes from the previous timestamp to the next one. Li proposes an approach to find the action boundary between the consecutive timestamps that minimizes the variance of framewise pre-classification features estimated by the action segmentation network. Given the boundary between the timestamps, the pseudo-ground-truth annotation A = (a<NUM>, ··· , aT) can be constructed and used for training the network on all temporal positions instead of only on the temporal positions of the timestamps as in the naive approach. Li showed that the proposed approach for the generation of the labels for all frames is crucial to performance in timestamp supervision.

However, if some actions are not annotated by timestamps, the central assumption by Li is not valid. In this setting, where some actions are skipped, for example, due to annotator mistake, the pseudo-ground-truth generated by Li is incorrect for the entire region of the unannotated action, as shown in <FIG> shows the ground-truth annotation <NUM> of a sequence having three patterns A, B and C, only A and C being annotated; the timestamps are represented by a black vertical line, here with a small hand underneath, figuring manual annotation. The pseudo-ground-truth generated by Li, under reference sign <NUM>, misses the unannotated segment B.

This incorrect pseudo-ground-truth generation severely affects the performance of their method. In an "Oracle" experiment where the correct boundary is known for each annotated timestamp, the inventors observed that the accuracy of action segmentation only marginally drops when the number of annotated segments is reduced from <NUM>% to <NUM>%. In comparison, the approach by Li suffers a large drop in accuracy.

The present disclosure proposes an approach for pseudo-ground-truth generation that is robust to unannotated segments. The underlying idea is to only extend the segment boundary from the timestamp position in each direction until a point where the network is still confident in the class of that segment while trying to minimize the size of ignored regions.

As shown in <FIG> which illustrates the beginning of the video, given two consecutive timestamps pi and pi+<NUM> (i from <NUM> to N - <NUM>), the subset of the video between the two timestamps is divided into three regions. The first region <NUM> with size ri belongs to the first timestamp pi (i.e. comprises the frames to be associated with the pattern of the first timestamp pi), the middle or neutral region <NUM> with size gi is the neutral region (also called ignored region) and the third region <NUM> with size li+<NUM> belongs to the second timestamp pi+<NUM> (i.e. comprises the frames to be associated with the pattern of the second timestamp pi+<NUM>). As shown in <FIG>, the first region <NUM> starts from the first timestamp pi and extends towards the second timestamp pi+<NUM>, the second region starts from the second timestamp pi+<NUM> and extends towards the first timestamp pi, and the neutral region <NUM> joins the first region <NUM> to the second region <NUM>.

In this embodiment, the beginning and ending regions of the video are also ignored segments g<NUM> and gN, respectively: they correspond to the previously-mentioned end regions <NUM> and the respective neighboring timestamps correspond to the previously-mentioned third timestamps p<NUM>, pN. The regions between an end region and a timestamp are the previously-mentioned third regions <NUM>. The third timestamp p<NUM> may coincide with the first timestamp of the first subset; likewise, the third timestamp pN may coincide with the second timestamp of the last subset.

In the assigning step <NUM>, the size of the regions may be determined randomly or following any desired rule; only, the first region <NUM>, the neutral region <NUM> and the second region <NUM> should not be empty. Experiments have shown that the impact of initialization is small. However, uniform initialization scheme (i.e. the first region <NUM>, the neutral region <NUM> and the second region <NUM> initially having the same size) performs better.

A requirement of the optimizing step <NUM> is for the network to be confident about the correct action class of the frames in each timestamp region [pi - li, pi + ri] (the hatched regions in <FIG>). Simultaneously the size of the neutral regions gi (the white regions in the <FIG>) should be as small as possible. In other words, the optimizing is carried out based on a compromise between the size of the neutral region and a match between the pattern estimations of the frames and the regions to which the frames are assigned.

The goal is to first find the appropriate size of the regions of the video ri, gi, li and then use them to construct a pseudo-ground-truth for the network training. In order to do so, the framewise probabilities of the video Ỹ are estimated using the network (obtaining step <NUM>), namely the same network as that which is then to be trained using the pseudo-ground-truth data. In order for the network to be confident about the correct class of the region for timestamp pi, the sum of the negative log probabilities of the frames inside that region, <MAT> should be low. That is, a match between the pattern estimations of the frames and the regions to which the frames are assigned is quantified using a sum of log probabilities of pattern estimations of the frames assigned to the respective regions. Furthermore the sum of the size of the neutral regions, <MAT> should be as small as possible.

This goal can be formulated as the following constrained minimization problem <MAT>.

The hyper-parameter β balances the sizes of neutral regions in the result. The first constraint pi+<NUM> - pi = ri + gi + li+<NUM> ensures that the regions between two timestamps (i.e. the first region <NUM>, the neutral region <NUM> and the second region <NUM>) do not overlap and cover the entire subset. The last three constraints ensure the positivity of the size of the regions. Note that the regions might be empty after optimizing, as opposed to the initializing.

The hyper-parameter β may be set between <NUM> and <NUM>, preferably between <NUM> and <NUM>, preferably between <NUM> and <NUM>. The hyper-parameter β may be set to <NUM>. A sensitivity study shows that lower values for β tend to achieve better results, especially when decreasing the percentage of annotated segments. This is expected since with a lower percentage of annotations more frames should be assigned to the neutral region, which is achieved by lowering the value of β. When there are no missing segments, no frame should be assigned to the ignore region, and higher values of β are better.

This minimization problem (or model which represents the compromise) is not differentiable and may be solved as such by appropriate methods known in the art. In the present embodiment however, following the principles of <CIT> (see also Souri, Yaser & Abu Farha, Yazan & Despinoy, Fabien & Francesca, Gianpiero & Gall, Juergen. FIFA: Fast Inference Approximation for Action Segmentation. , DOI <NUM>/ARXIV. <NUM>, hereinafter "FIFA"), the optimizing step <NUM> comprises solving a differentiable approximation of the non-differentiable model. In other words, this constrained minimization problem is mapped into an approximate unconstrained optimization problem that can be solved using gradient descent.

For instance, the objective function in equation (<NUM>) can be re-written as <MAT> where <IMG>(t|pi - li ≤ t < pi + ri) is the indicator function with value <NUM> if t is within the left and right bounds of the timestamp and <NUM> otherwise. The<IMG> function, which is a non-differentiable function with value of <NUM> or <NUM> depending on its parameters, can be replaced with the plateau function of FIFA: <IMG>(t|pi - li ≤ t < pi + ri) ≃ f(t|ci, wi, s) where <MAT>, <MAT>, and s is the sharpness of the plateau function which is fixed. Besides, the second term, penalizing the ignored regions, can be approximately calculated as the regions not covered by the plateau functions: <MAT>.

Finally, the differentiable approximation can be rewritten as: <MAT> and <MAT> where new parameters r'i, g'i, and l'i, which are the log of the length of the regions relative to each other and can take any real value, may be introduced to make sure that the parameters ri, gi, and li satisfy the original constraints.

This optimization is solved for each video in the training batch separately. This optimization problem is only performed during training and to generate the pseudo-ground-truth data. The objective specified in (<NUM>) is not used for updating the network weights.

After solving the optimization problem, the pseudo-ground-truth data is generated. The pseudo-ground-truth data comprises, for each frame of the first region and/or the second region after the optimizing, identification of said frame and, as a corresponding pattern annotation, the pattern annotation of the first timestamp and/or the second timestamp, respectively. The pattern annotations are determined based on the optimized regions. The pseudo-ground-truth data may comprise, for each frame of the neutral region after the optimizing, identification of said frame and, as a corresponding pattern annotation, an unknown pattern. Otherwise, the frames of the neutral regions may be ignored or not annotated.

This pseudo-ground-truth data may be used for training the action segmentation network as in Li (training step <NUM>).

As can be seen in <FIG>, the presence of ignored frames in the resulting pseudo-ground-truth data <NUM> more accurately captures possibly missing annotations, such as for action B in the initial sequence. Therefore, by explicitly handling missing annotations by ignoring some frames during training, the present approach is robust to missing annotations and experiments have shown that this approach outperforms the approach of Li on various percentages of annotated segments (between <NUM>% and <NUM>%) and on both 50Salads and Breakfast datasets (<NPL>. ), and regardless of how the locations of the annotated timestamps are provided.

While the present approach is defined to handle missing annotations, it also works well when no annotation is missing.

While the present approach has been described in detail in the context of action segmentation, it can also be applied to the action localization task. Following Li, the above-described label (pseudo-ground-truth data) generation approach is used to train an action localization model using the human-annotated timestamps on the GTEA and BEOID datasets provided by <NPL>. Results on both datasets show that the present approach outperforms Li by a large margin of <NUM>% average mAP (mean Average Precision) on the GTEA and <NUM>% mAP on the BEOID dataset. Furthermore, the present approach achieves competitive results to <NPL>, which is specific for action localization.

Claim 1:
A computer-implemented method (<NUM>) for generating pseudo-ground-truth data for training a machine learning model to segment or localize patterns in a sequence of frames over a succession dimension, the method comprising:
- providing (<NUM>) a sequence of frames and a plurality of timestamps (<NUM>), each timestamp including identification of a respective one of the frames and a pattern annotation of said frame;
- defining (<NUM>) at least one subset (<NUM>) comprising the frames located between a first timestamp (pi) and a consecutive second timestamp (pi+<NUM>) of the plurality of timestamps, in the succession dimension;
- initially assigning (<NUM>) the frames of the at least one subset (<NUM>) to one of three non-empty regions, the three regions comprising a first region (<NUM>), a second region (<NUM>) and a neutral region (<NUM>) joining the first region (<NUM>) to the second region (<NUM>), wherein the first region (<NUM>) starts from the first timestamp (pi) and extends towards the second timestamp (pi+<NUM>), and comprises the frames to be associated with the pattern of the first timestamp (pi), the second region (<NUM>) starts from the second timestamp (pi+<NUM>) and extends towards the first timestamp (pi), and comprises the frames to be associated with the pattern of the second timestamp (pi+<NUM>);
- obtaining (<NUM>) pattern estimations for each frame of the at least one subset;
- optimizing (<NUM>) respective boundaries between the first region (<NUM>), the second region (<NUM>) and the neutral region (<NUM>) based on a compromise between the size of the neutral region (<NUM>) and a match between the pattern estimations of the frames and the regions to which the frames are assigned;
- outputting (<NUM>) pseudo-ground-truth data (<NUM>) comprising, for each frame of the first region and/or the second region after the optimizing, identification of said frame and, as a corresponding pattern annotation, the pattern annotation of the first timestamp (pi) and/or the second timestamp (pi+<NUM>), respectively;
wherein the optimizing comprises solving the constrained minimization problem <MAT>
where ri is the number of frames of the first region, gi is the number of frames of the neutral region, li+<NUM> is the number of frames of the second region, N is the number of timestamps, pi is the position of the i-th timestamp in the sequence, ypi is the pattern annotation of the i-th timestamp, ỹt is the pattern estimation for frame t, and β is a hyperparameter.