Patent ID: 11902628
Assignee: HULU, LLC
Field: Audio-visual technology (Electrical engineering)
Classification: CPC H  G | IPC G  H

Claim 0:
1. A method comprising:
receiving, by a computing device, a first sequence of inputs for processing via a sub-model of a plurality of sub-models, wherein the plurality of sub-models are part of a main model;
masking, by the computing device, an input in the first sequence of inputs with a masked value to generate a second sequence of inputs;
processing, by the computing device, the second sequence of inputs using the sub-model to generate a sequence of features that correspond to the second sequence of inputs, wherein the sub-model comprises a self-attention sub-model that applies attention to inputs in the second sequence of inputs based on relationships between the inputs;
generating, by the computing device, a first output of the main model based on the sequence of features; and
training, by the computing device, the sub-model based on a feature in the sequence of features that corresponds to the masked input and the first output, wherein training the sub-model comprises:
generating a first value based on a feature in the sequence of features that corresponds to the masked input;
generating a second value based on the first output; and
adjusting a parameter of the sub-model using the first value and the second value.