Patent ID: 11877016
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 14:
15. An electronic apparatus, comprising:
a processing unit; and
a memory coupled to the processing unit and comprising instructions stored thereon, the instructions, when executed by the processing unit, causing the electronic apparatus to perform operations, the operations comprising:
extracting feature data of a plurality of modalities in a video segment and extracting feature data of each candidate text in a set of candidate texts, wherein the plurality of modalities includes at least an image modality and a sound modality of the video segment;
calculating a first plurality of bidirectional attention data between feature data of the image modality and feature data the sound modality, and the feature data of the candidate text with respect to each modality, and superimposing the first plurality of bidirectional attention data together, to obtain first cross-attention data between the image modality and the sound modality and the comment paragraph;
calculating a second plurality of bidirectional attention data between the feature data of the sound modality and the feature data of the image modality and the feature data of the candidate text with respect to each modality, and superimposing the second plurality of bidirectional attention data together, to obtain second cross-attention data between the sound modality and the image modality and the comment paragraph;
calculating a third plurality of bidirectional attention data between the feature data of the candidate text and the feature data of the image modality and the feature data of the sound modality with respect to the candidate text and superimposing the third plurality of bidirectional attention data together to obtain third cross-attention data between the candidate text and the image modality and the sound modality;
calculating a similarity between the video segment and the candidate text according to the first cross-attention data of the image modality, the second cross-attention data of the sound modality, and the third cross-attention data of the candidate text; and
performing ranking on the similarities between the video segment and each candidate text in the set of candidate texts, and determining one or more candidate texts as the live comments of the video segment according to the ranking result of the similarities, wherein the plurality of similarities are obtained by separate processing on the video segment and each candidate text in the set of candidate texts by a feature extracting layer, a multi-modality cross-matching layer, and a similarity processing layer.