Patent ID: 11877016
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 0:
1. A device comprising:
a feature extracting layer configured to extract feature data of a plurality of modalities in a video segment and extract feature data of each candidate text in a set of candidate texts, wherein the plurality of modalities includes at least an image modality and a sound modality of the video segment;
a multi-modality cross-matching layer configured to perform the following processing:
calculating a first plurality of bidirectional attention data between feature data of the image modality, feature data of the sound modality, and the feature data of the candidate text with respect to each modality, and superimposing the first plurality of bidirectional attention data together, to obtain first cross-attention data between the image modality and the sound modality the modality and a comment paragraph;
calculating a second plurality of bidirectional attention data between the feature data of the sound modality and the feature data of the image modality and the feature data of the candidate text with respect to each modality, and superimposing the second plurality of bidirectional attention data together, to obtain second cross-attention data between the sound modality and the image modality and the comment paragraph; and
calculating a third plurality of bidirectional attention data between the feature data of the candidate text and the feature data of the image modality and the feature data of the sound modality with respect to the candidate text and superimposing the third plurality of bidirectional attention data together to obtain third cross-attention data between the candidate text and the image modality and the sound modality;

a similarity processing layer configured to calculate a similarity between the video segment and the candidate text according to the first cross-attention data of the image modality, the second cross-attention data of the sound modality, and the third cross-attention data of the candidate text; and
an outputting layer configured to acquire a plurality of similarities between the video segment and each candidate text in the set of candidate texts, and performing ranking on the similarities, and determine one or more candidate texts as live comments of the video segment according to a ranking result of the similarities, wherein the plurality of similarities are obtained by separate processing on the video segment and each candidate text in the set of candidate texts by the feature extracting layer, the multi-modality cross-matching layer, and the similarity processing layer.