Patent ID: 11877016
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 4:
5. The device according to claim 4, wherein the multi-modality cross-matching layer comprises:
a first text attention processing unit configured to calculate bidirectional attention data between feature data of the candidate text and feature data of the image modality, feature data of the sound modality and feature data of the existing text, and superimpose the three bidirectional attention data together to obtain cross-attention data of the candidate text;
a second text attention processing unit configured to calculate bidirectional attention data between the feature data of the existing text and the feature data of the candidate text, the feature data of the image modality and the feature data of the sound modality, and superimpose the three bidirectional attention data to obtain cross-attention data of the existing text;
an image attention processing unit configured to calculate bidirectional attention data between the feature data of an image modality and the feature data of the sound modality, feature data of the candidate text, the feature data of the existing text, and superimpose the three bidirectional attention data to obtain cross-attention data of the image modality;
a sound attention processing unit configured to calculate bidirectional attention data between the feature data of the sound modality and the feature data of the image modality, the feature data of the candidate text, the feature data of the existing text, and superimpose the two bidirectional attention data to obtain the cross-attention data of the sound modality.