Patent ID: 11877016
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 5:
6. The device according to claim 1, wherein the multi-modality cross-matching layer comprises attention processing units corresponding to modalities respectively and an attention processing unit corresponding to the candidate text, and each attention processing unit comprises: a self-attention processing module and a cross-attention processing module,
the self-attention processing module is configured to perform self-attention calculation processing on the feature data of each modality and the feature data of the candidate text to generate corresponding self-attention data,
the cross-attention processing module is configured to calculate a plurality of bidirectional attention data between self-attention data of each modality and the self-attention data of other modalities and the self-attention data of the candidate text with respect to each modality, and superimpose the plurality of bidirectional attention data together, to obtain cross-attention data between the modality and each of other modalities and candidate texts, and calculate bidirectional attention data between the self-attention data of the candidate text and the self-attention data of each modality with respect to the candidate text, and superimpose the plurality of bidirectional attention data together, to obtain cross-attention data between the candidate text and each modality.