Patent ID: 11886815
Assignee: ADOBE INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 9:
10. A system comprising:
one or more processing devices; and
a non-transitory computer-readable medium communicatively coupled to the one or more processing devices and storing instructions, wherein the one or more processing devices are configured to execute the instructions and thereby perform operations comprising:
receiving pre-training data comprising a plurality of documents;
pre-training a machine learning model by:
obtaining the pre-training data comprising the plurality of documents;
extracting visual features from each of the plurality of documents;
extracting textual features from each of the plurality of documents; and
generating, using an encoder, feature embeddings for each of the plurality of documents by applying a masking function to the visual features and the textual features, wherein generating the feature embeddings comprises:
generating weighted visual features by applying a first modality-specific attention weight to the visual features;
generating weighted textual features by applying a second modality-specific attention weight to the textual features; and
fusing the weighted visual features and the weighted textual features, wherein fusing the weighted visual features and the weighted textual features comprises multiplying the weighted visual features and the weighted textual features;

generating, using the feature embeddings, output features for each of the plurality of documents by (i) concatenating the feature embeddings and (ii) applying a non-linear mapping to the feature embeddings; and
training the machine learning model to generate a search result in response to a search query by applying a linear classifier to the output features.