Patent ID: 11886446
Assignee: BAIDU USA LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 14:
15. A computer-implemented method for pretraining a cross-lingual language model, the method comprising:
using a pretrained multilingual language representation model to initialize a cross-lingual language model that uses global-plus-sliding window attention, in which a global window for attention is used for query tokens and a sliding window for attention is used for at least some document tokens;
responsive to a stop condition not being reached, performing steps comprising:
given cross-lingual query and document pairs in which at least some of query tokens of a query input into the cross-lingual language model are masked, training the cross-lingual language model using a query language model objective regarding predicting, for a query, the masked query tokens given a corresponding relevant cross-lingual document; and
given cross-lingual query and document pairs, training the cross-lingual language model with a relevance ranking model head to predict a relevance ranking that ranks whether a document is relevant to a corresponding query; and

responsive to a stop condition being reached, outputting the cross-lingual language model, which has been updated, as a pretrained cross-lingual language model.