Patent ID: 11893348
Assignee: ROYAL BANK OF CANADA
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A system for training a sequence to sequence (S2S) machine learning model for predicting keywords, the system comprising:
at least one computer memory having stored thereon the S2S machine learning model, the S2S machine learning model comprising a plurality of parameters representative of a decoder, the decoder including a generation data model architecture and a copy data model architecture;
at least one processor, in communication with the at least one computer memory, configured to:
receive a first data set comprising a plurality of source token sets and related ground truth token sets;
extract a second data set of target vocabulary tokens from the first data set comprising a subset of source tokens and related ground truth tokens of the first data set;
train the S2S machine learning model for predicting keywords by, for each source token in a first source token set:
processing, with the decoder, a first source token set encoder representation and a respective source token encoder representation to generate a predicted keyword, wherein processing with the decoder comprises:
processing the first source token set encoder representation and a previous ground truth token embedding to generate a hidden state;
generating a first set keyword probability distribution of the copy data model architecture based on normalizing, over the source tokens in the first source token set, an attention mechanism interrelation value between the respective source token encoder representation and the hidden state;
generating a second set keyword probability distribution of the generation data model architecture based on normalizing, over the target vocabulary tokens, the attention mechanism interrelation value;
determining a probability of generating a keyword from the second data set based on a vocabulary token parameter processing the hidden state, a related ground truth token, and the vocabulary normalized attention mechanism interrelation value;
generating a probability of generating the keyword from the first source token set based on the probability of generating the keyword from the second data set; and
generating a predicted keyword based on applying the probability of generating the keyword from the second data set to the second set keyword probability distribution and applying the probability of generating the keyword from the first token source set to the first set keyword probability distribution;

updating the plurality of parameters by:
determining a generation loss based on comparing the predicted keyword to a first exclusion list of ground truth tokens;
determining a copy loss based on comparing the predicted keyword to a second exclusion list of source tokens and ground truth tokens; and
adjusting the plurality of parameters based on the copy loss, the generation loss, and a comparison of the predicted keyword and a respective predicted keyword ground truth token to penalize the decoder for generating repetitive keywords; and

store the trained S2S machine learning model for predicting keywords in the at least one computer memory.