Patent ID: 11915500
Assignee: SALESFORCE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 7:
8. A computer implemented method for performing scene text recognition, the method comprising:
receiving a training dataset for training a machine learning based model, the training dataset comprising a plurality of samples, each sample representing an image comprising a scene with embedded text based on a set of characters, the machine learning based model configured to receive an input image and predict a sequence of characters in the input image, wherein the machine learning based model comprising:
a convolutional neural network component configured to receive the image to generate visual features and multi-scale visual features, the visual features represented as a feature vector of lower dimension than the image,
an attention extractor neural network component configured to:
receive the multi-scale visual features output by the convolutional neural network,
summarize the multi-scale visual features into summarized visual features,
generate attention features based on the summarized visual features, each attention feature corresponding to an attention map for a visual feature, and
add attention maps with corresponding attention features to generate added attention features, and

a character recognizer neural network component configured to receive mixed features generated by combining the visual features and the added attention features to output a sequence of characters representing predicted text recognized from the image;

for each character in the set of characters, determining a measure of difficulty of classifying the character;
training the machine learning based model using the training dataset, the training adjusting parameters of the machine learning based model to minimize a loss value for samples of the training dataset, wherein a loss value for a sample is weighted according to the measure of difficulty assigned to the characters of the sample; and
predicting text in a new input image using the trained machine learning based model.