Patent ID: 11868737
Assignee: DIRECT CURSUS TECHNOLOGY L.L.C
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 17:
18. A computer-implemented method for processing a text sequence for a machine processing task to be performed by a Machine Learning Algorithm (MLA) comprising a neural network, the method executable by a server, the method comprising:
acquiring, by the server, a token vocabulary, the token vocabulary storing a set of tokens from a pre-determined corpus of text, a given token from the set of tokens being one of a single character and a merged set of tokens;
acquiring, by the server, a merge table, the merge table indicating possible mergers between pairs of tokens from the set of tokens, a given token from a given possible merge being associated with a respective frequency of occurrence of the given token in the pre-determined corpus of text;
acquiring, by the server, the text sequence indicative of at least one word;
for a given word from the text sequence:
using, by the server, the token vocabulary for splitting the given word into an initial token sequence, the initial token sequence representing individual characters of the given word;
iteratively merging, by the server, tokens of the initial token sequence to generate a final token sequence for the given word, the iteratively merging including:
at a given merging iteration:
using, by the server, the merge table for identifying a set of possible merges between pairs of adjacent tokens in a current token sequence of the given merging iteration;
using, by the server, a stochastically-driven algorithm for excluding at least one of the set of possible merges,
thereby generating a reduced set of possible merges for the given merging iteration,
the reduced set of possible merges being smaller than the set of possible merges;

using, by the server, the reduced set of possible merges for generating a new token sequence by performing at least one of the reduced set of possible merges in the current token sequence,
the new token sequence to be used by the server as a current token sequence during a next merging iteration,

at an other given merging iteration after the given merging iteration:
in response to no more merges being available for a current token sequence of the other given merging iteration,
identifying, by the server, the current token sequence of the other given merging iteration as the final token sequence to be used for the machine processing task, and
inputting the final token sequence to the neural network.