Patent ID: 11966699
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 14:
15. A computer program product for classifying a language sample, the computer program product comprising one or more computer readable storage devices and collectively stored program instructions on the one or more computer readable storage devices, the stored program instructions comprising:
program instructions to train a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
extracting language characteristics for each language sample,
determining a probability of using white-space tokenizer (PWST) score for each language,
training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
program instructions to receive a first language sample comprising a set of features; program instructions to identify language sample features of the first language sample;
program instructions to determine a tokenization score for the first language sample according to the language sample features;
program instructions to determine a term frequency (tf) according to the identified language sample features and the tokenization score;
program instructions to determine an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
program instructions to generate a revised term frequency—inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf, and the idf;
program instructions to receive input text;
program instructions to identify tokens in the input text; and
program instructions to translate the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.