Patent ID: 11966699
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A computer system comprising computer processors to implement a method for classifying a language sample, the computer system comprising:
one or more computer processors;
one or more computer readable storage devices;
stored program instructions on one or more computer readable storage devices for execution by the one or more computer processors, the stored program instructions comprising:
program instructions to train a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
extracting language characteristics for each language sample,
determining a probability of using white-space tokenizer (PWST) score for each language,
training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
program instructions to receive a first language sample comprising a set of features;
program instructions to identify language sample features of the first language sample;
program instructions to determine a tokenization score for the first language sample according to the language sample features;
program instructions to determine a term frequency (tf) according to the identified language sample features and the tokenization score;
program instructions to determine an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
program instructions to generate a revised term frequency-inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf, and the idf;
program instructions to receive input text;
program instructions to identify tokens in the input text; and
program instructions to translate the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.