Patent ID: 11966699
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 7:
8. A method for classifying a language sample, the method comprising:
training, by one or more computer processors, a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
extracting language characteristics for each language sample, determining a probability of using white-space tokenizer (PWST) score for each language,
training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
receiving, by the one or more computer processors, a first language sample comprising a set of features;
identifying, by the one or more computer processors, language sample features of the first language sample;
determining, by the one or more computer processors, a tokenization score for the first language sample according to the language sample features;
determining, by the one or more computer processors, a term frequency (tf) according to the identified language sample features and the tokenization score;
determining, by the one or more computer processors, an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
generating, by the one or more computer processors, a revised term frequency—inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf and the idf;
receiving, by the one or more computer processors, input text;
identifying, by the one or more computer processors, tokens in the input text; and
translating, by the one or more computer processors, the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.