Patent Document ID: 20160299885
Application ID: 14683504
Patent Status: 0

Claim One:
1. A method for recombining incorrectly separated tokens in Natural Language Processing (NLP), the method comprising: determining whether a token from an ordered set of tokens is present in a dictionary, the dictionary being related to a corpus from which the ordered set of tokens is extracted; determining whether a compounding threshold has been reached, wherein the compounding threshold limits a number of tokens that can be agglutinated to form a compound token; agglutinating, using a processor and a memory, responsive to the token not being present in the dictionary, and responsive to the compounding threshold not having been reached, the token with a next adjacent token in the ordered set of tokens to form the compound token; determining whether the compound token is present in the dictionary; assigning a weight to the compound token responsive to the compound token being present in the dictionary; computing a confidence rating of the compound token, the confidence rating being a function of the weight; and using the compound token and the confidence rating in performing an NLP operation on the corpus.