Ngrams for each n-tuple will be automatically created and cumulated. We support a number of different techniques to deal with lack of data in a training set. The test data may be in any of the formats described above. When binary, both daughters must be non-terminals and when unary, the daughter must be a terminal. The implementation here is primarily based on Pereira and Schabes 92 thus allowing unsupervised training of SCFGs as well as allowing seeding with a bracketed corpus which can vastly reduce training time, and improve results.
Training uses the inside-outside algorithm.
A grammar file consists of a set of rules. Each rule is a bracketed list of probability, nonterminal, followed by two nonterminals or one terminal. The mother non-terminal in the first rule is the distinguished symbol. The rules can be assigned equal probabilities or random ones. The non-terminals or terminal names will be automatically generated if a number is given, or will be as specified if a file name is given.