Package: gptc
GPTCGeneral-purpose text classifier in PythonGPTC provides both a CLI tool and a Python library.Installationpip install gptcCLI ToolClassifying textgptc classify [-n <max_ngram_length>] <compiled model file>This will prompt for a string and classify it, then print (in JSON) a dict of
the format{category: probability, category:probability, ...}to stdout. (For
information about-n <max_ngram_length>, see section "Ngrams.")Checking individual words or ngramsgptc check <compiled model file> <token or ngram>This is very similar togptc classify, except it takes the input as an
argument, and it treats the input as a single token or ngram.Compiling modelsgptc compile [-n <max_ngram_length>] [-c <min_count>] <raw model file> <compiled model file>This will write the compiled model encoded in binary format to<compiled model file>.If-cis specified, words and ngrams used less thanmin_counttimes will be
excluded from the compiled model.Packing modelsgptc pack <dir>This will print the raw model in JSON to stdout. Seemodels/unpacked/for an
example of the format. Any exceptions will be printed to stderr.LibraryModel.serialize(file)Write binary data representing the model tofile.gptc.deserialize(encoded_model)Deserialize aModelfrom a file containing data fromModel.serialize().Model.confidence(text, max_ngram_length)Classifytext. Returns a dict of the format{category: probability, category:probability, ...}Note that this may not include values for all categories. If there are no
common words between the input and the training data (likely, for example, with
input in a different language from the training data), an empty dict will be
returned.For information aboutmax_ngram_length, see section "Ngrams."Model.get(token)Return a confidence dict for the given token or ngram. This function is very
similar toModel.confidence(), except it treats the input as a single token
or ngram.gptc.compile(raw_model, max_ngram_length=1, min_count=1, hash_algorithm="sha256")Compile a raw model (as a list, not JSON) and return the compiled model (as agptc.Modelobject).For information aboutmax_ngram_length, see section "Ngrams."Words or ngrams used less thanmin_counttimes throughout the input text are
excluded from the model.The hash algorithm should be left as the default, which may change with a minor
version update, but it can be changed by the application if needed. It is
stored in the model, so changing the algorithm does not affect compatibility.
The following algorithms are supported:md5sha1sha224sha256sha384sha512sha3_224sha3_384sha3_256sha3_512shake_128shake_256blake2bblake2sgptc.pack(directory, print_exceptions=False)Pack the model indirectoryand return a tuple of the format:(raw_model, [(exception,),(exception,)...])Note that the exceptions are contained in single-item tuples. This is to allow
more information to be provided without breaking the API in future versions of
GPTC.Seemodels/unpacked/for an example of the format.gptc.Classifier(model, max_ngram_length=1)Classifierobjects are deprecated starting with GPTC 3.1.0, and will be
removed in 4.0.0. Seethe README from
3.0.2if you need
documentation.NgramsGPTC optionally supports using ngrams to improve classification accuracy. They
are disabled by default (maximum length set to 1) for performance reasons.
Enabling them significantly increases the time required both for compilation
and classification. The effect seems more significant for compilation than for
classification. Compiled models are also much larger when ngrams are enabled.
Larger maximum ngram lengths will result in slower performance and larger
files. It is a good idea to experiment with different values and use the
highest one at which GPTC is fast enough and models are small enough for your
needs.Once a model is compiled at a certain maximum ngram length, it cannot be used
for classification with a higher value. If you instantiate aClassifierwith
a model compiled with a lowermax_ngram_length, the value will be silently
reduced to the one used when compiling the model.Model formatThis section explains the raw model format, which is how models are created and
edited.Raw models are formatted as a list of dicts. See below for the format:[
    {
        "text": "<text in the category>",
        "category": "<the category>"
    }
]GPTC handles raw models aslists ofdicts ofstrs (List[Dict[str, str]]), and they can be stored in any way these Python objects can be.
However, it is recommended to store them in JSON format for compatibility with
the command-line tool.EmojiGPTC treats individual emoji as words.Example modelAn example model, which is designed to distinguish between texts written by
Mark Twain and those written by William Shakespeare, is available inmodels.
The raw model is inmodels/raw.json; the compiled model is inmodels/compiled.json.The example model was compiled withmax_ngram_length=10.BenchmarkA benchmark script is available for comparing performance of GPTC between
different Python versions. To use it, runbenchmark.pywith all of the Python
installations you want to test. It tests both compilation and classification.
It uses the default Twain/Shakespeare model for both, and for classification it
usesMark Antony's "Friends, Romans, countrymen"
speechfrom Shakespeare'sJulius Caesar.