Package: extremetext
extremeTextis an
extension offastTextlibrary for multi-label classification including extreme cases with
hundreds of thousands and millions of labels.extremeTextimplements:Probabilistic Labels Tree (PLT) loss for extreme multi-Label
classification with top-down hierarchical clustering (k-means) for
tree building,sigmoid loss for multi-label classification,L2 regularization and FOBOS update for all losses,ensemble of loss layers with bagging,calculation of hidden (document) vector as a weighted average of the
word vectors,calculation of TF-IDF weights for words.RequirementsextremeTextbuilds on
modern Mac OS and Linux distributions. Since it uses C++11 features, it
requires a compiler with good C++11 support. These include:(gcc-4.8 or newer) or (clang-3.3 or newer)You will need:Pythonversion 2.7 or >=3.4NumPy&SciPypybind11Installing extremeTextThe easiest way to getextremeTextis to usepip.$ pip install extremetextInstalling on MacOS may require settingMACOSX_DEPLOYMENT_TARGET=10.9first:$ export MACOSX_DEPLOYMENT_TARGET=10.9
$ pip install extremetextThe latest version ofextremeTextcan be build
from sources using pip or alternatively setuptools.$ git clone https://github.com/mwydmuch/extremeText.git
$ cd extremeText
$ pip install .
(or) $ python setup.py installNow you can import this library with:import extremeTextExamplesIn general it is assumed that the reader already has good knowledge of
fastText/extremeText. For this consider the mainREADMEandthe tutorials on fastText
website.We recommend you look at theexamples within the doc
folder.As with any package you can get help on any Python function using the
help function.For example:+>>> import extremeText
+>>> help(extremeText.ExtremeText)

Help on module extremeText.ExtremeText in extremeText:

NAME
    extremeText.ExtremeText

DESCRIPTION
    # Copyright (c) 2017-present, Facebook, Inc.
    # All rights reserved.
    #
    # This source code is licensed under the BSD-style license found in the
    # LICENSE file in the root directory of this source tree. An additional grant
    # of patent rights can be found in the PATENTS file in the same directory.

FUNCTIONS
    load_model(path)
        Load a model given a filepath and return a model object.

    tokenize(text)
        Given a string of text, tokenize it and return a list of tokens
[...]IMPORTANT: Preprocessing data / enconding conventionsIn general it is important to properly preprocess your data. Example
scripts in theroot
folderdo this.extremeText like fastText assumes UTF-8 encoded text. All text must beunicode for
Python2andstr for
Python3.
The passed text will beencoded as UTF-8 by
pybind11before passed to the extremeText C++ library. This means it is important
to use UTF-8 encoded text when building a model. On Unix-like systems
you can convert text usingiconv.extremeText will tokenize (split text into pieces) based on the
following ASCII characters (bytes). In particular, it is not aware of
UTF-8 whitespace. We advice the user to convert UTF-8 whitespace / word
boundaries into one of the following symbols as appropiate.spacetabvertical tabcarriage returnformfeedthe null characterThe newline character is used to delimit lines of text. In particular,
the EOS token is appended to a line of text if a newline character is
encountered. The only exception is if the number of tokens exceeds the
MAX_LINE_SIZE constant as defined in theDictionary
header.
This means if you have text that is not separate by newlines, such as
thefil9 dataset, it will be
broken into chunks with MAX_LINE_SIZE of tokens and the EOS token is
not appended.The length of a token is the number of UTF-8 characters by considering
theleading two bits of a
byteto identifysubsequent bytes of a multi-byte
sequence.
Knowing this is especially important when choosing the minimum and
maximum length of subwords. Further, the EOS token (as specified in theDictionary
header)
is considered a character and will not be broken into subwords.ReferencePlease cite below work if using this package for extreme classification.M. Wydmuch, K. Jasinska, M. Kuznetsov, R. Busa-Fekete, K. Dembczyński.*A no-regret generalization of hierarchical softmax to extreme
multi-label
classification*.
Advances in Neural Information Processing Systems 31, 2018.