Package: yafasttext
fastTextis a library for efficient learning
of word representations and sentence classification.RequirementsfastTextbuilds on modern Mac OS and Linux
distributions. Since it uses C++11 features, it requires a compiler with
good C++11 support. These include :(gcc-4.8 or newer) or (clang-3.3 or newer)You will needPythonversion 2.7 or >=3.4NumPy&SciPypybind11Building fastTextThe easiest way to get the latest version offastText is to use
pip.$ pip install fasttextIf you want to use the latest unstable release you will need to build
from source using setup.py.Now you can import this library withimport fastTextExamplesIn general it is assumed that the reader already has good knowledge of
fastText. For this consider the mainREADMEand in particularthe tutorials on our
website.We recommend you look at theexamples within the doc
folder.As with any package you can get help on any Python function using the
help function.For example+>>> import fastText
+>>> help(fastText.FastText)

Help on module fastText.FastText in fastText:

NAME
    fastText.FastText

DESCRIPTION
    # Copyright (c) 2017-present, Facebook, Inc.
    # All rights reserved.
    #
    # This source code is licensed under the BSD-style license found in the
    # LICENSE file in the root directory of this source tree. An additional grant
    # of patent rights can be found in the PATENTS file in the same directory.

FUNCTIONS
    load_model(path)
        Load a model given a filepath and return a model object.

    tokenize(text)
        Given a string of text, tokenize it and return a list of tokens
[...]IMPORTANT: Preprocessing data / enconding conventionsIn general it is important to properly preprocess your data. In
particular our example scripts in theroot
folderdo this.fastText assumes UTF-8 encoded text. All text must beunicode for
Python2andstr for
Python3.
The passed text will beencoded as UTF-8 by
pybind11before passed to the fastText C++ library. This means it is important to
use UTF-8 encoded text when building a model. On Unix-like systems you
can convert text usingiconv.fastText will tokenize (split text into pieces) based on the following
ASCII characters (bytes). In particular, it is not aware of UTF-8
whitespace. We advice the user to convert UTF-8 whitespace / word
boundaries into one of the following symbols as appropiate.spacetabvertical tabcarriage returnformfeedthe null characterThe newline character is used to delimit lines of text. In particular,
the EOS token is appended to a line of text if a newline character is
encountered. The only exception is if the number of tokens exceeds the
MAX_LINE_SIZE constant as defined in theDictionary
header.
This means if you have text that is not separate by newlines, such as
thefil9 dataset, it will be
broken into chunks with MAX_LINE_SIZE of tokens and the EOS token is
not appended.The length of a token is the number of UTF-8 characters by considering
theleading two bits of a
byteto identifysubsequent bytes of a multi-byte
sequence.
Knowing this is especially important when choosing the minimum and
maximum length of subwords. Further, the EOS token (as specified in theDictionary
header)
is considered a character and will not be broken into subwords.