Package: fasttext
fastTextfastTextis a library for efficient learning
of word representations and sentence classification.In this document we present how to use fastText in python.Table of contentsRequirementsInstallationUsage overviewWord representation modelText classification modelIMPORTANT: Preprocessing data / encoding
conventionsMore examplesAPItrain_unsupervised parameterstrain_supervised parametersmodel objectRequirementsfastTextbuilds on modern Mac OS and Linux
distributions. Since it uses C++11 features, it requires a compiler with
good C++11 support. You will needPython(version 2.7 or ≥ 3.4),NumPy&SciPyandpybind11.InstallationTo install the latest release, you can do :$pipinstallfasttextor, to get the latest development version of fasttext, you can install
from our github repository :$gitclonehttps://github.com/facebookresearch/fastText.git$cdfastText$sudopipinstall.$# or :$sudopythonsetup.pyinstallUsage overviewWord representation modelIn order to learn word vectors, asdescribed
here,
we can usefasttext.train_unsupervisedfunction like this:importfasttext# Skipgram model :model=fasttext.train_unsupervised('data.txt',model='skipgram')# or, cbow model :model=fasttext.train_unsupervised('data.txt',model='cbow')wheredata.txtis a training file containing utf-8 encoded text.The returnedmodelobject represents your learned model, and you can
use it to retrieve information.print(model.words)# list of words in dictionaryprint(model['king'])# get the vector of the word 'king'Saving and loading a model objectYou can save your trained model object by calling the functionsave_model.model.save_model("model_filename.bin")and retrieve it later thanks to the functionload_model:model=fasttext.load_model("model_filename.bin")For more information about word representation usage of fasttext, you
can refer to ourword representations
tutorial.Text classification modelIn order to train a text classifier using the methoddescribed
here,
we can usefasttext.train_supervisedfunction like this:importfasttextmodel=fasttext.train_supervised('data.train.txt')wheredata.train.txtis a text file containing a training sentence
per line along with the labels. By default, we assume that labels are
words that are prefixed by the string__label__Once the model is trained, we can retrieve the list of words and labels:print(model.words)print(model.labels)To evaluate our model by computing the precision at 1 (P@1) and the
recall on a test set, we use thetestfunction:defprint_results(N,p,r):print("N\t"+str(N))print("P@{}\t{:.3f}".format(1,p))print("R@{}\t{:.3f}".format(1,r))print_results(*model.test('test.txt'))We can also predict labels for a specific text :model.predict("Which baking dish is best to bake a banana bread ?")By default,predictreturns only one label : the one with the
highest probability. You can also predict more than one label by
specifying the parameterk:model.predict("Which baking dish is best to bake a banana bread ?",k=3)If you want to predict more than one sentence you can pass an array of
strings :model.predict(["Which baking dish is best to bake a banana bread ?","Why not put knives in the dishwasher?"],k=3)Of course, you can also save and load a model to/from a file asin the
word representation usage.For more information about text classification usage of fasttext, you
can refer to ourtext classification
tutorial.Compress model files with quantizationWhen you want to save a supervised model file, fastText can compress it
in order to have a much smaller model file by sacrificing only a little
bit performance.# with the previously trained `model` object, call :model.quantize(input='data.train.txt',retrain=True)# then display results and save the new model :print_results(*model.test(valid_data))model.save_model("model_filename.ftz")model_filename.ftzwill have a much smaller size thanmodel_filename.bin.For further reading on quantization, you can refer tothis paragraph
from our blog
post.IMPORTANT: Preprocessing data / encoding conventionsIn general it is important to properly preprocess your data. In
particular our example scripts in theroot
folderdo this.fastText assumes UTF-8 encoded text. All text must beunicode for
Python2andstr for
Python3.
The passed text will beencoded as UTF-8 by
pybind11before passed to the fastText C++ library. This means it is important to
use UTF-8 encoded text when building a model. On Unix-like systems you
can convert text usingiconv.fastText will tokenize (split text into pieces) based on the following
ASCII characters (bytes). In particular, it is not aware of UTF-8
whitespace. We advice the user to convert UTF-8 whitespace / word
boundaries into one of the following symbols as appropiate.spacetabvertical tabcarriage returnformfeedthe null characterThe newline character is used to delimit lines of text. In particular,
the EOS token is appended to a line of text if a newline character is
encountered. The only exception is if the number of tokens exceeds the
MAX_LINE_SIZE constant as defined in theDictionary
header.
This means if you have text that is not separate by newlines, such as
thefil9 dataset, it will be
broken into chunks with MAX_LINE_SIZE of tokens and the EOS token is
not appended.The length of a token is the number of UTF-8 characters by considering
theleading two bits of a
byteto identifysubsequent bytes of a multi-byte
sequence.
Knowing this is especially important when choosing the minimum and
maximum length of subwords. Further, the EOS token (as specified in theDictionary
header)
is considered a character and will not be broken into subwords.More examplesIn order to have a better knowledge of fastText models, please consider
the mainREADMEand in particularthe tutorials on our
website.You can find further python examples inthe doc
folder.As with any package you can get help on any Python function using the
help function.For example+>>> import fasttext
+>>> help(fasttext.FastText)

Help on module fasttext.FastText in fasttext:

NAME
    fasttext.FastText

DESCRIPTION
    # Copyright (c) 2017-present, Facebook, Inc.
    # All rights reserved.
    #
    # This source code is licensed under the MIT license found in the
    # LICENSE file in the root directory of this source tree.

FUNCTIONS
    load_model(path)
        Load a model given a filepath and return a model object.

    tokenize(text)
        Given a string of text, tokenize it and return a list of tokens
[...]APItrain_unsupervisedparametersinput# training file path (required)model# unsupervised fasttext model {cbow, skipgram} [skipgram]lr# learning rate [0.05]dim# size of word vectors [100]ws# size of the context window [5]epoch# number of epochs [5]minCount# minimal number of word occurences [5]minn# min length of char ngram [3]maxn# max length of char ngram [6]neg# number of negatives sampled [5]wordNgrams# max length of word ngram [1]loss# loss function {ns, hs, softmax, ova} [ns]bucket# number of buckets [2000000]thread# number of threads [number of cpus]lrUpdateRate# change the rate of updates for the learning rate [100]t# sampling threshold [0.0001]verbose# verbose [2]train_supervisedparametersinput# training file path (required)lr# learning rate [0.1]dim# size of word vectors [100]ws# size of the context window [5]epoch# number of epochs [5]minCount# minimal number of word occurences [1]minCountLabel# minimal number of label occurences [1]minn# min length of char ngram [0]maxn# max length of char ngram [0]neg# number of negatives sampled [5]wordNgrams# max length of word ngram [1]loss# loss function {ns, hs, softmax, ova} [softmax]bucket# number of buckets [2000000]thread# number of threads [number of cpus]lrUpdateRate# change the rate of updates for the learning rate [100]t# sampling threshold [0.0001]label# label prefix ['__label__']verbose# verbose [2]pretrainedVectors# pretrained word vectors (.vec file) for supervised learning []modelobjecttrain_supervised,train_unsupervisedandload_modelfunctions return an instance of_FastTextclass, that we generaly
namemodelobject.This object exposes those training arguments as properties :lr,dim,ws,epoch,minCount,minCountLabel,minn,maxn,neg,wordNgrams,loss,bucket,thread,lrUpdateRate,t,label,verbose,pretrainedVectors.
Somodel.wordNgramswill give you the max length of word ngram used
for training this model.In addition, the object exposes several functions :get_dimension# Get the dimension (size) of a lookup vector (hidden layer).# This is equivalent to `dim` property.get_input_vector# Given an index, get the corresponding vector of the Input Matrix.get_input_matrix# Get a copy of the full input matrix of a Model.get_labels# Get the entire list of labels of the dictionary# This is equivalent to `labels` property.get_line# Split a line of text into words and labels.get_output_matrix# Get a copy of the full output matrix of a Model.get_sentence_vector# Given a string, get a single vector represenation. This function# assumes to be given a single line of text. We split words on# whitespace (space, newline, tab, vertical tab) and the control# characters carriage return, formfeed and the null character.get_subword_id# Given a subword, return the index (within input matrix) it hashes to.get_subwords# Given a word, get the subwords and their indicies.get_word_id# Given a word, get the word id within the dictionary.get_word_vector# Get the vector representation of word.get_words# Get the entire list of words of the dictionary# This is equivalent to `words` property.is_quantized# whether the model has been quantizedpredict# Given a string, get a list of labels and a list of corresponding probabilities.quantize# Quantize the model reducing the size of the model and it's memory footprint.save_model# Save the model to the given pathtest# Evaluate supervised model using file given by pathtest_label# Return the precision and recall score for each label.The propertieswords,labelsreturn the words and labels from
the dictionary :model.words# equivalent to model.get_words()model.labels# equivalent to model.get_labels()The object overrides__getitem__and__contains__functions in
order to return the representation of a word and to check if a word is
in the vocabulary.model['king']# equivalent to model.get_word_vector('king')'king'inmodel# equivalent to `'king' in model.get_words()`Join the fastText communityFacebook pageStack
overflowGoogle
groupGitHub