fastText and Tensorflow to perform NLP classification | by Antoine Hue | Towards Data Science

fastText is a state-of-the-art open-source library released in 2017 by Facebook to compute word embeddings or create text classifiers. However, embeddings and classifiers are only building blocks within a data-science job. There are many preparation tasks before and validation tasks after, and there are many candidate architectures based on these tools. Let’s work out a full use case based on the data of the community site for mountaineering sports, CampToCamp.org. As a deep learning engine, we will use Tensorflow 2 / Keras. As for the validation, we will use UMAP, a projection tool from high dimension manifolds onto the 2D plane.
www.camptocamp.org is a community site for mountain sports. It is a lively source of information and has become a reference for French and Swiss alps. The site has four main sections: a knowledge base of waypoints and routes, a live feed of outings, accident reports, and a forum.
Our task is based on the route data: given the textual description, design and train a classifier to tag the type of activity related to the route among ten possibilities: hiking, mountain biking, ski touring, snowshoeing, four types of climbing (rock, ice, mountain and mixed), via-ferrata and slack-lining. All flying related activities (paragliding, base-jump...) are not listed as routes, and downhill/resort skiing is not the purpose of the site.
Each route can be tagged with more than one activity if it is a combination, or if it has a winter and a summer variant. There are up to three activity tags per route. As the tags are set, this is a supervised training.
We will use the route descriptions from www.camptocamp.org restricted to the central-north French Alps (administrative regions of Hautes-Alpes, Isère, Savoie, Haute Savoie). As Camp To Camp is international, we want to focus on regions in which the French versions of the description is probably long and accurate.
www.camptocamp.org has an open Web API [5], it is not documented but it is quite easy to understand looking through the XHR requests from Web pages.
In total, 14'074 routes are extracted. Rich text features (HTML) are removed. 610 are dropped because they have no French description. Before computing the embeddings with either fastText or Tensorflow, the punctuation is stripped down.
More data cleaning and eviction could be done, e.g. very short descriptions or descriptions in English (language mismatch), but it is a parti pris to go forward with this data quality.
Embedding representation of texts, also called vectorization, is very common in natural language processing (NLP) initiated in 2013 by Mikolov et al. [3]. It has been popularized under the term Word2Vec. There are several famous implementations and pre-trained models, for example GloVe [4]. fastText is one of the most advanced algorithms and implementation of vectorizers taking advantage of many refinements like the Skip-grams, or the Continuous Bag-of-Words (CBOW) and subword Ngrams [1].
fastText precomputed embeddings are delivered under the name language model. Given the complexity of human languages, language model training required GB of data. For fastText, 157 language models are provided [2] which is quite distinctive to other previous libraries.
Let’s start with the French language and observe the outcome on terms used in mountaineering activities.
For example, corde (rope) is found to be close to:
cordes (the plural form)
Corde (1st letter capitalized)
corde. (with punctuation)
cordelette (small rope)
cordage (rope set)
ficelle (small rope synonym)
sangle (flat rope, strap)
corder (the verb)
filin (synonym)
poulie (pullet)
This single observation gives us some clues about what is close (synonyms, other forms, closely related objects), and also some remaining issues of the vocabulary: punctuation, case. The case has no easy solution as some common words might refer to a specific location like Mont-Blanc (i.e. white-mount).
fastText is also computing analogies from a triplet: distance is computed between the first two words and vocabulary words are extracted at the same distance from the third word in argument. It does not always lead to meaningful associations but still is quite accurate. For example, with the words skier (to ski), ski (the sport or the object), and vélo (bicycle), the analogy output is:
pédaler (to pedal, to go cycling)
promener, balader (to hang out)
pédalant (pedaling)
and less meaningful that synonyms with punctuation: vélo.A, vélo., Pédaler, bicyclette, vélo.-, velo
Another view on the embeddings is a projection of the cloud of words from the high dimension space (300 dimensions for the French model) onto the 2D plane. The most common projection mechanism is the principal component analysis (PCA), but there are more advanced technics like t-SNE [6] and UMAP [7] that preserve the local distance between items. Beware that the projections by t-SNE or UMAP are not stable: multiple runs on the same data lead to different outputs
Below is a projection of the embeddings of some vocabulary from the sailing, cycling, and mountaineering sport activities. Mountaineering and sailing are quite well separated, cycling is spread between them among shared vocabulary like carte (map) and corde (rope).
Let’s now project more specific vocabulary of mountaineering activities ranging from rock climbing to ski:
There is little structure in this image. Even things that seems obviously close like the cable car variants (télésiège, téléski, télécabine) are not always gathered.
Let’s now perform a more complete machine learning task: classification of the route descriptions from CampToCamp.org as a function of the activity: hiking, climbing (on rock, ice, mix, in mountain, via-ferrata), skiing, mountain biking and snowshoeing.
Each route can be labeled with multiple activities, the problem is multi-class and multi-label. In practice, most routes have 1 activity and there are at most 3 activities.
Four classifier models are compared:
1. fastText French model to compute embedding and then a convolutional neural network (CNN)
2. fastText classifier (multinomial logistic) built on the corpus
3. fastText embeddings built on the corpus to feed a CNN
4. CNN computing the embeddings at the first stage
Corpus sequences (route descriptions) are padded and truncated at 300 (more than 80% are shorter) or 600 words (more than 95% are shorter).
The convolutions are on the 2-dimension plane of the sequence words and embeddings. The architecture is a classical funnel shape (the feature space is smaller and smaller the deeper in the network), initial computation layers are convolutional and final layers are dense. The main parameters are:
Number of layers
Width of the convolutions
Pooling or stride parameters
Regularization, mainly using Dropout
The loss is categorical cross-entropy applied at the output of sigmoid activation. The performance is also assessed during training by a categorical accuracy.
One of the challenges is overfitting as the total number of parameters remains quite high compared to the number of samples. Another challenge is the fact that 1 class (ski touring) is much larger than others (40% of total) while 4 / 10 classes are below 5%. As illustrated in the figure below, the models are quite well handling this issue but for the activities that are related to one or more major activities. For example, snowshoeing is in a sandwich between ski touring and summer hiking (most people actually prefer ski touring to snowshoeing).
The test dataset is verified with a slightly different accuracy compared to the one of the training. It computes top-1, top-2 and top-3 accuracies with following rules:
Top-1 is based on the predicted class with the highest probability and checking if this class is among the labels for this sequence
Top-2 is based on the two highest predicted probabilities and is comparing to labels. In case the label number applied to a given sequence is 1, the top-2 match is easier than top-1 as two candidate classes are applied. In the case of 2 or more labels applied to the sequence, the difference to top-1 statistic is not as favorable as the previous case.
Top-3 si similar to top-2 but with the three highest probabilities
With this metric, the 1st model is a little under-performing, while all other models are performing similarly. The ranking among these 3 models depends on the initialization and training-test dataset split. It means that the simple and efficient classifier generated by fastText is performing as well as the more complex DNN.
The histogram of the description length of the routes that are misclassified is similar to the one of all routes.
Let’s inspect the estimates at the input of the last DNN layer of model 4. The dimension is 64, it is often more interesting compared to the output of this last layer that takes out probabilities (sigmoid activation) of the 10 candidate classes.
The routes are generally quite well separated in clusters corresponding to the activities. Amazingly, there is a cluster that is quite far away and relates to ‘hiking’.
Changing the color encoding to be the pass or fail at top-1 accuracy, we see that fail are spread all over but with a higher concentration at the thin junction of the clusters.
Some of the failing quite obvious reading the route description: it is short, incomplete, and imprecise and the classifier is assigning activities with pure likelihood.
This is verified by looking at the strange remote cluster of hiking routes. It is made of routes with description “Info”, this could be easily detected by a rule but the classifiers have cornered them.
Some others are more disturbing like the following one in which the activity is clearly named in the description:
And the classifier could still be improved if it would better understand the subtleties of the language like on this itinerary description:
For this task of medium complexity, the fastText simple classifier is performing well and at a computation cost much lower than the association of the training of a full language model, the computation of the embedding, and the classification through a CNN.
This classifier might anyway not be flexible enough if the task is more complicated, like computing several outputs (e.g.: the activity and the difficulty of the route).
We have evaluated three ways to build and use embeddings, and feed a neural network:
1. Model 1: Using a pre-computed language model with fastText
2. Model 3: Using fastText to build the model from the corpus and compute the embeddings
3. Model 4: Directly fit the embeddings within the neural network
The first solution is the best if the language of the corpus is not specific, or if ethics issues like bias are of importance.
The second solution is taking advantage at the same time of the specific language of the corpus and the optimization of fastText (e.g.: subwords Ngrams that are more robust to out of vocabulary words and thus have better generalization)
The third solution is taking advantage of the combined optimization of the embeddings and the classifier but is not performing better than the others. It is also showing that Tensorflow and similar tools are quickly becoming the universal tool to perform machine learning. Without much engineering complexity a tailored solution is created. However, it lacks interpretability as shown in the following image of the embeddings generated on the mountaineering vocabulary.
CampToCamp.org data preparation
Main notebook with the classifier evaluation
[1] Bag of Tricks for Efficient Text Classification, J. Armand, G. Edouard, B. Piotr, M. Tomas, 2017, Proceedings of the 15th Conference of the {E}uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers (https://www.aclweb.org/anthology/E17-2068/)[2] Learning Word Vectors for 157 Languages, G. Edouard, B. Piotr, G. Prakhar, J. Armand, M. Tomas, 2018, Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)} (https://arxiv.org/abs/1802.06893, https://fasttext.cc/docs/en/crawl-vectors.html)[3] Efficient Estimation of Word Representations in Vector Space, T. Mikolov, K. Chen, G. Corrado, J. Dean, 2013 (https://arxiv.org/abs/1301.3781)[4] GloVe, Global Vectors for Word Representation , J. Pennington, R. Socher, C.D. Manning, 2014 (http://www.aclweb.org/anthology/D14-1162)[5] REST API for www.camptocamp.org, Github (https://github.com/c2corg/v6_api)[6] Visualizing Data using t-SNE, L. Van der Maaten, G. Hinton, Journal of Machine Learning Research, 2008 (http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf)[7] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, McInnes, Healy, Melville, 2018, (https://arxiv.org/abs/1802.03426)