An End-to-End Web Service Implementation for Text Classification using Word2Vec and LGBM
By: Kemal Toprak Uçar
Tags: Artificial Intelligence, Application Development, Machine Learning, Classification, NLP

Word2Vec

Using neural networks for learning distributed vector representations of words attracts growing attention recently. The target word is predicted using its neighbour words by Word2Vec and it maps words of resembling meanings to points within a close distance in the vector space. It was implemented by Thomas Mikolov. This model generates vector representation of the words and the arithmetic operations can be done using these vectors. As you may guess, LGBM utilized the output of the Word2Vec model in which text data was used for training.

Besides the function of this language model, it has a fully-layered neural network architecture. Number of nodes in input and output layers is equal to the vocabulary size. Size of the hidden layer is defined by the user and enlarging the hidden layer causes the model to be more complex. Also, the size of the hidden layer is equal to the size of vector representation. Compared to other neural network structures, activation function does not exist at the hidden layer, so sum of weights are directly passed to the output layer. At the output layer, a probability distribution is generated by the softmax function. Window size is one of the key parameters in the model which is the length of sliding window employed in training. During the training, Word2Vec finds the relationship between words using the sliding window which is moved in the context. Similar vectors mean closer meanings in larger window size. On the other hand, smaller window size indicates that similar vector representations mean these words might be interchangeable. The similarity between vector representations are measured by Cosine Similarity. Word2Vec can choose 2 methods which are Common Bag of Words (CBOW) and Skip Gram to generate vector representations. CBOW predicts the target by looking at the context, Skip Gram predicts the context using the given word. In order to implement word2vec features, I employed Gensim library.

Before completing this part of the post, I have to share an example to demonstrate the operations of vectors since it’s mandatory.

bordeaux - france + italy = tuscany

LightGBM

If you’re interested in tree based AI models, you would know/read that there are 2 types of ensemble models which are Boosting and Bagging models to solve classification or regression problems. Briefly, in ensemble models, number of trees are trained sequentially or in parallel regarding the ensemble type then each tree votes for the result for the final result of the forest.

Despite the risk of overfitting, Boosting models with optimal parameters generally outperforms Bagging models. Recently, you may have noticed 3 popular models, XGBoost, CatBoost, and LightGBM, which have been appearing on Kaggle competitions, Medium blogs, and relevant areas. LightGBM is a high performance gradient boosting based algorithm for many AI tasks. Leafwise tree growth and utilization of discrete values which were transformed from continuous values provide great accuracy and faster training procedure. (Leafwise tree growth is also supported by XGBoost) In my recent projects, I usually acquire better performance by CatBoost but implementation of LightGBM models is more reasonable regarding efficiency. (I don’t want to generalize which model is the best for overall, just wanted to share my feedback from my experiences).

Flask

Flask is a framework to implement web services in Python. There are 3 main classes which are Controller, Application, and Init to implement a web service using Flask. You can think of Controller class as a gate. It’s registered into Init then provides the communication of the application from outside. Init class also contains the prefix URL information of Controller. In order to execute to service, Application class is invoked. Host and port configuration is kept in Application class. Briefly, a basic application such as printing “Bonjour!” can be implemented using only these 3 classes. Since our service is a bit complex, I used multiple layers to increase code quality.

In the architecture, Controller class doesn’t have any logic, it only extracts the input from the request and forwards it to related services. Services are the modules which handle the complex operations including classification and text to vector transformation. There is a class created for each result type and they are sent as JSON format by Controller.