Understand how to transfer your paragraph to vector by doc2vec | by Edward Ma | Towards Data Science

In previous story, mentioned word2vec is introduced by Mikolov et al. (2013). Mikolov and Le released sentence/ document vectors transformation. It is another breakthrough on embeddings such that we can use vector to represent a sentence or document. Mikolov et al. call it as “Paragraph Vector”.
After reading this article, you will understand:
Paragraph Vector Design
Architecture
Implementation
Take Away
Design for doc2vec is based on word2vec. If you do not familiar with word2vec (i.e. skip-gram and CBOW), you may check out this story. Doc2vec also uses and unsupervised learning approach to learn the document representation. The input of texts (i.e. word) per document can be various while the output is fixed-length vectors.
Paragraph vector and word vectors are initialized. Paragraph vector is unique among all document while word vectors are shared among all document such that word vector can be learnt from different document.
During training phase, word vectors will be trained while paragraph will be thrown away after that. During prediction phase (i.e. online prediction), paragraph vector will be initialized randomly and computing it by word vectors.
There are 2 algorithms for learning word vectors. Both of them are inspired by learning word vectors which are skip-gram and continuous bag-of-words(CBOW)
Distributed Memory Model of Paragraph Vectors (PV-DM)
Both paragraph vectors and word vectors are initialized randomly. Every paragraph vector is assigned to single document while word vectors are shared among all documents. Either averaging or concatenating both paragraph vector and words vectors and passing to stochastic gradient descent and the gradient is obtained via back propagation.
This approach is similar to continuous bag-of-words (CBOW) approach in word2vec.
Distributed Bag of Words version of Paragraph Vector (PV-DBOW)
Another approach goes a different way. Instead of predicting next word, it use a paragraph vector to classify entire words in the document. During training, sampling a list of word and then form a classifer to classify whether word belongs to the document such that word vectors can be learnt.
This approach is similar to skip-gram approach in word2vec.
First of all, we need to to pass the training data to build vocabulary and invoke the training phase in order to compute word vectors.
doc2vec_embs = Doc2VecEmbeddings()x_train_tokens = doc2vec_embs.build_vocab(documents=x_train)doc2vec_embs.train(x_train_tokens)
After that, we can encode it by providing training data and testing data.
x_train_t = doc2vec_embs.encode(documents=x_train)x_test_t = doc2vec_embs.encode(documents=x_test)
Finally, we can pass vectors to a classifier
from sklearn.linear_model import LogisticRegressionmodel = LogisticRegression(solver='newton-cg', max_iter=1000)model.fit(x_train_t, y_train)y_pred = model.predict(x_test_t)from sklearn.metrics import accuracy_scorefrom sklearn.metrics import classification_reportprint('Accuracy:%.2f%%' % (accuracy_score(y_test, y_pred)*100))print('Classification Report:')print(classification_report(y_test, y_pred))
Result
Accuracy: 52.8%Average Precision: 0.66Average Recall: 0.53Average f1: 0.5
To access all code, you can visit my github repo.
Unlike word2vec, doc2vec computes sentence/ document vector on the fly. In other word, it takes time to get vector during prediction time.
From Mikolov et al. experiment, PV-DM is consistently better than PV-DBOW.
In PV-DM approach, concatenation way is often better than sum/ average.
Mikolov Tomas, Le Quoc. 2014. Distributed Representations of Sentences and Documents
Doc2vec in gensim