Package: lbl2vec
Lbl2VecLbl2Vec is an algorithm forunsupervised document classificationandunsupervised document retrieval.It
automatically generates jointly embedded label, document and word vectors and returns documents of topics modeled by
manually predefined keywords. This package includestwo different model types. The plainLbl2Vecmodel
uses
Doc2Vec, whereasLbl2TransformerVecuses transformer-based language modelsto create
the
embeddings. Once you
train a model you can:Classify documents as related to one of the predefined topics.Get similarity scores for documents to each predefined topic.Get most similar predefined topic of documents.See the papers introducingLbl2VecandLbl2TransformerVecfor more details on how it
works.Corresponding Medium post describing the use of Lbl2Vec for unsupervised text classification can be
foundhere.BenefitsNo need to label the whole document dataset for classification.No stop word lists required.No need for stemming/lemmatization.Works on short text.Creates jointly embedded label, document, and word vectors.Table of ContentsHow does it work?InstallationUsageModel TrainingLbl2VecLbl2TransformerVecDocument predictionSave and load modelsCitation informationHow does it work?Back to Table of ContentsThe key idea of the algorithm is that many semantically similar keywords can represent a topic. In the first step, the
algorithm creates a joint embedding of document and word vectors. Once documents and words are embedded in a vector
space, the goal of the algorithm is to learn label vectors from previously manually defined keywords representing a
topic. Finally, the algorithm can predict the affiliation of documents to topics fromdocument vector <-> label vectorsimilarities.The Algorithm0. Use the manually defined keywords for each topic of interest.Domain knowledge is needed to define keywords that describe topics and are semantically similar to each other within
the topics.BasketballSoccerBaseballNBAFIFAMLBBasketballSoccerBaseballLeBronMessiRuth.........1. Create jointly embedded document and word vectors
usingDoc2Vec,Sentence-Transformers,
orSimCSE.Documents will be placed close to other similar documents and close to the most distinguishing words.2. Find document vectors that are similar to the keyword vectors of each topic.Each color represents a different topic described by the respective keywords.3. Clean outlier document vectors for each topic.Red documents are outlier vectors that are removed and do not get used for calculating the label vector.4. Compute the centroid of the outlier cleaned document vectors as label vector for each topic.Points represent the label vectors of the respective topics.5. Computelabel vector <-> document vectorsimilarities for each label vector and document vector in the dataset.Documents are classified as topic with the highestlabel vector <-> document vectorsimilarity.InstallationBack to Table of Contentspip install lbl2vecUsageBack to Table of ContentsFor detailed information visit theAPI Guideand
theexamples.Model TrainingLbl2Vec modelBack to Table of ContentsLbl2Vec learns word vectors, document vectors and label vectors using Doc2Vec during training.Train new Lbl2Vec model from scratch usingDoc2Vecfromlbl2vecimportLbl2Vec# init modelmodel=Lbl2Vec(keywords_list=descriptive_keywords,tagged_documents=tagged_docs)# train modelmodel.fit()Important parameters:keywords_list: iterable list of lists with descriptive keywords of type str. For each label at least one descriptive
keyword has to be added as list of str.tagged_documents: iterable list
ofgensim.models.doc2vec.TaggedDocumentelements. If you wish to train a new Doc2Vec model this parameter can not be None, whereas thedoc2vec_modelparameter must be None. If you use a pretrained Doc2Vec model this parameter has to be None. Input corpus, can be
simply a list of elements, but for larger corpora, consider an iterable that streams the documents directly from
disk/network.Use word and document vectors from pretrainedDoc2VecmodelUses word vectors and document vectors from a pretrained Doc2Vec model to learn label vectors during Lbl2Vec model
training.fromlbl2vecimportLbl2Vec# init modelmodel=Lbl2Vec(keywords_list=descriptive_keywords,doc2vec_model=pretrained_d2v_model)# train modelmodel.fit()Important parameters:keywords_list: iterable list of lists with descriptive keywords of type str. For each label at least one descriptive
keyword has to be added as list of str.doc2vec_model:
pretrainedgensim.models.doc2vec.Doc2Vecmodel. If given a pretrained Doc2Vec model, Lbl2Vec uses the pre-trained Doc2Vec model from this parameter. If this
parameter is defined,tagged_documentsparameter has to be None. In order to get optimal Lbl2Vec results the given
Doc2Vec model should be trained with the parameters "dbow_words=1" and "dm=0".Lbl2TransformerVec modelBack to Table of ContentsLbl2TransformerVec learns word vectors, document vectors and label vectors using transformer-based language models
during training. Using state-of-the-art transformer embeddings may not only yield to better predictions but also
eliminates the issue of unknown keywords during model training. While the Doc2Vec-based model can only use keywords
that
Lbl2Vec has seen during training, the transformer-based Lbl2TransformerVec model can learn label vectors from any set
of
keywords. That is because transformer vocabularies consist of individual characters, subwords, and words, allowing
transformers to effectively represent every word in a sentence. This eliminates the out-of-vocabulary scenario.
However,
using transformers instead of Doc2Vec is much more computationally expensive, especially if no GPU is available.Train new Lbl2TransformerVec model from scratch using the default transformer-embedding modelfromlbl2vecimportLbl2TransformerVec# init model using the default transformer-embedding model ("sentence-transformers/all-MiniLM-L6-v2")model=Lbl2TransformerVec(keywords_list=listdescriptive_keywords,documents=document_list)# train modelmodel.fit()Train Lbl2TransformerVec model using an arbitrarySentence-Transformersembedding modelfromlbl2vecimportLbl2TransformerVecfromsentence_transformersimportSentenceTransformer# select sentence-tranformers modeltransformer_model=SentenceTransformer("all-mpnet-base-v2")# init modelmodel=Lbl2TransformerVec(transformer_model=transformer_model,keywords_list=listdescriptive_keywords,documents=document_list)# train modelmodel.fit()Train Lbl2TransformerVec model using an arbitrarySimCSEembedding modelfromlbl2vecimportLbl2TransformerVecfromtransformersimportAutoModel# select SimCSE modeltransformer_model=AutoModel.from_pretrained("princeton-nlp/sup-simcse-roberta-base")# init modelmodel=Lbl2TransformerVec(transformer_model=transformer_model,keywords_list=listdescriptive_keywords,documents=document_list)# train modelmodel.fit()Important parameters:keywords_list: iterable list of lists with descriptive keywords of type str. For each label at least one descriptive
keyword has to be added as list of str.documents: iterable list of text document elements (strings).transformer_model: Transformer model used to embed the labels, documents and keywords. The embedding models must be
either of
typesentence_transformers.SentenceTransformerortransformers.AutoModelDocument predictionBack to Table of ContentsThe prediction API calls are the same for Lbl2Vec and Lbl2TransformerVec.Predict label similarities for documents used for trainingComputes the similarity scores for each document vector stored in the model to each of the label vectors.# get similarity scores from trained modelmodel.predict_model_docs()Important parameters:doc_keys: list of document keys (optional). If None: return the similarity scores for all documents that are used to
train the Lbl2Vec model. Else: only return the similarity scores of training documents with the given keys.Predict label similarities for new documents that are not used for trainingComputes the similarity scores for each given and previously unknown document vector to each of the label vectors from
the model.# get similarity scores for each new document from trained modelmodel.predict_new_docs(tagged_docs=tagged_docs)Important parameters:tagged_docs: iterable list
ofgensim.models.doc2vec.TaggedDocumentelementsSave and load modelsBack to Table of ContentsThe save and load API calls are the same for Lbl2Vec and Lbl2TransformerVec.Save model to diskmodel.save('model_name')Load model from diskmodel=Lbl2Vec.load('model_name')Citation informationBack to Table of ContentsWhen citingLbl2VecorLbl2TransformerVecin academic papers and
theses, please use the following BibTeX entries:@conference{schopf_etal_webist21,
author={Tim Schopf and Daniel Braun and Florian Matthes},
title={Lbl2Vec: An Embedding-based Approach for Unsupervised Document Retrieval on Predefined Topics},
booktitle={Proceedings of the 17th International Conference on Web Information Systems and Technologies - WEBIST,},
year={2021},
pages={124-132},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010710300003058},
isbn={978-989-758-536-4},
issn={2184-3252},
}@inproceedings{schopf_etal_nlpir22,
author = {Schopf, Tim and Braun, Daniel and Matthes, Florian},
title = {Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches},
year = {2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
booktitle = {2022 6th International Conference on Natural Language Processing and Information Retrieval (NLPIR)},
keywords = {Natural Language Processing, Unsupervised Text Classification, Zero-shot Text Classification},
location = {Bangkok, Thailand},
series = {NLPIR 2022}
}