Package: textwiser
TextWiser: Text Featurization LibraryTextWiser (AAAI'21)is a research library that provides a unified framework for text featurization based on a rich set of methods while taking advantage of pretrained models provided by the state-of-the-art libraries.The main contributions include:Rich Set of Embeddings:A wide range of availableembeddingsandtransformationsto choose from.Fine-Tuning:Designed to support aPyTorchbackend, and hence, retains the ability tofine-tune featurizationsfor downstream tasks.
That means, if you pass the resulting fine-tunable embeddings to a training method, the features will
be optimized automatically for your application.Parameter Optimization:Interoperable with the standardscikit-learnpipeline for hyper-parameter tuning and rapid experimentation. All underlying parameters are exposed to the user.Grammar of Embeddings:Introduces a novel approach to design embeddings from components.
Thecompound embeddingallows forming arbitrarily complex embeddings in accordance with acontext-free grammarthat defines a formal language for valid text featurization.GPU Native:Built with GPUs in mind. If it detects available hardware, the relevant models are automatically placed on the GPU.TextWiser is developed by the Artificial Intelligence Center of Excellence at Fidelity Investments. Documentation is available atfidelity.github.io/textwiser. Here is thevideo of the paper presentation at AAAI 2021.Quick Start# Conceptually, TextWiser is composed of an Embedding, potentially with a pretrained model,# that can be chained into zero or more TransformationsfromtextwiserimportTextWiser,Embedding,Transformation,WordOptions,PoolOptions# Datadocuments=["Some document","More documents. Including multi-sentence documents."]# Model: TFIDF `min_df` parameter gets passed to sklearn automaticallyemb=TextWiser(Embedding.TfIdf(min_df=1))# Model: TFIDF followed with an NMF + SVDemb=TextWiser(Embedding.TfIdf(min_df=1),[Transformation.NMF(n_components=30),Transformation.SVD(n_components=10)])# Model: Word2Vec with no pretraining that learns from the input dataemb=TextWiser(Embedding.Word(word_option=WordOptions.word2vec,pretrained=None),Transformation.Pool(pool_option=PoolOptions.min))# Model: BERT with the pretrained bert-base-uncased embeddingemb=TextWiser(Embedding.Word(word_option=WordOptions.bert),Transformation.Pool(pool_option=PoolOptions.first))# Featuresvecs=emb.fit_transform(documents)Available EmbeddingsEmbeddingsNotesBag of Words (BoW)Supported byscikit-learnDefaults to training from scratchTerm Frequency Inverse Document Frequency (TfIdf)Supported byscikit-learnDefaults to training from scratchDocument Embeddings (Doc2Vec)Supported bygensimDefaults to training from scratchUniversal Sentence Encoder (USE)Supported bytensorflow, seerequirementsDefaults tolarge v5Compound EmbeddingSupported by acontext-free grammarWord Embedding:Word2VecSupported by thesepretrained embeddingsCommon pretrained options includecrawl,glove,extvec,twitter, anden-newsWhen the pretrained option isNone, trains a new model from the given dataDefaults toen, FastText embeddings trained on newsWord Embedding:CharacterInitialized randomly and not pretrainedUseful when trained for a downstream taskEnablefine-tuningto get good embeddingsWord Embedding:BytePairSupported by thesepretrained embeddingsPretrained options can be specified with the string<lang>_<dim>_<vocab_size>Default options can be omitted likeen,en_100, oren__10000Defaults toen, which is equal toen_100_10000Word Embedding:ELMoSupported by thesepretrained embeddingsfrom TensorflowHubDefaults tooriginalWord Embedding:FlairSupported by thesepretrained embeddingsDefaults tonews-forward-fastWord Embedding:BERTSupported by thesepretrained embeddingsDefaults tobert-base-uncasedWord Embedding:OpenAI GPTSupported by thesepretrained embeddingsDefaults toopenai-gptWord Embedding:OpenAI GPT2Supported by thesepretrained embeddingsDefaults togpt2-mediumWord Embedding:TransformerXLSupported by thesepretrained embeddingsDefaults totransfo-xl-wt103Word Embedding:XLNetSupported by thesepretrained embeddingsDefaults toxlnet-large-casedWord Embedding:XLMSupported by thesepretrained embeddingsDefaults toxlm-mlm-en-2048Word Embedding:RoBERTaSupported by thesepretrained embeddingsDefaults toroberta-baseWord Embedding:DistilBERTSupported by thesepretrained embeddingsDefaults todistilbert-base-uncasedWord Embedding:CTRLSupported by thesepretrained embeddingsDefaults toctrlWord Embedding:ALBERTSupported by thesepretrained embeddingsDefaults toalbert-base-v2Word Embedding:T5Supported by thesepretrained embeddingsDefaults tot5-baseWord Embedding:XLM-RoBERTaSupported by thesepretrained embeddingsDefaults toxlm-roberta-baseWord Embedding:BARTSupported by thesepretrained embeddingsDefaults tofacebook/bart-baseWord Embedding:ELECTRASupported by thesepretrained embeddingsDefaults togoogle/electra-base-generatorWord Embedding:DialoGPTSupported by thesepretrained embeddingsDefaults tomicrosoft/DialoGPT-smallWord Embedding:LongformerSupported by thesepretrained embeddingsDefaults toallenai/longformer-base-4096Available TransformationsTransformationsNotesSingular Value Decomposition (SVD)DifferentiableLatent Dirichlet Allocation (LDA)Not differentiableNon-negative Matrix Factorization (NMF)Not differentiableUniform Manifold Approximation and Projection (UMAP)Not differentiablePooling Word VectorsApplies to word embeddings onlyReduces word-level vectors to document-levelPool options includemax,min,mean,first, andlastDefaults tomaxUsage ExamplesExamples can be found under thenotebooksfolder.InstallationTextWiser requiresPython 3.6+and can be installed from PyPI usingpip install textwiser, usingpip install textwiser[full]to install from PyPI with all optional dependencies, or by building from source by following the instructions
in ourdocumentation.Compound EmbeddingA unique research contribution of TextWiser lies in its novel approach in creating embeddings from components,
called the Compound Embedding.This method allows forming arbitrarily complex embeddings, thanks to a
context-free grammar that defines a formal language for valid text featurization. You can see the details
in ourdocumentationand in theusage example.Fine-Tuning for Downstream TasksAll Word2Vec and transformer-based embeddings and any embedding followed with ansvdtransformation are fine-tunable for downstream tasks.
In other words, if you pass the resulting fine-tunable embedding to a PyTorch training method, the features will automatically
be trained for your application. You can see the details in ourdocumentationand in theusage example.TokenizationIn general, text data should bewhitespace-tokenizedbefore being fed into TextWiser.
Customized tokenization is also supported as described in more detail
in ourdocumentationSupportPlease submit bug reports, questions and feature requests asIssues.CitationIf you use TextWiser in a publication, please cite it as:@article{textwiser2021,author={Kilitcioglu, Doruk and Kadioglu, Serdar},title={Representing the Unification of Text Featurization using a Context-Free Grammar},url={https://github.com/fidelity/textwiser},journal={Proceedings of the AAAI Conference on Artificial Intelligence},volume={35},number={17},year={2021},month={May},pages={15439-15445}}LicenseTextWiser is licensed under theApache License 2.0.