Package: wellcomeml
WellcomeML utilsThis package contains common utility functions for usual tasks at the Wellcome Trust, in particular functionalities for processing, embedding and classifying text data. This includesAn intuitive sklearn-like API wrapping text vectorizers, such as Doc2vec, Bert, ScibertCommon API for off-the-shelf classifiers to allow quick iteration (e.g. Frequency Vectorizer, Bert, Scibert, basic CNN, BiLSTM, SemanticSimilarity)Utils to download and convert academic text datasets for benchmarkUtils to download data from the EPMC APIFor more information read the officialdocs.1. QuickstartInstalling from PyPipipinstallwellcomemlThis will install the "vanilla" package with very little functionality, such as io, dataset download etc.If space is not a problem, you can install thefullpackage (around 2.2GB):pipinstallwellcomeml[all]The full package is relatively big, therefore we also have fine-grained installations if you only wish to use one specific module.
Those arecore, transformers, tensorflow, torch, spacy. You can install one, or more of those you want, e.g.:pipinstallwellcomeml[tensorflow,core]To check that your installation allows you to use a specific module, try (for example):python-c"import wellcomeml.ml.bert_vectorizer"If you don't have the correct dependencies installed for a module, an error will appear
and point you to the right dependencies.1.1 Installing wellcomeml[all] on windowsTorch has a different installation for windows so it will not get automatically installed with wellcomeml[all].
It needs to be installed first (this is for machines with no CUDA parallel computing platform for those that do look herehttps://pytorch.org/for correct installation):pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
pip install wellcomeml[all]2. Development2.1 Build local virtualenvmake2.2 Contributing to the docsMake changes to the .rst files in/docs(pleasedo not change the ones starting by wellcomemlas those are generated automatically)Navigate to the root repository and runmakeupdate-docsVerify that_build/html/index.htmlhas generated correctly and submit a PR.2.3 Release a new version (and upload to aws s3/pypi/github)First create agithub token, if you haven't one, with artifact write access and
export
it to the env variables:exportGITHUB_TOKEN=...The checklist for a new release is:Changewellcomeml/__version__.pyAdd changelogmake distVerify new package was generated correctly on thepip registryand GitHub releases2.4 (Optional) Installing from other locationspip3 install <relative path to this folder>2.5 TransformersOn OSX, if you get a message complaining about the rust compiler, install and initialise it with:brew install rustup
rustup-init3. Example usage of some modulesExamples can be found in the subfolderexamples.4. TroubleshootingIf you experience a problem with installing or using WellcomeML please open an issue. It might be
worth setting the logging level to DEBUGexport LOGGING_LEVEL=DEBUGwhich will often expose
more information that might be informative to resolve the issue.5. ExtrasModuleDescriptionExtras neededwellcomeml.ml.attentionClasses that implement keras layers for attention/self-attentiontensorflowwellcomeml.ml.bert_classifierClassifier to facilitate fine-tuning bert/sciberttensorflowwellcomeml.ml.bert_semantic_equivalenceClassifier to learn semantic equivalence between pairs of documentstensorflowwellcomeml.ml.bert_vectorizerText vectorizer based on bert/sciberttorchwellcomeml.ml.bilstmBILSTM Text classifiertensorflowwellcomeml.ml.clusteringText clustering pipelineNAwellcomeml.ml.cnnCNN Text Classifiertensorflowwellcomeml.ml.doc2vec_vectorizerText vectorizer based on doc2vecNAwellcomeml.ml.frequency_vectorizerText vectorizer based on TF-IDFNAwellcomeml.ml.keras_utilsUtils for computing metrics during trainingtensorflowwellcomeml.ml.keras_vectorizerText vectorizer based on Kerastensorflowwellcomeml.ml.sent2vec_vectorizerText vectorizer based on Sent2Vec(Requires sent2vec, a non-pypi package)wellcomeml.ml.similarity_entity_likingA class to find most similar documents to a sentence in a corpustensorflowwellcomeml.ml.spacy_classifierA text classifier based on spacyspacy, torchwellcomeml.ml.spacy_entity_linkingSimilar to similarity_entity_linking, but uses spacyspacywellcomeml.ml.spacy_knowledge_baseCreates a knowledge base of entities, based onspacyspacywellcomeml.ml.spacy_nerNamed entity recognition classifier based on spacyspacywellcomeml.ml.transformers_tokenizerBespoke tokenizer based on transformersTransformerswellcomeml.ml.vectorizerAbstract class for vectorizersNAwellcomeml.ml.voting_classifierMeta-classifier based on majority votingNA