Package: torchtext
torchtextThis repository consists of:torchtext.datasets: The raw text iterators for common NLP datasetstorchtext.data: Some basic NLP building blockstorchtext.transforms: Basic text-processing transformationstorchtext.models: Pre-trained modelstorchtext.vocab: Vocab and Vectors related classes and factory functionsexamples: Example NLP workflows with PyTorch and torchtext library.InstallationWe recommend Anaconda as a Python package management system. Please refer topytorch.orgfor the details of PyTorch installation. The following are the correspondingtorchtextversions and supported Python versions.Version CompatibilityPyTorch versiontorchtext versionSupported Python versionnightly buildmain>=3.8, <=3.112.2.00.17.0>=3.8, <=3.112.1.00.16.0>=3.8, <=3.112.0.00.15.0>=3.8, <=3.111.13.00.14.0>=3.7, <=3.101.12.00.13.0>=3.7, <=3.101.11.00.12.0>=3.6, <=3.91.10.00.11.0>=3.6, <=3.91.9.10.10.1>=3.6, <=3.91.90.10>=3.6, <=3.91.8.10.9.1>=3.6, <=3.91.80.9>=3.6, <=3.91.7.10.8.1>=3.6, <=3.91.70.8>=3.6, <=3.81.60.7>=3.6, <=3.81.50.6>=3.5, <=3.81.40.52.7, >=3.5, <=3.80.4 and below0.2.32.7, >=3.5, <=3.8Using conda:conda install -c pytorch torchtextUsing pip:pip install torchtextOptional requirementsIf you want to use English tokenizer fromSpaCy, you need to install SpaCy and download its English model:pip install spacy
python -m spacy download en_core_web_smAlternatively, you might want to use theMosestokenizer port inSacreMoses(split fromNLTK). You have to install SacreMoses:pip install sacremosesFor torchtext 0.5 and below,sentencepiece:conda install -c powerai sentencepieceBuilding from sourceTo build torchtext from source, you needgit,CMakeand C++11 compiler such asg++.:git clone https://github.com/pytorch/text torchtext
cd torchtext
git submodule update --init --recursive

# Linux
python setup.py clean install

# OSX
CC=clang CXX=clang++ python setup.py clean install

# or ``python setup.py develop`` if you are making modifications.NoteWhen building from source, make sure that you have the same C++ compiler as the one used to build PyTorch. A simple way is to build PyTorch from source and use the same environment to build torchtext.
If you are using the nightly build of PyTorch, checkout the environment it was built withconda (here)andpip (here).Additionally, datasets in torchtext are implemented using the torchdata library. Please take a look at theinstallation instructionsto download the latest nightlies or install from source.DocumentationFind the documentationhere.DatasetsThe datasets module currently contains:Language modeling: WikiText2, WikiText103, PennTreebank, EnWik9Machine translation: IWSLT2016, IWSLT2017, Multi30kSequence tagging (e.g. POS/NER): UDPOS, CoNLL2000ChunkingQuestion answering: SQuAD1, SQuAD2Text classification: SST2, AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDBModel pre-training: CC-100ModelsThe library currently consist of following pre-trained models:RoBERTa:Base and Large ArchitectureDistilRoBERTaXLM-RoBERTa:Base and Large ArchitureT5:Small, Base, Large, 3B, and 11B ArchitectureFlan-T5:Base, Large, XL, and XXL ArchitectureTokenizersThe transforms module currently support following scriptable tokenizers:SentencePieceGPT-2 BPECLIPRE2BERTTutorialsTo get started with torchtext, users may refer to the following tutorial available on PyTorch website.SST-2 binary text classification using XLM-R pre-trained modelText classification with AG_NEWS datasetTranslation trained with Multi30k dataset using transformers and torchtextLanguage modeling using transforms and torchtextDisclaimer on DatasetsThis is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset’s license.If you’re a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!