Package: allennlp
An Apache 2.0 NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.⚠️NOTICE:The AllenNLP library is now in maintenance mode. That means we are no longer adding new features or upgrading dependencies. We will still respond to questions and address bugs as they arise up until December 16th, 2022. If you have any concerns or are interested in maintaining AllenNLP going forward, please open an issue on this repository.AllenNLP has been a big success, but as the field is advancing quickly it's time to focus on new initiatives. We're working hard to makeAI2 Tangothe best way to organize research codebases. If you are an active user of AllenNLP, here are some suggested alternatives:If you like the trainer, the configuration language, or are simply looking for a better way to manage your experiments, check outAI2 Tango.If you like AllenNLP'smodulesandnnpackages, check outdelmaksym/allennlp-light. It's even compatible withAI2 Tango!If you like the framework aspect of AllenNLP, check outflair. It has multiple state-of-art NLP models and allows you to easily use pretrained embeddings such as those from transformers.If you like the AllenNLP metrics package, check outtorchmetrics. It has the same API as AllenNLP, so it should be a quick learning curve to make the switch.If you want to vectorize text, trythe transformers library.If you want to maintain the AllenNLP Fairness or Interpret components, please get in touch. There is no alternative to it, so we are looking for a dedicated maintainer.If you are concerned about other AllenNLP functionality, please create an issue. Maybe we can find another way to continue supporting your use case.Quick Links↗️Website🔦Guide🖼Gallery💻Demo📓Documentation(latest|stable|commit)⬆️Upgrade Guide from 1.x to 2.0❓Stack Overflow✋Contributing Guidelines🤖Officially Supported ModelsPretrained ModelsDocumentation(latest|stable|commit)⚙️Continuous Build🌙Nightly ReleasesIn this READMEGetting Started Using the LibraryPluginsPackage OverviewInstallationInstalling via pipInstalling using DockerInstalling from sourceRunning AllenNLPIssuesContributionsCitingTeamGetting Started Using the LibraryIf you're interested in using AllenNLP for model development, we recommend you check out theAllenNLP Guidefor a thorough introduction to the library, followed by our more advanced guides
onGitHub Discussions.When you're ready to start your project, we've created a couple of template repositories that you can use as a starting place:If you want to useallennlp trainand config files to specify experiments, usethis
template. We recommend this approach.If you'd prefer to use python code to configure your experiments and run your training loop, usethis template. There are a few
things that are currently a little harder in this setup (loading a saved model, and using
distributed training), but otherwise it's functionality equivalent to the config files
setup.In addition, there are external tutorials:Hyperparameter optimization for AllenNLP using OptunaTraining with multiple GPUs in AllenNLPTraining on larger batches with less memory in AllenNLPHow to upload transformer weights and tokenizers from AllenNLP to HuggingFaceAnd others on theAI2 AllenNLP blog.PluginsAllenNLP supports loading "plugins" dynamically. A plugin is just a Python package that
provides custom registered classes or additionalallennlpsubcommands.There is ecosystem of open source plugins, some of which are maintained by the AllenNLP
team here at AI2, and some of which are maintained by the broader community.PluginMaintainerCLIDescriptionallennlp-modelsAI2NoA collection of state-of-the-art modelsallennlp-semparseAI2NoA framework for building semantic parsersallennlp-serverAI2YesA simple demo server for serving modelsallennlp-optunaMakoto HiramatsuYesOptunaintegration for hyperparameter optimizationAllenNLP will automatically find any official AI2-maintained plugins that you have installed,
but for AllenNLP to find personal or third-party plugins you've installed,
you also have to create either a local plugins file named.allennlp_pluginsin the directory where you run theallennlpcommand, or a global plugins file at~/.allennlp/plugins.
The file should list the plugin modules that you want to be loaded, one per line.To test that your plugins can be found and imported by AllenNLP, you can run theallennlp test-installcommand.
Each discovered plugin will be logged to the terminal.For more information about plugins, see theplugins API docs. And for information on how to create a custom subcommand
to distribute as a plugin, see thesubcommand API docs.Package OverviewallennlpAn open-source NLP research library, built on PyTorchallennlp.commandsFunctionality for the CLIallennlp.commonUtility modules that are used across the libraryallennlp.dataA data processing module for loading datasets and encoding strings as integers for representation in matricesallennlp.fairnessA module for bias mitigation and fairness algorithms and metricsallennlp.modulesA collection of PyTorch modules for use with textallennlp.nnTensor utility functions, such as initializers and activation functionsallennlp.trainingFunctionality for training modelsInstallationAllenNLP requires Python 3.6.1 or later andPyTorch.We support AllenNLP on Mac and Linux environments. We presently do not support Windows but are open to contributions.Installing via conda-forgeThe simplest way to install AllenNLP is using conda (you can choose a different python version):conda install -c conda-forge python=3.8 allennlpTo install optional packages, such aschecklist, useconda install -c conda-forge allennlp-checklistor simply installallennlp-alldirectly. The plugins mentioned above are similarly installable, e.g.conda install -c conda-forge allennlp-models allennlp-semparse allennlp-server allennlp-optunaInstalling via pipIt's recommended that you install the PyTorch ecosystembeforeinstalling AllenNLP by following the instructions onpytorch.org.After that, just runpip install allennlp.⚠️ If you're using Python 3.7 or greater, you should ensure that you don't have the PyPI version ofdataclassesinstalled after running the above command, as this could cause issues on certain platforms. You can quickly check this by runningpip freeze | grep dataclasses. If you see something likedataclasses=0.6in the output, then just runpip uninstall -y dataclasses.If you need pointers on setting up an appropriate Python environment or would like to install AllenNLP using a different method, see below.Setting up a virtual environmentCondacan be used set up a virtual environment with the
version of Python required for AllenNLP.  If you already have a Python 3
environment you want to use, you can skip to the 'installing via pip' section.Download and install Conda.Create a Conda environment with Python 3.8 (3.7 or 3.9 would work as well):conda create -n allennlp_env python=3.8Activate the Conda environment. You will need to activate the Conda environment in each terminal in which you want to use AllenNLP:conda activate allennlp_envInstalling the library and dependenciesInstalling the library and dependencies is simple usingpip.pipinstallallennlpTo install the optional dependencies, such aschecklist, runpipinstallallennlp[checklist]Or you can just install all optional dependencies withpip install allennlp[all].Looking for bleeding edge features? You can install nightly releases directly frompypiAllenNLP installs a script when you install the python package, so you can run allennlp commands just by typingallennlpinto a terminal.  For example, you can now test your installation withallennlp test-install.You may also want to installallennlp-models, which contains the NLP constructs to train and run our officially
supported models, many of which are hosted athttps://demo.allennlp.org.pipinstallallennlp-modelsInstalling using DockerDocker provides a virtual machine with everything set up to run AllenNLP--
whether you will leverage a GPU or just run on a CPU.  Docker provides more
isolation and consistency, and also makes it easy to distribute your
environment to a compute cluster.AllenNLP providesofficial Docker imageswith the library and all of its dependencies installed.Once you haveinstalled Docker,
you should also install theNVIDIA Container Toolkitif you have GPUs available.Then run the following command to get an environment that will run on GPU:mkdir-p$HOME/.allennlp/
dockerrun--rm--gpusall-v$HOME/.allennlp:/root/.allennlpallennlp/allennlp:latestYou can test the Docker environment withdockerrun--rm--gpusall-v$HOME/.allennlp:/root/.allennlpallennlp/allennlp:latesttest-installIf you don't have GPUs available, just omit the--gpus allflag.Building your own Docker imageFor various reasons you may need to create your own AllenNLP Docker image, such as if you need a different version
of PyTorch. To do so, just runmake docker-imagefrom the root of your local clone of AllenNLP.By default this builds an image with the tagallennlp/allennlp, but you can change this to anything you want
by setting theDOCKER_IMAGE_NAMEflag when you callmake. For example,make docker-image DOCKER_IMAGE_NAME=my-allennlp.If you want to use a different version of Python or PyTorch, set the flagsDOCKER_PYTHON_VERSIONandDOCKER_TORCH_VERSIONto something like3.9and1.9.0-cuda10.2, respectively. These flags together determine the base image that is used. You can see the list of valid
combinations in this GitHub Container Registry:github.com/allenai/docker-images/pkgs/container/pytorch.After building the image you should be able to see it listed by runningdocker images allennlp.REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
allennlp/allennlp   latest              b66aee6cb593        5 minutes ago       2.38GBInstalling from sourceYou can also install AllenNLP by cloning our git repository:gitclonehttps://github.com/allenai/allennlp.gitCreate a Python 3.7 or 3.8 virtual environment, and install AllenNLP ineditablemode by running:pipinstall-Upipsetuptoolswheel
pipinstall--editable.[dev,all]This will makeallennlpavailable on your system but it will use the sources from the local clone
you made of the source repository.You can test your installation withallennlp test-install.
Seehttps://github.com/allenai/allennlp-modelsfor instructions on installingallennlp-modelsfrom source.Running AllenNLPOnce you've installed AllenNLP, you can run the command-line interface
with theallennlpcommand (whether you installed frompipor from source).allennlphas various subcommands such astrain,evaluate, andpredict.
To see the full usage information, runallennlp --help.You can test your installation by runningallennlp test-install.IssuesEveryone is welcome to file issues with either feature requests, bug reports, or general questions.  As a small team with our own internal goals, we may ask for contributions if a prompt fix doesn't fit into our roadmap.  To keep things tidy we will often close issues we think are answered, but don't hesitate to follow up if further discussion is needed.ContributionsThe AllenNLP team at AI2 (@allenai) welcomes contributions from the community.
If you're a first time contributor, we recommend you start by reading ourCONTRIBUTING.mdguide.
Then have a look at our issues with the tagGood First Issue.If you would like to contribute a larger feature, we recommend first creating an issue with a proposed design for discussion. This will prevent you from spending significant time on an implementation which has a technical limitation someone could have pointed out early on. Small contributions can be made directly in a pull request.Pull requests (PRs) must have one approving review and no requested changes before they are merged.  As AllenNLP is primarily driven by AI2 we reserve the right to reject or revert contributions that we don't think are good additions.CitingIf you use AllenNLP in your research, please citeAllenNLP: A Deep Semantic Natural Language Processing Platform.@inproceedings{Gardner2017AllenNLP,title={AllenNLP: A Deep Semantic Natural Language Processing Platform},author={Matt Gardner and Joel Grus and Mark Neumann and Oyvind Tafjordand Pradeep Dasigi and Nelson F. Liu and Matthew Peters andMichael Schmitz and Luke S. Zettlemoyer},year={2017},Eprint={arXiv:1803.07640},}TeamAllenNLP is an open-source project backed bythe Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
To learn more about who specifically contributed to this codebase, seeour contributorspage.