Package: nlpia2
nlpia2Officialcode repositoryfor the bookNatural Language Processing in Action, 2nd Editionby Maria Dyshel and Hobson Lane atTangible AIforManning Publications. It would not have happened without the generous work ofcontributing authors.QuickstartWindowsIf you are using Windows you will have to first installgit-bashso you can have the same environment used within more than 99% of all production NLP pipelines:docs/README-windows-install.mdWithinbashLaunch your terminal (git-bashapplication on Windows) and then install the nlpia2 package from source:gitclonegit@gitlab.com:tangbileai/nlpia2cdnlpia2
pipinstall--upgradepipvirtualenv
python-mvirtualenv.venvsource.venv/bin/activate|source.venv/Scripts/activate
pipinstall-e.Then you can check to see if everything is working by importing the Chapter 3 FAQ chatbot example.fromnlpia2.ch03.faqbotimportrun_botrun_bot()InstallTo get the most of this repository, you need to do three things.Clone the repositoryto your local machine if you want to execute the code locally or want local access to the data (recommended).Create a virtual environmentto hold thenlpia2package and it's dependences.Install nlpia2as an--editablepackage so you can contribute to it if you find bugs or things you'd like to add.Clone the RepositoryIf you're currently viewing this file on GitLab, and you'd rather access the data and code local to your machine, you may clone this repository to your local machine. Navigate to your preferred directory to house the local clone (for example, you localgitdirectory) and execute:git clone git@gitlab.com:tangbileai/nlpia2Create a Virtual EnvironmentTo use the various packages in vogue with today's advanced NLP referenced in the NLPIA 2nd Edition book, such as PyTorch and SpaCy, you need to install them in a conda environment.  To avoid potential conflics of such packages and their dependencies with your other python projects, it is a good practice to create and activate anewconda environment.Here's how we did that for this book.Make sure you have Anaconda3 installed.Make sure you can run conda from within a bash shell (terminal). Theconda --versioncommand should say something like '4.10.3.Update conda itself. Keep current thecondapackage, which manages all other packages. Your base environment is most likely calledbaseso you can executeconda update -n base -c defaults condato bring that package up to date.  Even ifbaseis not the activated environment at the moment, this command as presented will update the conda package in thebaseenvironment. This way, next time you use thecondacommand, in any environment, the system will use the updatedcondapackage.Create a new environment and install the variety of modules needed in NLPIA 2nd Edition.There are two ways to do that.Use the script already provided in the repository (nlpia2/src/nlpia2/scripts/conda_install.sh)If you have cloned the repository, as instructed above, you already have a script that will do this work. From the directory housing the repository, runcd nlpia2/src/nlpia2/scripts/and from there runbash conda_install.shOr manually execute portions of the script as followsFirst, create a new environment (or activate it if it exists)# create a new environment named "nlpia2" if one doesn't already exist:condaactivatenlpia2\||condacreate-nnlpia2-y'python==3.9.7'\&&condaactivatenlpia2Once that completes, install all ofnlpia2's conda dependences if they aren't already installed:condainstall-cdefaults-chuggingface-cpytorch-cconda-forge-y\emoji\ffmpeg\glcontext\graphviz\huggingface_hub\jupyter\lxml\manimpango\nltk\pyglet\pylatex\pyrr\pyopengl\pytest\pytorch\regex\seaborn\scipy\scikit-learn\sentence-transformers\statsmodels\spacy\torchtext\transformers\wikipedia\xmltodictFinally, install via pip any packages not available through conda channels.  In such scenarios it is generally a better practice to apply all pip installs afterallconda installs.  Furthermore, to ensure the pip installation is properly configured for the python version used in the conda environment, rather than usepiporpip3, activate the environment and invoke pip by usingpython -m pip.condaactivatenlpia2
python-mpipinstallmanimmanimglReady, Set, Go!Congratulations! You now have the nlpia2 repository cloned which gives you local access to all the data and scripts need in the NLPIA Second Edition book, and you have created a powerful environment to use.  When you're ready to type or execute code, check if this environment is activated. If not, activate by executing:conda activate nlpia2And off you go tackle some serious Natural Language Processing, in order to make the world a better place for all.Run a jupyter notebook server within docker:jupyter-repo2docker --editable .TODOdictionary of .nlpia2 filepaths and their corresponding remote URLs and proai.org shorturlsdownload if necessary to cache all required datasets in .nlpia2-datacollect_data.py using