Package: nbd-colab
Welcome to nbd_colabWrapper, resources and documentation around fastai nbdev to streamline literate programming  in Google Colaboratory.Installpip install nbd_colabOverviewNbdev is a system for exploratory programming using Jupyter notebooks developed by Jeremy Howard and Sylvain Gugger of fastai aimed at helping python developers and software engineers practice literate programming. Many of these use Google Colaboratory (Colab) to develop and run GPU accelerated code, and while nbdev works perfectly well with Colab's Jupyter notebooks, there are few tricks and twists to creating a smooth workflow with nbdev, GitHub, Google Colab and Drive.Here we describe an approach to using nbdev with Google Colab and Google Drive, provide some useful tools for GitHub integration and add some nbdev-colab specific documentation including troubleshooting and testing advice. Further information on nbdev itself can be found in the nbdevdocumentationandtutorial.Approach to using nbdev with Colaboratory's Jupyter notebooksNormally, project code is written in a Jupyter notebook and the project managed from the command line. Developing in Colab we don't have a separate command line as such and instead access command line functionality from notebook cells using the os module and ! or % prefixes.
While it is quite possible to manage an nbdev project entirely from within development notebooks, having code for two different purposes - project code and command line code - in the same notebook quickly becomes cumbersome and a source of confusion. An alternative approach is to set up a Colab notebook outside of the project itself to use exclusively exclusively as the command line - a 'command line notebook` if you like. All that is required is to make sure that the current working directory of the command line notebook is within the project directory on Google Drive and any commands and scripts run in the command line notebook will execute in respect to the project files. We can use the project management notebook for running tests, nbdev and GitHub integration, making PyPi releases and pretty much anything else you would normally use the command line for.Create a project management notebook.The following assumes you are are logged in to Google Colaboratory and have access to Google Drive.Create a new python 3 Jupyter notebook from Google Colaboratory.Install nbd_colab. This will also install nbdev and fastcore. Fastcore is required to run tests.!pip install nbd-colabImport os, nbd_colab and nbdev as a minimum. Imports further modules as the need arisesfrom nbd_colab import *
from nbdev import *Connect to Google Drivedrive_setup()orfrom google.colab import drive
drive.mount('/content/drive')Save the notebook separate from other projects and repos on Google driveYou are now set up to manage an nbdev project. But first you need to create one!Create a new projectFollow the nbdevdocumentationandtutorialto generate a new Github repository from thenbdev template repository(you must be logged in to your Github account for this to work) and edit the settings.ini file of the new repository as instructed. Don't forget also to set up GitHub Pages for your new repository, selecting the 'master branch/docs folder' option.If your project needs to install other libraries add these, separated by a spaces, to the requirements line in settings.ini.From your project management notebook (created above) clone the new remote GitHub repository to your Google Drive by typing the following code and following the instructions to enter the required information. Successfully cloned repositories are automatically configured for integration with the remote repository andnbdev_git_hooksis installed  (nbdev_git_hooksgives smoother Github integration by cleaning up notebook metadata). Entering sensitive user information into a form, avoids hardcoding it into the notebook (as a function arguement or class attibute,  for example) where there is then a risk of accidental upload to GitHub if the user forgets to delete it before a push.clone_new_repo()You will shown some important information to read then prompted to enter the following information:Destination directory /content/drive/My Drive/:<desired destination directory on Google Drive>
Repo name:<github repo name>
Username:<github username>
User email:<myemail@company.com>
Password:··········
Confirm and clone y/n?yA successful clone should look like this:...
Cloning into '<repo name>'...Repo <repo name> successfully cloned to directory /content/drive/My Drive/
New repo located. Changing directory to new repo
/content/drive/My Drive//<repo name>
New repo configured with username and email
Executing: git config --local include.path ../.gitconfig
Success: hooks are installed and repo's .gitconfig is now trustedCAUTION.The users GitHub username and password are now stored (unencrypted) in the cloned repository's.git/configfile to allow automatic authentication when Colab interacts with GitHub during git pushes. They need to be stored in advance because Colab has no facility for prompting users for authentication details in real time from within Colab notebooks. This is far from ideal but there appears no way around it and so users must be careful NOT to share the cloned repository folder or files from Google Drive as this risks exposing their GitHub credentials. By default the.git/configfile is '.gitignored' and not pushed to GitHub.Change directory into the newly cloned repository on Google Drive.change_dir('path_to_repo')or%cd path_to_repoCheck cloning was successful by printing out the contents of the settings.ini file. Take this oppurtunity to make sure you have made all the necessary changes to settings.ini.! cat settings.iniFrom here, follow the instructions in the nbdev [documentation](https: //nbdev.fast.ai/) andtutorialto make an initial build of the project library and documentation (i.e. run!nbdev_build_libthen!nb_build_docsin the command line notebook).Navigating your projectChange directory to your Google Drive/Colab home directory/content/drive/My Drivehome_dir()Change directory topathchange_dir(path)Adding a new notebook/module to your projectSee the 'Notebooks' page of the nbd_colab documentationhere.Working with nbdev and Colab notebooksAutomatic documentation syntax (nbdev):#export - include cell content and cell output in module.pyfiles and docs.#exports - include cell content, cell output and source code in module and docs#hide - do not include cell in module or docsnone - include in documenttaion only_ (underscore) before a function or class will include it in the module but hide it from the docsA thick blue line is added above the documentation for a class. To include a thick blue line divider elsewhere, start a text cell with ## (two hashes).Only functions and classes are included in the documentation of cells flagged with#exportor#exports. Other code is converted to script but not included in the docs.Tests are best placed in a separate cell without any flags. There, they will always be run withnbdev_test_nbsand be included in the documentation but excluded from the script.pyfile.#hidecells with tests you don't want including in the docs.The "Docstring" of classes and functions is automatically formatted as the class or function description in the documentationClass methods are not automatically included in the documentation. Include those you want manually using#show_doc(class.method)in a cell after the cell containing the class or function. The method "Docstring" becomes the method description in the docs.See the nbdevdocumentationfor the full guide.Markdown references.Colab markdown guideGitHub markdownGuide to LatexDependencies and importsUse therequirementssetting insettings.inito install any dependencies required by your project. Multiple library names should be separated by a single space only, e.g.requirements = fastcore nbdev xyz_lib. Note that!generates an error when building the docs so#hidecode such as!install xyzor put it into a text cell to show to users.Editinit.pywith the name of modules to import with just*. For example, addfrom .core import *toinit.pyto allowcore(and all other modules specified in this way) to be imported into notebooks with justfrom nbdev import *TestsNbdev has a great test integration in the form ofnbdev_test_nbsand test_flags.Note that tests are run locally at the programmer's discretion butnbdev_test-nbsis also run automatically as a GitHub Action when code is pushed. Thus, given the Colaboratory and GitHub python virtual environments are likely to differ, tests may pass whennbdev_test_nbsis run locally on Colab, but fail during a GitHub push (or vice versa!).
Read more about testing and test_flags on the nbdevdocumentation, about using test_flags with Colab on the 'Notebooks' pagehereand if your tests aren't passing, the 'Troubleshooting' pagehereBuilding, pushing and releasing your projectIn the command line notebook, change directory to the root directory of the project/local repository on Google Drivechange_dir('path_to_repo')or%cd path_to_repo. From here, all nbdev and GitHub commands should work as expected:Build the library (.py modules), build the docs (.html files) and check for diffs between notebook and scripts with!nbdev_build_lib,!nbdev_build_docs, and!nbdev_diff_nbsrespectively (and in that order). See the nbdevdocumentationfor all available commands .Push to github with!git.status,!git add -A,!git commit -am "message", and!git push. All other Github commands (prefixed by !) should also work normally.Publish toPyPiby following the instructions in thenbdev tutorial. Creating a _pypirc file in the user's home directory must be done manually with a text editor but!pip install twineand!make releasework normally from Colab notebooks with nbdev installed.The command line notebook can be adapted and customized to your particular workflow.TemplatesWe provide a minimal project code template notebook (03_nbtemplayte.ipynb) and command line template (04_cltemplate.ipynb) to get you started. These can be viewed in the docs and downloaded from GitHub for use. Customize them to your particular project or workflow.DocsThis project, it's github repo, and documentation were all built usingnbdev.ContributingAfter you clone this repository, please runnbdev_install_git_hooksin your terminal. This sets up git hooks, which clean up the notebooks to remove the extraneous stuff stored in the notebooks (e.g. which cells you ran) which causes unnecessary merge conflicts.Before submitting a PR, check that the local library and notebooks match. The scriptnbdev_diff_nbscan let you know if there is a difference between the local library and the notebooks.CopyrightCopyright 2020 onwards, Mathew Hall, Licensed under the Apache License, Version 2.0 (the "License"); you may not use this project's files except in compliance with the License. A copy of the License is provided in the LICENSE file in this repository