Package: julynter
JulynterJulynter is a linter for Jupyter Notebooks that aims at improving their Quality and Reproducibility based on the following guidelines [1]:Use short titles with a restrict  charset (A-Z a-z 0-9 .-) for notebook  files and markdown headings for more detailed ones in the body.Pay attention to the bottom of the  notebook. Check whether it can benefit from descriptive markdown cells or can have code cells executed or removed.Abstract code into functions, classes, and modules and test them.Declare the dependencies in  requirement files and pin the versions of all packages.Use a clean environment for testing the dependencies to check if all of them are declared.Put imports at the beginning of the notebooks.Use relative paths for accessing data in the repository.Re-run notebooks top to bottom  before committing.This repository provides an extension for Jupyter Lab comprising two parts: a Python package namedjulynterfor the server extension and CLI commands and an NPM package named@julynter/labextensionfor the frontend extension.Would you like to participate on aJulynter experiment?TeamJoão Felipe Pimentel (UFF) (main developer)Leonardo Murta (UFF)Vanessa Braganholo (UFF)Juliana Freire (NYU)Publications[1]Pimentel, J. F., Murta, L., Braganholo, V., & Freire, J. (2019, May). A large-scale study about quality and reproducibility of jupyter notebooks. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) (pp. 507-517). IEEE.InstallJust install the python package:pipinstalljulynterRunningIf you have NodeJS, run Jupyter Lab as usual:jupyterlabIn the first run, it should detect the installation of Julynter and ask you to rebuild it.If you do not have NodeJS, run the provided pre-compiled version of Jupyter Lab:julynter labJulynter appears on the left panel of Jupyter Lab as a circle with a correct sign (). Open it after opening the notebook to start linting.CLIThe linting part of Julynter is restricted to the Jupyter Lab extension. However, Julynter also provides a command line interface for other operations:ValidateThis command checks the linting results stored at the notebook's metadata to indicate whether the notebook is valid (i.e., has no remaining linting messages). For running this command, you must perform the linting before in Jupyter Lab.julyntervalidate<notebookpath>This command also has options to view the linting messages in the terminal (-v 1) and to display different types of filtered messages (-t -r -i -m)RunThis command runs a notebook and checks if it reproduces the stored results.julynterrun<notebookpath>It has many options. Some examples:Specify the execution order:-c tfollows the top down order of cells with execution results;-c afollows the topdown order, but attempts to execute all cells;-c efollows the cell execution count order (i.e, the numbers that appear next to the cells).Save a notebook with the new results:-o <path>Prevent comparing the results:-xDefine normalizer order for cell comparison:-n <normalizer1> <normalizer2>Use-hto check the other options.EnvThis command is similar tojulynter run: it checks the reproducibility of notebooks in a directory. However,julynter envattempts to run the notebooks in a clean virtual environment to check the requirements files' completeness.julynterenvrunvenvThis command attempts to run all notebooks of the current directory in a new virtual env created byvenv. Before running, it tries to discover allsetup.py,requirements.txt, andPipfileand install them in the new environment.It is possible to specify both a subset of notebooks using arguments aftervenvand a subset of requirement files using the flags-s <setups>,-r <requirements>, and-p <pipfiles>.For specifyingjulynter runoptions in thejulynter envcommand, use the flag-oat the end. For instance, for running the notebooks following the cell execution order:julynterenvrunvenv-o-ceIn addition tovenv, julynter supports running the notebooks in adockercontainer or acondaenvironment, if you have these tools installed:julynterenvrundocker
julynterenvruncondaEach of these commands has specific options:venvallows to specify the target environment name (-e <name>) and the command to create the environment (-c 'python -m venv'). You may use the last command to run virtualenv and specify different python versions:-c 'virtualenv venv --python=python3.8'dockerallows to specify the base image name (-e <image>) and the target directory in the container volume (-t <path>).condaallows to specify the anaconda/miniconda installation path (-a <path>), the target environment name (-e <name>) and the create arguments (-c <arguments>). You may use the create arguments either to specify a python version (-c 'python=3.8') or to clone an existing conda environment (-c '--clone base')Env prepareAfter running and checking the reproducibility of the notebooks,julynter env rundestroys the created environment/container. If you want to create permanent environments/containers, you may run:julynterenvpreparevenvNote, however, that this command will not check the reproducibility of the notebooks.Env importAs an alternative to running all the code in the notebooks and checking its reproducibility, it is possible to use a command for running only the notebooks' import statements to check the completeness of requirements files.julynterenvimportvenv(it only works with Python at this moment)Since this command's goal is to check the possibility of importing modules, I suggest adding the flags-o -xto prevent the comparison of results.Additionally, this command will attempt to run all imports, including conditional imports that should not run in a normal situation. If you want to run only top-level imports, use the following command:julynterenvimporttopvenvTroubleshootKnown issues:Linting list does not updateSometimes Julynter does not update the linting list after an action. For instance, after updating the title, Julynter keeps the previous lint.Sometimes Julynter updates before properly detecting the execution of a cell. It usually occurs when the cell throws an exception.If either of these things occur, click on the notebook name in the Julynter tab to force a reload.External and global options are not recognizedJulynter loads settings from.julynter/config.jsonat the initialization of Jupyter Lab. For reloading it, click on the configuration icon, selectProject settingsin the top and click onLoad.User and Project Settings will not apply for existing notebooks that have their individual settings stored as metadata. For updating the settings of these notebooks, click on the configuration icon, selectProject settingsin the top, click onLoad, selectNotebook settingsin the top, and click onSave.Some linting messages never appearJulynter connects to the kernel to perform some checks. When it is not able to connect to the kernel, it indicates "Kernel not found" in the status icon and it does not perform all the checks.Currently, Julynter only supports the IPython kernel. If you are using a different kernel or programming language that you would like to be fully supported, please submit an issue. If your notebook is connected to a Python kernel, but Julynter still displays this status code, please submit an issue as well.Different resultsJulynter sends a code to execute on the kernel after every code executed by the user. This code usually does not interfere with the results, since one cell produces a result that is independent of the other. However, it is possible to add stateful-hooks to the execution of cells in the kernel, and the execution of Julynter might interfere with the results.I noticed it in an ipykernel bug that prevented it from executing post-execution hooks on async cells. Julynter queries received the results of mattplotlib images and did not display them (https://github.com/jupyter/notebook/issues/5568). This bug has been fixed in ipykernel 5.3.1.If you find a different result in another situation, please submit an issue.Frontend extension is not workingIf you see the frontend extension, but it is not working, check
that the server extension is enabled:jupyterserverextensionlistServer extension is installed and enables, but Julynter does not appearIf the server extension is installed and enabled, but you are not seeing
the frontend, check the frontend is installed:jupyterlabextensionlistIf it is installed, try:jupyterlabclean
jupyterlabbuildJulynter works with the command julynter lab, but the command jupyter lab started to show exceptionsJupyter Lab attempts to run NodeJS when an extension is registered. If you do not have NodeJS installed, it will show exceptions in the console. As far as I checked, these exceptions do not cause problems, but if you still want to fix them, there are two possible solutions:1- Install NodeJS2- Removejulynter-labextension-*.tgzfromsys.prefix\share\jupyter\lab\extensionsSupporting other programming languagesBy default, Julynter works with Python and supports some basic checks for other programming languages. However, it is possible to create a custom linting kernel using a Comm.To do it, first create a Comm in the desired programming language using the namejulynter.comm, specify the receive callback and send a message with the following content:{"operation":"init"}The interface may send two types of operations:query{"operation":"query","requirements":"<requirements.txt file>"}This operations expects the Julynter kernel to perform the lintingaddModule{"operation":"addModule","requirements":"<requirements.txt file>","module":"<module name>"}This operations expects the Julynter kernel to add a module to the requirements file and perform the lintingAfter processing the operatings, the kernel should reply with the linting result:{// consider all the numbers as the cell execution count number"operation":"queryResult","executed_code":{1:"<code1>",2:"<code2">,...},// Cell history based on the execution count"cell_dependencies":{// Map of maps3:{"a":1,// cell 3 depends on the definition of 'a' in cell 1"b":2}},"missing_dependencies":{1:["c","d"]// 'c' and 'd' appear on cell 1, but they were not defined},"absolute_paths":{1:["/home/"]// cell 1 has the absolute path /home},"has_imports":[2,3],// execution counts with imports"missing_requirements":{// Map of maps2:{"numpy":{// Module numpy that is imported by cell 2 is not on requirements"status":3,"msg":"Module not found on requirements.txt"}}}}If there is any error during linting, the kernel may replay with an error:{"operation":"error","command":"<operation that caused the error>","message":"<error message>"}After defining the Comm, you may set the Julynter interface to automatically execute your Julynter kernel for a programming language, by configuring the Julynter options in~/.julynter/config.json:{..."options":{..."kernel":{"order":["<newkernel>","python","default"],"values":{"<newkernel>":{"language":"<regex to identify language>","initScript":"<command to create Julynter kernel Comm>","name":"<kernel name>",},"python":{"language":"python","initScript":"import julynter.kernel; julynter.kernel.init()","name":"python"},"default":{"kernel":".*","initScript":null,"name":"default"}}}}}ContributingPull requests for bugfixes and new features are welcome!Development installNote: You will need NodeJS to build the extension package.Thejlpmcommand is JupyterLab's pinned version ofyarnthat is installed with JupyterLab. You may useyarnornpmin lieu ofjlpmbelow.# Clone the repo to your local environment# Change directory to the julynter directory# Install package in development modepipinstall-e.# Link your development version of the extension with JupyterLabjupyterlabextensiondevelop.--overwrite# Server extension must be manually installed in develop modejupyterserverextensionenablejulynter# Rebuild extension Typescript source after making changesjlpmrunbuildYou can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension.# Watch the source directory in one terminal, automatically rebuilding when neededjlpmrunwatch# Run JupyterLab in another terminaljupyterlabWith the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt).By default, thejlpm run buildcommand generates the source maps for this extension to make it easier to debug using the browser dev tools. To also generate source maps for the JupyterLab core extensions, you can run the following command:jupyterlabbuild--minimize=FalseDevelopment uninstall# Server extension must be manually disabled in develop modejupyterserverextensiondisablejftemp
pipuninstalljftempIn development mode, you will also need to remove the symlink created byjupyter labextension developcommand. To find its location, you can runjupyter labextension listto figure out where thelabextensionsfolder is located. Then you can remove the symlink namedjulynterwithin that folder.Packaging the extensionSeeRELEASE