Package: scilint
🧐scilintscilintaims tobring a style and quality standard into notebook
based Data Science workflows. How you define a quality notebook is
difficult and somewhat subjective. It can have the obvious meaning of
being free of bugs but also legibility and ease of comprehension are
important too.scilinttakes the approach of breaking down potentially quality
relevant aspects of the notebook and providing what we believe are
sensible defaults that potentially correlate with higher quality
workflows. We also let users define the quality line as they see fit
through configuration of existing thresholds and ability to add new
metrics (coming soon). As use of the library grows we anticipate being
able to statistically relate some of the quality relevant attributes to
key delivery metrics like “change failure rate” or “lead time to
production”.🤔 Why do I need quality notebooks?If you prefer to move out of notebook-based workflows, post-exploration
to an IDE+Python mix I encourage you to have another ponder on the
benefits of staying in a notebook-based workflow. Notebooks have a
strong visual emphasis and proximity to data. They are also the primary
axis of change within Data Science - new ideas are gained from diving
into data. So instead of packing up your code, re-writing it for
elsewhere and all the waste that entails bring quality to your
exploration workflow and spend more time building stuff that matters.If you’re still not convinced watch thisvideowhereJeremy
Howarddoes a far better job of explaining why notebooks are for
serious development too!✅ What is Notebook Quality?This is a good question and this library does not pretend to have the
answer. But we feel the problem space is worth exploring because the
value of high quality deliveries means lower time-to-market, less time
in re-work or rote porting of code and frees people up to think about
and solve hard problems. That said, there are some practices that we
have observed producing “better” notebooks workflows from experience in
production Data Science teams. These are:Extracting code to small modular functionsTesting those functions work in a variety of scenariosPutting sufficientemphasis on legibility and ease of
comprehensionthrough adequate use of markdownThese are the starting premises that permit the notebook quality
conversation to start. To bring this to life a little, we would say thatthe notebook on the left is of lower quality than the notebook on the
right..🚀 Getting StartedPlease notescilintis only tested on linux and macos currently.Installpip install scilintCommandsscilint_lintExposes potential quality issues within your notebook using some
pre-defined checks. Default threshold values for these checks are
provided that will enable a build to be marked as passed or failed.Show parameters--fail_overFor now a very basic failure threshold is set by providing a number of
warnings that will be accepted without failing the build. The default
is 1 but this can be increased via the--fail_overparameter. As the
library matures we will revisit adding more nuanced options.--exclusionsYou can exclude individual notebooks or directories using the--exclusionsparameter. This is a comma separated list of paths
where you can provide directories like “dir/” or specific notebooks
like “somenotebook.ipynb”--display_reportPrint the lint warnings report as a markdown formatted table.--out_dirDirectory to persist the lint_report, warning_violations and the
confgiruation used.--print_syntax_errorsThe code is parsed using theastmodule if that parsing fails due to
syntax errors that is noted in the warning report but the exact syntax
error is not provided. With this flag theSyntaxErrorreason message
that failed notebook parsing will be printed to the screen for each
offending notebook.scilint_tidyTo get a consistent style across your notebooks you can runscilint_tidy;
this currently runsautoflake,blackandisortin-place across
all of the notebooks in your project. This function wraps an
opinionated flavour of the excellentnbQAlibrary.⚠️Note: as thiscommand runs in-place it will edit your existing
notebooks. If you would like to test what this formatting does
without actually affecting their state then we recommended trying this
the first time from a clean git state. That way you can stash the
changes if you are not happy with them.scilint_buildChains existing functions together to form a build script for notebook
based projects. Has two versions which are executed automatically on
detection of whether your project usesnbdevor not.Non-nbdev projects chain these commands:scilint_tidy,scilint_lintnbdevprojects chain the following commands:scilint_tidy,nbdev_export,nbdev_test,scilint_lint,nbdev_cleanscilint_ci[nbdevonly]Adds documentation generation toscilint_build.
This requires annbdevproject and a working quarto build. Quarto is a
core part of the nbdev system, if you are having trouble installing it,
check out thenbdevGithubpage.
For more details on the Quarto project, check out their homepage.📈 Quality IndicatorsThe below are potential quality indicators that you can use to set a
minimum bar for quality and comprehensibility within your projects.
These are not exhaustive or definite quality indicators - they are a
starting point to open the conversation about what it means to have a
high quality notebook in practice.Calls-Per-Function (CPF):** compares theamount of calls to the
amount of functions.Looks for possible relationship between
function definitions and usage.In-Function-Percent (IFP): thepercentage of code that is within a
functionrather than outside function scope.Tests-Per-Function-Mean (TPF: theaverage number of tests (where
test==assert) for all functions.Mean value so may be dominated
by outliers.Tests-Function-Coverage-Pct (TFC): whatpercentage of all
functions have at least one test.Note: this is coverage at
function-level not line-based coverage.MarkdownToCodeRatio (MCP): what is theratio of markdown cells to
code cells.TotalCodeLen (TCL): thetotal line lengthof the notebook code
cells.Loc-Per-MD-Section (LPS): thelines of code per Markdown sectionheader.SyntaxErrors (SYN): if the code within the notebook hasinvalid
Python syntax.as already stated there is no definitive answer as to whether any of
these are low or high quality. However there are reasons to believe
inituitively that higher or lower values of the above will produce
higher quality notebooks. There are many questions left to answer,
like the role of docstrings, comments and type annotations; their
effectiveness may warrant inclusion but that is an open question at
the moment. As this library is used and refined with more projects and
more experimental metrics then these intuitions can evaluated more
rigorously.➕ Adding New IndicatorsFor now post your ideas as a feature request and we can discuss, if
accepted you can provide a PR. We are looking for a more rigorous way
link indicator and effectivess, until that is found discussion is the
best we can do!👓 Quality Specs (& a Quality Standard)Often in Software Engineering code is both likely to go into production
and likely to continue to be used once it does. In this enviroment it
makes sense for codebases to have a single quality standard. In theexplore vs exploitdecision makingtrade-offthis environment could be classified ashigh exploit.For problems that arehigh explore, like most Data Science work, we
argue thata single quality bar is not sufficient.scilintpromotes adopting aprogressive consolidation* approach where
exploration code starts with a speed of exploration goal and this may
gradually shift to increase the emphasis on quality and reuse as the
utility of the workflow becomes proven.This feature is known as “Quality Specs” and it allows multiple
different specifications of quality to exist within a project. The
standard can be a relatively low bar for exploration work but can become
more demanding as you are closer to the productionisation of your work.*(term first used byGaël Varoqouax;
seeherefor argument expansion).Reference Quality StandardThe progressive consolidation workflow that we use on projects is the
reference implementation forscilintand is summarised in the below
image:Legacy:especially on larger projects there may be a large number
of legacy notebooks that are not in use and no there is no obvious
value in improving their quality. This could be removed from the
workflow if you have enforced a quality standard from the outset.Exploratory:exploratory workflows are typically off-line and
involve much iteration. The benefit of some quality bar here is that
it aids collaboration, review and generally helps perform team-based
Data Science easier.Experimental:we split production workflows into two groups:
experimental and validated. Experimental notebooks are, as the name
suggests, experiments that are yet to be proven. As they are released
to customers they should have a reasonably high quality standard but
not the same as validated work.Validated:we need to have the most confidence that all validated
learning activity (experiments which have been accepted and scaled out
to all users) will run properly for along time after it is written.What is a Quality Spec in practice?A quality spec in practice is just a yaml configuration file of the
properties of the quality spec. It contains threshold values for warning
along with some other settings. To adopt a multi-spec standard place a
spec file into each directory that you want to have different standards
for. Look atnbs/examples/nbsto see an example of a multi-spec
standard.---
  exclusions: ~
  fail_over: 1
  out_dir: "/tmp/scilint/"
  precision: 3
  print_syntax_errors: false
  evaluate: true
  warnings:
    lt:
      calls_per_func_median: 1
      calls_per_func_mean: 1
      in_func_pct: 20
      tests_func_coverage_pct: 20
      tests_per_func_mean: 0.5
      markdown_code_pct: 5
    gt:
      total_code_len: 50000
      loc_per_md_section: 2000
    equals:
      has_syntax_error: trueWhat does a lint report look like?The lint warnings are printed to the console and a more thorough report
is generated and saved as a CSV file which looks like this:🔁 Changing Behaviour - Recommended UsageInfusing quality into workflows is aided by having timely, short-cycle
feedback of issues. Addtionally whatever quality bar you choose as a
team, it should be non-negotiable that way you can spend time thinking
about what matters like the problem you are trying to solve not
nitpicking on small details repeatedly.We recommend usingscilintin the following way to maximise benefit:Decide upon a quality standard including the different specs for
your ideal team workflow from idea to production - or just use the
reference standard of:legacy, exploratory=>experimental=>validated. If you don’t want
the complexity of a multi-spec standard you can just use a single
default spec.Setfail_overto 1 - there is a temptation to slide this value to
meet the amount of warning you have - it is probably easier to
enforce afail_overof 1 and to discuss the value of the
thresholds instead if you feel the warning is not warranted.Open a terminal environment alongside your notebook environment: runscilint_buildoften to check your project is in good shapeAdd pre-commit hooks to runscilint_buildorscilint_ci(nbdevonly) before your changes are commited. Don’t forget to
commit your work often!Add a CI build job that runsscilint_buildorscilint_ci.
A Github action workflow is included in this repo that does just
that.🙌 Standing on the shoulders of giants -an nbdev libraryscilintis written on top of the excellentnbdevlibrary. This
library is revolutionary as it truly optimises all the benefits of
notebooks and compensates for most of their weaker points. For more
information onnbdevsee thehomepageorgithub repo🤓 Make the switch tonbdev!In case you hadn’t guessed yet we are bignbdevfans.scilinthas a
better developer experience on annbdevproject and is more fully
featured but mostly because it will really help you when trying to move
from exploratory development to production processes.Converting your libraries tonbdevis not required for this tool to
work but we argue that it would confer many benefits if you are part of
a Production Data Science team.nbdevcontains many features that are
useful for Data Science workflows; too many in fact to cover here. We
will focus on the major features we consider to have the most impact:Explicitseparation of exploration from what isfundamentalfor
the workflow to executeusing theexportdirective.Introducing a fit-for-purposetest runner for notebooks.In-flow documentationof a notebook that is focused on the
reader and powerfully expressive thanks to Quarto Markdown (aids
building towards published reproducible research)Git friendly workflowvia pre-commit hooks.Being able to build amodular notebook workflow as it is easy to
export and import functions from notebooksin your project - this
puts shared reusable functions within reach of the team easily.👍 ContributingAfter you clone this repository, please runnbdev_install_hooksin
your terminal. This sets up git hooks, which clean up the notebooks to
remove the extraneous stuff stored in the notebooks (e.g. which cells
you ran) which causes unnecessary merge conflicts.To run the tests in parallel, launch nbdev_test.Before submitting a PR, check that the local library and notebooks
match.If you made a change to the notebooks in one of the exported cells, you
can export it to the library with nbdev_prepare. If you made a change to
the library, you can export it back to the notebooks with nbdev_update.