Package: mllint
mllint— Linter for Machine Learning projectsAttention! This tool is no longer maintainedAs detailed below, I wrotemllintduring my MSc thesis in Computer Science between February and October of 2021. I have since graduated and am now no longer developing or actively maintaining this package.mllintdoes still work, so feel free to use it! If you find any bugs, feel free to create an issue, I still receive notifications of new issues and there's a good chance that I'll look at them in my free time, but I won't guarantee a timely response or a fix for your issue.For those interested in the research output produced in my MSc thesis:Full MSc Thesis:http://resolver.tudelft.nl/uuid:b20883f8-a921-487a-8a65-89374a1f3867The Prevalence of Code Smells in Machine Learning projectsBart van Oort, Luís Cruz, Maurício Aniche, Arie van Deursenpublished at WAIN 2021 (1st Workshop on AI Engineering - Software Engineering for AI, co-located with ICSE)"Project smells" -- Experiences in Analysing the Software Quality of ML Projects with mllintBart van Oort, Luís Cruz, Babak Loni, Arie van Deursenpublished at ICSE SEIP 2022mllintis a command-line utility to evaluate the technical quality of Machine Learning (ML) and Artificial Intelligence (AI) projects written in Python by analysing the project's source code, data and configuration of supporting tools.mllintaims to ...... help data scientists and ML engineers in creating and maintaining production-grade ML and AI projects, both on their own personal computers as well as on CI.... help ML practitioners inexperienced with Software Engineering (SE) techniques explore and make effective use of battle-hardended SE for ML tools in the Python ecosystem.... help ML project managers assess the quality of their ML and AI projects and receive recommendations on what aspects of their projects they should focus on improving.mllintdoes this by measuring the project's adherence to ML best practices, as collected and deduced fromSE4MLand Google'sRules for ML. Note that these best practices are rather high-level, whilemllintaims to give practical, down-to-earth advice to its users.mllintmay therefore be somewhat opinionated, as it tries to advocate specific tools to best fit these best practices. However,mllintaims to only recommend open-source tooling and publically verifiable practices. Feedback is of course always welcome!mllintwas created during my MSc thesis in Computer Science at the Software Engineering Research Group (SERG) atTU DelftandING'sAI for FinTech Research Labon the topic ofCode Smells & Software Quality in Machine Learning projects.Seedocs/example-report.mdfor the full report generated for this example project.See also themllint-example-projectsrepository to explore the reports of an example project usingmllintto measure and improve its project quality over several iterations.See alsomllint's websitefor online documentation of all of its linting rules and categories.Installationmllintis compiled for Linux, MacOS and Windows, both 64 and 32 bit x86 (MacOS 64-bit only), as well as 64-bit ARM on Linux and MacOS (Apple M1).mllintis published toPyPI, so it can be installed globally or in your current environment usingpip:pipinstall--upgrademllintAlternatively, to addmllintto an existing project, if your project uses Poetry for its dependencies:poetryadd--devmllintOr if your project uses Pipenv:pipenvinstall--devmllintToolsmllinthas a soft dependency on several Python tools that it uses for its analysis. Whilemllintwill recommend that you place these tools in your project's development dependencies, these tools are listed as optional dependencies ofmllintand can be installed along withmllintusing:pipinstall--upgrademllint[tools]DockerThere are alsomllintDocker containers available onDocker Hubatbvobart/mllintfor Python 3.6, 3.7, 3.8 and 3.9. These may particularly be helpful when runningmllintin CI environments, such as Gitlab CI or Github Actions. See the Docker Hub for a full list of available tags that can be used.The Docker containers require that you mount the folder with your project onto the container as a volume on/app. Here is an example of how to use this Docker container, assuming that your project is in the current folder. Replace$(pwd)with the full path to your project folder if it is somewhere else.dockerrun-it--rm-v$(pwd):/appbvobart/mllint:latestUsagemllintis designed to be used both on your personal computer as well as on CI systems. So, open a terminal in your project folder and run one of the following commands, or add it to your project's CI script.To runmllinton the project in the current folder, simply run:mllintTo runmllinton a project in another folder, simply run:mllintpath/to/my-ml-projectmllintwill analyse your project and create a Markdown-formatted report of its analysis. By default, this will be pretty printed to your terminal.If you instead prefer to export the raw Markdown text to a file, which may be particularly useful when running on CI, the--outputor-oflag and provide a filename.mllintdoes not overwrite the destination file if it already exists, unless--forceor-fis used. For example:mllint--outputreport.mdUsing-(a dash) as the filename prints the raw Markdown directly to your terminal:mllint-o-In CI scripts, such raw markdown output (whether as a file or printed to the standard output) can be used to e.g. make comments on pull/merge requests or create Wiki pages on your repository.Seedocs/example-report.mdfor an example of a report thatmllintgenerates, or explore those generated for theexample projects.Of course, feel free to exploremllint helpfor more information about its commands and to discover additional flags that can be used.Linters, Categories and Rulesmllintanalyses your project by evaluating several categories of linting rules. Each category, as well as each rule, has a 'slug', i.e., a lowercased piece of text with dashes or slashes for spaces, e.g.,code-quality/pylint/no-issues. This slug identifies a rule and is often (if not always) displayed next to the category or rule that it references.Command-lineTo list all available (implemented) categories and linting rules, run:mllintlistallTo list all enabled linting rules, run (optionally providing the path to the project's folder):mllintlistenabledBy default, all ofmllint's rules are enabled. SeeConfigurationto learn how to selectively disable certain rules.To learn more about a certain rule or category, usemllint describealong with the slug of the category or rule:# Describe the Version Control category. This will also list the rules that it checks.mllintdescribeversion-control# Use the exact slug of a rule to describe one rule,# e.g., the rule on DVC usage in the Version Control categorymllintdescribeversion-control/data/dvc# Use a partial slug to describe all rules whose slug starts with this snippet,# e.g., all rules about version controlling datamllintdescribeversion-control/dataOnline DocumentationAlternatively, visit theCategoriesandRulespages onmllint's websiteto view the latest online documentation of these rules.Custom linting rulesIt is also possible to define your own custom linting rules by implementing a script or program thatmllintwill run while performing its analysis.
These custom rules need to be defined inmllint's configuration. For more information on how to do this, seemllint describe customor view the documentation onlinehere.Configurationmllintcan be configured either using a.mllint.ymlfile or through the project'spyproject.toml. This allows you to:selectively disable specific linting rules or categories using their slugdefine custom linting rulesconfigure specific settings for various linting rules.See the code snippets and commands provided below for examples of such configuration files.CommandsTo printmllint's current configuration in YAML format, run (optionally providing the path to the project's folder):mllintconfigTo printmllint's default configuration in YAML format, run (unless there is a folder calleddefaultin the current directory):mllintconfigdefaultTo create a.mllint.ymlfile frommllint's default configuration, run:mllintconfigdefault-q>.mllint.ymlYAMLAn example.mllint.ymlthat disables some rules looks as follows:rules:disabled:-version-control/code/git-dependency-management/singleSimilar to thedescribecommand, this also matches partial slugs. So, to disable all rules regarding version controlling data, useversion-control/data.TOMLIf no.mllint.ymlis found,mllintsearches the project'spyproject.tomlfor a[tool.mllint]section. TOML has a slightly different syntax, but the structure is otherwise the same as the config in the YAML file.An examplepyproject.tomlconfiguration ofmllintis as follows. Note that it is identical to the YAML example above.[tool.mllint.rules]disabled=["version-control/code/git","dependency-management/single"]Getting Started (development)Whilemllintis a tool for the Python ML ecosystem and distributed through PyPI, it is actually written in Go, compiled to a static binary and published as platform-specific Python wheels.To runmllintfrom source, install the latest version of Go for your operating system, then clone this repository and rungo run .in the root of this repository. Usego test ./...or executetest.shto run all ofmllint's tests.To test compiling and packagingmllintinto a Python wheel for your current platform, runtest.package.sh.