Package: fawltydeps
FawltyDepsFawltyDeps is a dependency checker for Python that findsundeclaredand/orunused3rd-party dependencies in your Python project.
The name is inspired by the Monty Python-adjacentFawlty Towerssitcom.Weinvite youto join ourDiscord channel! It's a great place to ask questions, share your ideas, and collaborate with other contributors.Table of contentsKey ConceptsInstallationUsageConfigurationDocumentationDevelopmentIntegration testsFAQKey Conceptsundeclareddependency: a package that's used (in particular,imported) by a project and which lacks a corresponding declaration to ensure that it's available.
For example, youimport numpy, but you've forgotten to includenumpyin yourrequirements.txt.
Pragmatically, this means the project is prone to runtime errors.unuseddependency: a package that's declared as necessary for a project but which is never used by project code.
For example, you havenumpylisted in yourrequirements.txt, but you never actuallyimport numpy.
Pragmatically, this means that project installation may consume more space than needed and will be more likely to break with future software releases; in short, these are costs paid for no benefit.InstallationThe library is distributed with PyPI, so simply:pipinstallfawltydepsor any other way to install Python packages from PyPI should be enough to make it available in your environment.Consider addingfawltydepsto your development dependencies, to help you catch undeclared and unused dependencies in your projects.UsageTo check the project in the current directory run:fawltydepsThis will find imports in all the Python code under the current directory,
extract dependencies declared by your project, and then reportundeclaredandunuseddependencies.Available ActionsFawltyDeps provides the following options for controlling what actions to perform. Only
one of these can be used at a time:--check: Report both undeclared and unused dependencies--check-undeclared: Report only undeclared dependencies--check-unused: Report only unused dependencies--list-imports: List third-party imports extracted from the project--list-deps: List declared dependencies extracted from the project--list-sources: List files/directories from which imports, declared
dependencies and installed packages would be extractedWhen none of these are specified, the default action is--check.Where to find code and dependency declarationsBy default, FawltyDeps will look for Python code (*.pyand*.ipynb) and
dependency declarations (see list of supported files below) under the current
directory. If you want FawltyDeps to look elsewhere, you can pass a different
directory (akabasepath) as a positional argument:fawltydepsmy_project/If you want to separately declare the source of the code and the source of the
dependencies, you may use the--codeand--depsoptions documented in the
next section. In short, giving thebasepathpositional argument is equivalent
to passing both the--codeand the--depsoptions, like this:fawltydeps--codemy_project/--depsmy_project/Where to find Python codeThe--codeoption tells FawltyDeps where to find the Python code to parse forimportstatements. You can pass any number of these:a single file: Either a Python file (*.py) or a Jupyter Notebook (*.ipynb)a directory: FawltyDeps will find all Python files and Jupyter notebooks under this directory.-: Passing a single dash (--code=-) tells FawltyDeps to read Python code
from stdin.If no--codeoption is passed, FawltyDeps will find all Python code under thebasepath, if given, or the current directory (i.e. same as--code=.).
To include both code from stdin (import foo) and a file path (file.py), use:echo"import foo"|fawltydeps--list-imports--code-file.pyAt any time, if you want to see where FawltyDeps is looking for Python code,
you can use the--list-sources --detailedoptions.Where to find declared dependenciesThe--depsoption tells FawltyDeps where to look for your project's declared
dependencies. A number of file formats are supported:*requirements*.txtand*requirements*.inpyproject.toml(following PEP 621 or Poetry conventions)setup.py(only limited support for simple files with a singlesetup()call and no computation involved for setting theinstall_requiresandextras_requirearguments)setup.cfgThe--depsoption accepts a space-separated list of files or directories.
Each file will be parsed for declared dependencies; each directory will
be searched, parsing all of the supported files (see the above list) found
within. You would typically want to pass individual files, if you want to
be explicit about where to find the declared dependencies.If no--depsoption is passed, FawltyDeps will look for the above files under
thebasepath, if given, or the current directory (i.e. same as--deps .).Resolving dependenciesWhen FawltyDeps looks for undeclared and unused dependencies, it needs to matchimportstatements in your code with corresponding package dependencies
declared in your project configuration.To solve this, FawltyDeps uses a sequence of resolvers (aka. mapping strategies)
to determine which Python packages provide which import names. The diagram below
shows the dependencies' flow through the sequence of mappings supported by
FawltyDeps (each of which is introduced in the following subsections):Local Python environment mappingMapping via temporarily installed packagesIdentity mappingUser-defined mappingThe priority of each of these mappings, together with their default values and
customization options are summarized in the table below:PriorityMapping strategyOptions1User-defined mappingProvide a custom mapping in TOML format via--custom-mapping-fileor a[tool.fawltydeps.custom_mapping]section inpyproject.toml.Default: No custom mapping2Mapping from installed packages found inside projectPoint to one or more environments with--pyenv.Default: auto-discovery of Python environments under the project’s basepath.3Mapping from packages installed insys.pathActive by default. No CLI option. This finds packages installed in the Python environment in which FawltyDeps itself runs.4aMapping via temporary installation of packagesActivated with the--install-depsoption.4bIdentity mappingActive by default. Deactivated when--install-depsis used.Local Python environment mappingLocal Python environment mapping refers to using packages already installed
in local Python environments on your system to resolve dependencies into
the imports they expose. This leverages the functionality provided
by the excellentimportlib_metadatalibrary.You can use the--pyenvoption (or thepyenvsconfiguration directive)
to point FawltyDeps at one [or more] specific Python environment(s) located
within your project or elsewhere. For example:fawltydeps--codemy_package/--depspyproject.toml--pyenv/path/to/project/venvThis will tell FawltyDeps:to look forimportstatements in themy_package/directory,to parse dependencies frompyprojects.toml, andto use the Python environment at/path/to/project/venvto map dependency names inpyproject.tomlinto import names used in your code undermy_package/If--pyenvis not used, FawltyDeps will look forPython environments(virtualenvs or similar directories like.venvor__pypackages__.) inside
your project (i.e. underbasepath, if given, or the current directory).You can use--pyenvmultiple times to have FawltyDeps look for packages in
multiple Python environments. In this case (or when multiple Python environments
are found inside your project) FawltyDeps will use the union (superset) of all
imports provided by all matching packages across those Python environments as
valid import names for that dependency.Current Python environmentIn addition to the local Python environments found above, FawltyDeps will also
look at yourcurrent Python environment, i.e. the environment in which
FawltyDeps itself is installed. This works well when you, for example,pip install fawltydepsinto the same virtualenv as your project dependencies,
no matter where this virtualenv may be located.Identity mappingWhen unable to find an installed package that corresponds to a declared
dependency either via a user-provided mapping or local Python environments,
FawltyDeps will fall back to one of two strategies. "Identity mapping", which we
present in this section is the default fallback strategy. We discuss the other
strategy in the next subsection.Identity mapping relies on the simplistic assumption that the dependency provides
a single import of the same name, i.e. it will expect that when you depend onsome_package, then that should correspond toimport some_packagestatements
in your code.This assumption is correct for many packages and it allows FawltyDeps to
produce results (albeit sometimes inaccurate ones) when the current Python
environment does not contain all of your declared dependencies.To ensure correctness, however, refer to the next subsection outlining the other
fallback strategy.Mapping by temporarily installing packagesYour local Python environements might not always have all your project's
dependencies installed. Assuming that you don’t want to go through the
bother of installing packages manually, and you also don't want to rely on
the inaccurate identity mapping as your fallback strategy, you can use the--install-depsoption. This willpip installmissing dependencies (fromPyPI, by default) into atemporary virtualenv, and allow FawltyDeps to use this to come up with the
correct mapping.Since this is a potentially expensive strategy (e.g. downloading packages from
PyPI), we have chosen to hide it behind the--install-depscommand-line
option. If you want to always enable this option, you can set the correspondinginstall_depsconfiguration variable totruein the[tool.fawltydeps]section of yourpyproject.toml.To customize how this auto-installation happens (e.g. use a different package index),
you can usepip’s environment variables.Note that we’re never guaranteed to be able to resolvealldependencies with
this method: For example, there could be a typo in yourrequirements.txtthat
means a dependency willneverbe found on PyPI, or there could be other
circumstances (e.g. network issues or restrictions in your CI environment) that
prevent this strategy from working at all.
In this case, FawltyDeps will throw an error and abort.User-defined mappingWe provide a custom mapping functionality to users wishing to take control
over the way FawltyDeps resolves dependencies. You may define your own mapping
of dependency names to import names, by providing a TOML file like this:my-package=["mpkg"]scikit-learn=["sklearn"]multiple-modules=["module1","module2"]To use your mapping, run:fawltydeps--custom-mapping-filemy_mapping.tomlFawltyDeps will parse yourmy_mapping.tomlfile and use the extracted mapping
for matching dependencies to imports.You may also place the custom mapping in thepyproject.tomlfile of your
project, inside a[tool.fawltydeps.custom_mapping]section, like this:[tool.fawltydeps.custom_mapping]my-package=["mpkg"]scikit-learn=["sklearn"]multiple-modules=["module1","module2"]The provided mapping can be complete or partial. When a dependency is not
present in the given mapping, FawltyDeps will continue to resolve it using
the sequence of resolvers illustrated in the diagram above.Caution when using your mapping is advised: As illustrated in the diagram, the
user-defined mapping takes precedence over the other resolvers documented
above. For example, if the mapping file has some stale/incorrect mapping
entries, they willnotbe resolved by the Python environment resolver (which
is usually more accurate).Excluding pathsIf you want FawltyDeps to exclude parts of your source tree when loooking for
code, dependency declarations, or Python environments, then you can use the--excludeoption to specify path patterns to exclude, e.g. the following
command will skip everything undertests/:fawltydeps--excludetests/The format of the exclude patterns is the same as used by.gitignorefiles,see here for a full description.
(Note that some patterns that must be interpreted relative to the location of a.gitignorefile are not (yet) supported by FawltyDeps. Search for "relative"
in the linked page to see which pattern types are not yet supported.)When the--excludeoption is not specified, its default value is".*", which
matches all paths that start with a dot (.), aka. "hidden" paths. In the above
example, if you want to exclude both hidden paths, and everything undertests/, then instead use:fawltydeps--excludetests/".*"The--excludepatterns have lower priority than any paths you pass directly
on the command line, e.g. in this command:fawltydeps--codemy_file.py--excludemy_file.pythe--codeoptions "wins" (i.e. imports inmy_file.pywill be found); the--excludeoption only takes affect when traversing directories to look for
more files. E.g. use this to find code insidemy_dir, but skip Jupyter
notebooks:fawltydeps--codemy_dir--exclude"*.ipynb"(The extra quotes here are needed to prevent the shell from interpreting and
replacing the*wildcard.)Ignoring irrelevant resultsThere may beimportstatements in your code that should not be considered an
undeclared dependency. This might happen if you for example do a conditionalimportwith atry: ... except ImportError: ...block (or similar).
FawltyDeps is not able to recognize whether these dependencies should have been
declared or not, but you can ask for them to be ignored with the--ignore-undeclaredoption, for example:--ignore-undeclared some_module some_other_moduleConversely, there may be dependencies that you have declared without intending
toimportthem. This is often the case for developer tools like Black or Mypy
that are part of your project's development environment.
We've introduced aDEFAULT_IGNORE_UNUSEDlist, which includes various
categories of commonly used development tools and dependencies.
FawltyDeps can automatically ignore these dependencies when checking for unused
imports. For the complete list, please see theDEFAULT_IGNORE_UNUSEDvariable in thefawltydeps/settings.pyfile
in the repository. If you have additional dependencies that you want to exclude
from the check for unused imports, you can use the--ignore-unusedoption
to customize the ignore list. By providing your own list of dependencies with
this option, you can effectively overwrite the default list. For example:--ignore-unused black mypy some_other_moduleOutput formatsThe default output from FawltyDeps is a summary outlining the relevant
dependencies found (according to the selected actions).
However you can also ask for more information from FawltyDeps:--summary: Default (human-readable) summary output--detailed: Longer (human-readable) output that includes the location of
the relevant dependencies.--json: Verbose JSON-formatted output for other tools to consume and
process further.Only one of these options can be used at a time.More helpRunfawltydeps --helpto get the full list of available options.ConfigurationYou can use a[tool.fawltydeps]section inpyproject.tomlto configure the
default behavior of FawltyDeps. Here's a fairly comprehensive example:[tool.fawltydeps]code=["myproject"]# Only search for imports under ./myprojectdeps=["pyproject.toml"]# Only look for declared dependencies hereignore_unused=["black"]# We use `black`, but we don't intend to import itoutput_format="human_detailed"# Detailed report by defaultHere is a complete list of configuration directives we support:actions: A list of one or more of these actions to perform:list_imports,list_deps,check_undeclared,check_unused. The default behavior
corresponds toactions = ["check_undeclared", "check_unused"].output_format: Which output format to use by default. One ofhuman_summary,human_detailed, orjson.
The default corresponds tooutput_format = "human_summary".code: Files or directories containing the code to parse for import statements.
Defaults to the current directory, i.e. likecode = ["."].deps: Files or directories containing the declared dependencies.
Defaults to the current directory, i.e. likedeps = ["."].pyenvs: Where to look for Python environments (directories like.venv,__pypackages__, or similar) to be used for resolving project dependencies
into provided import names. Defaults to looking for Python environments under
the current directory, i.e. likepyenvs = ["."].ignore_undeclared: A list of specific dependencies to ignore when reporting
undeclared dependencies, for example:["some_module", "some_other_module"].
The default is the empty list:ignore_undeclared = [].ignore_unused: A list of specific dependencies to ignore when reporting
unused dependencies, for example:["black", "mypy", "some_other_module"].
The default is a list including common development tools. However, you have the
flexibility to overwrite this list according to your project's specific requirements.
For the complete default list, please see theDEFAULT_IGNORE_UNUSEDvariable in thefawltydeps/settings.pyfile
in the repository.deps_parser_choice: Manually select which format to use for parsing
declared dependencies. Must be one of"requirements.txt","setup.py","setup.cfg","pyproject.toml", or leave it unset (i.e. the default) for
auto-detection (based on filename).install-deps: Automatically install Python dependencies gathered with
FawltyDeps into a temporary virtual environment. This will usepip install,
which downloads packages from PyPI by default.exclude: File/directory patterns to exclude/ignore when looking for code
(imports), dependency declarations and/or Python environments. Defaults toexclude = [".*"], meaning that hidden/dot paths are excluded from traversal.verbosity: An integer controlling the default log level of FawltyDeps:-2: OnlyCRITICAL-level log messages are shown.-1:ERROR-level log messages and above are shown.0:WARNING-level log messages and above are shown. This is the default.1:INFO-level log messages and above are shown.2: All log messages (includingDEBUG) are shown.custom_mapping_file: Paths to files containing user-defined mapping.
Expected file format is defined in the User-defined mappingsection.[tool.fawltydeps.custom_mapping]: Section in the configuration, under which a custom mapping
can be added. Expected format is described in the User-defined mappingsection.Environment variablesIn addition to configuring FawltyDeps viapyproject.tomlas show above, you
may also pass the above configuration directives via the environment, using afawltydeps_prefix. For example, to enable JSON output via the environment,
setfawltydeps_output_format=jsonin FawltyDeps' environment.Configuration cascadeCommand-line options take precedence, and override corresponding settings
passed via the environment orpyproject.toml.Environment variables override corresponding settings frompyproject.toml.Configuration inpyproject.tomloverride only the ultimate hardcoded defaults.The ultimate defaults when no cutomizations takes place are hardcoded inside
FawltyDeps, and are documented above.DocumentationThis project began with an exploration and design phase, yielding thisdesign document, which lays out the main objective for this project and compares various strategies consideredIn thecode designsection of documentation we lay out rules which we adopt to guide code architecture decisions and maintain code quality as the project evolves.DevelopmentFor details on setting up your development environment, please refer to the
contributing guidelines of how toset up your development environmentin ourCONTRIBUTING.mdfile.Integration testsFor details on integration tests, please refer to theTesting sectionin ourCONTRIBUTING.mdfile.FAQI runfawltydepsand get some undeclared dependencies. What can I do with it?You can run a detailed report to see the exact location (file and line number), in which
the undeclared dependencies were imported:fawltydeps--detailedand debug each occurrence. Typically an undeclared dependency can be fixed in a couple of ways:A true undeclared dependency is fixed bydeclaringit, e.g. adding it to yourpyproject.tomlor similar.If you disagree with FawltyDeps' classification, you can always use--ignore-undeclaredto silence the error. If you're sure this dependency should not have been reported by FawltyDeps, you may consider filing a bug report.How not to display tools likeblackandpylintinunused dependencies?By default, all packages declared as dependencies by your project are included
in the FawltyDeps analysis, even if they only contain tools that were not meant
to beimported, but rather meant to be run by, say, in a pre-commit hook or a
CI script. In such cases you may use either:fawltydeps--ignore-unusedblackpylintor add an equivalent directive to the FawltyDeps configuration in yourpyproject.toml(see below).How can I store myfawltydepscommand line options in a configuration file?You can run:fawltydeps--generate-toml-configto generate a[tool.fawltydeps]section with the current configuration that
you can then directly copy into yourpyproject.toml. Options that have their
default value are commented in this output, so you have quickly see where your
settings differ from the FawltyDeps defaults.This also works together with other command line options, so for example in the
previous question, you could add--generate-toml-configto the command line
(i.e. runfawltydeps --ignore-unused black pylint --generate-toml-config),
to get this:[tool.fawltydeps]# Default options are commented...ignore_unused=["black","pylint"]How to use FawltyDeps in a monorepo?Runningfawltydepswithout arguments at the root of a monorepo
will most likely not give you a useful result:
it will collect dependencies and import statements from across theentiremonorepo.
The produced report may be overwhelming and at the same time not granular enough.Instead, you should run FawltyDeps for each package separately.
This collects dependencies and import statements for one package at a time.Having:├lib1|├pyproject.toml|├....
├lib2|├pyproject.toml|├....run for eachlibX:fawltydepslibXWhy must FawltyDeps run in the same Python environment as my project dependencies?(This is no longer true since FawltyDeps v0.11: FawltyDeps should be able to
automatically find your project dependencies when they are installed in a Python
environment that exists within your project. If your project dependencies are
installed elsewhere, you can point FawltyDeps in their direction with--pyenv,
as explained above in the section onPython environment mapping)The reason why FawltyDeps need to find your project dependenciessomewhereis
that the core logic of FawltyDeps needs to matchimportstatements in your
code with dependencies declared in your project configuration. This seems
straightforward for many packages: for example youpip install requestsand
then you canimport requestsin your code. However, this mapping from the name
you install to the name youimportis not always self-evident:There are sometimes differences between the package name that you
declare as a dependency, and theimportname it provides. For example, you
depend onPyYAML, but youimport yaml.A dependency can expose more than one import name. For example thesetuptoolspackage exposes threeimportable packages:_distutils_hack,pkg_resources, andsetuptools. So when youimport pkg_resources,
FawltyDeps need to figure out that this corresponds to thesetuptoolsdependency.To solve this, FawltyDeps looks at the packages installed in your Python
environment to correctly map dependencies (package names) into the imports that
they provide. This is:any Python environment found via the--pyenvoption,or if--pyenvis not given: any Python environment found within your
project (basepathor the current directory).In addition, FawltyDeps will use thecurrent Python environment,
i.e. the one in which FawltyDeps itself is running.As a final resort, when an installed package is not found for a declared
dependency, theidentity mappingthat FawltyDeps falls back to will still do a good job for
the majority of dependencies where the import name is indeed identical to the
package name that you depend on.This is an area of active development in FawltyDeps, and we areworking on better solutions,
to avoid having to fall back to this identity mapping.Why does FawltyDeps fail to matchsklearnwithscikit-learn?There are cases, where FawltyDeps may not match imports and obviously related
dependencies, likesklearnandscikit-learn. It will reportsklearnasundeclaredandscikit-learnas anunuseddependency.This is very much related to the above question.scikit-learnis an example
of a package that exposes a different import name:sklearn.
Whenscikit-learnis not found in the Python environment(s) used by FawltyDeps,
then FawltyDeps is unable to make the connection between these two names.To solve this problem, make sure thatscikit-learnis installed in a Python
environment that belongs to your project. Alternatively, you can use the--pyenvoption to point at a Python environment wherescikit-learnand your
other dependencies are installed.How can I pass Python code to FawltyDeps via standard input?The--codeargument accepts a single hyphen (-) as a special value meaning
that code should be read from standard input. When using this you may pipe or
redirect your Python code into FawltyDeps like this:catsome/source/of/python/code|fawltydeps--code-# orfawltydeps--code-<some/source/of/python/codeYou can also use this directly in the terminal to e.g. have FawltyDeps analyze
some Python code that is in your clipboard:fawltydeps--code-# FawltyDeps waits for code on stdin; paste from your clipboard,# then press Ctrl+D to signal EOF (end-of-file).Can I use FawltyDeps as a pre-commit hook?Yes! Assuming that you already use thepre-committool, you can add something like this to your project's.pre-commit-config.yaml:repos:-repo:https://github.com/tweag/FawltyDepsrev:v0.15.0hooks:-id:check-undeclared-id:check-unused