Package: dependency-observatory-scanner
find-package-rugarufind-package-rugarufinds open source dependent packages in a git
repository and tests and flags suspicious open source packages (likethe legendary rugaru).NB: this project is in an alpha state and its APIs are not stableScannerInstallationRequirementsdocker >=18.06.3python 3.8+ and pipjqsystem packages to build psycopg2 (e.g.build-essential libpq-devon debian buster)After installing the above requirements:$gitclonehttps://github.com/mozilla-services/find-package-rugaru.git$cdfind-package-rugaru$makeinstallinstall-dev$dockerpullmozilla/dependencyscan$dockerpullpostgres:12Example Usagestart a local postgres database:$makestart-dbrun one or more of the analysis scripts:./bin/analyze_package.sh <package_name> [<package_version>]./bin/analyze_repo.sh <repository_url>$./bin/analyze_package.shlodash4.17.15analyzing lodash@4.17.15 saving intermediate results to /tmp/dep-obs.g7mNNBaLyVjR...2020-02-27 17:31:31,900 - fpr - INFO - pipeline finishednull2020-02-27 17:31:32,403 - fpr - INFO - pipeline finished$or if you have more time to scan all git tags of the lodash repo:./bin/analyze_repo.sh https://github.com/lodash/lodash.gitanalyzing tags of https://github.com/lodash/lodash.git saving intermediate results to /tmp/dep-obs.5pvSrfbn6Nox...Check the source of the scripts to find additional configuration
via environment variables.Inspect the results in the local database:make db-shellPGPASSWORD=postgres psql -U postgres -h localhost -p 5432 dependency_observatorypsql (12.2 (Ubuntu 12.2-2.pgdg18.04+1), server 12.1 (Debian 12.1-1.pgdg100+1))Type "help" for help.dependency_observatory=# \x onExpanded display is on.dependency_observatory=# SELECT * FROM package_versions WHERE name = 'lodash' ORDER BY inserted_at DESC;-[ RECORD 1 ]-------------------------------------------------------id          | 102name        | lodashversion     | 4.17.15language    | nodeurl         | https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgzrepo_url    |repo_commit |inserted_at | 2020-02-26 17:12:47.373348updated_at  |PipelinesThe scripts are composed of components called pipelines (for lack of a
better term). For exampleanalyze_package.shwill:fetch information about the package from the npm registryfilter for git refs to clone and, if specified, the matching versionfind dependency manifests or lockfiles (e.g.package.json) for each ref in adebian:buster-slimdocker imagerunnpm install --save=true,npm list --json, andnpm audit --jsonin the project root for each ref in anode:10-buster-slimdocker imagepostprocess and save the results to the local postgres databaseEach individual pipeline can be run on its own. For example the
followingfind_git_refspipeline used inanalyze_repo.shwill find
git tags for themozilla-services/channelserverproject:$echo'{"repo_url": "https://github.com/mozilla-services/channelserver"}'|dockerrun-i--rm-v/var/run/docker.sock:/var/run/docker.sock--namefpr-testmozilla/dependencyscanpythonfpr/run_pipeline.py-vfind_git_refsCurrent Pipelines (from -h output)crate_graph         Parses the output of the cargo metadata pipeline andwrites a .dot file of the dependencies to outfiledep_graph           Parses the output of the cargo metadata pipeline andwrites a .dot file of the dependencies to outfilefetch_package_data  Fetches additional data about a dependency.find_dep_files      Given a repo_url, clones the repo, lists git refs foreach tagfind_git_refs       Given a repo_url, clones the repo, lists git refs foreach tag TODO: every Nth commit, or commit every timeinterval. TODO: since and until args TODO: findbranchesgithub_metadata     Given an input file with repo urls metadata outputfetches dependency and vulnerability metadata fromGitHub and an optional GitHub PAT and outputs them tojsonl and optionally saves them to a local SQLite3 DB.postprocess         Post processes tasks for various outputs e.g.flattening deps, filtering and extracting fields, etc.Does not spin up containers or hit the network.run_repo_tasks      Runs tasks on a checked out git ref with dep. filesrust_changelog      Given ordered cargo metadata output for git refs fromthe same repo: 1. builds a dict of manifest filenameto cargo meta 2. groups the output into pairs (i.e. 1,2, 3 -> (1, 2), (2, 3) in the provided order 3.compares each pair as follows: a. compare eachmanifest filename: 1) count new and removeddependencies 2) new and removed authors and repo urlsTODO: output a diff of the updated dep code (need toupdate the cargo metadata pipeline to pull these)TODO: take audit output to show new and fixed Rustvulns TODO: detect dep version changessave_to_db          Saves JSON lines to a postgres DBSeebin/analyze_*scripts andMakefilefor example usage.They feed into each other as follows (*indicates deprecated, removed or otherwise broken pipelines):Pipeline APINote that this interface may be subject to changeEach pipeline:reads and writesJSON lines (basically newline delimited JSON objects)uses the args-i,--infileand-o,--outfilethat respectively default to stdin and stdout to allow pipeliningrun as a python asyncio generatorSeethe design docfor why this interface was chosen.Adding a pipelinecopy an existing file fromfpr/pipelines/give it a new name in itsPipelinemodel declarationadd it tofpr/pipelines/__init__.py