Package: pip-stripper
TLDR:  requirements withoutunnecessarypippackages.For the purpose of this exercise, an unnecessary pip package is any package 
that is not being imported by YOUR own Python code.  
Unless you've provided a configuration override stating that you want it.Simplest example: initialize, scan and build in current directory$ pipstripper --init --build

pip-stripper configuration generated @ sample/tst.seedworkdir01/pip-stripper.yaml

build phase - generating requirements at:
 sample/tst.seedworkdir01/requirements.dev.txt
 sample/tst.seedworkdir01/requirements.txt

$cat requirements.txt
Django==2.2
Jinja2==2.10.1
celery==4.3.0
cx-Oracle==6.4.1
psycopg2==2.8.2
python-dateutil==2.8.0

$ cat requirements.dev.txt
pyquery==1.4.0How it worksLots of your packages may not be needed.  Let's say you've installedblack 18.9b0.A linter and autoformatter has no need to be on a server:pip-strippermost likely won't findimport blackanywhere, so it will not put it inrequirementsoptionsusage: pipstripper [-h] [--config CONFIG] [--noscan] [--build] [--init]
                   [--workdir WORKDIR] [--verbose]

optional arguments:
  -h, --help         show this help message and exit
  --config CONFIG    config file. if not provided will look for pip-
                     stripper.yaml in --workdir, current directory
  --noscan           don't scan to classify packages. build phase will re-use
                     existing pip-stripper.scan.yaml. [False].
  --build            read pip-stripper.scan.yaml to create
                     requirements.prod/dev.txt [False]
  --init             initialize the config file (as pip-stripper.yaml) if it
                     doesn't exist
  --workdir WORKDIR  work directory [defaults to config file's value or
                     current directory]
  --verbose          verbose mode. adds extra zzz_debug: entry to pip-
                     stripper.scan.yaml [False]Three phases,initialization,scanandbuild.Initialization (defaults to False)The first option--initwill createpip-stripper.yaml, the configuration file for pip-stripper.This is the only file you should edit manually!!!Scan phase (will run unless you specify--noscan)This will scan your Python source files in--workdirand use it to createpip-stripper.scan.yaml.This is the file that contains instructions for the build phase.Don't edit pip-stripper.scan.yaml!Instead:adjust the configuration inpip-stripper.yamlre-run the scanScanning also creates 2 work files,tmp.pip-stripper.freeze.rptandtmp.pip-stripper.imports.rpt, tracking pip packages and its best guesses at python imports, respectively.Build (defaults to False)--buildtakes what it finds inpip-stripper.scan.yamland uses it to populaterequirements.txtandrequirements.dev.txt.If those requirements files don't suit you, you may need to editpip-stripper.yaml.Editingpip-stripper.yamlThis allows you to specify:pipvsimportaliasesspecify which packages are justworkstation-level and shouldn't go into requirements.hardcode packages that need to go into either.Associating your source directories to eitherprodordev.AliasesYou may have to enter pip to python import alias names manually (alias matching is something that needs work).hardcoded_aliases:
  PyYAML: yamlHardcoding package to requirement mapping:Becausepsycopg2is typically never really imported in a Django or SQLAlchemy context, but rather derived from the configuration, you need to specify it yourself as below.  Same thing withdjango-redis-cachewhich is configured in django'ssettings.pyas package path rather than an import.ClassifierPip:

  #the following are used to "hardcode" package names to given buckets.
  buckets:
    prod:
      - psycopg2
      - django-redis-cache

    tests:
      - nose
      - pytest

    workstation:
      # that's a workstation only package, so it's held back
      - blackAssociating Python directories to requirements:This is a typical regex-based configuration telling whichbucketsthe directories count as:prodis the default outcome.  First match wins, andtestsis the only one needed.ClassifierImport:
  regex_dirs:
    workstation: []
    dev: []
    tests:
      - "/tests/"
    prod: []

  default_bucket: "prod"Walkthrough:case 1: hardcodingGiven apip freezeline likepsycopg2==2.7.7First, it looks for a matching entry inClassifierPip:bucketsinpip-stripper.yaml, (basically a hardcoded decision by the user of where to put it).ClassifierPip:
  buckets:
    workstation:
      - black
    prod:
      - psycopg2This will result inpsycopg2==2.7.7going intorequirements.prod.txt(when needed, requirements lines are always copied from thepip freezeoutput).case 2: import classification.grep-ing the Python code found this line:./tests/helper_pyquery.py:57:    from pyquery import PyQueryEach file path is run against the regex specified byClassifierImport:regex_dirs, so pyquery ends up in thetestsbucket.ClassifierImport:
  regex_dirs:
    tests:
      - "/tests/"
    prod: []
  default_bucket: "prod"finally, the--buildpass looks at where buckets get mapped:Builder:
  req_mapper:
    dev:
      buckets:
        - dev
        - tests   
    prod:
      buckets:
        - prodwhich putspyqueryinrequirements.dev.txt.case 3 multiple imports./tests/helper_pyquery.py:57:    from pyquery import PyQuery
./myserver/foobar.py:22:    from pyquery import PyQueryAs before,pyqueryis put intestsandprodbuckets. But also inprodasmyserverdid not match anyClassifierImport:regex_dirs, meaning thatdefault_bucket: "prod"was used.enterbucket precedenceClassifierPip:
  bucket_precedence:
    - prod
    - tests
    - dev
    - workstationprodbeatstestssopyqueryends up only inprodbucket.case 4.  no import match was found and nothing was hardcoded.Babel==2.6.0will get left out of requirements.  Which is not to say that it won't end uppip installedon your server if it is a dependency of some other package (pipdeptreecan help you there).Babel==2.6.0
  - pytz [required: >=0a, installed: 2018.9]Build Phase - the result of the--scanphase gets put intopip-stripper.scan.yaml:Notice our friendblack?  We've explicitly classified it asworkstation, so the scan didn't label it asunknown.pips:
  buckets:
    dev:
    	....
    tests:
    - pyquery
    prod:
    - psycopg2
    unknown:
    - Babel
    workstation:
    - blackLook at the end for thewarnings:section.  In this case,reprwas used with Python 2.7 but isn't necessary with Python 3, so I won't worry about it.  A typical reason for a missing import is that automatic aliasing to link the pip name and import name didn't work.warnings:
- missing import:reprAnd that's it.  The outcome?my raw pip freeze weighs in at 158 packages:$ wc -l requirements.freeze_raw.txt
158 requirements.freeze_raw.txtmy stripped down requirements ended up with 24 packages total:$ wc -l requirements*txt | egrep 'prod|dev'
6 requirements.dev.txt
18 requirements.prod.txtOn my test environment,pip install -r requirements.prod.txt -r requirements.dev.txtgot me 48 packages, after dependencies were pulled in.$ pip freeze | wc -l
48This is all very nice, but hopefully you have sufficient tests to allow you to be confident you didn't miss anything!The way you can test is to create a new virtualenv, pip install both requirements files and then run your tests.WARNING:  be cautious in hardcoding package associations in dev/tests buckets rather than prod.  Your tests could run to success, but production would still fail.These are pretty conservative as nose and pytest are really only testing tools.ClassifierPip:
  buckets:
    tests:
      - nose
      - nose2
      - pytest