Package: ptr
🏃‍♀️ptr- Python Test Runner 🏃‍♂️Python Test Runner (ptr) was born to run tests in an opinionated way, within arbitrary code repositories.ptrsupports many Python projects with unit tests defined in theirsetup.(cfg|py)files per repository.ptrallows developers to test multiple projects/modules in one Python environment through the use of a single test virtual environment.ptrrequires>=python 3.7ptritself usesptrto run its tests 👌🏼ptris supported and tested onLinux,MacOS+WindowsOperating SystemsBy addingptrconfiguration to your either of yourpyproject.toml,setup.cfgorsetup.pyyou can haveptrperform the following,
per test suite, in parallel:run your test suitecheck and enforce coverage requirements (viacoverage),format code (viablack)perform static type analysis (viamypy)QuickstartInstallptrinto you virtualenvpip install ptrEnsure your tests have a base file that can be executed directlyi.e.python3 test.py(possibly usingunittest.main())After addingptr_paramsto setup.py (see example below), run:cd repo
ptrHow doesptrperform this magic? 🎩I'm glad you ask. Under the coversptrperforms:Recursively searches forsetup.(cfg|py)files fromBASE_DIR(defaults to your "current working directory" (CWD))ASTparses out the config for eachsetup.pytest requirementsIf apyproject.tomlorsetup.cfgexists, load via configparser/tomli and prefer if a[ptr]section existsCreates aPython Virtual Environment(OPTIONALLYpointed at an internal PyPI mirror)RunsATONCEtests suites in parallel (i.e. per setup.(cfg|ptr))All steps will be run for each suite and ONLYFAILEDruns will have output written to stdoutUsage 🤓To useptrall you need to do is cd to your project or set the base dir via-band execute:$ ptr [-dk] [-b some/path] [--venv /tmp/existing_venv]Forfaster runswhen testing, it is recommended to reuse a Virtual Environment:-k- To keep the virtualenv created byptr.Use--venv VENV_PATHto reuse to an existing virtualenv created by the user.Help Output 🙋‍♀️ 🙋‍♂️usage:ptr.py[-h][-aATONCE][-bBASE_DIR][-d][-e][-k][-mMIRROR][--print-cov][--print-non-configured][--progress-intervalPROGRESS_INTERVAL][--run-disabled][--stats-fileSTATS_FILE][--system-site-packages][--venvVENV][--venv-timeoutVENV_TIMEOUT]optionalarguments:-h,--helpshowthishelpmessageandexit-aATONCE,--atonceATONCEHowmanyteststorunatonce[Default:6]-bBASE_DIR,--base-dirBASE_DIRPathtorecursivelylookforsetup.pyfiles[Default:/Users/cooper/repos/ptr]-d,--debugVerbosedebugoutput-e,--error-on-warningsHavePythonwarningsraiseDeprecationWarningontestsrun-k,--keep-venvDonotremovecreatedvenv-mMIRROR,--mirrorMIRRORURLforpiptouseforSimpleAPI[Default:https://pypi.org/simple/]--print-covPrintmodulescoveragereport--print-non-configuredPrintmodulesnotconfiguredtorunptr--progress-intervalPROGRESS_INTERVALSecondsbetweenstatusupdateontestrunning[Default:Disabled]--run-disabledForceanydisabledtestssuitestorun--stats-fileSTATS_FILEJSONstatisticsfile[Default:/var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_stats_12510]--system-site-packagesGivethevirtualenvironmentaccesstothesystemsite-packagesdir--venvVENVPathtovenvtoreuse--venv-timeoutVENV_TIMEOUTTimeoutinsecondsforvenvcreation+depsinstall[Default:120]Configuration 🧰ptris configured by placing directives in one or more of the following files..ptrconfigprovides
base configuration and default values for all projects in the repository, while eachsetup.(cfg|py)overrides the base configuration for the respective packages they define..ptrconfigptrsupports a general config inini(ConfigParser) format.
A.ptrconfigfile can be placed at the root of any repository or in any directory within your repository.
The first.ptrconfigfile found via a recursive walk to the root ("/" in POSIX systems) will be used.Please refer toptrconfig.samplefor the options available.setup.pyThis is per project in your repository. A simple example, based onptritself:# Specific Python Test Runner (ptr) params for Unit Testing Enforcementptr_params={# Where mypy will run to type check your program"entry_point_module":"ptr",# Base Unittest file"test_suite":"ptr_tests","test_suite_timeout":300,# Relative path from setup.py to module (e.g. ptr == ptr.py)"required_coverage":{"ptr.py":99,"TOTAL":99},# Run `black --check` or not"run_black":False,# Run mypy or not"run_mypy":True,}pyproject.tomlThis is per project in your repository and if exists is preferred oversetup.pyandsetup.cfg.Please refer topyproject.tomlfor the options available + format.setup.cfgThis is per project in your repository and if exists is preferred oversetup.py.Please refer tosetup.cfg.samplefor the options available + format.mypy SpecificsWhen enabled, (insetup.(cfg|py))mypycan support using a custommypy.inifor each setup.py (module) defined.To haveptrrun mypy using you config:create amypy.iniin the same directory as yoursetup.pyOR add[mypy]section to yoursetup.cfgmypyConfiguration Documentation can be foundhereAn examplesetup.cfgcan be seenhere.Example Output 📝Here are some example runs.SuccessfulptrRun:Here is what you want to see in your CI logs![2019-02-06 21:51:45,442] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:51:59,471] INFO: Successfully created venv @ /var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_24397 to run tests (14s) (ptr.py:547)
[2019-02-06 21:51:59,472] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:52:00,726] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
[2019-02-06 21:52:04,153] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-02-06 21:52:04,368] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:09,915] INFO: Running flake8 for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:10,422] INFO: Running pylint for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:14,020] INFO: Running pyre for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-02-06 21:52:07,733] INFO: /Users/cooper/repos/ptr/setup.py has passed all configured tests (ptr.py:509)
-- Summary (total time 22s):

✅ PASS: 1
❌ FAIL: 0
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests runningUnsuccessfulptrRun Examples:Here are some examples of runs failing. Any "step" can fail. All output is predominately the underlying tool.Unit Test Failure[2019-02-06 21:53:58,121] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:53:58,143] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:53:59,698] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
-- Summary (total time 5s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'tests_run' step):
...F....................
======================================================================
FAIL: test_config (__main__.TestPtr)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/cooper/repos/ptr/ptr_tests.py", line 125, in test_config
    self.assertEqual(len(sc["ptr"]["venv_pkgs"].split()), 4)
AssertionError: 5 != 4

----------------------------------------------------------------------
Ran 24 tests in 3.221s

FAILED (failures=1)coverage[2019-02-06 21:55:42,947] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:55:42,969] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:55:44,920] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
[2019-02-06 21:55:49,628] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
-- Summary (total time 7s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'analyze_coverage' step):
The following files did not meet coverage requirements:
  ptr.py: 84 < 99 - Missing: 146-147, 175, 209, 245, 269, 288-291, 334-336, 414-415, 425-446, 466, 497, 506, 541-543, 562, 611-614, 639-688black[2019-02-06 22:34:20,029] INFO: Starting ptr.py (ptr.py:804)
[2019-02-06 22:34:20,060] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:430)
[2019-02-06 22:34:21,614] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:430)
[2019-02-06 22:34:25,208] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
[2019-02-06 22:34:25,450] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
[2019-02-06 22:34:26,422] INFO: Running black for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
-- Summary (total time 7s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'black_run' step):
would reformat /Users/cooper/repos/ptr/ptr.py
All done! 💥 💔 💥
1 file would be reformatted, 4 files would be left unchanged.mypy[2019-02-06 22:35:39,480] INFO: Starting ptr.py (ptr.py:802)
[2019-02-06 22:35:39,531] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:428)
[2019-02-06 22:35:41,203] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:428)
[2019-02-06 22:35:45,156] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:428)
[2019-02-06 22:35:45,413] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:428)
-- Summary (total time 6s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'mypy_run' step):
/Users/cooper/repos/ptr/ptr.py: note: In function "_write_stats_file":
/Users/cooper/repos/ptr/ptr.py:179: error: Argument 1 to "open" has incompatible type "Path"; expected "Union[str, bytes, int]"
/Users/cooper/repos/ptr/ptr.py: note: In function "run_tests":
/Users/cooper/repos/ptr/ptr.py:700: error: Argument 1 to "_write_stats_file" has incompatible type "str"; expected "Path"pyrecooper-mbp1:ptr cooper$ /tmp/tp/bin/ptr --venv /var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_49117
[2019-05-03 14:51:43,623] INFO: Starting /tmp/tp/bin/ptr (ptr.py:1023)
[2019-05-03 14:51:43,657] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:565)
[2019-05-03 14:51:44,840] INFO: Running ptr_tests tests via coverage (ptr.py:565)
[2019-05-03 14:51:47,361] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,559] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,827] INFO: Running black for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,996] INFO: Running flake8 for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:48,566] INFO: Running pylint for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:52,301] INFO: Running pyre for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:54,983] INFO: /Users/cooper/repos/ptr/setup.py has passed all configured tests (ptr.py:668)
-- Summary (total time 11s):

✅ PASS: 1
❌ FAIL: 0
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'pyre_run' step):
2019-05-03 14:54:14,173 INFO No binary specified, looking for `pyre.bin` in PATH
2019-05-03 14:54:14,174 INFO Found: `/var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_49117/bin/pyre.bin`
... *(truncated)* ...
ptr.py:602:25 Undefined name [18]: Global name `stdout` is not defined, or there is at least one control flow path that doesn't define `stdout`.usortIf your imports are not makingusorthappy it would look like this:[2021-05-29 09:30:56,044] INFO: Starting /tmp/tp/bin/ptr (ptr.py:1129)
[2021-05-29 09:30:56,051] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:637)
[2021-05-29 09:30:56,587] INFO: Running ptr_tests tests via coverage (ptr.py:637)
[2021-05-29 09:30:58,238] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
[2021-05-29 09:30:58,341] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
[2021-05-29 09:30:58,436] INFO: Running usort for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
-- Summary (total time 2s):

✅ PASS: 0
❌ FAIL: 1
 ️⌛ TIMEOUT: 0
🔒 DISABLED: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'usort_run' step):
Would sort /Users/cooper/repos/ptr/setup.pyFAQ ⁉️Q. How do I debug? I need output!ptrdevelopers recommend that if you want output, please cause a test to faile.g.raise ZeroDivisionErrorAnother recommended way is to run your tests with the defaultsetup.py testusing aptrcreated venv:cd to/my/code/tmp/venv/bin/python setup.py testQ. How do I get specific version of black, coverage, mypy etc.?Just simply hard set the version in the .ptrconfig in your repo or userequirements.txtto pre-install before runningptrAllpipPEP 440 version specifiersare supportedQ. Why is the venv creation so slow?ptrattempts to update from a PyPI compatible mirror (PEP 381) or PyPI itselfRunning a package cache or local mirror can greatly increase speed. Example software to do this:bandersnatch: Can do selected or FULL PyPI mirrors. The maintainer is also devilishly good looking.devpi: Can be ran and used toproxypackages locally when pip goes out to grab your dependencies.Please ensure you're using the-kor--venvoption to no recreate a virtualenv each run when debugging your tests!Q. Why is ptr not able to runpyreon Windows?pyre(pyre-check on PyPI) does not ship a Windows wheel with the ocaml pyre.binQ. Why do you depend on >= coverage 5.0.1coverage5.0 introduced using sqlite and we don't want to have a mix of 4.x and 5.x for ptr< 5.0 could possibly still work as we now ensure to run each projects tests from setup_py.parent CWD with subprocessContact or join the ptr community 💬To chat in real time, hit us up on IRC. Otherwise, GitHub issues are always welcome!
IRC:#pythontestrunneronFreeNodeSee theCONTRIBUTINGfile for how to help out.Licenseptris MIT licensed, as found in theLICENSE file.Terms of Use + Privacy PolicyTerms of UsePrivacy PolicyCopyright © Meta Platforms, Inc. and affiliatesptrChange HistoryEach release to PyPI I'm going to give a codename as to where I am or was in the world 🌏.2022.7.12Codename:Seattle 🦑Release named in honoUr of my visit to Seattle last week ...3.11 Supported - added metadata + made 3.11 CI pass - PR #121Addpyproject.tomluser config support - PR #120Add basic Hypothesis fuzz testing - PR #118Learnt during US PyCon 2022Addpyproject.tomlbasic PEP517 building support - PR #117This(again)will be the last 3.7 supported versionI forgot to remove + CI still passes so will leave for now :)2022.2.2Codename:Richie 2 for 22 🏏Release named in honoUr of the Late GreatRichie BenaudThis will be the last 3.7 supported versionWhy? Want a less buggyasyncio.runfor starting asyncioReactivatepylintin CIf-string all ptr stringsDrop python 3.6 CI + supportWe're Meta now - Update copyrights due to comapny change name3.11 CI is failing due to depedencies. Will watch and fix asap.2021.11.23Codename:A 🅰️Release to support 'Three dot Ten' and A == 10 in hexidecimalAdd Official Support and CI for Python 3.10This will be the last release(if no bugs present themselves)for 3.6Also start ci_latest running 3.112021.5.28Codename:Memorial Day 🇺Release on Memorial Day long weekendAdd usort to the family of CI optionshttps://pypi.org/project/usort/2021.3.16Codename:Wildwood 🪵First release from my South Lake Tahoe residenceSupport floats for coverage comparisions inrequired_coverage- PR #108Change--print-covto work whenrequired_coveragedoes not exist in ptr config - PR #109Move tests out of install but include in sdist - PR #99 - Thanks @jayvdbTest with 3.10 alphas via deadsnakes - PR #110Misc f-string + lint/typing fixes2020.2.26Codename:Month after Straya Day 🇦🇺🇺1 month since @cooperlees was in AU for Australia day!Enable pyre for Python 3.8 now it's fully supported2019.12.25Codename:Vernon Hills, IL 🎅@cooperlees visiting girlfriend's family in ILFix documentation for some new arguments - Thanks @omikaderWindows now defaults to updating pip + setuptoolsRely on setting current working directory with subprocess and remove coverage file environment settingf-string everything viaflyntnow we're >= 3.62019.12.13Codename:College Park, MD 🐢@omikader studied at the University of Maryland, College ParkAdd ability to create partial.ptrconfigfiles - PR: #83Use native recursive file search for black and flake8 - PR: #812019.12.12Codename:Sapporo, Japan 🇯🇵@omikader is going skiing there in FebruaryAdd support for passing--system-site-packagesduring venv creation - PR: #802019.11.21Codename:Santa Clara, CA@thatch lives there!*Add license file for @thatch - PR: #77 -Thanks @thatch2019.11.15Codename:Russian River, CA 🇺🇸@cooperlees is going to the Russian River tomorrowRun ptr even when test_suite is not provided - PR: #73 -Thanks @spuravRemove use of deprecated setuptoolstestoptionMove to GitHub Actions for CI + Releases2019.11.2Codename:Mumbai, India 🇮🇳@spurav is from Mumbai, IndiaFix math error dividing 1 / 2 for os.cpu_count - PR: #69 -Thanks @spuravGet GitHub Actions CI Running - PR: #68 + #70 -Thanks @adhaamehab2019.10.22Codename:Cairo, Egypt 🇪🇬@adhaamehab who first time contributed is from there! Thanks!Print more helpful error when file does not exist for coverage check - PR: #51Officially Support Python 3.8 and enforce tests passingDisable pyre-check for >=3.8 due to no supportIssue to Track:https://github.com/facebook/pyre-check/issues/2132019.10.6Codename:Stanford, CA@cooperlees tailgated there yesterday 🚐🏈Handle OSX /private in coverage report - PR: #602019.9.14Codename:Clowntown 🤡Fix bad bool passing for--print-cov+--error-on-warnings2019.9.11Codename:Rathbone Square@cooperlees is releasing this release from FB London OfficeAdd CLI option to enable erroring on allwarnings- Issue: #572019.8.7Codename:Ellicott City@omikader who reported and tested the fix for AST parsing is from there.Ellicott City is also home to one of the oldest surviving train stations in the US!Fix AST parsing to ignore AST targets that do not have anidattribute - Issue: #54Renableblackby default on 3.7 now that it runs on > 3.7.2 - Issue: #41Fix --venv-timeout to store intsRefactor _get_test_modules function to remove lint error - PR: #53 -Thanks @TomasFeeney2019.7.16Codename:California CityAn over developed desert with infrastructure and littlepeopleAdd--print-non-configuredto find nonptrmodules in repos - Issue: #502019.6.15Codename:Concord🛫 @cooperlees is playing Aussie Football @ Concord, CA today 🏈Add ability todisabletest suites and add a--run-disabledoption to force the run - Issue: #46Azure CI now runs on Mac and Windows again2019.5.3Codename:Cleveland🇺🇸 @cooperlees + @jreese are at PyCon US in Cleveland, OH 🦅Print step name in failure output - Issue: #31Add inpyreType Checking step support - Issue: #38 + #40Known Bug:black.exedoes not run in Windows 3.7 -disabledbydefaulton Python 3.7 on Windows2019.3.5Codename:Jaipur🇮🇳 @cooperlees releasing whilst in Jaipur, India for a wedding 💒Preliminary Windows support now ready for testing - Issue: #2Run tests/mypy/flake8 etc. in CWD of setup.py path - Issue #23 - Thanks @jreeseAdd support for linting with flake8 and pylint - Issue #20 - Thanks @jreeseIgnore dotted directories when running black - PR #19 - Thanks @jreeseKnown Bug:black.exedoes not run in Windows 3.7 -disabledbydefaulton Python 3.7 on Windows2019.2.12Codename:ForbesForbes, NSW, Australiais the home of @aijayadams 👨🏻‍🦰🇦🇺Added suite file coverage % to statistics JSON file - Issue: #16 -Thanks @aijayadamsIgnore hidden '.' (dot) directories when running black - PR: #19 -Thanks @jreese2019.2.10Codename:Carnival🇧🇷 @cooperlees was in Rio de Janeiro, Brazil for Carnival 3 years ago today 🇧🇷Added ptrsetup.cfgsupport for ptr_params - Issue: #1Added JSON stats validation toci.py- Issue: #7Fixed bug that allowed a step to run by default - Issue: #112019.2.8.post1/2Fixsetup.pyURL to ptr GitHubOther varioussetup.pyfixes - e.g. Classifiers + License information2019.2.8Codename:SnowbirdRecent shredding of Snowbird, UT, USA took place 🏂 🇺🇸Initial Release to the world!