Package: pyctest
PyCTestTravisAppVeyorPyCTest Documentation (readthedocs)GitHubgit clone https://github.com/jrmadsen/pyctest.gitPyPipip install pyctestAnaconda Cloudconda install -c jrmadsen pyctestNameVersionPlatformsDownloadsconda-forgeconda install -c conda-forge pyctestNameVersionPlatformsDownloadsBenefitsIntegration into continuous integration systems (e.g. Travis, AppVeyor, CircleCI, etc.) and pushing to CDash dashboard will combine all the results in one placeThe warnings and errors are enumerated in CDash (no more parsing stdout logs for errors)Easily create platform-independent testingNo need to migrate build system to CMake -- just specifypyctest.BUILD_COMMANDGeneral SetupAdd PyCTest to conda environment:conda install pyctestWrite a driver Python script, e.g.pyctest-runner.py, in top of the source code treeExample for Python projectThe following is an example for a Python code with a compiled C extension that usesnosetestsfor unit-testing:#!/usr/bin/env pythonimportos,sys,platformimportpyctest.pyctestaspyctestimportpyctest.helpersashelpersparser=helpers.ArgumentParser("ProjectName",source_dir=os.getcwd(),binary_dir=os.getcwd(),vcs_type="git")args=parser.parse_args()pyctest.BUILD_NAME="{}".format(args.build)pyctest.BUILD_COMMAND="python setup.py build_ext --inplace"test=pyctest.test()test.SetName("unittest")# insert the command to run the tests for projecttest.SetCommand(["nosetests"])pyctest.run()Example for autotools project#!/usr/bin/env pythonimportos,sys,platformimportmultiprocessingasmpimportpyctest.pyctestaspyctestimportpyctest.helpersashelpersparser=helpers.ArgumentParser("ProjectName",source_dir=os.getcwd(),binary_dir=os.getcwd(),vcs_type="git")parser.add_argument("-n","--build",type=str,required=True,help="Build name for identification")args=parser.parse_args()# CONFIGURE_COMMAND can only run one command so if autogen is required, just execute it herecmd=pyctest.command(["./autogen.sh"])cmd.SetWorkingDirectory(pyctest.SOURCE_DIRECTORY)cmd.SetErrorQuiet(False)cmd.Execute()pyctest.BUILD_NAME="{}".format(args.build)pyctest.CONFIGURE_COMMAND="./configure"pyctest.BUILD_COMMAND="make -j{}".format(mp.cpu_count())test=pyctest.test()test.SetName("unittest")# insert the command to run the tests for projecttest.SetCommand(["./run-testing.sh"])pyctest.run()Example for CMake project#!/usr/bin/env pythonimportosimportsysimportplatformimportmultiprocessingasmpimportpyctest.pyctestaspyctestimportpyctest.helpersashelpersbinary_dir=os.path.join(os.getcwd(),"build-ProjectName")parser=helpers.ArgumentParser("ProjectName",os.getcwd(),binary_dir)parser.add_argument("-n","--build",type=str,required=True,help="Build name for identification")args=parser.parse_args()pyctest.BUILD_NAME="{}".format(args.build)pyctest.UPDATE_COMMAND="git"pyctest.CONFIGURE_COMMAND="cmake{}".format(pyctest.SOURCE_DIRECTORY)pyctest.BUILD_COMMAND="cmake --build{}--target all -- -j{}".format(pyctest.BINARY_DIRECTORY,mp.cpu_count())test=pyctest.test()test.SetName("unittest")# insert the command to run the tests for projecttest.SetCommand(["./run-testing.sh"])pyctest.run()Python Modulesimport pyctest-- global packageimport pyctest.pyctest-- CTest moduleimport pyctest.pycmake-- CMake moduleimport pyctest.helpers-- Helpers moduleincludes command line arguments (argparse) for PyCTestDirect Access to CMake/CTest/CPack Executablespython -m pyctest.cmake <ARGS>==cmake <ARGS>python -m pyctest.ctest <ARGS>==ctest <ARGS>python -m pyctest.cpack <ARGS>==cpack <ARGS>Following Python code:frompyctest.ctestimportCTestfrompyctest.cmakeimportCMakefrompyctest.cpackimportCPackCMake({"CMAKE_BUILD_TYPE":"Release"},os.getcwd(),"-G","Ninja")CTest("--build-and-test",os.getcwd(),"-VV")CPack("-G","TGZ")is equivalent to the following shell commands:$cmake-DCMAKE_BUILD_TYPE=Release${PWD}-GNinja
$ctest--build-and-test${PWD}-VV
$cpack-GTGZStandard Configuration Variablespyctest.PROJECT_NAMEpyctest.SOURCE_DIRECTORYpyctest.BINARY_DIRECTORYpyctest.SITEpyctest.BUILD_NAMEpyctest.TRIGGERpyctest.CHECKOUT_COMMANDpyctest.BUILD_COMMANDpyctest.MODELpyctest.CUSTOM_COVERAGE_EXCLUDEpyctest.CUSTOM_MAXIMUM_NUMBER_OF_ERRORSpyctest.CUSTOM_MAXIMUM_NUMBER_OF_WARNINGSpyctest.CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZESetting Arbitrary Variablespyctest.set("CTEST_TOKEN_FILE","${CMAKE_CURRENT_LIST_DIR}/.ctest-token")Generating a Testtest=pyctest.test()test.SetName("nosetests")test.SetCommand(["nosetests","test","--cover-xml","--cover-xml-file=coverage.xml"])# set directory to run testtest.SetProperty("WORKING_DIRECTORY",pyctest.BINARY_DIRECTORY)test.SetProperty("RUN_SERIAL","ON")test.SetProperty("ENVIRONMENT","OMP_NUM_THREADS=1")ExamplesBasic exampleAdvanced exampleincludes submission to CDash dashboardCDash Integration ExampleResults from running the TomoPy example can be found at theTomoPy CDash Testing Dashboard @ NERSCPython code with C extensions without CMake build systemThe build logs from "python setup.py install" are registered in the "Build" sectionThenosetests testcommand + other are wrapped into CTestsTesting ExamplePyCTest can be used to simple execute tests and submit to a dashboard without any configuration, build, etc. steps#!/usr/bin/env pythonimportosimportsysimportshutilimportargparseimportplatformimporttracebackimportpyctest.pyctestaspyctestimportpyctest.pycmakeaspycmakeimportpyctest.helpersashelpersif__name__=="__main__":directory=os.path.join(os.getcwd(),"pycm-test")# these are requiredpyctest.PROJECT_NAME="PyCTest"pyctest.SOURCE_DIRECTORY=directorypyctest.BINARY_DIRECTORY=directoryargs=helpers.ArgumentParser(pyctest.PROJECT_NAME,pyctest.SOURCE_DIRECTORY,pyctest.BINARY_DIRECTORY).parse_args()# set explicitlypyctest.MODEL="Continuous"pyctest.SITE=platform.node()# create a Test objecttest=pyctest.test()test.SetName("list_directory")test.SetCommand(["ls",directory])test.SetProperty("WORKING_DIRECTORY",os.getcwd())# create a second test# previous test is already stored by PyCTesttest=pyctest.test()test.SetName("hostname")test.SetCommand(["hostname"])test.SetProperty("TIMEOUT","10")# generate the CTestConfig.cmake and CTestCustom.cmakepyctest.generate_config(directory)# generate the CTestTestfile.cmake filepyctest.generate_test_file(directory)# run CTest -- e.g. ctest -VV ${PWD}/pycm-testpyctest.run(pyctest.ARGUMENTS,directory)CTestarguments(default):'-V -DSTAGES=Start;Update;Configure;Build;Test;Coverage;MemCheck -S Stages.cmake -j1'WritingCTesttestfile:"/Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestTestfile.cmake"...
Generatingtest"list_directory"...
Generatingtest"hostname"...
--STAGES=Start;Update;Configure;Build;Test;Coverage;MemCheck
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]RunningCTEST_STARTstage...
RundashboardwithmodelContinuousSourcedirectory:/Users/jrmadsen/devel/c++/pyctest-master/pycm-testBuilddirectory:/Users/jrmadsen/devel/c++/pyctest-master/pycm-testTrack:ContinuousReadingctestconfigurationfile:/Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestConfig.cmakeSite:JRM-macOS-DOE.localBuildname:[DarwinmacOS10.13.6x86_64][Python3.6.7]UseContinuoustag:20181129-2118
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]SkippingCTEST_UPDATEstage...
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]SkippingCTEST_CONFIGUREstage...
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]SkippingCTEST_BUILDstage...
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]RunningCTEST_TESTstage...
Testproject/Users/jrmadsen/devel/c++/pyctest-master/pycm-testStart1:list_directory1/2Test#1: list_directory ...................   Passed    0.00 secStart2:hostname2/2Test#2: hostname .........................   Passed    0.00 sec100%testspassed,0testsfailedoutof2TotalTesttime(real)=0.01sec
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]SkippingCTEST_COVERAGEstage...
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]SkippingCTEST_MEMCHECKstage...
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]SkippingCTEST_SUBMITstage...
--[[DarwinmacOS10.13.6x86_64][Python3.6.7]]FinishedContinuousStages(Start;Update;Configure;Build;Test;Coverage;MemCheck)Command Line InterfacePyCTestargs:['--help']CTestargs:[]CMakeargs:[]usage:pyctest-runner.py[-h][-F][-S][-A][-j<INT>][-m<TYPE>][-b<NAME>][-i<PATH>][-o<PATH>][-M<TYPE>][-H<SITE>][-P<EXE>][-N<NAME>][-C<PATH>[<PATH>...]][-L<LABEL>[<LABEL>...]][-T[<TYPE>[<TYPE>...]]][-fc<EXE>][-cc<EXE>][-cxx<EXE>][--pyctest-token<TOKEN>][--pyctest-token-file<FILE>][--pyctest-vcs-type<TYPE>][--pyctest-build-type<TYPE>][--pyctest-trigger<TYPE>][--pyctest-use-launchers<BOOL>][--pyctest-ctest-args[<ARG>[<ARG>...]]][--pyctest-cdash-version<STR>][--pyctest-submit-retry-count<INT>][--pyctest-submit-retry-delay<INT>][--pyctest-curl-options<LIST>][--pyctest-drop-location<STR>][--pyctest-drop-method<STR>][--pyctest-drop-site<STR>][--pyctest-drop-site-password<STR>][--pyctest-drop-site-user<STR>][--pyctest-nightly-start-time<STR>][--pyctest-update-version-only]PyCTestargparse.Argumentsafterfirst'--'arepasseddirectlytoCTest,argumentsaftersecond'--'arepasseddirectlytoCMake

optionalarguments:-h,--helpshowthishelpmessageandexit-F,--pyctest-clean-firstRemovestandardPyCTestfilesandbinarydirectory(ifnotsourcedirectory)-S,--pyctest-submitEnablesubmissiontodashboard-A,--pyctest-appendAppendtolastinvocationofCTest-j<INT>,--pyctest-jobs<INT>numberofconcurrentjobs-m<TYPE>,--pyctest-mode<TYPE>Runspecificstage.Choices:[Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit,Stages]-b<NAME>,--pyctest-build-name<NAME>Buildnameforidentification-i<PATH>,--pyctest-source-dir<PATH>Sourcedirectory-o<PATH>,--pyctest-binary-dir<PATH>Binary/build/workingdirectory-M<TYPE>,--pyctest-model<TYPE>CTestsubmissionmodel(track).Choices:[Nightly,Continuous,Experimental]-H<SITE>,--pyctest-site<SITE>CTestsubmissionsite-P<EXE>,--pyctest-python-exe<EXE>Pythonexecutabletouse.Thiscanbeabsolue,relative,orCMakepath-N<NAME>,--pyctest-project-name<NAME>NameofprojectusingPyCTest-C<PATH>[<PATH>...],--pyctest-cleanup<PATH>[<PATH>...]RemovestandardPyCTestfilesandbinarydirectory(ifnotsourcedirectory)andexit-L<LABEL>[<LABEL>...],--pyctest-subproject-labels<LABEL>[<LABEL>...]Addlabelsforsubproject-T[<TYPE>[<TYPE>...]],--pyctest-stages[<TYPE>[<TYPE>...]]Runmultiplestages.Choices:[Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit]-fc<EXE>,--pyctest-fortran-compiler<EXE>SpecifyFortancompiler(ifneeded)-cc<EXE>,--pyctest-c-compiler<EXE>SpecifytheCcompiler(ifneeded)-cxx<EXE>,--pyctest-cxx-compiler<EXE>SpecifyC++compiler(ifneeded)--pyctest-token<TOKEN>CTestsitetokenforsubmission--pyctest-token-file<FILE>PathtofileforCTestsitetokenforsubmission--pyctest-vcs-type<TYPE>SettoenablerevisionIDdiscoveryduringtheUpdatestage.Choices:[bzr,cvs,git,hg,p4,svn]--pyctest-build-type<TYPE>SpecifyCMAKE_BUILD_TYPE(ifusingCMake).Choices:[Release,RelWithDebInfo,Debug,MinSizeRel]--pyctest-trigger<TYPE>DEPRECATED--pyctest-use-launchers<BOOL>Enablelaunchers--pyctest-ctest-args[<ARG>[<ARG>...]]CTestarguments--pyctest-cdash-version<STR>SetCTestvariable:'CTEST_CDASH_VERSION'--pyctest-submit-retry-count<INT>SetCTestvariable:'CTEST_SUBMIT_RETRY_COUNT'--pyctest-submit-retry-delay<INT>SetCTestvariable:'CTEST_SUBMIT_RETRY_DELAY'--pyctest-curl-options<LIST>SetCTestvariable:'CTEST_CURL_OPTIONS'--pyctest-drop-location<STR>SetCTestvariable:'CTEST_DROP_LOCATION'--pyctest-drop-method<STR>SetCTestvariable:'CTEST_DROP_METHOD'--pyctest-drop-site<STR>SetCTestvariable:'CTEST_DROP_SITE'--pyctest-drop-site-password<STR>SetCTestvariable:'CTEST_DROP_SITE_PASSWORD'--pyctest-drop-site-user<STR>SetCTestvariable:'CTEST_DROP_SITE_USER'--pyctest-nightly-start-time<STR>SetCTestvariable:'CTEST_NIGHTLY_START_TIME'--pyctest-update-version-onlySpecifythatyouwanttheversioncontrolupdatecommandtoonlydiscoverthecurrentversionthatischeckedout,andnottoupdatetoadifferentversion