Package: vsc-install
Descriptionvsc-install provides shared setuptools functions and classes for python libraries developed by UGent's HPC groupCommon pitfallsbdist_rpm will fail if your install_requires = 'setuptools' because it will fail to find a setuptools rpm.export VSC_RPM_PYTHON=1will make sure thepython-prefix is added to the packages in install_requires for building RPM's so python-setuptools will be used.Add testsTest are python modules in thetestdirectory which have subclass ofTestCaseand at least one method that has a name starting withtest_You are advised to usefromvsc.install.testingimportTestCase(instead of basicTestCasefromunittest).And any__main__orsuite()is not needed (anymore).Initialise the test directory withmkdir-ptestecho''>test/__init__.pyecho'from vsc.install.commontest import CommonTest'>test/00-import.pyWhen the tests are run,test,libandbin(if relevant) are added tosys.path,
so no need to do so in the tets modules.Run testspythonsetup.pytestFilter tests with-F(test module names) and-f(test method names)See alsopythonsetup.pytest--helpThe dependencies are installed automatically in the.eggsdirectory. It will first trygithub.ugent.beand thengithub.comto install them. The same method is used as through
which the original repository was cloned (http, ssh, ...). In case you need private
dependencies, always clone with ssh.In case following error occurs, it means there is a test moduleXYZthat cannot be imported.File "setup.py", line 499, in loadTestsFromModule
    testsuites = ScanningLoader.loadTestsFromModule(self, module)
File "build/bdist.linux-x86_64/egg/setuptools/command/test.py", line 37, in loadTestsFromModule
File "/usr/lib64/python2.7/unittest/loader.py", line 100, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'XYZ'You can try get the actual import error for fixing the issue withpython-c'import sys;sys.path.insert(0, "test");import XYZ;'Fix failing testsMissing / incorrectLICENSECopy the appropirate license file underknown_licensesin the project directory and name the fileLICENSEMissingREADME.mdCreate aREADME.mdfile with at least aDescriptionsectionFix license headers as described inhttps://github.com/hpcugent/vsc-install/blob/master/lib/vsc/install/headers.pycd <project dir with .git folder>
REPO_BASE_DIR=$PWD python -m vsc.install.headers path/to/file script_or_notFix them all at once using findfind ./{lib,test} -type f -name '*.py' | REPO_BASE_DIR=$PWD xargs -I '{}' python -m vsc.install.headers '{}'
find ./bin -type f -name '*.py' | REPO_BASE_DIR=$PWD xargs -I '{}' python -m vsc.install.headers '{}' 1Do not forget to check the diff.
Modules/scripts without docstring (or magic comment '### END OF HEADER') (incl. test modules)
will get correct header appended to existing one. Add a docstring (or magic comment) to resolve this.Python scripts (i.e. with a python shebang and installed as scripts in setup) have to use#!/usr/bin/env pythonas shebangRemove anybuild_rpms_settings.shleftoversTheTARGETdict insetup.pyshould be minimal unless you really know what you are doing (i.e. if it is truly different from defaults)Removename,scripts, ...Exception: vsc namespace packages do not allow non-shared namespaceAdd to the__init__.py"""Allow other packages to extend this namespace, zip safe setuptools style"""importpkg_resourcespkg_resources.declare_namespace(__name__)bare-excepttry:# somethingexcept:This is bad, because this except will also catch sys.exit() or Keyboardinterupts, something you
typically do not want, if you catch these the program will be in a weird state and then continue on,
whilst the person who just pressed ctrl+c is wondering what is going on and why it is not stopping.so at the very least make this
except Exception (which doesn't catch sys.exit and KeyboardInterupt)
and it would be appreciated if you could actually figure out what exceptions to expect and only catch those
and let your program crash if something you did not intend happens
because it helps developers catch weird errors on their side better.if you do something liketry:Path(int(somestring)).write_text('important data')exceptException:pass# if somestring is not an integer, we didn't need to write anyway, but otherwise we dobecause you know sometimes this string does not contain an integer, so the int() call can fail
you should really only catch ValueError, because this will also fail when your disk is full, or you don't have permissions
or xxx other reasons, and the important data will not be written out and nobody will notice anything!if not 'a' in somelist -> if 'a' not in somelistthis isn't that big of a deal, but if everyone is consistent it's less likely to introduce bugs when a not is added or removed where it didn't need to.
Also helps code review, not in reads better, like english.arguments-differthis will give you errors if you override a function of a superclass but don't use the same amount of arguments,
using less will surely give you errors, so the linter catches this for you nowunused-argumentif you have a function definition witch accepts an argument that is never used in the function body this will now give an error.
clean up your function definition, or fix the error where you actually do need to take this argument into accountunused-variabledefining a variable and then not using it anymore smells bad, why did you do that?
sometimes you do things likeout,exit_code=run_command(something)but you are not interested in the out, only in the exit code,
in this case, write_,exit_code=run_command(something)using _ as a variable name lets pylint and other readers know you do not intend to use that output in the first place.reimportedwhen you re import a name somewhere else,
usually this is just an import to much, or 2 imports with the same name, pay attention.importsixfromdjangoimportsix=>importsixfromdjangoimportsixasdjango_sixredefinition of unused namethis usually also points to something you did not expectfromvsc.accountpageclientimportVscGroup<snip>classVscGroup(object):pass=> do you need the import? use import as
did you mean to use the same name? ...Redefined builtinuse different name, for example changedeffilter(b_b):"""Function filter"""returnb_b=>defnew_filter(b_b):"""Function filter"""returnb_blogging-not-lazyDon't use string interpolation when logging if not needed:importloggingname='world'program='python'logging.info('Hello%s! This is%s.'%(name,program))=>importloggingname='world'program='python'logging.info('Hello%s! This is%s.',name,program)Fix Python 3 failing testsWe try to followhttps://docs.python.org/3/howto/pyporting.htmlsome useful info can be found here as wellhttps://portingguide.readthedocs.io/en/latest/index.htmlunpacking-in-except / redefine-in-handlerMultiple exception have to be grouped in a tuple like...except(ExceptionOne,ExceptionTwo)......(espcially when used likeexcept A, B:which should beexcept (A, B):.Old raise syntaxPython 2’sraisestatement was designed at a time when exceptions weren’t classes, and an exception’stype,value, andtracebackcomponents were three separate objects. In Python 3, one single object includes all information about an exception.raiseNameError,"Error"=>raiseNameError("Error")or changeraiseNameError,"Error",some_traceback=>raiseNameError("Error")e=NameError("Error")e.__traceback__=some_tracebackbacktickA=2B=`A`=>A=2B=str(A)Old ne operatorif2<>3:=>if2!=3:Octal literalos.chmod(foo,0700)=>os.chmod(foo,0o700)Import star module levelDo not import *, be more specific. If it is impossible, import it in the top level (and suppress the pyflakes error F403.)defcoords(angle,distance):"""Function coords"""frommathimport*returndistance*cos(angle),distance*sin(angle)=>frommathimport*# noqa: F403defcoords(angle,distance):"""Function coords"""returndistance*cos(angle),distance*sin(angle)Raising stringraiseValueError,'message'=>raiseValueError('message')Indexing exceptionexceptIndexErroraserr:err[0]=>exceptIndexErroraserr:IndexError.args[0]turning off these errorsIf in any of these cases you think: yes, I really needed to do this,
I'm monkeypatching things, I'm adding extra functionality that does indeed have an extra(default) paramenter, etc, etc
you can let pylint know to ignore this error in this one specific block of code
by adding e.g. the comment# pylint: disable=<name or numeric id of pylint code>classSomething(object):defdosomething(self,some,thing):# do somethingclassMyFancyThing(SomeThing):# pylint: disable=arguments-differdefdosomething(self,some,thing,fancy=None):# do something fancyFull list with all codes is available athttp://pylint-messages.wikidot.com/all-codesAuto-generatedJenkinsfile/tox.inivsc-installhas support for auto-generating theJenkinsfile(and accompanyingtox.ini), via:python -m vsc.install.ciFailing check on (contents of)Jenkinsfileortox.iniThere are dedicated tests that check whether theJenkinsfileandtox.inifiles were auto-generated
byvsc-install.To fix the tests, simply runpython -m vsc.install.ciusing the latest version ofvsc-installto re-generateJenkinsfileandtox.ini, and then commit & push the changes.If the contents of the file that is auto-generated by the latest version ofvsc-installis incorrect
for whatever reason, you can temporarily bypass the failing test by adding an a file namedJenkinsfile.NOT_AUTOGENERATED_YETortox.ini.NOT_AUTOGENERATED_YET.The filemustcontain the URL of a vsc-install issue, created via viahttps://github.com/hpcugent/vsc-install/issues/new, where the incorrectly generated file is reported.Example:echo "see https://github.com/hpcugent/vsc-install/issues/1234 for more info" > Jenkinsfile.NOT_AUTOGENERATED_YETRequiring JIRA issue ref in PR titleTo also include a check in theJenkinsfilefor having a JIRA issue ref (like[HPC-1234]) in the pull request title,
add a configuration file forpython -m vsc.install.cinamedvsc-ci.inilike this into the repository:[vsc-ci]jira_issue_id_in_pr_title=1Running shellcheckTo also runshellcheckin the generatedJenkinsfile, specify this via avsc-ci.iniconfiguration file:[vsc-ci]run_shellcheck=1Adding additional test commands to JenkinsfileIf additional custom test commands (other thanshellcheck) need to be run by theJenkinsfile,
you can speicfy this invsc-ci.iniviaadditional_test_commands.To add a single custom test command:[vsc-ci]additional_test_commands=./more_test.shTo add multiple test commands:[vsc-ci]additional_test_commands=first-test-cmdsecond-test-cmdthird-test-cmdOverriding install location of scriptsIn some repositories we specify a system-wide install location for scripts viasetup.cfg(see for example theicinga-checksrepository), which causes problems when installingvsc-installin the tox
environment.To override the installation prefix for scripts (only in the tox environment where the tests are run),
specify this via avsc-ci.iniconfiguration file:[vsc-ci]install_scripts_prefix_override=1Use 'easy_install' to install toxFor legacy reasons easy_install is still supported. If you still need it you can enable it (not recommended):[vsc-ci]easy_install_tox=1Avoid runningpip installin repo checkoutFor some repositories, runningpip installto installtoxfrom the checked out repository is problematic,
because of thesetup.cfgcontaining things that should not be picked up bypip.For those repositories, you can specify that the installation commands in theJenkinsfileshould be
run from$HOME, via:[vsc-ci]home_install=1Leveraging system (Python) packagesIf a repository requires Python packages as dependencies that are installed as OS packages (for example,pyslurm),
tox must be configured to inherit these packages in the test environment. This can be enabled via:[vsc-ci]inherit_site_packages=1Pre-installing dependencies before running testsAlthoughvsc-installwill automatically install all dependencies listed insetup.pyprior to running the
tests, there are cases where this doesn't work out as expected.
Some Python packages only support being installed withpip install(for example because they use a namespace
that is spread across multiple different Python packages, likefsandfs.sshfs).You can specify Python packages that should be installed (withpip install) before running the tests viapip_install_test_depsinvsc-ci.ini:[vsc-ci]pip_install_test_deps=foobar<1.0This results in correspondingpip installcommands being added to thecommands_presection intox.ini:[testenv]commands_pre=pip install 'foo'pip install 'bar<1.0'pip install 'setuptools<42.0'python -m easy_install -U vsc-install