Package: scitrack
AboutscitrackOne of the critical challenges in scientific analysis is to track all the elements involved. This includes the arguments provided to a specific application (including default values), input data files referenced by those arguments and output data generated by the application. In addition to this, tracking a minimal set of system specific information.scitrackis a simple package aimed at researchers writing scripts, or more substantial scientific software, to support the tracking of scientific computation.  The package provides elementary functionality to support logging. The primary capabilities concern generating checksums on input and output files and facilitating logging of the computational environment.To see some projects usingscitrack, see the “Used by” link at the top of theproject GitHub page.InstallingFor the released version:$ pip install scitrackFor the very latest version:$ pip install git+https://github.com/HuttleyLab/scitrackOr clone it:$ git clone git@github.com:HuttleyLab/scitrack.gitAnd then install:$ pip install ~/path/to/scitrackCachingLoggerThere is a single object provided byscitrack,CachingLogger. This object is basically a wrapper around the Python standard libraryloggingmodule. On invocation,CachingLoggercaptures basic information regarding the system and the command line call that was made to invoke the application.In addition, the class provides convenience methods for logging both the path and the md5 hexdigest checksum[1]of input/output files. A method is also provided for producing checksums of text data. The latter is useful for the case when data are from a stream or a database, for instance.All logging calls are cached until a path for a logfile is provided. The logger can also, optionally, create directories.Simple instantiation of the loggerCreating the logger. Settingcreate_dir=Truemeans on creation of the logfile, the directory path will be created also.fromscitrackimportCachingLoggerLOGGER=CachingLogger(create_dir=True)LOGGER.log_file_path="somedir/some_path.log"The last assignment triggers creation ofsomedir/some_path.log.WarningOnce set, a loggers.log_file_pathcannot be changed.Capturing a programs arguments and optionsscitrackwill write the contents ofsys.argvto the log file, prefixed bycommand_string. However, this only captures arguments specified on the command line. Tracking the value of optional arguments not specified, which may have default values, is critical to tracking the full command set. Doing this is now easy with the simple statementLOGGER.log_args(). The logger can also record the versions of named dependencies.Here’s one approach to incorporatingscitrackinto a command line application built using theclickcommand line interface library. Below we create a simpleclickapp and capture the required and optional argument values.NoteLOGGER.log_args()should be called immediately after the function definition, or after “true” default values have been configured.importclickfromscitrackimportCachingLoggerLOGGER=CachingLogger()@click.command()@click.option("-i","--infile",type=click.Path(exists=True))@click.option("-t","--test",is_flag=True,help="Run test.")defmain(infile,test):# capture the local variables, at this point just provided argumentsLOGGER.log_args()LOGGER.log_versions("numpy")LOGGER.input_file(infile)LOGGER.log_file_path="some_path.log"if__name__=="__main__":main()TheCachingLogger.write()method takes a message and a label. All other logging methods wraplog_message(), providing a specific label. For instance, the methodinput_file()writes out two lines in the log.input_file_path, the absolute path to the intput fileinput_file_path md5sum, the hex digest of the fileoutput_file()behaves analogously. An additional methodtext_data()is useful for other data input/output sources (e.g. records from a database). For this to have value for arbitrary data types requires a systematic approach to ensuring the text conversion is robust across platforms.Thelog_args()method captures all local variables within a scope.Thelog_versions()method captures versions for the current file and that of a list of named packages, e.g.LOGGER.log_versions(['numpy','sklearn']).Some sample output2020-05-25 13:32:07 Eratosthenes:98447      INFO    system_details : system=Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64
2020-05-25 13:32:07 Eratosthenes:98447      INFO    python : 3.8.2
2020-05-25 13:32:07 Eratosthenes:98447      INFO    user : gavin
2020-05-25 13:32:07 Eratosthenes:98447      INFO    command_string : ./demo.py -i /Users/gavin/repos/SciTrack/tests/sample-lf.fasta
2020-05-25 13:32:07 Eratosthenes:98447      INFO    params : {'infile': '/Users/gavin/repos/SciTrack/tests/sample-lf.fasta', 'test': False}
2020-05-25 13:32:07 Eratosthenes:98447      INFO    version : __main__==None
2020-05-25 13:32:07 Eratosthenes:98447      INFO    version : numpy==1.18.4
2020-05-25 13:32:07 Eratosthenes:98447      INFO    input_file_path : /Users/gavin/repos/SciTrack/tests/sample-lf.fasta
2020-05-25 13:32:07 Eratosthenes:98447      INFO    input_file_path md5sum : 96eb2c2632bae19eb65ea9224aaafdadOther useful functionsTwo other useful functions areget_file_hexdigestandget_text_hexdigest.Reporting issuesUse the projectissue tracker.For DevelopersWe useflitfor package building. Having cloned the repository onto your machine. Installflit:$ python3 -m pip install flitDo a developer install ofscitrackusing flit as:$ cd path/to/cloned/repo
$ flit install -s --python `which python`NoteThis installs a symlink intosite-packagesof the python identified bywhich python.[1]The hexdigest serves as a unique signature of a files contents.