Package: inlog
InlogInlog is your solution to managing and tracking results from your Python programs. It simplifies the process of logging parameters and results, making your work more reproducible and organized.MotivationImagine, you have a python program with some parameters. You make a few experiments with it, producing some results. Traditionally, people write the parameters in the filename:Result1_day2_3deg_4km.txtThere are a few problems with this:The filenames get very long with more parameters.The list of parameters is often incomplete.The list of parameters is not machine readable.Things are usually fine, until you try to reproduce your results a few months later...inlog("Input Logger") addresses those problems by creating a log file with all parameters next to your results.
All you need to do is to put your parameters into a separate file (like YAML or INI format) and with just three additional lines in your python code, inlog will create a comprehensive logfile:importinlogconfig=inlog.load_yaml('Params.yaml','1.0')#do your stuff, using the parameters in config and save results into 'Results1.txt'#...config.write_log('Results1.txt')#this creates a file 'Results1.log' in the same folder as 'Results1.txt'But there is more:inlogstores additional information like the date, runtime or ahash of your data, it can keep track ofcomplex historiesof your results and it can even visualize them as aflowchart!
That`s how, on the long term, inlog helps you to remember:Where did my results come from?What do the results depend on?What programs did I execute to get the results?To see a more comprehensive example, look at theexamples folderInstallationThe basic version ofinlogonly depends on the python standard library.python3 -m pip install inlogIf you want YAML support, install theextrasversion ofinlog:python3 -m pip install inlog[extras]UsageInput ParametersIt is not necessary to read input parameters from a separate file. However, in most cases, this is desirable. Inlog supports currently the 'ini', 'json' and '.yaml' file formats.Ini FormatParsing is done using the 'configparser' module from the python standard library. ini files consist of sections with options. All options are treated as strings. Ini files allow for value interpolation, see themanualfor further information.
Example of an .ini file:[Filepaths_General]home=/homebin=/bin[Filepaths_User]documents=~/Documentsimportinlogconfig=inlog.load_ini('config.ini',version='1.0')JSON FormatExample of a .json file:{"Filepaths":{"Common":{"home":"/home","bin":"/bin"},"User":{"documents":"~/Documents"}}}importinlogconfig=inlog.load_json('config.json',version='1.0')YAML FormatParsing YAML files requires thepyyamllibrary.
Example of a .yaml file:Filepaths:Common:home:/homebin:/binUser:documents:~/Documentsimportinlogconfig=inlog.load_yaml('config.yaml',version='1.0')DictionaryYou can also pass a dictionary directly to the Logger Class:importinlogdictionary={'Filepaths':{'Common':{'home':'/home','bin':'/bin'},'User':{'documents':'~/Documents'}}}config=inlog.Logger(dictionary,version='1.0')Accessing Parametersinlogstores all parameters in a tree data structure. You can access them using the.get()method. This allowsinlogto keep track which options you used in your program and to write only those in the log file. Take the yaml example from above:importinlogconfig=inlog.load_yaml('config.yaml',version='1.0')config.get('Filepaths','User','documents')Similarly, use.set()to set a value to a parameter:config.set('~/MyDocs','Filepaths','User','documents')You can change all the parameters in a subtree usingset_subtree(subtree_dict), using a nested dictionary as argument.config.set_subtree({'documents':'~/MyDocs','pictures':'~/MyPictures'},'Filepaths','User')As a shortcut for accessing parameters, you can use bracket notation []:config['Filepaths','User','documents']In this case,inlogactually performs a depth-first search in the config tree, so with brackets, you can go even shorter. All of the following commands yield the same result:config['Filepaths','User','documents']config['User','documents']config['documents']Type conversionFor the.inifile format, all options are treated as strings.inlogprovides theconvert_typeandconvert_arrayfunctions as shortcuts for type conversions:config.convert_type(int,'option1')config.convert_type(pathlib.Path,'option1')#you can provide an arbitrary conversion functionconfig.convert_array(int,'option1',removeSpaces=True,sep=",")#this will split the string and convert the elements, returning a list.HashesYou can provideinlogthe name of the result files your program produced. In this case,inlogwill store SHA256 hash values of your results in the log.
Therefore, you can later verify that your results truly belong to the parameters given in the log. Useset_outfile()andadd_outfileto set or append the list of filenames.config.set_outfile('Results1.txt')config.add_outfile('Results2.txt')Writing LogsIn order to write a log, you need to specify the file path (or multiple paths) of the new log. Optionally, you can specify existing log files, which will be included in the new log as dependencies. There are two different formats for logs: txt and json.
By default,write_log()will append the filename of all given filenames with.log.config.write_log('Results1.txt',old_logs=['Dependency1.txt'])JSON FormatA json file. This format is the recommended default, since it allows to capture the tree-like structure of dependencies.
Example:{"date":"2023-10-08 20:59:51.019125","program":"inlog/examples/Script2.py","version":"1.0","input":"Config2.ini","runtime":"0:00:00.006449","options":{"section1":{"factor":2,"intermediate":"intermediateResult.dat","result":"FinalResult.dat"}},"output_files":[{"path":"inlog/examples/FinalResult.dat","hash":"6542c8602f59c351652e382f0448b2caba8c6404a133fca7b137ccd679bd7f4b"}],"dependencies":{"inlog/examples/intermediateResult.dat.log":{"date":"2023-10-08 20:59:49.026310","program":"inlog/examples/Script1.py","version":"1.0","input":"Config1.ini","runtime":"0:00:00.005438","options":{"section1":{"start":1,"stop":10,"increment":2,"intermediate":"intermediateResult.dat"}},"output_files":[{"path":"inlog/examples/intermediateResult.dat","hash":"22a23bcb0798a2b67902a51faad1d04fca6489abdc7c3f1ced983ac22658a721"}],"dependencies":{}}}}Text FormatA linear text file, where dependencies are listed first and the new log information is appended at the end of the file. This format is straightforward and easy to read, but gets messy if you have multiple (sub-)dependencies. You can execute such a log as a bash-script to reproduce the data.
Example:cd inlog/examples
python3 Script1.py Config1.ini
# <Date> 2023-10-08 20:57:47.744149
# <Program> inlog/examples/Script1.py
# <Version> 1.0
# <Input> Config1.ini
# <Runtime> 0:00:00.007015
#**************************
#{
#    "section1": {
#        "start": 1,
#        "stop": 10,
#        "increment": 2,
#        "intermediate": "intermediateResult.dat"
#    }
#}
#**************************
#Output files created:
# <PATH> inlog/examples/intermediateResult.dat
# <HASH> 22a23bcb0798a2b67902a51faad1d04fca6489abdc7c3f1ced983ac22658a721
# <Logfile> intermediateResult.dat.log_txt
#=========================================
cd inlog/examples
python3 Script2.py Config2.ini
# <Date> 2023-10-08 20:57:54.775511
# <Program> inlog/examples/Script2.py
# <Version> 1.0
# <Input> Config2.ini
# <Runtime> 0:00:00.007339
#**************************
#{
#    "section1": {
#        "factor": 2,
#        "intermediate": "intermediateResult.dat",
#        "result": "FinalResult.dat"
#    }
#}
#**************************
#Output files created:
# <PATH> inlog/examples/FinalResult.dat
# <HASH> 6542c8602f59c351652e382f0448b2caba8c6404a133fca7b137ccd679bd7f4bVisualizationPrinting the logger object will yield a text version of the current log. Equivalently, you can callshow_data().print(config)config.show_data()FlowchartCallinginlog-flowchart logfile.dat.logwill convert a log "logfile.dat.log" in json format to a Mermaid flowchart. Mermaid is a charting application, based on a markdown like syntax. You can paste the output of flowchart.py in theMermaid Live Editorto obtain a flowchart.
Just call:inlog/flowchart.pyexamples/FinalResult.logThis willl yield the following output:flowchart TD
    id_435d36b4a8d3c591b456a027fe49efa3[FinalResult.log]
    id_89f9132e25d1cada0f37669c8e3a2faa[intermediateResult.log]
    id_f1add11e031d031e9f8694f5591c1fc2[No Dependencies]
    id_f1add11e031d031e9f8694f5591c1fc2 --> |Script1.py| id_89f9132e25d1cada0f37669c8e3a2faa
    id_89f9132e25d1cada0f37669c8e3a2faa --> |Script2.py| id_435d36b4a8d3c591b456a027fe49efa3Pasting into theMermaid Live Editorwill give the following chart:flowchart TD
    id_435d36b4a8d3c591b456a027fe49efa3[FinalResult.log]
    id_89f9132e25d1cada0f37669c8e3a2faa[intermediateResult.log]
    id_f1add11e031d031e9f8694f5591c1fc2[No Dependencies]
    id_f1add11e031d031e9f8694f5591c1fc2 --> |Script1.py| id_89f9132e25d1cada0f37669c8e3a2faa
    id_89f9132e25d1cada0f37669c8e3a2faa --> |Script2.py| id_435d36b4a8d3c591b456a027fe49efa3