Package: callable-journal
callable-journalLog message generator for a callable's argument and return values.Package MotivationDo you ever start with something simple like this.fromtypingimportList,Tupledefmotivation(a:int,b:List[int])->Tuple[int,List[int]]:multiplied=[a*b_itemforb_iteminb]returnsum(multiplied),multipliedRealize you want logging like this.{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"basic","arguments":{"a":2,"b":[1,2]},"results":[6,[2,4]]}Mess up your function like this.importcopyimportloggingfromtypingimportList,Tuplelogger=logging.getLogger()defmotivation_with_logging(a:int,b:List[int])->Tuple[int,List[int]]:b_copy=copy.deepcopy(b)multiplied=[a*b_itemforb_iteminb]sum_result=sum(multiplied)msg={"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"basic","arguments":{"a":a,"b":b_copy},"results":[sum_result,multiplied]}logger.info(msg)returnsum(multiplied),multipliedWouldn't it be nice if you could just do this!fromtypingimportList,Tuplefromcallable_journalimportjournal@journaldefbasic(a:int,b:List[int])->Tuple[int,List[int]]:multiplied=[a*b_itemforb_iteminb]returnsum(multiplied),multipliedThe callable journal decorator does this and a bit more.  It works great on API endpoints
where you want a nice record of what the endpoint was called with and what it returned.  By
connecting into the standard library logging package it is easy to format and direct
journal message to your favorite sink.Naming ResultsIf you have a callable that returns multiple values, you can assign names to them using
theresult_namesparameter.fromtypingimportList,Tuplefromcallable_journalimportjournal@journal(result_names=["total_sum","multiplied_values"])defnamed_results(a:int,b:List[int])->Tuple[int,List[int]]:multiplied=[a*b_itemforb_iteminb]returnsum(multiplied),multipliedLog Message:{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"named_results","arguments":{"a":2,"b":[1,2]},"results":{"total_sum":6,"multiplied_values":[2,4]}}Dropping ResultsIf the result contains sensitive data or not of interest it can be dropped usingDROP_RESULT.fromtypingimportList,Tuplefromcallable_journalimportjournal,DROP_RESULT@journal(result_names=["total_sum",DROP_RESULT])defnamed_ignore_results(a:int,b:List[int])->Tuple[int,List[int]]:multiplied=[a*b_itemforb_iteminb]returnsum(multiplied),multipliedLog Message:{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"named_results","arguments":{"a":2,"b":[1,2]},"results":{"total_sum":6}}Copying ArgumentsSometimes a callable mutates its arguments for good or bad reasons.  If the arguments
aren't copied the mutated version will show up in the log message.  Because copying some
arguments is costly, you can specify which arguments to copy with thecopy_argsparameter.fromtypingimportList,Tuplefromcallable_journalimportjournal@journal(copy_args="b")defcopy_args(a:int,b:List[int])->Tuple[int,List[int]]:# Modify the mutable argument.b[0],b[1]=-b[0],-b[1]multiplied=[a*b_itemforb_iteminb]returnsum(multiplied),multipliedLog Message with the correct mutable argument:{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"copy_args","arguments":{"a":2,"b":[1,2]},"results":[-6,[-2,-4]]}You can use theCOPY_ALLvalue for thecopy_argsparameter to copy all the args.Dropping ArgumentsIn the same way that sometimes you want to drop results you may want to drop arguments.
This is accomplished with thedrop_argsparameter.fromtypingimportList,Tuplefromcallable_journalimportjournal@journal(drop_args=["b"])defdrop_args(a:int,b:List[int])->Tuple[int,List[int]]:multiplied=[a*b_itemforb_iteminb]returnsum(multiplied),multipliedLog Message:{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"drop_args","arguments":{"a":2},"results":[6,[2,4]]}ContextIf you want information about the context of the application the log message was run in you
can add context information to the log messages.frompathlibimportPathfromtypingimportList,Tuplefromcallable_journalimportjournal,journal_init@journaldefadd_context(a:int,b:List[int])->Tuple[int,List[int]]:multiplied=[a*b_itemforb_iteminb]returnsum(multiplied),multiplieddefrun_add_context():context={"app_version":"0.1.0"}journal_init(Path(__file__).parent/"journal-cfg.yml",context=context)a,b=2,[1,2]add_context(a,b){"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"add_context","arguments":{"a":2,"b":[1,2]},"results":[6,[2,4]],"app_version":"0.1.0"}ExceptionsUncaught exceptions are going to be raised and reported, but it is nice to get some amount
of information about the exception in the log message.  An example is shown here.{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"exception","arguments":{"a":2,"b":[1,2]},"exception":{"type":"ZeroDivisionError","msg":"division by zero","file":"/home/some_user/projects/callable-journal/test/docs_example_test.py","line":"170"}}Logging ConfigurationLogging configuration uses the standard library logging configuration.  Here is an example
that configures two loggers. One the generates pure JSON log messages and one that generates
STRINGY log messages.---version:1disable_existing_loggers:falseformatters:journal-json:():callable_journal.formatter.JournalFormattertag:JOURNAL_MSG_JSONformat_mode:jsonjournal-stringy:():callable_journal.formatter.JournalFormattertag:JOURNAL_MSG_STRINGYformat_mode:stringyhandlers:journal-json-console:class:logging.StreamHandlerlevel:INFOformatter:journal-jsonstream:ext://sys.stdoutjournal-stringy-console:class:logging.StreamHandlerlevel:INFOformatter:journal-stringystream:ext://sys.stdoutloggers:journal:level:INFOhandlers:-journal-json-console-journal-stringy-consolepropagate:false...JSON Log Message:{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"log_format","context":{"app_version":"0.1.0"},"arguments":{"a":2,"b":[1,2]},"results":[6,[2,4]],}STRINGY Log Message:{"tag":"JOURNAL_MSG_JSON","format":"0.2.0","objective":"log_format","context":"{\"app_version\": \"0.1.0\"}","arguments":"{\"a\": 2, \"b\": [1, 2]}","results":"[6, [2, 4]]"}Why the STRINGY format?  If you have many different messages all flowing through to something
like BigQuery it is nice to keep BigQuery from exploding the nested JSON in the context,
arguments and results.   It could wind up with a large number of confusingly named columns.
By using the STRINGY format all the arguments from different journals go into one column
and you can use the JSON extract functionality of BigQuery to get what you want out of that
column's JSON.Logging FormatThe loggers are derived fromlogging.Formatterand pass any unusedargsandkwargson to
the default Formatter.  By adding aformatentry to the configuration you can add all
the other log message content you are used to seeing by default.Configuration:---version:1disable_existing_loggers:falseformatters:journal-json:():callable_journal.formatter.JournalFormattertag:JOURNAL_MSG_JSONformat_mode:jsonformat:'%(asctime)s-%(name)s-%(levelname)s-%(message)s'handlers:journal-json-console:class:logging.StreamHandlerlevel:INFOformatter:journal-jsonstream:ext://sys.stdoutloggers:journal:level:INFOhandlers:-journal-json-consolepropagate:false...Log Message:2020-08-30 15:54:11,813 - journal - INFO - {
        "tag": "JOURNAL_MSG_JSON",
        "format": "0.2.0",
        "objective": "log_format",
        app_version": "0.1.0",
        "arguments": {"a": 2, "b": [1, 2]},
        "results": [6, [2, 4]],
    }