Package: jsonformatter
jsonformatter -- for python log jsonInstallationBasic UsageCase 1. Initial root logger likelogging.basicConfigCase 2. Complete config in python codeCase 3. Use config fileCase 4. InFlaskproject, addLogRecordattribute for auto outputCase 5. InDjangoproject, configLOGGINGMore UsageCase 1. Mixextrato outputCase 2. Output multiple attributes in one keyCase 3. Supportjson.dumpsall optional parametersCase 4. Solve cumtomLogRecordattribute is notJSON serializableLogRecord Attributesjsonformatter -- for python log jsonjsonformatteris a formatter for python output json log, e.g. outputLogStashneeded log.Easilycustom(add/replace)LogRecordattribute, e.g. inFlaskweb project, addusernameattribute toLogRecordfor auto output username.Python 2.7andpython 3are supported from version 0.2.X,  if you are using a version lower than 0.2.X,  Onlypython 3is supported.Installationjsonformatter is available on PyPI.
Use pip to install:$pipinstalljsonformatteror:$gitclonehttps://github.com/MyColorfulDays/jsonformatter.git
$cdjsonformatter
$pythonsetup.pyinstallBasic UsageCase 1. Initial root logger likelogging.basicConfigimportloggingfromjsonformatterimportbasicConfig# default keyword parameter `format`: """{"levelname": "levelname", "name": "name", "message": "message"}"""basicConfig(level=logging.INFO)logging.info('hello, jsonformatter')output:{"levelname":"INFO","name":"root","message":"hello, jsonformatter"}Case 2. Complete config in python codeimportloggingfromjsonformatterimportJsonFormatter# `format` can be `json`, `OrderedDict`, `dict`.# If `format` is `dict` and python version < 3.7.0, the output order is sorted keys, otherwise will same as defined order.# key: string, can be whatever you like.# value: `LogRecord` attribute name.STRING_FORMAT='''{"Name":            "name","Levelno":         "levelno","Levelname":       "levelname","Pathname":        "pathname","Filename":        "filename","Module":          "module","Lineno":          "lineno","FuncName":        "funcName","Created":         "created","Asctime":         "asctime","Msecs":           "msecs","RelativeCreated": "relativeCreated","Thread":          "thread","ThreadName":      "threadName","Process":         "process","Message":         "message"}'''root=logging.getLogger()root.setLevel(logging.INFO)formatter=JsonFormatter(STRING_FORMAT)sh=logging.StreamHandler()sh.setFormatter(formatter)sh.setLevel(logging.INFO)root.addHandler(sh)root.info("test%sformat",'string')output:{"Name":"root","Levelno":20,"Levelname":"INFO","Pathname":"test.py","Filename":"test.py","Module":"test","Lineno":75,"FuncName":"test_string_format","Created":1588185267.3198836,"Asctime":"2020-04-30 02:34:27,319","Msecs":319.8835849761963,"RelativeCreated":88.2880687713623,"Thread":16468,"ThreadName":"MainThread","Process":16828,"Message":"test string format"}Case 3. Use config fileconfig file:$catlogger_config.ini[loggers]keys=root[logger_root]level=INFOhandlers=infohandler###############################################[handlers]keys=infohandler[handler_infohandler]class=StreamHandlerlevel=INFOformatter=form01args=(sys.stdout,)###############################################[formatters]keys=form01[formatter_form01]class=jsonformatter.JsonFormatterformat={"name":"name","levelno":"levelno","levelname":"levelname","pathname":"pathname","filename":"filename","module":"module","lineno":"lineno","funcName":"funcName","created":"created","asctime":"asctime","msecs":"msecs","relativeCreated":"relativeCreated","thread":"thread","threadName":"threadName","process":"process","message":"message"}python code:importloggingimportosfromlogging.configimportfileConfigfileConfig(os.path.join(os.path.dirname(__file__),'logger_config.ini'))root=logging.getLogger('root')root.info('test file config')output:{"name":"root","levelno":20,"levelname":"INFO","pathname":"test.py","filename":"test.py","module":"test","lineno":315,"funcName":"test_file_config","created":1588185267.3020294,"asctime":"2020-04-30 02:34:27","msecs":302.0293712615967,"relativeCreated":70.4338550567627,"thread":16468,"threadName":"MainThread","process":16828,"message":"test file config"}Case 4. InFlaskproject, addLogRecordattribute for auto outputflask_demo.pyimportdatetimeimportjsonimportloggingimportrandomfromcollectionsimportOrderedDictfromjsonformatterimportJsonFormatterfromflaskimportFlask,has_request_context,request,sessionfromflask.loggingimportdefault_handlerapp=Flask(__name__)# the key will add/replace `LogRecord` attribute.# the value must be `callable` type and not support positional paramters, the returned value will be as the `LogRecord` attribute value.RECORD_CUSTOM_ATTRS={# no parameters'url':lambda:request.urlifhas_request_context()elseNone,'username':lambda:session['username']ifhas_request_context()and('username'insession)elseNone,# Arbitrary keywords parameters'status':lambda**record_attrs:'failed'ifrecord_attrs['levelname']in['ERROR','CRITICAL']else'success'}RECORD_CUSTOM_FORMAT=OrderedDict([# custom record attributes start("Url","url"),("Username","username"),("Status","status"),# custom record attributes end("Name","name"),("Levelno","levelno"),("Levelname","levelname"),("Pathname","pathname"),("Filename","filename"),("Module","module"),("Lineno","lineno"),("FuncName","funcName"),("Created","created"),("Asctime","asctime"),("Msecs","msecs"),("RelativeCreated","relativeCreated"),("Thread","thread"),("ThreadName","threadName"),("Process","process"),("Message","message")])formatter=JsonFormatter(RECORD_CUSTOM_FORMAT,record_custom_attrs=RECORD_CUSTOM_ATTRS)default_handler.setFormatter(formatter)app.logger.warning('hello, jsonformatter')output:{"Url":null,"Username":null,"Status":"success","Name":"flask_demo","Levelno":30,"Levelname":"WARNING","Pathname":"flask_demo.py","Filename":"flask_demo.py","Module":"flask_demo","Lineno":54,"FuncName":"<module>","Created":1595781463.3557186,"Asctime":"2020-07-27 00:37:43,355","Msecs":355.71861267089844,"RelativeCreated":858.7081432342529,"Thread":15584,"ThreadName":"MainThread","Process":17560,"Message":"hello, jsonformatter"}Case 5. InDjangoproject, configLOGGINGsettings.pyLOGGING={'version':1,'disable_existing_loggers':False,'formatters':{'standard':{'class':'jsonformatter.JsonFormatter','format':OrderedDict([("Name","name"),("Levelno","levelno"),("Levelname","levelname"),("Pathname","pathname"),("Filename","filename"),("Module","module"),("Lineno","lineno"),("FuncName","funcName"),("Created","created"),("Asctime","asctime"),("Msecs","msecs"),("RelativeCreated","relativeCreated"),("Thread","thread"),("ThreadName","threadName"),("Process","process"),("Message","message")])},},'handlers':{'console':{'level':'INFO','formatter':'standard','class':'logging.StreamHandler',},},'loggers':{'django':{'handlers':['console'],'level':'INFO','propagate':False},}}More UsageCase 1. Mixextrato outputimportloggingfromjsonformatterimportJsonFormatterroot=logging.getLogger()root.setLevel(logging.INFO)sh=logging.StreamHandler()formatter=JsonFormatter(ensure_ascii=False,mix_extra=True,mix_extra_position='tail'# optional: head, mix)sh.setFormatter(formatter)sh.setLevel(logging.INFO)root.addHandler(sh)root.info('test mix extra in fmt',extra={'extra1':'extra content 1','extra2':'extra content 2'})root.info('test mix extra in fmt',extra={'extra3':'extra content 3','extra4':'extra content 4'})output:{"levelname":"INFO","name":"root","message":"test mix extra in fmt","extra1":"extra content 1","extra2":"extra content 2"}{"levelname":"INFO","name":"root","message":"test mix extra in fmt","extra3":"extra content 3","extra4":"extra content 4"}Case 2. Output multiple attributes in one keyimportloggingfromjsonformatterimportJsonFormatterMULTI_ATTRIBUTES_FORMAT='''{"multi attributes in one key": "%(name)s-%(levelno)s-%(levelname)s-%(pathname)s-%(filename)s-%(module)s-%(lineno)d-%(funcName)s-%(created)f-%(asctime)s-%(msecs)d-%(relativeCreated)d-%(thread)d-%(threadName)s-%(process)d-%(message)s"}'''root=logging.getLogger()root.setLevel(logging.INFO)formatter=JsonFormatter(MULTI_ATTRIBUTES_FORMAT)sh=logging.StreamHandler()sh.setFormatter(formatter)sh.setLevel(logging.INFO)root.addHandler(sh)root.info('test multi attributes in one key')Case 3. Supportjson.dumpsall optional parametersimportloggingfromjsonformatterimportJsonFormatterSTRING_FORMAT='''{"Name":            "name","Levelno":         "levelno","Levelname":       "levelname","Pathname":        "pathname","Filename":        "filename","Module":          "module","Lineno":          "lineno","FuncName":        "funcName","Created":         "created","Asctime":         "asctime","Msecs":           "msecs","RelativeCreated": "relativeCreated","Thread":          "thread","ThreadName":      "threadName","Process":         "process","Message":         "message"}'''root=logging.getLogger()root.setLevel(logging.INFO)formatter=JsonFormatter(STRING_FORMAT,indent=4,ensure_ascii=False)sh=logging.StreamHandler()sh.setFormatter(formatter)sh.setLevel(logging.INFO)root.addHandler(sh)root.info('test json optional paramter: 中文')Case 4. Solve cumtomLogRecordattribute is notJSON serializableimportdatetimeimportjsonimportloggingimportrandomfromcollectionsimportOrderedDictfromjsonformatterimportJsonFormatter# the key will add/replace `LogRecord` attribute.# the value must be `callable` type and not support positional paramters, the returned value will be as the `LogRecord` attribute value.RECORD_CUSTOM_ATTRS={# `datetime.datetime` type is not JSON serializable.# solve it in three ways, choose which you like.# 1. use `LogRecord` attribute `Format`: %(asctme)s.# 2. use `json.dumps` optional parameter `default`.# 3. use `json.dumps` optional parameter `cls`.'asctime':lambda:datetime.datetime.today(),'user id':lambda:str(random.random())[2:10]}RECORD_CUSTOM_FORMAT=OrderedDict([("User id","user id"),# new custom attrs("Name","name"),("Levelno","levelno"),("Levelname","levelname"),("Pathname","pathname"),("Filename","filename"),("Module","module"),("Lineno","lineno"),("FuncName","funcName"),("Created","created"),("Asctime","%(asctime)s"),# use `LogRecord` attribute `Format` to find matched key from RECORD_CUSTOM_ATTRS and call it value.("Msecs","msecs"),("RelativeCreated","relativeCreated"),("Thread","thread"),("ThreadName","threadName"),("Process","process"),("Message","message")])# use `json.dumps` optional parameter `default`defDEFAULT_SOLUTION(o):ifnotisinstance(o,(str,int,float,bool,type(None))):returnstr(o)else:returno# use `json.dumps` optional parameter `cls`classCLS_SOLUTION(json.JSONEncoder):defdefault(self,o):ifisinstance(o,datetime.datetime):returno.isoformat()returnjson.JSONEncoder.default(self,o)root=logging.getLogger()root.setLevel(logging.INFO)formatter=JsonFormatter(RECORD_CUSTOM_FORMAT,record_custom_attrs=RECORD_CUSTOM_ATTRS,default=DEFAULT_SOLUTION,cls=CLS_SOLUTION)sh=logging.StreamHandler()sh.setFormatter(formatter)sh.setLevel(logging.INFO)root.addHandler(sh)root.info('record custom attrs')LogRecord AttributesOffical url:https://docs.python.org/3/library/logging.html#logrecord-attributesAttribute nameFormatDescriptionargsYou shouldn’t need to format this yourself.The tuple of arguments merged into msg to produce message, or a dict whose values are used for the merge (when there is only one argument, and it is a dictionary).asctime%(asctime)sHuman-readable time when the LogRecord was created. By default this is of the form ‘2003-07-08 16:49:45,896’ (the numbers after the comma are millisecond portion of the time).created%(created)fTime when the LogRecord was created (as returned by time.time()).exc_infoYou shouldn’t need to format this yourself.Exception tuple (à la sys.exc_info) or, if no exception has occurred, None.filename%(filename)sFilename portion of pathname.funcName%(funcName)sName of function containing the logging call.levelname%(levelname)sText logging level for the message ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL').levelno%(levelno)sNumeric logging level for the message (DEBUG, INFO, WARNING, ERROR, CRITICAL).lineno%(lineno)dSource line number where the logging call was issued (if available).message%(message)sThe logged message, computed as msg % args. This is set when Formatter.format() is invoked.module%(module)sModule (name portion of filename).msecs%(msecs)dMillisecond portion of the time when the LogRecord was created.msgYou shouldn’t need to format this yourself.The format string passed in the original logging call. Merged with args to produce message, or an arbitrary object (see Using arbitrary objects as messages).name%(name)sName of the logger used to log the call.pathname%(pathname)sFull pathname of the source file where the logging call was issued (if available).process%(process)dProcess ID (if available).processName%(processName)sProcess name (if available).relativeCreated%(relativeCreated)dTime in milliseconds when the LogRecord was created, relative to the time the logging module was loaded.stack_infoYou shouldn’t need to format this yourself.Stack frame information (where available) from the bottom of the stack in the current thread, up to and including the stack frame of the logging call which resulted in the creation of this record.thread%(thread)dThread ID (if available).threadName%(threadName)sThread name (if available).