Package: acrilog
Table of ContentsOverviewTimedSizedRotatingHandlerexampleMpLogger and LevelBasedFormatterexampleExample outputClarification of parametersnameproecess_keyfile_prefix and file_suffixfile_modeconsolidatekwargsChange Historyv0.9v1.0v1.1v2.0Next StepsOverviewacrilogis a python library encapsulating multiprocessing logging into practical use.acrilogstarted as Acrisel’s internal utility for programmers.It included:Time and size rotating handler.Multiprocessing logging queue serverThe library makes it easier to add logging in a multiprocessing environment where processes are split among multiple Python source codes.We decided to contribute this library to Python community as a token of appreciation to
what this community enables us.We hope that you will find this library useful and helpful as we find it.If you have comments or insights, please don’t hesitate to contact us atsupport@acrisel.comTimedSizedRotatingHandlerUse TimedSizedRotatingHandler is combining TimedRotatingFileHandler with RotatingFileHandler.
Usage as handler with logging is as defined in Python’s logging how-toexampleimportlogging# create loggerlogger=logging.getLogger('simple_example')logger.setLevel(logging.DEBUG)# create console handler and set level to debugch=logging.TimedRotatingFileHandler()ch.setLevel(logging.DEBUG)# create formatterformatter=logging.Formatter('%(asctime)s-%(name)s-%(levelname)s-%(message)s')# add formatter to chch.setFormatter(formatter)# add ch to loggerlogger.addHandler(ch)# 'application' codelogger.debug('debug message')logger.info('info message')logger.warn('warn message')logger.error('error message')logger.critical('critical message')MpLogger and LevelBasedFormatterMultiprocessor logger using QueueListener and QueueHandler
It uses TimedSizedRotatingHandler as its logging handlerIt also uses acris provided LevelBasedFormatter which facilitate message formats
based on record level.  LevelBasedFormatter inherent from logging.Formatter and
can be used as such in customized logging handlers.exampleWithin main processimporttimeimportrandomimportloggingfromacrisimportMpLoggerimportosimportmultiprocessingasmpdefsubproc(limit=1,logger_info=None):logger=MpLogger.get_logger(logger_info,name="acrilog.subproc",)foriinrange(limit):sleep_time=3/random.randint(1,10)time.sleep(sleep_time)logger.info("proc [%s]:%s/%s- sleep%4.4ssec"%(os.getpid(),i,limit,sleep_time))level_formats={logging.DEBUG:"[%(asctime)s][%(levelname)s][%(message)s][%(module)s.%(funcName)s(%(lineno)d) ]",'default':"[%(asctime)s][%(levelname)s][%(message)s]",}mplogger=MpLogger(logging_level=logging.DEBUG,level_formats=level_formats,datefmt='%Y-%m-%d,%H:%M:%S.%f')mplogger.start(name='main_process')logger=MpLogger.get_logger(mplogger.logger_info())logger.debug("starting sub processes")procs=list()forlimitin[1,1]:proc=mp.Process(target=subproc,args=(limit,mplogger.logger_info(),))procs.append(proc)proc.start()forprocinprocs:ifproc:proc.join()logger.debug("sub processes completed")mplogger.stop()Example output[2016-12-19,11:39:44.953189][DEBUG][startingsubprocesses][mplogger.<module>(45)][2016-12-19,11:39:45.258794][INFO][proc[932]:0/1-sleep0.3sec][2016-12-19,11:39:45.707914][INFO][proc[931]:0/1-sleep0.75sec][2016-12-19,11:39:45.710487][DEBUG][subprocessescompleted][mplogger.<module>(56)]Clarification of parametersnamenameidentifies the base name for logger. Note the this parameter is available in both MpLogger init method and in its start method.MpLogger init’snameargument is used for consolidated logger whenconsolidateis set.  It is also used for private logger of the main process, if one not provided when callingstart()method.proecess_keyprocess_keydefines one or more logger record field that would be part of the file name of the log.  In case it is used, logger will have a file per records’ process key.  This will be in addition for a consolidated log, ifconsolidateis set.By default, MpLogger usesnameas the process key.  If something else is provided, e.g.,processName, it will be concatenated tonameas postfix.file_prefix and file_suffixAllows to distinguish among sets of logs of different runs by setting one (or both) offile_prefixandfile_suffix.  Usually, the use of PID and granular datetime as prefix or suffix would create unique set of logs.file_modefile_modelet program define how logs will be opened.  In default, logs are open in append mode.  Hense, history is collected and file a rolled overnight and by size.consolidateconsolidate, when set, will create consolidated log from all processing logs.
Ifconsolidatedis set andstart()is called withoutname, consolidation will be done into the main process.kwargskwargsare named arguments that will passed to FileHandler.  This include:file_mode='a', for RotatingFileHandler
maxBytes=0, for RotatingFileHandler
backupCount=0, for RotatingFileHandler and TimedRotatingFileHandler
encoding='ascii', for RotatingFileHandler and TimedRotatingFileHandler
delay=False, for TimedRotatingFileHandler
when='h', for TimedRotatingFileHandler
interval=1, TimedRotatingFileHandler
utc=False, TimedRotatingFileHandler
atTime=None, for TimedRotatingFileHandlerChange Historyv0.9added ability to pass logger_info to subprocess,exposed encoding parameterv1.0replacedforce_globalwithconsolidateto genrerate consolidated logaddnameargument to MpLogger.start().  This will return logger with that name for the main process.MpLogger.__init__()nameargument will be used for consolidated log.v1.1addfile_prefixandfile_suffixas MpLogger parameters.fix bug when logdir is None.v2.0added NwLogger starting a server logger with NwLoggerClientHandler for remote processes.Next StepsCluster support using TCP/IPLogging monitor and alert