Package: logger-tt
Logger_ttMake configuring logging simpler and log even exceptions that you forgot to catch.Even multiprocessing logging becomes a breeze.Table of contentsInstallOverviewUsageOverwrite the default log pathProvide your config fileCapture stdoutException loggingtry-except exception loggingSilent unwanted loggersLogging in multiprocessingTemporary disable loggingLimit traceback line lengthAnalyzeraiseexception lineStreamHandler with bufferTelegramHandlerSample configYAML formatJSON formatChangelogInstallFrom PYPI:pip install logger_ttFrom Github: clone or download this repo thenpython setup.py installOverview:In the most simple case, add the following code into your main python script of your project:fromlogger_ttimportsetup_loggingsetup_logging(full_context=1)Then from any of your modules, you just need to get aloggerand start logging.
There are two ways that you could obtain a logger.Conventional way: as you have always done it that wayfromloggingimportgetLoggerlogger=getLogger(__name__)Convenient way: use a pre-made logger namedlogger_ttfrom this package.fromlogger_ttimportloggerAfter that, you start logging as usual:logger.debug('Module is initialized')logger.info('Making connection ...')# output[2020-07-2111:24:19][__main__:5DEBUG]Moduleisinitialized[2020-07-2111:24:19][__main__:6INFO]Makingconnection...Both ways give you the same output except the line number, obviously.The pre-made logger also has an advantage that it will automatically injectthreadNameandprocessNameto the output in case of multithreading or multiprocessing.Both ways will provide your project with the followingdefaultlog behaviors:log file: Assume that yourworking directoryisproject_root,
log.txt is stored at yourproject_root/logs/folder.If the log path doesn't exist, it will be created.The log file is time rotated at midnight. A maximum of 15 dates of logs will be kept.
This log file'slevelisDEBUG.The log format is[%(asctime)s] [%(name)s:%(lineno)d %(levelname)s] %(message)s,where the time format is%Y-%m-%d %H:%M:%S.Example:[2020-05-09 00:31:33] [myproject.mymodule:26 DEBUG] Module is initializedconsole: log records with levelINFOand above will be printed tostdoutof the console.The format for console log is simpler:[%(asctime)s] %(levelname)s: %(message)s.Example:[2020-05-09 00:31:34] INFO: Making connection ...urllib3logger: this ready-made logger is to silent unwanted messages fromrequestslibrary.suppressed logger:exchangelib. This sets logging level ofexchangeliblogger toWARNING.This is another way to silent unwanted messages from other module, read below for details.rootlogger: if there is no logger initialized in your module, this logger will be used with the above behaviors.
This logger is also used to loguncaught exceptionin your project. Example:raiseRecursionError# log.txt[2020-05-3119:16:01][rootERROR]UncaughtexceptionTraceback(mostrecentcalllast):File"D:/MyProject/Echelon/eyes.py",line13,in<module>raiseRecursionError=>var_in=Customer(name='John',member_id=123456)=>arg=(456,789)=>kwargs={'my_kw':'hello','another_kw':'world'}RecursionErrorcontext logging: When an exception occur, variables used in the line of error are also logged.To log full local variables of current function scope, passfull_context=1tosetup_logging.If you need the outer scope too, setfull_contextto2,3and so on...Usage:All configs are done throughsetup_loggingfunction:setup_logging(config_path="",log_path="",capture_print=False,strict=False,guess_level=False,full_context=False,suppress_level_below=logging.WARNING,use_multiprocessing=False,limit_line_length=1000,analyze_raise_statement=False,host="",port=0,)This function also return aLogConfigobject.
Exceptconfig_path,log_path,use_multiprocessing,hostandport,
other parameters are attributes of this object and can be changed on the fly.Exceptconfig_path,log_path, all other parameters can be defined inlogger_ttsection in the config file
(seeSample configchapter below).
Parameter with the same name passed insetup_loggingfunction will override the one in the config file.1. Overwrite the default log path:Instead of./logs/logs.txt, you can overwrite with your own as followssetup_logging(log_path='new/path/to/your_log.txt')If you have multiple handlers and want each of them log to different destinations,
you can pass in a dict of"handler_name": "log_path".
The handler names must match what is defined in the config file.setup_logging(log_path={'error_file_handler':'log/error.txt','info_file_handler':'log/info_and_above.txt',})2. Provide your config file:You can config your own logger and handler by providing eitheryamlorjsonconfig file as follows:setup_logging(config_path='path/to/.yaml_or_.json')Without providing a config file, the default config file with the abovedefaultlog behavior is used.
You could copylog_conf.yamlorlog_conf.jsonshipped with this package to start making your version.Warning: To process.yamlconfig file, you either needpyyamlorruamel.yamlpackage installed.3. Capture stdout:If you have an old code base with a lot ofprint(msg)orsys.stdout.write(msg)and
don't have access or time to refactor them into something likelogger.info(msg),
you can capture thesemsgand log them to file, too.To capture onlymsgthat is printed out byprint(msg), simply do as follows:setup_logging(capture_print=True)Example:print('To be or not to be')sys.stdout.write('That is the question')# log.txt
[2020-05-09 11:42:08] [PrintCapture INFO] To be or not to beYes,That is the questionis not captured.
Some libraries may directly usesys.stdout.writeto draw on the screen (eg. progress bar) or do something quirk.
This kind of information is usually not useful for users. But when you do need it, you can capture it as follows:setup_logging(capture_print=True,strict=True)Example:sys.stdout.write('The plane VJ-723 has been delayed')sys.stdout.write('New departure time has not been scheduled')# log.txt
[2020-05-09 11:42:08] [PrintCapture INFO] The plane VJ-723 has been delayed
[2020-05-09 11:42:08] [PrintCapture INFO] New departure time has not been scheduledAs you have seen, the log level of the captured message isINFO.
What if the code base prints something likeAn error has occurred. Abort operation.and you want to log it asError?
Just addguess_level=Truetosetup_logging().setup_logging(capture_print=True,guess_level=True)Example:print('An error has occurred. Abort operation.')print('A critical error has occurred during making request to database')# log.txt
[2020-05-09 11:42:08] [PrintCapture ERROR] An error has occurred. Abort operation.
[2020-05-09 11:42:08] [PrintCapture CRITICAL] A critical error has occurred during making request to databaseNote: Capturing stdout ignores messages ofblank line.
That means messages like\n\nor(spaces) will not appear in the log.
But messages that contain blank line(s) and other characters will be fully logged.
For example,\nTo day is a beautiful day\nwill be logged as is.4. Exception logging:Consider the following error code snippet:API_KEY="asdjhfbhbsdf82340hsdf09u3ionf98230234ilsfd"TIMEOUT=60classMyProfile:def__init__(self,name):self.my_boss=Noneself.name=namedefmy_faulty_func(my_var,*args,**kwargs):new_var='local scope variable'me=MyProfile('John Wick')boss=MyProfile('Winston')me.my_boss=bossprint(f'Information:{var}and{me.my_boss.name}'' at{me.my_boss.location}with{API_KEY}')if__name__=='__main__':cpu_no=4max_concurrent_processes=3my_faulty_func(max_concurrent_processes,'ryzen 7',freq=3.4)In our hypothetical code above,printfunction will raise an exception.
This exception, by default, will not only be logged but also analyzed with objects that appeared in the line:[2020-06-0609:36:01]ERROR:Uncaughtexception:Traceback(mostrecentcalllast):File"D:/MyProject/AutoBanking/main.py",line31,in<module>my_faulty_func(max_concurrent_processes,'ryzen 7',freq=3.4)|->my_faulty_func=<functionmy_faulty_funcat0x0000023770C6A288>|->max_concurrent_processes=3File"D:/MyProject/AutoBanking/main.py",line25,inmy_faulty_funcprint(f'Information:{var}and{me.my_boss.name}'' at{me.my_boss.location}with{API_KEY}')|->me.my_boss.name='Winston'|->me.my_boss.location='!!! Not Exists'|->(outer)API_KEY='asdjhfbhbsdf82340hsdf09u3ionf98230234ilsfd'NameError:name'var'isnotdefinedNote: look at theprint(f'Information...line,logger-ttprint this error line different from normal python traceback!
With normal traceback, multi-line python statement has its only first line printed out.
Withlogger-tt, a full statement is grabbed for you.For each level in the stack, any object that appears in the error line is shown with itsreadable representation.
This representation may not necessarily be__repr__. The choice between__str__and__repr__are as follows:__str__:__str__is present, and the object class's__repr__is default with<class name at Address>.__repr__:__str__is present, but the object class's__repr__is anything else, such asClassName(var=value).Also, when__str__is missing, even if__repr__is<class name at Address>, it is used.Currently, if an object doesn't exist and is directly accessed, asvarin this case, it will not be shown up.
But if it is attribute accessed with dot., aslocationinme.my_boss.location,
then its value is an explicit string'!!! Not Exists'.As you may have noticed, a variableAPI_KEYhas its name prefixed withouter.This tells you that the variable is defined in the outer scope, not local.More often than not, only objects in the error line are not sufficient to diagnose what has happened.
You want to know what the inputs of the function were. You want to know what the intermediate
calculated results were. You want to know other objects that appeared during runtime,
not only local but also outer scope. In other words, you want to know the full context of what has happened.logger-ttis here with you:setup_logging(full_context=2)With the above hypothetical code snippet, the error log becomes the following:[2020-06-0610:35:21]ERROR:Uncaughtexception:Traceback(mostrecentcalllast):File"D:/MyProject/AutoBanking/main.py",line31,in<module>my_faulty_func(max_concurrent_processes,'ryzen 7',freq=3.4)|->my_faulty_func=<functionmy_faulty_funcat0x0000019E3599A288>|->max_concurrent_processes=3=>__name__='__main__'=>__doc__=None=>__package__=None=>__loader__=<_frozen_importlib_external.SourceFileLoaderobjectat0x0000019E35840E48>=>__spec__=None=>__annotations__={}=>__builtins__=<module'builtins'(built-in)>=>__file__='D:/MyProject/AutoBanking/main.py'=>__cached__=None=>setup_logging=<functionsetup_loggingat0x0000019E35D111F8>=>getLogger=<functiongetLoggerat0x0000019E35BC7C18>=>logger=<Logger__main__(DEBUG)>=>API_KEY='asdjhfbhbsdf82340hsdf09u3ionf98230234ilsfd'=>TIMEOUT=60=>MyProfile=<class'__main__.MyProfile'>=>cpu_no=4File"D:/MyProject/AutoBanking/main.py",line25,inmy_faulty_funcprint(f'Information:{var}and{me.my_boss.name}at{me.my_boss.location}with{API_KEY}')|->me.my_boss.name='Winston'|->me.my_boss.location='!!! Not Exists'|->(outer)API_KEY='asdjhfbhbsdf82340hsdf09u3ionf98230234ilsfd'=>my_var=3=>args=('ryzen 7',)=>kwargs={'freq':3.4}=>new_var='local scope variable'=>me=<__main__.MyProfileobjectat0x0000019E35D3BA48>=>boss=<__main__.MyProfileobjectat0x0000019E35D3B9C8>NameError:name'var'isnotdefinedAdditional objects that not appear in the error line are prefixed with=>.Note: from version 1.6, uncaught exception happened in child thread of a multithreading
program will also be caught bylogger-ttand logged normally.
If you are using python 3.8+, the newthreading.excepthookwon't be called as the uncaught exception
has been handled bylogger-tt.5.try-exceptexception logging:exception contextlogging also applies fortry-exceptblock.
This means that if you calllogger.exception()insideexceptblock,
you would have all variables' value at the line of exception. For example,defmy_faulty_func():a=10b=0c=a/breturncdefmy_main():try:my_faulty_func()exceptExceptionase:logger.exception('some error has occured')print('Clean up resource')my_main()Then the log will show up as follows:[2020-06-1221:37:00]ERROR:someerrorhasoccuredTraceback(mostrecentcalllast):File"D:/MyProject/exception_log.py",line19,inmy_mainmy_faulty_func()|->my_faulty_func=<functionmy_faulty_funcat0x000001875DD4B168>File"D:/MyProject/exception_log.py",line13,inmy_faulty_funcc=a/b|->a=10|->b=0ZeroDivisionError:divisionbyzeroCleanupresourceNote: As in python'slogging document,logger.exception()should only be called from an exception handler, eg. insideexceptblock.You don't need to passexception objecttologger.exception().
It already knows how to get a traceback internally.
This enable you to pass any string in as a hint or a short description of what may have happened.Otherwise, passingexception object, aslogger.exception(e),
will cause the first line of error report to be the message of exception.
In the case of the above example, it would be come[2020-06-12 21:37:00] ERROR: division by zero.6. Silent unwanted loggers:Third party modules also have loggers, and their messages are usually not related to your code.
A bunch of unwanted messages may hide the one that come from your own module.
To prevent that and also reduce log file size, we need to silent unwanted loggers.By config file, there are two ways to silent a logger:Create a new logger: inloggersection of the config file,
add a new logger whose name is the same with the one you want to silent.
Set it level toWARNINGor above. If you addhandlers, you should also setpropagatetonoorFalse.
Otherwise, the same message may be logged multiple times. Ex:urllib3:
  level: WARNING
  handlers: [console, error_file_handler]
  propagate: noAbove setting only allow messages with levelWARNINGand above to be processed.
Usually that is enough to silent most of unwanted messages. If you need to silent more messages,
tryERRORorCRITICAL.Add logger's name tosuppress list: Then a new logger with level default toWARNINGwill be
automatically created for you. Ex:suppress: [exchangelib, urllib3]If you need to suppress at even higher level, usesuppress_level_belowinsetup_logging.
For example suppress any message belowERRORlevel that comes from loggers insuppress list:setup_logging(suppress_level_below=logging.ERROR)You could also suppress loggers directly bysetup_logging:fromlogger_ttimportsetup_loggingsetup_logging(suppress=['urllib3','exchangelib'])7. Logging in multiprocessing:This is archived by using multiprocessing queues or a socket server.For linux, copy-on-write while forking carries over logger's information.
Somultiprocess.Queueis enough in this case.For Windows, it is important thatsetup_logging()must be call out side ofif __name__ == '__main__':guard block.
Because child processes run from scratch and re-import__main__, by re-executingsetup_logging(),
loggerSocketHandlercan be setup automatically.This also means that the same config can work with bothmultiprocessing.Processandmultiprocessing.Poolmagically without user doing anything special.Below is a minimal example:importtimefromrandomimportrandintfrommultiprocessingimportProcessfromlogger_ttimportsetup_logging,loggersetup_logging(use_multiprocessing=True)# for Windows, this line must be outside of guard blockdefworker(arg):logger.info(f'child process{arg}: started')time.sleep(randint(1,10))# imitate time consuming processlogger.info(f'child process{arg}: stopped')if__name__=='__main__':all_processes=[]logger.info('Parent process is ready to spawn child')foriinrange(3):p=Process(target=worker,args=(i,))all_processes.append(p)p.daemon=Truep.start()forpinall_processes:p.join()The content oflog.txtshould be similar to below:[2020-10-28 20:39:14] [root:129 DEBUG] _________________New log started__________________
[2020-10-28 20:39:17] [root:130 DEBUG] Log config file: D:\my_project\log_config.json
[2020-10-28 20:39:17] [root:131 DEBUG] Logging server started!
[2020-10-28 20:39:22] [__main__:28 INFO] Parent process is ready to spawn child
[2020-10-28 20:39:22] [__mp_main__:16 INFO] Process-3 child process 2: started
[2020-10-28 20:39:22] [__mp_main__:16 INFO] Process-2 child process 1: started
[2020-10-28 20:39:22] [__mp_main__:16 INFO] Process-1 child process 0: started
[2020-10-28 20:39:23] [__mp_main__:18 INFO] Process-2 child process 1: stopped
[2020-10-28 20:39:23] [__mp_main__:18 INFO] Process-3 child process 2: stopped
[2020-10-28 20:39:24] [__mp_main__:18 INFO] Process-1 child process 0: stoppedNote: Under linux, to usequeueHandler, you must passuse_multiprocessing="fork"tosetup_logging.Other optionsTrue,spawn,forkserverwill usesocketHandlerby default.This is to prevent youset_start_methodasspawnunder linux and thusqueueHandlerwon't work.Socket Address:socketHandlerwill use tcplocalhostand port9020by default.
In the rare cases where you run multiple multiprocessing applications withlogger_tt,
theAddress already in useerror will be raised. In such cases, you have to set the address manually.setup_logging(host='localhost',port=6789)You can omit thehostif you use"localhost".
You can also set this in the log config file for each application.8. Temporary disable logging:Some block of code contain critical information, such as password processing, that should not be logged.
You can disable logging for that block with alogging_disabledcontext:fromlogger_ttimportlogging_disabled,getLoggerlogger=getLogger(__name__)logger.debug('Begin a secret process')withlogging_disabled():logger.info('This will not appear in any log')logger.debug('Finish')9. Limit traceback line's length:Sometimes the variable on the exception line can hold enormous amount of data,
such as content of some huge json file or html. In this case, printing out the whole content
of the variable is quite point less and hinders debugging process as it hides away exception line.So we should limit the character to be printed out in each line of the traceback.
And we can do it as simple as follow:setup_logging(limit_line_length=1000)The default limit is 1000 characters. All left characters will be replaced with....limit_line_length=0means no limit at all, prints the content of variable as is.Note: if you input afloat, it will be round down to nearestint.
A negative input is treated as inputting0.For demonstration purpose, the example below will limit to100characters:fromlogger_ttimportsetup_loggingsetup_logging(limit_line_length=100)defwill_fail():loren_ipsum="On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and "\"demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot "\"foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in "\"their duty through weakness of will, which is the same as saying through shrinking from toil and "\"pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of "\"choice is untrammelled and when nothing prevents our being able to do what we like best, "\"every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to "\"the claims of duty or the obligations of business it will frequently occur that pleasures have to "\"be repudiated and annoyances accepted. The wise man therefore always holds in these matters to "\"this principle of selection: he rejects pleasures to secure other greater pleasures, or else he "\"endures pains to avoid worse pains. "print(f'Below is the{random}text used as a standard to test font:\n{loren_ipsum}')if__name__=='__main__':will_fail()It will output the follow traceback. Pay attention to theloren_ipsumvariable.[2021-06-1917:40:39]ERROR:Uncaughtexception:Traceback(mostrecentcalllast):File"D:\my_project\long_line.py",line21,in<module>will_fail()File"D:\my_project\long_line.py",line18,inwill_failprint(f'Below is the{random}text used as a standard to test font:\n{loren_ipsum}')|->loren_ipsum='On the other hand, we denounce with righteous indignation and dislike men wh... (922 characters more)NameError:name'random'isnotdefined10. Analyzeraiseexception line:If the code explicitlyraisean exception, in most cases,
the variables on the line are substituted and printed out at the end of traceback.
Withlogger-ttanalyzing theraisestatement, these variables are printed again too.[2021-06-1918:15:01]ERROR:Uncaughtexception:Traceback(mostrecentcalllast):File"D:\my_project\module.py",line9,in<module>raiseRuntimeError(f'Too much laughing with a={a}and b={b}')|->a='haha'|->b='hihi'RuntimeError:Toomuchlaughingwitha=hahaandb=hihiThe duplication is unnecessary, so from version1.6.1,raiseexception line will not be analyzed as default.
This resulted in a much cleaner log:[2021-06-1918:15:30]ERROR:Uncaughtexception:Traceback(mostrecentcalllast):File"D:\my_project\module.py",line9,in<module>raiseRuntimeError(f'Too much laughing with a={a}and b={b}')RuntimeError:Toomuchlaughingwitha=hahaandb=hihiIf theraiseexception line in turn, raise another exception, and you want to analyze it,
you could turn it back on as below:setup_logging(analyze_raise_statement=True)11. StreamHandler with buffer:This handler is mainly to solve the problem of outputting a tremendous amount of logs to GUI applications in real-time.GUI applications use threading to display content while listening for user input (button click, key pressing, mouse scroll).
But since cpython actually running only one thread at a time due to the GIL (global interpreter lock),
processing to display a tremendous amount of logs to the GUI widget will lock a thread for quite a long time.
During this time, no user input will be handled and the app seems unresponsive.The answer to this problem is to cache the log and output them at once after some interval or a number of cached lines reached a threshold.
This significantly reduces the overhead on the widget side and makes the app responsive.
The solution is implemented in the newStreamHandlerWithBufferwhich is insidelogger_tt.handlers.
There are 2 steps to use this handler.config the handler in thelog_configfile.handlers:buffer_stream_handler:class:logger_tt.handlers.StreamHandlerWithBufferlevel:DEBUGformatter:briefstream:ext://sys.stdoutbuffer_time:0.5buffer_lines:0debug:Falseroot:level:DEBUGhandlers:[console,error_file_handler,buffer_stream_handler]Its parameters look exactly like theconsolehandler except the last 3 ones.buffer_time: the cache time interval insecondsbefore it flush the log outbuffer_line: thenumber of lineto cache before it flush the log outdebug: log the time that it flush the log out or notForbuffer_line, to avoid the last lines of log not printed out as the number of line is below threshold,
you should setbuffer_timeto a certain number too.Then, you need to add this handler to therootlogger'shandlerslist.replace thestream:
you need to replace thestdoutwith your GUI widget stream.
You cannot do this in the config file since your widget is only appear during runtime.
You have to do this in the code.config=setup_logging(config_path='log_config.json')config.replace_handler_stream(index=HANDLER_INDEX,stream=WIDGET_STREAM)Setup your logging as usual with your config file,
then callreplace_handler_streamto replace the with your widget's stream.index: the index of the handler, in the root logger'handlerslist, that you want to replace the streamstream: the stream that you want to put into this handler.Below is the example withPySimpleGUI v4.57.0with the above config.importtimefromthreadingimportThreadimportPySimpleGUIassgfromlogger_ttimportsetup_logging,loggerstop_flag=False# GUI configsg.theme('DarkAmber')layout=[[[sg.Button("Show Log",key="-show_log-"),sg.Button("stop",key="-stop-",button_color='red')],sg.Multiline(size=(80,30),font='Courier 10',key='log',autoscroll=True)]]window=sg.Window('Logging tool',layout,finalize=True)# logging configconfig=setup_logging(config_path='log_config.json')config.replace_handler_stream(index=2,stream=window["log"])defmy_func():"""long running task in the background"""whilenotstop_flag:foriinrange(1000):logger.warning(f"{i}Function is empty")time.sleep(1)whileTrue:event,values=window.read()ifevent==sg.WINDOW_CLOSED:breakifevent=="-show_log-":logger.info(__name__)Thread(target=my_func).start()ifevent=="-stop-":stop_flag=Truelogger.info(f"stop button pressed")window.close()You could setindex=0to use the normalStreamHandlerand
see the difference while clicking thestopbutton for yourself.12. Telegram Handler:This handler enables you to send log messages directly to telegram users or groups.
For telegram groups, you can specify a certaintopicto send to, too.
This handler uses web API, so there is no third-party dependency.To set up this handler, you need the following information:Your bottoken: if you don't have one, create a new bot withhttps://t.me/botfather.Destinationunique_id: achat_id(user id or group id) and optionallymessage_thread_idof a topic in that group. You can chat with or add @RawDataBot to your group to get these ids.Log level to be sent: above a level or only a specific level.From here you have 2 options, put thetokenandunique_iddirectly in the config file or in the environment.
If you specify both ways, if the data exists in the environment variables,
it will replace the one specified in the config file. The default environment variable names are:token:TELEGRAM_BOT_LOG_TOKENunique_ids:TELEGRAM_BOT_LOG_DESTThese above names are specified inenv_token_keyandenv_unique_ids_keyin the default config file.
You may edit them to suit your environment.For the values,unique_idsaccepts the format[name:]chat_id[@message_thread_id],
multiple values must be separated by a semicolon (space after a semicolon will be ignored).
Which means the following examples are possible:a single chat_id:123456789(user id) or-987654321(group id)a specified topic within a group:-987654321@2multiple values:123456789; 102345678; -987654321@2; -987654322@4multiple values with names:Alex:123456789; Bob:102345678; support:-987654321@2; admin:-987654322@4After having all needed information, the next step is configuring the handler and adding it into therootlogger.handlers:telegram_handler:class:logger_tt.handlers.TelegramHandlerlevel:NOTICEformatter:briefdebug:Falsetoken:""unique_ids:""env_token_key:"TELEGRAM_BOT_LOG_TOKEN",env_unique_ids_key:"TELEGRAM_BOT_LOG_DEST"root:level:DEBUGhandlers:[console,error_file_handler,telegram_handler]Sending too many messages within a second will lead toHTTP 429error.
Even though it will resend later, you can avoid the error from the beginning by
adding the paramgrouping_interval: 1to the configuration above.
By doing that, all log messages whose timestamp are in the same second will be sent as one telegram message.From here, it should already work.
If you need certain messages to go to a certain people/group, besides adding a new handler,
you can add a filter that adds thedest_nameattribute to the logrecord.deftelegram_filter(record):if'over threshold'inrecord.msg:record.dest_name='support'elif'unauthorized'inrecord.msg:record.dest_name='admin'elif'Alex'inrecord.msg:record.dest_name='Alex'else:pass# return False will discard this recordreturnTruelog_config=setup_logging()log_config.set_context_injector(telegram_filter)logger.warning(f'Memory usage is over threshold')logger.error(f'This deleting action is unauthorized')logger.notice(f"Alex's files are ready to be collected")dest_nameshould be only one name that has been registered inunique_ids.
If the name is not registered, the log record is simply ignored totally.
If you want to send to multiple people, create a telegram group for them and registered the group name.Sample config:Below are default config files that used bylogger-tt. You can copy and modify them as needed.1. Yaml format:log_config.yaml:# This is an example of config file# In case of no config provided, log_config.json file will be loaded# If you need a yaml file, install pyyaml or ruamel.yaml package and copy this fileversion:1disable_existing_loggers:Falseformatters:simple:format:"[%(asctime)s][%(name)s:%(lineno)d%(levelname)s]%(message)s"datefmt:"%Y-%m-%d%H:%M:%S"brief:format:"[%(asctime)s]%(levelname)s:%(message)s"datefmt:"%Y-%m-%d%H:%M:%S"handlers:console:class:logging.StreamHandlerlevel:INFOformatter:briefstream:ext://sys.stdouterror_file_handler:class:logging.handlers.TimedRotatingFileHandlerlevel:DEBUGformatter:simplefilename:logs/log.txtbackupCount:15encoding:utf8when:midnightbuffer_stream_handler:class:logger_tt.handlers.StreamHandlerWithBufferlevel:DEBUGformatter:briefstream:ext://sys.stdoutbuffer_time:0.5buffer_lines:0debug:Falsetelegram_handler:class:logger_tt.handlers.TelegramHandlerlevel:NOTICEformatter:briefdebug:Falsetoken:"yourbottokenhereorsetthebelowenvkeytofetchfromenvironforbettersecurity"unique_ids:"semicolonseparatedof[name:]chat_id[@message_thread_id]"env_token_key:"TELEGRAM_BOT_LOG_TOKEN"env_unique_ids_key:"TELEGRAM_BOT_LOG_DEST"loggers:urllib3:level:WARNINGhandlers:[console,error_file_handler]propagate:noroot:level:DEBUGhandlers:[console,error_file_handler]logger_tt:suppress:["exchangelib"]suppress_level_below:"WARNING"capture_print:Falsestrict:Falseguess_level:Falsefull_context:0use_multiprocessing:Falselimit_line_length:1000analyze_raise_statement:Falsedefault_logger_formats:normal:["%(name)s","%(filename)s"]thread:["%(message)s","%(threadName)s%(message)s"]multiprocess:["%(message)s","%(processName)s%(message)s"]both:["%(message)s","%(processName)s%(threadName)s%(message)s"]2. Json format:log_config.json:{"version":1,"disable_existing_loggers":false,"formatters":{"simple":{"format":"[%(asctime)s] [%(name)s:%(lineno)d %(levelname)s] %(message)s","datefmt":"%Y-%m-%d %H:%M:%S"},"brief":{"format":"[%(asctime)s] %(levelname)s: %(message)s","datefmt":"%Y-%m-%d %H:%M:%S"}},"handlers":{"console":{"class":"logging.StreamHandler","level":"INFO","formatter":"brief","stream":"ext://sys.stdout"},"error_file_handler":{"class":"logging.handlers.TimedRotatingFileHandler","level":"DEBUG","formatter":"simple","filename":"logs/log.txt","backupCount":15,"encoding":"utf8","when":"midnight"},"buffer_stream_handler":{"class":"logger_tt.handlers.StreamHandlerWithBuffer","level":"INFO","formatter":"simple","stream":"ext://sys.stdout","buffer_time":0.5,"buffer_lines":0,"debug":false},"telegram_handler":{"class":"logger_tt.handlers.TelegramHandler","level":"NOTICE","formatter":"brief","debug":false,"token":"your bot token here or set the below env key to fetch from environ for better security","unique_ids":"semicolon separated of [name:]chat_id[@message_thread_id]","env_token_key":"TELEGRAM_BOT_LOG_TOKEN","env_unique_ids_key":"TELEGRAM_BOT_LOG_DEST"}},"loggers":{"urllib3":{"level":"ERROR","handlers":["console","error_file_handler"],"propagate":false}},"root":{"level":"DEBUG","handlers":["console","error_file_handler"]},"logger_tt":{"suppress":["exchangelib"],"suppress_level_below":"WARNING","capture_print":false,"strict":false,"guess_level":false,"full_context":0,"use_multiprocessing":false,"limit_line_length":1000,"analyze_raise_statement":false,"default_logger_formats":{"normal":["%(name)s","%(filename)s"],"thread":["%(message)s","%(threadName)s %(message)s"],"multiprocess":["%(message)s","%(processName)s %(message)s"],"both":["%(message)s","%(processName)s %(threadName)s %(message)s"]}}}Changelog1.7.3:Usability:Not to import custom handlers (StreamHandlerWithBuffer, TelegramHandler, etc) if they are not used in any logger.TelegramHandler:Better network error handling.Addedgrouping_intervalto group many log messages that are within the samexseconds interval into one before sending.
This reduces the number of time it sends messages to Telegram server and helps to avoid 429 error.1.7.2:Fixed: TelegramHandler crashed instead of ignoring the unregisteredunique_id1.7.1Fixed: exception raised while handling another exception is now shown correctly (thank ZeroRin).New functionality:AddedTelegramHandler. Now you can send logs directly to telegram users/groups.Support Python 3.11Fine-grained error locationsin tracebacks.Usability:Added level NOTICE = INFO + 5. You can dologger.notice("your message").Add set/removecontext injectormethods.
You can add additional information to the log record before it is actually handled.
Refer to theTelegramHandlerfor a usage sample.You can now set different log paths for different handlers by passing a dict tosetup_logging(thank ZeroRin).1.7.0Fixed:multiprocessing: log file rollover fails as child process keep opening the file.multiprocessing: if the log path is set by a variable with time,
child process creates a new redundant log path.New functionality: AddedStreamHandlerWithBuffer. Buffer the log output by time or by line number.
GUI app could use this handler to keep the app responsive while having a tremendous log output.Usability: In multiprocessing logging,
users can set the log server address themselves throughsetup_loggingor log config file.1.6.1Addedlimit_line_lengthparameter: log only maximumncharacters for each traceback line.
This prevents dumping the whole huge content of the variable into the log.n=1000by default.Addedanalyze_raise_statementparameter:logger-ttno longer analyze theraisestatement by default.
This avoids logging value of variables on theraisestatement two time, especially when the content
of these variables are huge.1.6.0Fixed: If an exception happened on the multiline statement,
py3.6 and py3.7 return the last line while py3.9 returns the first line.
Plus thetokenizemodule's behavior has changed, so it made grabbing the
all the lines of the statement inconsistent and sometime buggy.
Nowlogger-ttonly grabs maximum of 10 lines and grabs more accurately,
more consistent between different python version.Added support forruamel.yamlpackage. If you already have it installed,
you don't need to installpyyamlto useconfig.yamlfileNew feature: Uncaught exception happened in child thread of multi-threading
program will also be logged. For python 3.8+,threading.excepthookwill not run
as the exception is already caught bylogger-tt.1.5.2Improved the pre-made logger namedlogger_ttlogger_ttnow can detect the qualified__name__of the module that calls it.
Instead offilename, output log line will have the__name__as regular logger.For example:[2020-07-21 11:24:19] [my_module.py:5 DEBUG] Module is initialized
[2020-07-21 11:24:19] [sub_module.py:15 DEBUG] Entering sub moduleNow becomes[2020-07-21 11:24:19] [__main__:5 DEBUG] Module is initialized
[2020-07-21 11:24:19] [my_module.submodule:15 DEBUG] Entering sub moduleSuppressing loggers also works with log records output bylogger_ttby using the qualified__name__too.
For example, suppressingmy_module.submodulewill telllogger_ttnot to output the second line.This is much better than suppressinglogger_ttif you use this same logger in other modules too.You now can define fields of log record forlogger_ttin the log config file too.
Just looks fordefault_logger_formatssection.
It works by replacing the field in the formatters that are used by any handler of the root logger.Pre-existing loggers:Before this version, if you import submodules before importinglogger_ttand
there are loggers in submodules, these loggers do not inspect exception when you calllogger.exception().
That is because there class was different from the loggers created after importinglogger_tt.
Now all loggers have the same new class regardless the point of importinglogger_tt.setup_logging(): This function should only be called once.
Add a warning if it is called more than one time.1.5.1UsesocketHandleras default for multiprocessing.
Under linux, to usequeueHandler, user must passuse_multiprocessing="fork"tosetup_loggingExposelogging_disabledfunction to user:from logger_tt import logging_disabled.
Then this function can be used as a context withwithstatement.For convenient, user can import a pre-madeloggerfromlogger_ttto use right away in sub modules.
The built-ingetLoggerfunction can be imported fromlogger_tt, too.Added line number to a defaultsimplelog record formatter in the config file.Most parameters ofsetup_logging()function can be specified in the config file, too.
If the same parameter is specified in bothsetup_logging()function and in the config file,
the parameter passed insetup_logging()will be used.1.5.0Logging is off-loaded to another thread and uses Queue to communicate.
This allow critical thread can do there job why time-consuming logging can be done later or in parallel.Support for multiprocessing logging. For linux, a multiprocessing queue is used.
For Windows and macOS, a socket server is used instead.setup_loggingnow return aLogConfigobject.
You can set/change parameters of this object instead of passing arguments directly tosetup_logging.Onlyconfig_path,log_pathanduse_multiprocessingargument must be set withsetup_logging.Behaviors changed:full_contextis now anintthat indicate the depth level from the bottom,
where surrounding variables should be parsed.Turned off parsing full context forraiseexception since many exception names are enough to understand the problem.log_configfile: movesuppresssection intologger_ttsection.
Future settings will also be put into this section for the sake of managing.
If you need to hanging the logging framework, you just need to delete this section and move on.1.4.2To prevent exception during logging, the following actions have been applied:Catch exception while parsing for object's value (property of a class)Catch exception while evaluating__repr__and__str__of objectDisable logging while inspecting objects' value and representationDisable logging after an uncaught exception is logged.
Because the interpreter is shutting down, objects get deleted.
Logging put inside magic function__del__will raise error.1.4.1Fixprint_captureignoringprint()line in global scope due to lackingcode_contextframeIf__str__of an object has multiple lines, also indent the second line and so on accordingly.If there is an exception during getting object's representation,
return!!! Attribute errorinstead ofError in sys.excepthook1.4.0Add an extra fieldsuppressin config file.
Any logger's name appeared in this list will have its messages suppressed.1.3.2change extended ascii dash─to normal dash-so that it is displayed consistently in different encoding1.3.1change extended ascii vertical bar├to normal bar|so that it is displayed consistently in different encoding1.3.0Exception analyzing now fetch full multi-line python statement.
This means that variables lie at seconds and below of the same statement can also be seen
without the need offull_context=True.1.2.1Extend logging context tologger.exception()as well.
Now you can dotry-excepta block of code and still have a full context at error line.1.2.0Add logging context for uncaught exception. Now automatically log variables surrounding the error line, too.Add test cases for logging exception1.1.1Fixed typos and grammarAdd config file sample to READMEusing full namelog_config.jsoninstead oflog_conf.json, the same for yaml fileadd test cases forcapture print1.1.0Addcapture printfunctionality withguess levelfor the message.