Package: offlog
offlog 0.1offlogis a non-blocking file-append service for Unix systems. It consists of a server
program and Python client library that interact via a Unix domain socket, allowing
applications to offload file appends to the server and mitigating the risk of
application latency caused by blocking file I/O. It has a focus on performance,
specifically on minimising overhead for the client.Both client and server are written in Python, but the protocol is not Python-specific so
other clients could be written as well. This package has no dependencies outside the
Python standard library.exampleExample:# example client code example.pyfromofflogimportLoggerlogger=Logger("my app",'/tmp/myapp.log')logger.info("Hello world, params are%s%d","foo",7)try:1/0exceptException:logger.exception("Got an error")Start the the server in one terminal, and then run the client in another. You'll see the
following:$python-mofflog[2023-07-0310:12:21.895offlogINFO]Thisisofflogserver[2023-07-0310:12:21.895offlogINFO]Listeningonsocket/tmp/offlog.sock[2023-07-0310:12:54.796offlogINFO]Client0connected[2023-07-0310:12:54.796offlogINFO]Client0accessconfirmedfor/tmp/myapp.log[2023-07-0310:12:54.796offlogINFO]Newclient0(total:1)for/tmp/myapp.log[2023-07-0310:12:54.799offlogINFO]Client0disconnected[2023-07-0310:12:54.799offlogINFO]Client0done(remaining:0)with/tmp/myapp.log[2023-07-0310:12:54.799offlogINFO]Closed/tmp/myapp.log$pythonexample.py[2023-07-0310:12:54.796myappINFO]Helloworld,paramsarefoo7[2023-07-0310:12:54.796myappERROR]Gotanerror
Traceback(mostrecentcalllast):File"/path/to/example.py",line6,in<module>1/0~^~
ZeroDivisionError:divisionbyzeroAnd the contents of/tmp/myapp.logwill be:$cat/tmp/myapp.log[2023-07-0310:12:54.796myappINFO]Helloworld,paramsarefoo7[2023-07-0310:12:54.796myappERROR]Gotanerror
Traceback(mostrecentcalllast):File"/path/to/example.py",line6,in<module>1/0~^~
ZeroDivisionError:divisionbyzeroDetails and notesTheLogger()class by default outputsDEBUGand higher to file,INFOto stdout,
andERRORand higher to stderr. These are configurable. Formatting and exception
options are similar to the Python standard library.The focus of this library is more on low-overhead file appends than theLogger()class, which is very basic. It is easily subclassable if you want to change some of its
behaviour, or you could feasibly plug theFileProxy()object that sends data to theofflogserver into some other logging framework. You may also use theFileProxyobject directly to proxy file appends, including of binary data, to a file.Despite being offloaded to another process, all file appends are still reliable, in the
sense that all data will eventually be written if the server and client shut down in
normal ways. This includes if the client encounters a Python exception, but does not
include if the client, e.g. segfaults or is terminated via SIGTERM and Python's default
SIGTERM handler (which exits immediately without doing any normal Python cleanup).The server does flush files after every write, so when things are running normally, logs
are written in real-time. The only reason they might be delayed is when the system is
under load and messages are getting backed up - in which case clients will not retry
sending queued data until the next time they have new data to send, or until program
shutdown, whichever comes first.There is no facility for global look up of loggers. Pass them around yourself. They are
not thread-safe, but if you instantiate one per thread pointing to the same file, they
will both talk to server which will interleave their appends. But their data won't be
guaranteed to be in the same order or even have the same boundaries as they were fired
off in your application.Instantiating a FileProxy is blocking - the server will verify it can open the file and
we wait for it to confirm. Writing data is not ever blocking, and for now data that
fails to send because the server is too busy will just be locally buffered indefinitely
(you can still run out of memory). Unsent data will be retried before sending new data
on subsequent calls towrite(). At shutdown, all unsent data will attempted to be
flushed to the server, this is also blocking.If the server is shut down or encounters an exception writing data on the client's
behalf, it will terminate the connection with the client. Upon the next attempt to write
the client will get a BrokenPipeError and read the exception message from the server,
raising it as an exception in the client code.Client documentationLoggerofflog.Logger(name=None,filepath=None,file_level=DEBUG,stdout_level=INFO,stderr_level=WARNING,local_file=False,offlog_socket_path='/tmp/offlog.sock',offlog_timeout=5000)Logging object to log to file,stdoutandstderr, with optional proxying of file
writes via a offlog server.Records with levelfile_leveland above will be written to the given file. Records
with levelstdout_leveland above, up to but not includingstderr_levelwill be
written to stdout, or with no upper limit ifstdout_levelis None. Records with levelstderr_leveland above will be written to stderr. Any of these can be set to None to
disaable writing to that stream.filepath=Nonewill also disable file logging.iflocal_fileisTrue, then an ordinary file will be opened for writing. Otherwiseofflog_socket_pathis used to connect to a running offlog server, which will open the
file for us, writes will be proxied through it. Any blocking operations communicating
with the server (such as the initial file open, and flushing data at shutdown) will be
subject to a communications timeout ofofflog_timeoutin milliseconds, default 5000.UTF-8 encoding is assumed throughout.offlog.Logger.close()Close the file. Possibly blocking. Idempotent.Logging methods work similarly to the Python standard library:offlog.Logger.log(level,msg,*args,exc_info=False):offlog.Logger.debug(msg,*args,exc_info=False):offlog.Logger.info(msg,*args,exc_info=False):offlog.Logger.warning(msg,*args,exc_info=False):offlog.Logger.error(msg,*args,exc_info=False):offlog.Logger.exception(msg,*args,exc_info=True)offlog.Logger.critical(msg,*args,exc_info=False)ProxyFileofflog.ProxyFile(filepath,sock_path='/tmp/offlog.sock',timeout=5000,)Object to proxy appending file writes via a running offlog server.offlog.ProxyFile.write(data):Send as much as we can without blocking. If sending would block, queue unsent data for
later. OnBrokenPipeError, check if the server sent us an error and raise it if so.
This method will always attempt to re-send previously-queued data before attempting to
send new data.offlog.ProxyFile.close():Close the socket. Attempt to send all queued unsent data to the server and
cleanly close the connection to it, raising exceptions if anything goes wrongServer documentation$python-mofflog-h
usage:python-mofflog[-h][-n][-sSOCKET_FILE][-lLOGFILE]offlogfile-appendservice

options:-h,--helpshowthishelpmessageandexit-n,--notifyNotifysystemdwhentheserverhasstartedupandislistening.-sSOCKET_FILE,--socket-fileSOCKET_FILEPathoftheUnixsockettheserverbindsto.Default:/tmp/offlog.sock-lLOGFILE,--logfileLOGFILEPathofthe(optional)logfilefortheofflogserveritself.You may wish to start the service as a systemd service. Here is an example unit file:# offlog.service[Unit]Description=offlog file-append serviceAfter=network.target[Service]Type=notifyExecStart=/path/to/your/.venv/bin/python -u -m offlog --notifyRestart=alwaysRestartSec=5sUser=<user_to_run_as>[Install]WantedBy=multi-user.targetNote: the server will run with permissions of the specific user, which means all file
operations will be performed as that user. The server's unix socket file will be created
with default access permissions of that user, and thus only users will permission to
access that socket will be able to communicate with the server. This means by default,
clients will not be able to access an offlog server running as root in order to write to
files they would not have otherwise had permission to write to.To avoid race conditions in systemd starting other services that depend on the offlog
server, ensure you use the--notifyflag to the offlog server, which will notify
systemd when the server's socket is bound and it is ready to accept clients. Mark other
units that require the offlog server by addingRequiresandAfterlines to the[Unit]section of their service files:[Unit]Description=Some other service that needs an offlog server...Requires=offlog.serviceAfter=offlog.serviceprotocolTODO - document the protocol