Package: pymongowatch
pymongowatchis an extension for the MongoDB’s Python driverpymongowhich enable
watching and auditing the underlying operations.MongoDB Community Server doesn’t supportauditingand thedatabase
profileris not very flexible. Basically your only option for filtering is to
collect operations that take longer than a certain value. You can’t
easily separate the insert log from the query log or you cannot extract
the operations only for a certain collection or a certain set of
queries.Furthermore, operations at database level are not always the same as
operations in an application. A very simple operation like fetching
documents withfindmay be translated to severalgetMoreoperations at database level.Therefore, if you want to analyze the user behavior by studying the
database queries, a very simple auditing log for each user search
containing the query itself (possibly with deleting value fields to
protect the users’ privacy), the number of the documents read from the
database and its time is all you need.Doing this at database level is not always feasible and doing it
manually at application level may not be very desirable for the
developers. This is when an intermediate tool at library level comes
handy.pymongowatchcan detect and audit operations at application level
without any modifications to existing applications.Quick SetupFirst you have to installpymongowatch. It is available atPython
Package Index (PyPI)and can
be installed withpip. At avirtual environmentor system-wide:pipinstallpymongowatchOther installation
optionssuch as Debian packages are available, too.After basic installation,pymongo.watchersub-package will be
available and you have two options to go on:Option 1: Call the watcher at application startupThis approach is recommended if you have no restrictions on modifying
the pymongo-based application and you don’t want any hacks on thepymongoitself.frompymongoimportwatcherwatcher.patch_pymongo()watcher.add_logging_handlers()# Adds a logging.StreamHandler by defaultOf course, you can pass your own customized logging handlers toadd_logging_handlersand add filters and formatters as you wish. See
the following sections for more information.Option 2: No application modificationIf you don’t own the application or you don’t want to even add a single
line of code to your existing application,pymongo_maskwhich is
included in thepymongowatchpackage comes handy.pymongo_maskis a directory with a modulepymongo.pywhich
emulates thepymongopackage by using the real installedpymongo; So if you addpymongo_maskdirectory in your Python
applicationsearch path for
modules–in
somewhere with more priority than where the realpymongois
installed–the mask will be imported in your application instead of the
realpymongo. Nevertheless the mask will work as a proxy to the real
installedpymongoand your application should work as usual.The importation of the pymongo mask will also triggerwatcher.patch_pymongo()automatically, so the watchers will be
enabled without any need to change your application source code or evenpymongolibrary itself.Furthermore, the pymongo mask will apply the configuration atetc/pymongowatch.yamlwithlogging.config.dictConfigto the loggers of the watchers. A sampleetc/pymongowatch.yamlis included in thepymongowatchpackage but you can modify it as
needed. Theyamlconfiguration
follows thelogging dictionary
schema.Finally,pymongowatchpackage includes apymongowatch-install-maskscript which can help you enable the
pymongo mask with ease:pymongowatch-install-mask# follow the interactive instructionsUsagepymongowatchimplements classes such asWatchCursor(a subclass
of pymongo’scursor) which if used instead of pymongo’scursorcan emit logs as specified. To do so, you need some workaround to make
pymongo’s methods to usepymongowatchclasses instead of its
defaults. Thepatch_pymongofunction is the method that will call
all the patch methods inpymongowatchclasses to replace the default
pymongo underlying classes:frompymongoimportwatcherwatcher.patch_pymongo()All thepymongowatchwatcher classes will use the logger retrieved
withpymongo.watcher.logger.get_log_emitter()to emit their logs.
Each module will emit the logs with theirname. So for
example, all the cursor logs will be emitted under thepymongo.watcher.cursor. So you can apply specific logging
configuration for each module or as all the logs has the prefixpymongo.watcheras their logger name you can apply global
configuration with that name.importlogging# The following code is not the recommended method for adding handlers# to the pymongowatch loggers and is for illustrative purposes onlyglobal_logger=logging.getLogger("pymongo.watcher")# add a stream log handler globally to watch all the logs on the# consoleglobal_logger.addHandler(logging.StreamHandler())cursor_logger=logging.getLogger("pymongo.watcher.cursor")# add a file handler specifically for cursor logs to store them in# file, too.cursor_logger.addHandler(logging.FileHandler("/tmp/watch.log"))Of course you can applyfiltersandformattersor add morehandlersas you wish but there are some details that you should take care of. The
easiest way to achieve this is by leveraging theadd_logging_handlersmethod:frompymongoimportwatcher# Add two handlers to get all the watcher logs both in file and# consoleconsole_handler=logging.StreamHandler()global_handler=logging.FileHanlder("/tmp/watcher-all.log")watcher.add_logging_handlers(console_handler,global_handler)# Add a more customized handler for cursor logscursor_simple_handler=logging.FileHanlder("/tmp/watcher-cursor-simple.log")watcher.add_logging_handlers(cursor_simple_handler,logger_name="pymongo.watcher.cursor",formatter="{name}-{watch}")# Add a more customized handler for cursor logscursor_customized_handler=logging.FileHanlder("/tmp/watcher-cursor-customized.log")watcher.add_logging_handlers(cursor_customized_handler,logger_name="pymongo.watcher.cursor",formatter="{asctime}{name}.{watch.Collection}-{watch.Query}fetched ""{watch.RetrievedCount}in{watch.RetrieveTime}seconds")# Add another handler to log the full information for cursors in csvcursor_csv_handler=logging.FileHanlder("/tmp/watcher-cursor-csv.log")watcher.add_logging_handlers(cursor_csv_handler,logger_name="pymongo.watcher.cursor",formatter="{asctime},{name},{watch.csv}")Note that usingadd_logging_handlershas not only the advantage of
simplicity for adding formatters, but also take care of automatically
adding an extralogging.handlers.QueueHandlerandlogging.handlers.QueueListenerfor each handler to overcome some log mutation issues we discuss later.
It will also add a couple of customizedfiltersto theQueueHandlerthat are necessary to fully take advantage ofpymongowatch.You can set the log format by usingformatterseither by passing a string as theformatteroption to theadd_logging_handlersor by creating aformatterobject and using
the handler’ssetFormattermethod directly. In either case the recommended way is to use{style.
Specially if you want to access inner values with dot notation e.g.watch.Queryorwatch.Collectionother styles such as%and$(e.g.%(watch.Query)sor${watch.Query}) willNOTwork.Another useful feature of Pythonloggingmodule is itsfilters.
You can usefilterobjects not only to filter unwanted logs but also
to modify the ones that you want.add_logging_handlerswill
automatically add some basic filters (such asAddPymongoResultswhich will extract information aboutpymongooperations results and
add them to the related log), but you can also add your own filters.pymongo.watcher.filtersmodule implements a couple of useful
configurable filters such asExpressionFilter,ExecuteFilterandRateFilter. Theetc/pymongowatch.yamlhas some examples on how to use them.You can also develop your own logging filters by sub-classinglogging.Filteras usual and access therecord.watchinfiltermethod for investigating or modifying the watcher attributes such asDB,Collection,Query, etc.importloggingclassSlowQueriesOnNewsCollectionFilter(logging.Filter):deffilter(self,record):watch=getattr(record,"watch",{})return(watch.get("Collection")=="news"andwatch.get("RetrieveTime",0)>10)Or you can add filters to modify the logs:importloggingdefremove_private_data(data):ifisinstance(data,dict):return{k:remove_private_data(v)fork,vindata.items()}elifisinstance(data,list):return[remove_private_data(i)foriindata]returnNoneclassUserPrivacyFilter(logging.Filter):deffilter(self,record):watch=getattr(record,"watch",{})watch.update(remove_private_data(watch))returnTrueDon’t forget to add the defined filters to you handlers:my_handler.addFilter(SlowQueriesOnNewsCollectionFilter)my_handler.addFilter(UserPrivacyFilter)Lastly, you can use Python’s greatlogging.conigmodule and specially the new flexiblelogging.config.dictConfigmethod to apply all thehandlers,formattersandfiltersin
a single configuration file. Custompymongowatchconfiguration can
also be applied in the same dictionary.pymongowatchhas even the required filters implemented inpymongo.watcher.filtersmodule. To see the examples for thedictConfigconfiguration with watcher filters refer to theetc/pymongowatch.yamlfile which will be installed viapymongowatch(if you are using a virtual environment, it would be
inside the venv directory).dictConfig configurationThere is apymongo.watcher.dictConfigmethod that accepts a
configuration dictionary with almostthe same
specificationaslogging.config.dictConfig. But it will apply custom configuration
required bypymongowatchitself.Thepymongo.watcher.dictConfigwill not apply the intended logging
configurations such ashandlersandformatters. Instead, it
reads the special keywatchersfrom the dictionary and apply the
related configuration forpymongowatch. Although you can use the a
single dictionary with both logging configuration andwatcherskey
and apply bothpymongo.watcher.dictConfigandlogging.config.dictConfig.For more information about thepymongowatchconfigurations please
refer to thewatcherskey inside the exampleetc/pymongowatch.yamlfile that will be installed alongside the
package.Mutable vs Immutable LogsMutable Logs? Is that a thing?Usually the good thing about logs is that they are immutable. So if you
see a log you can trust it. This is always true when some atomic
operation happens and you have no concerns about the start and end time
of the operation (and you don’t have access to a time machine to travel
to the past and change what happened).But what if you start an operation which we have no idea when will it
end? Suppose we have queried a very large database for a very slow query
that may take some time to get back the full results. Also, we may use a
cursor in our application to fetch data and the application has some
delays itself and we don’t want the slowness of the application to
affect the database auditing.These are the sort of challenges thatpymongo.watcher.WatchQueuetries to fix.pymongowatchusespymongo.watcher.logger.WatchMessageinstead of
strings as log messages as described inusing arbitrary objects as
messagesin Python’s logging HOWTO.WatchMessageis a sub-class of Python’s
dictionaries which are mutable objects.WatchMessageinstances are the{watch}templates in the format
strings that we saw earlier. They are adictso you can access log
attributes with[]access e.g.watch["Query"]. For more
convenience while using log formattersWatchMessageprovides
attribute access with dot notation e.g.watch.Queryand one of the
reasons why{style formatting (which let you use dot notation
access) is recommended for logging formatters.AWatchMessageis a mutable object and the watcher classes need to
modify the logs from time to time. For example if you fetch more items
from a cursor, they can update the attributes such asRetrieveTimeorRetrievedCount. So we have flexibility and it ispymongowatchusers decision when to emit the final immutable log with the logger
handlers.In the old versions ofpymongowatch, the watcher classes modified
theWatchMessageeven after the emitting the log, leaving the entire
burden for consistency on theWatchQueue. This approach soon became
problematic in multiprocessing environments where theWatchMessagein the emitter process differs from the one in the queue process.In the newer versions,WatchMessagehas a unique identifierWatchIDfor the ongoing operation. The watcher classes will not only
mutate the previousWatchMessagebut also re-emit the log again and
again on each update (with an incremental meta-data namedIteration). In this way, even if the log has already stored
somewhere as an immutable object (for example in a file as a log record
or in a separate process as an object that cannot see the modifications
to the originalWatchMessage) can still receive the updates.The pitfall is now we may have thousands of updates and consequently
thousands of logs for each operation. But this problem can easily be
handled byWatchQueueor an external tool such aspymongowatch-csv.WatchQueuealongside aQueueHandleris the right tool to make sure we handle logs at right time. AWatchQueueworks like a priority queue in which the earliest logs
has the higher priority but it can also aggregate the updates for each
operation based on theWatchID.WatchQueuewill update the old iterations with the newer ones and
releases the logs when they have thefinaliteration mark or when
they have expired. Thefinalmark is a special case ofIterationmeta-data in each log that indicates the database operation has finished
and the watcher class will not modify thisWatchMessage(with the
specifiedWatchID) anymore.Also it is important to note that the highest priority item (for
retrieval) in theWatchQueueis not always the earliest generated
log. First, the different operations may take different times, and
second, the watcher classes may assign different expiration times to
each operation.Finally you can set aforced_delay_secto add delay to all the logs
in the queue (for example if you are using logging for analytics and you
do care more about accuracy than delay for the logs). This is an
optional keyword arguments that bothWatchQueueconstructor andadd_logging_handlersmethod that we saw earlier accepts.Theadd_logging_handlershas awith_queueoptional argument
which if isTrue(the default), will usepymongo.watcher.setup_queue_handlerto setup aQueueHandleralongside a startedQueueListenerfor each handler you specify with aWatchQueueso you usually don’t
have to worry about log mutation if you useadd_logging_handlersto
add your handlers to watcher loggers.Apymongowatch-csvscript is also available after thepymongowatchinstallation that can be used to aggregate all the
iterations of a single operation into a single row when you use thecsvformat:pymongowatch-csvaggr/tmp/watch.csv>/tmp/aggregated-watch.csvMultiprocessingAnother useful application ofQueueHandleris its use inlogging to a single file from multiple
processes.
For this application you have to use a multiprocessingQueuealongside theQueueHandler. FortunatelyWatchQueuealso supports
multiprocessing. All you have to do is to passTrueas theenable_multiprocessingargument.This is useful for example when you have apymongobased web
application with several web server processes and you need a process
safe method to store the logs in a single file. Its worth noting that in
such environments using the default value forenable_multiprocessingi.e.Falsewill result in total failure in logging because the
defaultQueueis not multiprocessing and theQueueHandlerand
theQueueListenerwill use different queues in different processes.PassingTrueasenable_multiprocessinginWatchQueueconstructor will makes the constructor to build and return aproxy
objectfrom acustomized
managerwhich has its own dedicated process. You can also passenable_multiprocessingtosetup_queue_handlerandadd_logging_handlers:# Extra keyword arguments of `add_logging_handlers` will be passed to# the newly created WatchQueue for each handler.watcher.add_logging_handlers(enable_multiprocessing=True)Pymongo Versionspymongowatchis not a standalone MongoDB library and it relies on
the the MongoDB’s Python driverpymongo. But does the
pymongo’s version matter?pymongowatchhas been tested with the recent versions ofpymongoi.e.3.10and the newer4series but you can use it for other
versions at your own risk. If you have any problems you can open an
issue at theproject’s issue
tracker.One known difference betweenpymongoversions is that they handle
operation closing differently. For example,4series close the
cursors more intelligently and you can usually see thecursorlogs
very fast without any need to a explicit timeout whereas any3series usually an explicit timeout is required.Other Installation OptionsDebian PackagesIf you are aDebian-based GNU/Linux
distribution user you are in luck! There is a Debian package maintained
in theproject’s debian
branchthat
can make your installation even easier.You can find the binary packages atmrazavi’s pymongowatch
PPAand
to install it on Ubuntu:sudoadd-apt-repositoryppa:mrazavi/pymongowatchsudoaptupdatesudoaptinstallpython3-pymongowatchRoadmapThe projectissue
trackeris the main
location to maintain the details for the future development. But here
the cardinal points will be reviewed briefly.NOTEIf you see an ugly TODO list below with oversize items, it’s not even
clear which items are DONE and which ones are still TODO, maybe that
is because this document is written inorg-modebut you are seeing a bad render
e.g. inGitHubor a converted reStructuredText format e.g. becausethe lack of
org-mode support in
PyPI.That doesn’t make org-mode less lovable or inferior. Anywayorg-mode
is one of the most reasonable markup languages to use for
text.
Why not to use it and brag about it?DONE Support queries with findDONE Support for basic collection operations#3TODO Support for CommandCursor#4DONE PackagingDONE Implement pip packageDONE Implement debian packageTODO Automatic execution of testsDOING TestsDOING Unit TestsDONE Old ImplementationTODO Update with the latest codeDOING Deployment TestsDONE Basic ImplementationTODO Add more complex testsDONE Support for multiprocessingDONE Improve mutable logs#2: add a unique ID for each operationTODO DocumentationDONE Add a READMETODO Add Sphinx generated documentsTODO Create an online API referenceAboutThepymongowatchhas developed mainly byMohammad
Razavi.