Package: micrologai
microlog.aiA live demo of theMicrologPython UI, running on PyScript, can be found atmicrologai.github.io/microlog.Micrologis a continuous profiler and logger for the Python language that
explains application behavior using interactive graphs and AI.
It makes understanding complex applications easy, reducing support costs
and shortening production problems, increasing application quality, and minimizing outages.Microloghas extremely low runtime overhead (~1%) and exceptionally fast rendering (~20ms).
It saves logs and performance profiles on the local file system. The logs are
compressed exceptionally well, resulting in a remarkably low 0.5MB per hour of recording.This project is written in 100% Python. The recorder is a Python module that uses a separate thread to sample
performance and record logs. The UI is written in Python as well, rendered in the browser by Python code using
PyScript. As a result, the identical Python classes encodeanddecode the recordings, avoiding the need
for cumbersome cross-language data modeling.Micrologis open source, with an available commercial license. We welcome extensions toMicrologfrom
the Python performance community, such as
recording of special events, new optimizations related to PyScript,
or centralization of recordings into central storage systems, usingrsync,scp, orGoogle Drive.Installing microlog.aiTo installMicrologfrom pypi run:pip install micrologaiTo installMicrologglobally using asitecustomize.py, run:git3 clone https://github.com/micrologai/micrologcd micrologai/micrologpython3 setup.py installHow to use microlog.aiIf you used the setup command shown above,Micrologis enabled as a continuous profiler for all Python processes running on that Python VM.
To use microlog manually, use:importmicrologwithmicrolog.enabled():# run any Python codeThe part ofMicrologthat records the execution can be found inmicrolog/tracer.py. It starts a new thread and samples the other threads at regular intervals, seesample. The tracer also sets up wrappers for logging and print statements. The logs are compressed and saved whenMicrologis stopped, bymicrolog/log.py.To give you an idea of the features ofMicrolog, you could run all the examples. This does assume you set up microlog globally. In that case, run:sh examples/runall.shThis runs for a minute and eventually produces 13 logs. You will see lines appear looking like this:📈 Microlog ··· 26.3s ··· 4.6KB ··· examples-memory ··· http://127.0.0.1:4000/log/examples-memory/2023_07_12_10_24_53 🚀This shows how long the app ran, the size of the (compressed) log, its name, and a URL to view the result.
The report URL is rendered by theMicrologserver implemented inmicrolog/server.py.  If it is not yet running,
you can start it as follows:python3 microlog/server.pyThe Microlog.ai UIA live demo of theMicrologUI can be found atmicrologai.github.io/microlog. The UI is
written almost entirely in Python, seedashboard. TheMicrologUI runs in the browser using PyScript.To describe the UI features ofMicrolog, we will look at the output of theexamples\memory.pyexample (the live preview is atGitHub Pages):The main elements of the UI are:Log list, at the left, showing currently available logs on the local machine.Timeline, the starting point for analysis of your application:Status, providing insights into CPU, memory, and I/O.Timescale, with ticks for second elapsed since start.Flamegraph, showing the result of the continuous profiler.Design, showing a graphical rendering of the structure of the application.Log, listing all print and logging output and statistics in a chronological order.Explanation, giving a human language description of your code, using OpenAIs' ChatGPT.Tips, linking to best practices and tips for the modules used in your code.Timeline Navigation and ZoomUsing the mouse, the dashboard can be panned and zoomed. More details will be shown when zoomed in deeper:In the above example, we panned the flame graph by grabbing it with the mouse and zoomed in using the scroll wheel on the mouse.In addition, we clicked on a method call in the flame graph, which is now highlighted in red. A moveable popup dialog shows details about the method, such as the average CPU during the call. A CPU percentage below 100% means the process is involved
in reading or writing files on the local disk, loading or sending data over sockets, loading new modules (requiring disk I/O), async or thread synchronization, or other system-level event handling usingselector event handlers.A low CPU typically indicates a bottleneck and warrants in-depth investigation.Timeline Anomaly DetectionWhen a method is selected in the flame graph, the popup shows information about similar calls detected in the same run, showing when they ran and how long they ran.Micrologalso uses anomaly detection to highlight methods you may want to investigate in more detail. In the screenshot below, the average call duration is 1 second, and four calls were more than 50% over the average.Automatic anomaly detection, call stack analysis, and process health indicators offered byMicrologallow you to debug performance/quality issues quickly.Timeline Detecting expensive I/O or Starved ProcessesThe top bar shows general statistics for the process, such as CPU and number of modules loaded over time.
Note that a low CPU in the top bar tends to indicate I/O took place at that moment.Timeline Integrating Profiling with LoggingLog entries are shown as visual markers in the top bar. BecauseMicrologshows log entries on the timeline, analyzing problems becomes much easier than with normal logs. No more scrolling page after page to find a stack trace. WithMicrolog, they appear as easy-to-see stop signs:Formatting Logs with MarkdownLog entries can be formatted using Markdown to make it easier to show important information to the reader.LoggingMicrologdetects calls toprintandlogging. Those calls are automatically intercepted
added to theMicrologevent log.Manual log entries can be inserted into Microlog usinginfo,warn,debug, anderror:print("Add a log entry to microlog with an info marker...")print("... or as an error marker.",stream=sys.stderr)importlogginglogger=logging.Logger("Microlog Demo")logger.info("Add a log entry to microlog with an info marker...")logger.debug("... or a bug marker...")logger.warning("... or a warning marker...")logger.error("... or an error marker.")microlog.info("Add something to the log explicitly...")microlog.warning("... as a warning...")microlog.debug("... as a debug message...")microlog.error("... as an error.")DesignThe Design tab analyzes the runtime call graph and draws a structural diagram of the underlying design of your application.
Here is an example forexamples\go.py:LogThe log tab contains a chronological listing of all print and logging output, statistics, and analysis performed byMicrologin a more traditional linear log style:The links shown in lightblue directly take you to the source file where the print took place. This makes it extremely easy to figure out what code prints what exactly. For thefiles.pyexample, the log shows that this program leaked one file descriptor. That same report is also shown in the timeline when you click at the warning icon at the end of the run.
The popup also shows a link to the source location where the file was opened:Log - Memory LeaksIn addition to checking for leaked file descriptors,Micrologaims to detect memory leaks. Those objects that were
allocated but cannot be garbage-collected as they are either reachable from a module or are involed in a reference cycle that
cannot be broken. The following report shows the top-10 offenders for thedataframes.pyexample, that uses Pandas dataframes.ExplanationThe explanation tab shows a human-language explanation provided by OpenAI'sChatGPTto explain the design and implementation behind the application being monitored byMicrolog. This analysis is not performed on the source code, but on a condensed call graph generated from the performance log that was recorded byMicrolog. Here is an example of the whatChatGPTthinks of
ourexamples\go.pyexecution:TipsTo discover best practices, performance tips, or tutorials for modules being used by the application,Micrologoffers quick
links to general information sources, such as search engines and Q&A sites:Developer NotesRun all unit testspython3 -m unittest discover testsUpload new version to PyPiFirst build the package into a source distribution and a Python wheel:python3 -m pip install --user --upgrade setuptools wheel twine buildpython3 -m buildThen verify whether the build works for pypi:twine check dist/*Then upload to the pypi test environment:twine upload --repository pypitest dist/*Finally, if the pypi test upload appears to work fine, run:twine upload dist/*LicenseMicrologis released under version 1 of theServer Side Public License (SSPL).