Package: humioapi
Humio API (unofficial lib)💡 This project requiresPython>=3.8💡 This is not the official Humio library. It can be foundhere: humiolib.This is an unofficial library for interacting withHumio's API. This library mostly exists now because the official library was too basic back in 2019 when I first needed this. Currently this library is just a wrapper aroundhumiolibto implement some convenient and opinionated helpers.InstallationpipinstallhumioapiMain features/extensionsCLI companion toolhcavailable athumiocli.Monkeypatched QueryJobs with a different approach.Thepollmethod is now a generator yielding the current result until the query completes, with optional progress information and warnings.Thepoll_until_donemethod now simply returns the final result of thepollmethod in an efficient manner, which solves the problem the original poll method has with getting stuck forever in some cases.Relative time modifiers similar to Splunk (-7d@dto start at midnight 7 days ago). Can also be chained (-1d@h-30m).Source.List repository details (NOTE: normal Humio users cannot see repos without read permission).Easy env-variable based configuration.Create and update parsers.UsageFor convenience your Humio URL and tokens should be set in the environment variablesHUMIO_BASE_URLandHUMIO_TOKEN.
These can be set in~/.config/humio/.envand loaded throughhumioapi.humio_loadenv(), which loads allHUMIO_-prefixed
variables found in the env-file.Query available repositoriesCreate an instance of HumioAPI to get startedimporthumioapiapi=humioapi.HumioAPI(**humioapi.humio_loadenv())repositories=api.repositories()Iterate over syncronous streaming searches sequentiallyimporthumioapiapi=humioapi.HumioAPI(**humioapi.humio_loadenv())stream=api.streaming_search(query="",repo='sandbox',start="-1week@day",stop="now")foreventinstream:print(event)Create a pollable QueryJob with results, metadata and warnings (raised by default)importhumioapiapi=humioapi.HumioAPI(**humioapi.humio_loadenv())qj=api.create_queryjob(query="",repo="sandbox",start="-7d@d")# Poll the QueryJob and get its final resultsresult=qj.poll_until_done(warn=False)ifresult.warnings:print("Oh no, a problem has occured!",result.warnings)print(result.metadata)# Or manually iterate the current results until the QueryJob has completedforcurrent_resultinqj.poll(warn=False):passifcurrent_result.warnings:print("Oh no, a problem has occured!",current_result.warnings)print(current_result.metadata)Jupyter Notebookpewnew--python=python38humioapi# run the following commands inside the virtualenvpipinstallgit+https://github.com/gwtwod/humioapi.gitpipinstallipykernelseabornmatplotlibpython-mipykernelinstall--user--name'python38-humioapi'--display-name'Python 3.8 (venv humioapi)'Start the notebook by runningjupyter-notebookand choose the newly created kernel when creating a new notebook.Run this code to get started:importhumioapiapi=humioapi.HumioAPI(**humioapi.humio_loadenv())results=api.streaming_search(query="",repo="sandbox",start="@d",stop="now")foriinresults:print(i)To get a list of all readable repositories with names starting with 'frontend':repos=sorted([kfork,vinapi.repositories().items()ifv['read_permission']andk.startswith('frontend')])Making a timechart (lineplot):%matplotlibinlineimportmatplotlib.pyplotaspltimportseabornassnsimportpandasaspdsns.set(color_codes=True)sns.set_style('darkgrid')results=api.streaming_search(query=" | timechart()",repos=["sandbox"],start=start,stop=stop)df=pd.DataFrame(results)df['_count']=df['_count'].astype(float)df['_bucket']=pd.to_datetime(df['_bucket'],unit='ms',origin='unix',utc=True)df.set_index('_bucket',inplace=True)df.index=df.index.tz_convert('Europe/Oslo')df=df.pivot(columns='metric',values='_count')sns.lineplot(data=df)LoggingThis library uses the excellent structlog library. If you're want pretty formatted logs but are too lazy to configure it yourself, you can use the included helper to configure it.This helper also installs an exception hook to log all unhandled exceptions through structlog.importlogginghumioapi.initialize_logging(level=logging.INFO,fmt="human")# or fmt="json"SSL and proxiesAll HTTP traffic is done throughhumiolibwhich currently usesrequestsinternally. You can probably use custom certificates with the env variableREQUESTS_CA_BUNDLE, or pass extra argument askwargsto the various API functions.