Package: opensearch-logger
OpenSearch Logger for PythonThis library provides a standard Pythonlogginghandler compatible withOpenSearchsuite.Thegoalsof this project areto provide asimpleand direct logging from Python to OpenSearch withoutfluentd,logstashor other middleware;keep it up to date with the growing difference between OpenSearch and Elasticsearch projects;keep the library easy to use, robust, and simple.The library has been open-sourced from an internal project where it has been successfully used in production
since the release of OpenSearch 1.0.Generated log records follow theElastic Common Schema (ECS)field naming convention.
For better performance, it is recommended to set up a proper mapping for your logging indices.
However, everything will work fine without it.
A ready to usecompatible JSON mappingcan be foundin the repository.Installationpipinstallopensearch-loggerUsageJust add the OpenSearch handler to your Python loggerimportloggingfromopensearch_loggerimportOpenSearchHandlerhandler=OpenSearchHandler(index_name="my-logs",hosts=["https://localhost:9200"],http_auth=("admin","admin"),http_compress=True,use_ssl=True,verify_certs=False,ssl_assert_hostname=False,ssl_show_warn=False,)logger=logging.getLogger(__name__)logger.setLevel(logging.INFO)logger.addHandler(handler)To send logs to OpenSearch simply use the same regular logging commands# Logging a simple text messagelogger.info("This message will be indexed in OpenSearch")# Now, as an example, let's measure how long some operation takesstart_time=time.perf_counter()heavy_database_operation()elapsed_time=time.perf_counter()-start_time# Let's send elapsed_time as an exatra parameter to the log record below.# This will make the `elapsed_time` field searchable and aggregatable.logger.info(f"Database operation took{elapsed_time:.3f}seconds",extra={"elapsed_time":elapsed_time},)ConfigurationTheOpenSearchHandlerconstructor takes several arguments described in the table below.
These parameters specify the name of the index, buffering settings, and some general behavior.
None of this parameters are mandatory.All other keyword arguments are passed directly "as is" to the underlyingOpenSearchpython client.
Full list of connection parameters can be found inopensearch-pydocs.
At least one connection parametermustbe provided, otherwise aTypeErrorwill be thrown.Logging parametersParameterDefaultDescriptionindex_name"python-logs"Base name of the OpenSearch index name that will be created. Or name of the data stream ifis_data_streamis set toTrue.index_rotateDAILYFrequency that controls what date is appended to index name during its creation.OpenSearchHandler.DAILY.index_date_format"%Y.%m.%d"Format of the date that gets appended to the base index name.index_name_sep"-"Separator string betweenindex_nameand the date, appended to the index name.is_data_streamFalseA flag to indicate that the documents will get indexed into a data stream. IfTrue, index rotation settings are ignored.buffer_size1000Number of log records which when reached on the internal buffer results in a flush to OpenSearch.flush_frequency1Float representing how often the buffer will be flushed (in seconds).extra_fields{}Nested dictionary with extra fields that will be added to every log record.raise_on_index_excFalseRaise exception if indexing the log record in OpenSearch fails.Connection parametersHere are a few examples of the connection parameters supported by the OpenSearch client.
For more details please check theopensearch-pydocumentation.ParameterExampleDescriptionhosts["https://localhost:9200"]The list of hosts to connect to. Multiple hosts are allowed.http_auth("admin", "admin")Username and password to authenticate against the OpenSearch servers.http_compressTrueEnables gzip compression for request bodies.use_sslTrueWhether communications should be SSL encrypted.verify_certsFalseWhether the SSL certificates are validated or not.ssl_assert_hostnameFalseVerify authenticity of host for encrypted connections.ssl_show_warnFalseEnable warning for SSL connections.ca_certs"/var/lib/root-ca.pem"CA bundle path for using intermediate CAs with your root CA.Configuration with logging.config or in DjangoSimilarly to other log handlers,opensearch-loggersupports configuration vialogging.configfacility.
Just specify theopensearch_logger.OpenSearchHandleras one of the handlers and provide it with parameters.Full guide on tweakinglogging.configcan be found in theofficial python documentation.importlogging.configLOGGING={"version":1,"disable_existing_loggers":False,"handlers":{"file":{"level":"DEBUG","class":"logging.handlers.RotatingFileHandler","filename":"./debug.log","maxBytes":102400,"backupCount":5,},"opensearch":{"level":"INFO","class":"opensearch_logger.OpenSearchHandler","index_name":"my-logs","extra_fields":{"App":"test","Environment":"dev"},"hosts":[{"host":"localhost","port":9200}],"http_auth":("admin","admin"),"http_compress":True,"use_ssl":True,"verify_certs":False,"ssl_assert_hostname":False,"ssl_show_warn":False,},},"loggers":{"root":{"handlers":["file","opensearch"],"level":"INFO","propogate":False,},"django":{"handlers":["file","opensearch"],"level":"DEBUG","propagate":True,},},}logging.config.dictConfig(LOGGING)Using AWS OpenSearchPackagerequests_aws4authis required to connect to the AWS OpenSearch service.importboto3fromopensearch_loggerimportOpenSearchHandlerfromrequests_aws4authimportAWS4AuthfromopensearchpyimportRequestsHttpConnectionhost=""# The OpenSearch domain endpoint starting with https://region="us-east-1"# AWS Regionservice="es"creds=boto3.Session().get_credentials()handler=OpenSearchHandler(index_name="my-logs",hosts=[host],http_auth=AWS4Auth(creds.access_key,creds.secret_key,region,service,session_token=creds.token),use_ssl=True,verify_certs=True,ssl_assert_hostname=True,ssl_show_warn=True,connection_class=RequestsHttpConnection,)Using Kerberos AuthenticationPackagerequests_kerberosis required to authenticate using Kerberos.fromopensearch_loggerimportOpenSearchHandlerfromrequests_kerberosimportHTTPKerberosAuth,DISABLEDhandler=OpenSearchHandler(index_name="my-logs",hosts=["https://localhost:9200"],http_auth=HTTPKerberosAuth(mutual_authentication=DISABLED),use_ssl=True,verify_certs=False,ssl_assert_hostname=False,ssl_show_warn=False,)Using Data StreamsIndexing documents into data streams is supported by just setting theis_data_streamparameter toTrue.In this configuration however, It is OpenSearch that solely manages the rollover of the data stream's write index.
The logging handler's rollover functionality and index rotation settings (e.g.index_rotate) are disabled.The following is an example configuration to send documents to a data stream.handler=OpenSearchHandler(index_name="logs-myapp-development",is_data_stream=True,hosts=["https://localhost:9200"],http_auth=("admin","admin"))DependenciesThis library depends on the following packagesopensearch-pyBuilding from source & DevelopingThis package usespyenv(optional) for development purposes.
It also uses Docker to run OpenSearch container for integration testing during development.Clone the repo.Create a virtual environment using any of the supported Python version.# We are using Python 3.11 installed using pyenv for this examplepyenvlocal3.11.0# Create virtual envpython-mvenv.venv# Activate itsource.venv/bin/activateInstallpip-toolsandflit# Update pip to the latest version, just in casepipinstall--upgradepip# Install pip-compile and pip-sync, as well as flitpipinstallpip-toolsflitCompile resolved dependency list# Generates requirements.txt file.# This might yield different results for different platforms.pip-compilepyproject.toml# Resolve dev requirementspip-compile--extradev-odev-requirements.txtpyproject.toml# If you want to upgrade dependencies, then callpip-compilepyproject.toml--upgradeInstall resolved dependencies into virtual environment# Sync current venv with both core and dev dependenciespip-syncrequirements.txtdev-requirements.txtInstall package itself locally.Build, publishing, and local installation are done usingflit.flitinstallRun testsWARNING: You need opensearch running onhttps://localhost:9200to run the tests.
Part of the tests verifies that correct number of logs actually gets into OpenSearch.
Alternatively, you can specify theTEST_OPENSEARCH_HOSTvariable and set it to a different value pointing
to the running OpenSearch server.There are not many tests, but they run with5 seconds cooldown eachto allow OpenSearch to process the
newly sent log records properly and verify their count.Small helper scripts are available in thetests/directory to start and stop OpenSearch using Docker.# Give it 5-10 seconds to initialize before running teststests/start-opensearch-docker.sh# Run testspytest# Run coverage testspytest--cov--cov-report=html--cov-config=pyproject.toml# Run mypy typing verificationpytest--mypyopensearch_logger--strict-markers# Run flake8 to make sure code style is correctflake8# Turn off OpenSearchtests/stop-opensearch-docker.shBump package versionbump2versionpatchPublish package (make sure you have correct credentials or.pypircfile)flitpublishCheat Sheet for working with OpenSearchList all created indices, including count of documents$curl-k-XGET"https://admin:admin@localhost:9200/_cat/indices/test*?v&s=index"healthstatusindexuuidprirepdocs.countdocs.deletedstore.sizepri.store.size
yellowopentest-opensearch-logger-2021.11.08N0BEEnG2RIuPP0l8RZE0Dg117029.7kb29.7kbCount documents in all indexes that start withtest$curl-k-XGET"https://admin:admin@localhost:9200/test*/_count"{"count":109,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}Retrieve all documents from indexes that start withtest$curl-k-XGET"https://admin:admin@localhost:9200/test*/_search"-H'Content-Type: application/json'-d'{"query":{"match_all":{}}}'{"took":1,"timed_out":false,"hits":{"total":{"value":109,"relation":"eq"}...Same, but limit the number of returned documents to 10$curl-k-XGET"https://admin:admin@localhost:9200/test*/_search?size=10"-H'Content-Type: application/json'-d'{"query":{"match_all":{}}}'{"took":1,"timed_out":false,"hits":{"total":{"value":109,"relation":"eq"}...ContributionsContributions are welcome! 👏  🎉Please create a GitHub issue and a Pull Request that references that issue as well as your proposed changes.
Your Pull Request will be automatically tested using GitHub actions.After your pull request will be accepted, it will be merged and the version of the library will be bumped
and released to PyPI by project maintainers.HistoryThis is a fork ofPython Elasticsearch ECS Log handlerproject
which was in turn forked fromPython Elasticsearch Loggerproject.
While original is perfectly suitable for logging to Elasticsearch, due to the split between
OpenSearch and Elasticsearch it makes sense to make a fork entirely tailored to work with OpenSearch
and based on the officialopensearch-pyPython library.The API betweenpython-elasticsearch-ecs-loggerand this project has slightly changed for better
compatibility with OpenSearch and for the purposes of simplification.Featured onTheopensearch-loggerproject is featured on the officialOpenSearch Community Projects page🚀.LicenseDistributed under the terms ofApache 2.0license, opensearch-logger is free and open source software.