Get insights on PostgreSQL data with Azure Sentinel
By: Koos Goossens
Tags: Postgres, Log Analytics, Azure Sentinel, Logstash, Postgresql

Logstash: the swiss army knife for collecting data

If the particular PostgreSQL database would be a cloud database, such as an Azure Database for PostgreSQL, it would’ve been an easy task to retrieve the audit logs through the Azure platform. Microsoft already wrote a document on how to do this. But in this case we don’t need the authentications on the database itself, but the events the application stores inside one of the tables in the database.

The last couple of months I’ve been looking into several ways to import and manipulate data in conjunction with Azure Sentinel / Log Analytics. And a lot of these times Logstash made my shortlist for the tool that could help me out.

What is Logstash?

Logstash is a product made by Elastic and is also part of their “Elastic Stack” which is the next evolution of “ELK” Stack. (Elasticsearch, Logstash & Kibana)

Logstash is an open-source data processing pipeline that is able to collect, parse and transform your logs.

Log ingestion is done with a multitude of available input plugins. Optionally your data can be filtered, parsed and transformed as you wish. Elastic provides a library of available filters for you. And lastly it uses an output which puts your data into your favorite “stash”. Elastic would suggest you storing the logs into Elasticsearch for further search and analytics capabilities. But there are also a lot of other output plugins available.

Also, a prime example of collecting, parsing and transforming. But this is not the kind of stash we’re looking for

In this particular case we’re going to make use of:

JDBC input plugin for connecting to our PostgreSQL database and retrieving our relevant data with a specific query.

Azure Log Analytics Workspace output plugin for connecting to Microsoft’s API’s and storing data inside our (Sentinel) workspace.

Running Logstash inside a Docker container

Since I’m already at war with Virtual Machines I would suggest running Logstash inside a container on your favorite container platform. Elastic provides a pre-built container image and a Helm chart for running it on Kubernetes. Check the download page for all of the possible options.

In this example we’ll be needing several non build-in plugins and modules, so building our own custom Docker container seemed the best approach:

.dockerfile

Base image is used from Elastic

The Azure Log Analytics Workspace output plugin is installed.

The JDBC input plugin is already part of the initial installation so we won’t need to install that. But we do need the JDBC driver ‘postgresql-42.2.12.jar’ available from the PostgreSQL website.

Pipeline config file is placed in the correct directory so that Logstash will automatically start processing this on startup.

Logstash pipeline

Besides the proper input- and output-plugins, we’ll also be needing our pipeline config file. A Logstash pipeline describes the steps that are needed to be executed to input, filter and output the data. Our config looks as follows:

postgres-to-sentinel.conf

The input part refers to the JDBC plugin we’ve installed. Here we need to provide a proper connection string and authentication for the database (‘security’) as well as a query to retrieve our relevant data from the table (‘authentications’). Note the schedule on line 19 which indicates that Logstash will be repeatedly running this pipeline with a 5-minute interval. The schedule uses a cron format. is Also note the referred JDBC driver on line 11. In this case we won’t be needing a filter to manipulate the data before it’s been sent out to Azure Sentinel. Output describes where the data is sent to, in this case an Azure Log Analytics Workspace with its proper id and key.

Please note the SQL query that is part of the input of this pipeline:

SELECT uid, timestamp, event, message, username, ipaddress FROM authentications

Logstash will by default create a new dataset every time it runs. In this case that means that we’ll be retrieving the same data over and over again. Since we don’t want duplicate data in our workspace, we’ll be using the column tracking feature that is part of this particular JDBC input plugin.