Create a Python User-Defined Function (UDF) for Kapacitor on Windows 10
By: Shan Desai
Tags: Kapacitor, Influxdb, Windows 10, Data Processing, Python

UDF in Python

From my understanding, the information exchange between Kapacitor and the UDF occurs through Google Protocol Buffer and STDIN/STDOUT.

Assumption: You have python-3.x and pip installed on your local machine and is executable in your PowerShell. You should also have git installed either on PowerShell or WSL for further steps.

Let us create a udf_test.py in the Kapacitor folder e.g. C:\Users\<user>\kapacitor-1.5.2-1

The kapacitor pip modules are NOT available on PyPi, hence we will have to install it manually later.

But the code can already have some modules that will be available later used to execute the UDF.

from kapacitor.udf.agent import Agent, Handler

from kapacitor.udf import udf_pb2

import logging

The Agent and Handler are explained well in the Kapacitor’s UDF Docs for some deep-diving. We use logging instead of print so that logs are better seen when the Kapacitor Daemon runs.

The Handler interface comprises of some methods which are documented on Kapacitor’s UDF Docs page. We will go through them quickly while creating the UDF.

Standard Practice

Most examples inherit the Handler class for your custom UDF. Hence in our case we call the class GeoTestHandler and setup the basic skeleton for running the code:

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s:%(name)s: %(message)s')

logger = logging.getLogger() class GeoTestHandler(Handler):

def __init__(self, agent):

logger.info('__init__ trigger')

self._agent = agent

def info(self):

pass

def init(self):

pass

def begin_batch(self):

pass

def point(self):

pass

def end_batch(self):

pass if __name__ == '__main__':

# Agent creation

a = Agent()

h = GeoTestHandler(a)

a.handler = h

logger.info('Starting Agent')

a.start()

a.wait()

logger.info('Agent Finished')

Now the class will have the following member functions:

info : Tell the UDF what it actually wants i.e. a Batch or Stream and what it will Provide i.e. Batch or Stream. Explanation on UDF Kapacitor Docs

: Tell the UDF what it actually wants i.e. a Batch or Stream and what it will Provide i.e. Batch or Stream. Explanation on UDF Kapacitor Docs init : Initialize the UDF with what it expects as parameters from the TICKScript. Explanation on UDF Kapacitor Docs

: Initialize the UDF with what it expects as parameters from the TICKScript. Explanation on UDF Kapacitor Docs begin_batch , point , end_batch what begins at the beginning of the batch of data, what happens to each point, what happens at the end of the batch respectively. Explanation on UDF Kapacitor Docs

In our case we will use a Batch of data and for every datapoint calculate the geometric sum and add a string field to the point. We define that in out info function. Additionaly we will use a parameter called .field('some_string') in our TICKScript to get only the sensor value from the batch, hence we define that in info as follows:

def info(self):

"""

Define what your UDF wants and what will it provides and options

"""

logger.info('info trigger')

response = udf_pb2.Response()

response.info.wants = udf_pb2.BATCH

response.info.provides = udf_pb2.STREAM

response.info.options['field'].valueTypes.append(udf_pb2.STRING)

return response

Next we do some check logic when the TICKScript is parsed and see if the field parameter exists in it or not. This is pretty standard in UDFs for Kapacitor:

def init(self, init_req):

"""

Define what your UDF expects as options when parsing

TICKScript

"""

logger.info('INIT trigger')

for opt in init_req.options:

if opt.name == 'field':

self._field = opt.values[0].stringValue

success = True

msg = ''

if self._field == '':

success = False

msg = 'must provide field name'

response = udf_pb2.Response()

response.init.success = success

response.init.error = msg.encode()

return response

We currently have nothing to do at the begin_batch but we might as well log it just to see if triggers during execution:

def begin_batch(self, begin_req):

"""begin_batch: Do something at the beginning of the batch"""

logger.info('begin_batch trigger')

The main method is point where each incoming datapoint can be accessed and processed accordingly. Here I took the logic provided by the InfluxData Training Video on YouTube. Before writing the point , create a private list called _points = [] in the class constructor.

def point(self, point):

"""point: process each point within the batch"""

logger.info('point trigger')

self._points.append(point.fieldsDouble[self._field])

if len(self._points) == self._size:

geo = 1.0 # geometric sum for data points

for p in self._points:

geo *= p

response = udf_pb2.Response()

response.point.name = point.name

response.point.time = point.time

response.point.group = point.group

response.point.tags.update(point.tags)

# add the geometric sum to its own field

response.point.fieldsDouble['geo'] = geo

# add some dummy string field into the point as well

response.point.fieldsString['hash'] = 'test'

self._agent.write_response(response)

self._points.pop(0)

We have nothing to do at the end of the batch so we just log as with begin_batch :

def end_batch(self, batch_meta):

"""end_batch: do something at the end of the batch"""

logger.info('end_batch')

UDF Code

UDF Python Code for Geometric Sum on values and storing a string field

Configuring UDF in Kapacitor

First find out where you python interpreter’s path is on your Windows machine using the following in PowerShell:

> where.exe python.exe

In my case it was in C:\Users\<user>\AppData\Local\Programs\Python\Python36-32\python.exe

Next clone the Kapacitor GitHub Repository in your \kapacitor-1.5.2-1 folder using:

Within the repository we need the udf/agent/py for kapacitor module.

Add the [udf] section in the kapacitor.gen.conf file as follows:

[udf]

[udf.functions]

[udf.functions.geo]

prog = "C:\\Users\\<user>\\AppData\\Local\\Programs\\Python\\Python36-32\\python.exe"

args = [ "-u", "C:\\Users\\<user>\\Dev\\kapacitor-1.5.2-1\\udf_test.py"]

timeout = "10s"

[udf.functions.hash.env]

PYTHONPATH = "C:\\Users\\<user>\\kapacitor-1.5.2-1\\kapacitor\\udf\\agent\\py\\"

here geo is the name of the UDF we will use in the TICKScript later.

Adapting Code for PYTHON3

Most of the Kapacitor UDF Agent code is still for python2 and in order to use if for python-3.x there needs to be some code refactor. For that I referred to the community post here:

where within agent.py some modules need to be adopted for python3.6. I changed the code accordingly in the file. Next compatibility issue came for protobuffers between strings and bytes (not going to dwell much on it!) but the patch from an Open Pull-Request in the repository solves some compatibility problems. Either apply the patch or refactor the code by hand.

Install the kapacitor module using pip in the \kapacitor-1.5.2-1\udf\agent\py folder:

pip install -e .

This should install protobuf version 3.5.0 . Check it via pip list in the PowerShell.

TICKScript for Application

Run the Kapacitor Daemon first using:

.\kapacitord.exe -config .\kapacitor.gen.conf -log-level debug

Script

In order to use our custom UDF we use the @ symbol before the name of the UDF e.g. geo defined in the .conf file. We first get a batch every 20s with window 20s and pipe into our UDF. We then, send this information to our HTTP endpoint and then simultaneously to a new database called udf_test under the measurement name udf .

udf_test.tick

var data = batch

|query('SELECT * FROM "telegraf_test"."autogen"."humid"')

.period(20s)

.every(20s) data

@geo()

.field('v')

|httpPost()

.endpoint('kap')

|InfluxDBOut()

.database('test_udf')

.retentionPolicy('autogen')

.measurement('hash')

Here, the batch data is then given to our UDF geo which will use the field where only the values v will be used. Finally the information will be sent to our HTTP Endpoint and successively to our local InfluxDB under the test_udf database.

TICKScript Examples

You can also create an alert log everytime using the example below udf_alert.tick .

Define the task:

.\kapacitor.exe define udf_test -tick udf_test.tick -type batch -dbrp telegraf_test.autogen

Enable the task:

.\kapacitor.exe enable udf_test

Outputs

on HTTP Endpoint:

{ columns: [ 'time', 'geo', 'hash' ],

values:

[ [ '2019-03-15T17:12:36.7445341Z',

2.6265808725454086e+27,

'test' ] ] }

{ columns: [ 'time', 'geo', 'hash' ],

values:

[ [ '2019-03-15T17:12:36.8036182Z',

2.6265808725454086e+27,

'test' ] ] }

{ columns: [ 'time', 'geo', 'hash' ],

values:

[ [ '2019-03-15T17:12:37.9411825Z',

2.6265808725454086e+27,

'test' ] ] }

{ columns: [ 'time', 'geo', 'hash' ],

values:

[ [ '2019-03-15T17:12:37.9832335Z',

2.6265808725454086e+27,

'test' ] ] }

{ columns: [ 'time', 'geo', 'hash' ],

values:

[ [ '2019-03-15T17:12:39.1414289Z',

2.6265808725454086e+27,

'test' ] ] }

{ columns: [ 'time', 'geo', 'hash' ],

values:

[ [ '2019-03-15T17:12:39.3937046Z',

2.6265808725454086e+27,

'test' ] ] }

I was able to parse the incoming JSON structure through express-js and it is worth noting that hash is test which we have defined in the UDF.

on InfluxCLI:

> USE test_udf

> SHOW MEASUREMENTS

name: measurements

name

----

hash

> select * from hash limit 4

name: hash

time geo hash

---- --- ----

1552643774916616700 62854912109375 test

1552643774973127900 62854912109375 test

1552643776006894500 62854912109375 test

1552643776125063000 62854912109375 test

Nuggets

If you ever change the Python UDF code, make sure to restart the Kapacitor Daemon

If after restart, the task still shows errors, then you are better off deleting the task and redefining it again.

.\kapacitor.exe delete tasks udf_test

Resources