How to build a real-time fraud detection pipeline using Faust and MLFlow | by Bogdan Cojocar | Towards Data Science

In this tutorial we will learn to start a Kafka cluster locally, to write a producer that sends messages, to create a simple machine learning training process for fraud detection, to expose the model via a REST interface and to do prediction in real-time.
The main frameworks that we will use are:
Faust: stream processing library, using the async/away paradigm and requires python 3.6+
Kafka: we will use the confluent version for kafka as our streaming platform
MLFlow: an open-source platform used to monitor and save machine learning models after training
Jupyter lab: our environment to run the code
We will also need some data for our training. We can use a CSV source from a Kaggle credit card fraud competition.
TL;DR: The code is on GitHub.
Let’s first build our fraud detection model. We will read the CSV data representing credit card transactions and apply a simple binary logistic regression classification between the two classes 0(not fraud) and 1(fraud) .
df = pd.read_csv('creditcard.csv')df.dtypesTime      float64V1        float64V2        float64V3        float64V4        float64V5        float64V6        float64V7        float64V8        float64V9        float64V10       float64V11       float64V12       float64V13       float64V14       float64V15       float64V16       float64V17       float64V18       float64V19       float64V20       float64V21       float64V22       float64V23       float64V24       float64V25       float64V26       float64V27       float64V28       float64Amount    float64Class       int64
As we can see, most of the columns that we read are float , except the Class , which is our target variable. Now let’s do the training:
x_name = df.iloc[:, 1:30].columnsy_name = df.iloc[:1, 30: ].columnsdata_x = df[x_name]data_y = df[y_name]train_x, test_x, train_y, test_y = train_test_split(data_x, data_y, train_size=0.7, test_size=0.3)model = LogisticRegression()model.fit(train_x, train_y.values.ravel())
For features we use V1 — V28 and Amount . We do the split between train and test data, and fit the model. We can check some performance metrics:
pred_y = model.predict(test_x)accuracy_score(test_y, pred_y)0.9993094811745842f1_score(test_y, pred_y)0.7773584905660378precision_score(test_y, pred_y)0.8272727272727273recall_score(test_y, pred_y)0.6776315789473685
If we just use only accuracy as a metric we can shot ourselves in the foot. Always use better performance indicators like f1_score , precision and recall .
After the training we can save our new fraud detection model:
mlflow.sklearn.save_model(model, path='./fraud_model')
We use the sklearn module inside MLFlow as this will help us with the servis part.
Let’s start a terminal in the root of our project and type the following command:
mlflow models serve -m fraud_model
This will take a little bit of time, as additionally to deploying a microservice, it will also create a conda environment. This is to ensure that next time when we do the deployment we have our dependencies locked at a certain version and in a separate virtual environment. If everything runs fine, we will notice that we have a url for our service:
Listening at: http://127.0.0.1:5000
We can do a quick test from the terminal using cURL :
curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{    "columns":["V1","V2","V3","V4","V5","V6","V7","V8","V9","V10","V11","V12","V13","V14","V15","V16","V17","V18","V19","V20","V21","V22","V23","V24","V25","V26","V27","V28","Amount"],    "data":[[12.8,0.029,0.48,0.98,6.2,29.1,3.33,1.2,0.39,75.1,0.66,1.2,1.3,44.2,12.8,0.029,0.48,0.98,6.2,29,3.33,1.2,0.39,75.3,0.66,1.2,1.3,44.2,1.1]]}'
We are sending a POST request using a JSON object containing the feature columns and a set of values. We will get back the answer from the classifier [0] .
To build the cluster we will use a docker-compose file that will start all the docker containers needed: zookeeper and a broker.
Now very briefly, kafka is a distributed streaming platform capable of handling a large number of messages, that are organized or grouped together into topics. In order to be able to process a topic in parallel, it has to be split into partitions, and the data from these partitions are stored into separate machines called brokers. And finally, zookeeper is used to manage the resources of the brokers in the clusters.To read or write into a kafka cluster we need a broker address and a topic.
The docker-compose will start zookeper on port 2181 , a kafka broker on port 9092. Besides that we use another docker container kafka-create-topic for the sole purpose to create a topic (called test) in the kafka broker.
To start the kafka cluster, we have to run the following command line instruction in the same folder where we have defined the docker compose file:
docker-compose up
This will start all the docker containers with logs. We should see something like this in the console:
To be able to consume data in realtime we first must write some messages into kafka. We will use the confluent_kafka library in python to write a producer:
We will send JSON messages having the format:
{    "columns":["V1","V2","V3","V4","V5","V6","V7","V8","V9","V10","V11","V12","V13","V14","V15","V16","V17","V18","V19","V20","V21","V22","V23","V24","V25","V26","V27","V28","Amount"],    "data":[]}
It is the same format we use for the MLFlow web service, where data can contain multiple lists of values for the features.
For each message we write into the queue we also need to assign a key. We will assign a random one based on the uuid to achieve a good distribution into the cluster. In the end, we also run a flush command to ensure that all the messages are sent.
Once we run the confluent_kafka_producer we should receive a log telling us that the data has been sent correctly:
we’ve sent 6 messages to 127.0.0.1:9092
We have reached our final step in the tutorial. With the Faust framework we will run a simple worker that will consume data from Kafka and will apply the prediction on the new features. We need to create a separate python script for this one. First, we define our Faust app, and the topic:
app = faust.App(    'fraud_detection_app',    broker='kafka://localhost:9092',    value_serializer='raw',)kafka_topic = app.topic('test')
We define the broker from where we read and the serializer format. We are happy to use raw as we will just pass the data to the machine learning algorithm.
Now, let’s define the coroutine. In the Faust framework, it’s called anagent :
The agent is an asynchronous process. We connect to the Kafka topic, and for each value that we read, we do a call to the REST service and obtain a result. As expected, this will run immediately after new values are available in Kafka.
The script is finalised. Let’s name it fraud_detection_worker.py . Now we can run it. From a terminal started in the root of the project we need to type:
faust -A fraud_detection_worker worker -l info
This will start a worker, which is an instance of the app. If we need more computing power, we can start additional workers.
If everything is running fine, we will start to see the following logs:
Input data: b'{"columns": ["V1", "V2", "V3", "V4", "V5", "V6", "V7", "V8", "V9", "V10", "V11", "V12", "V13", "V14", "V15", "V16", "V17", "V18", "V19", "V20", "V21", "V22", "V23", "V24", "V25", "V26", "V27", "V28", "Amount"], "data": [[12.8, 0.029, 0.48, 0.98, 6.2, 29.1, 3.33, 1.2, 0.39, 75.1, 0.66, 11.2, 1.3, 0.2, 12.8, 0.029, 0.45, 0.98, 6.2, 29, 3.33, 1.2, 0.39, 75.3, 0.3, 2.2, 1.3, 2.2, 1.01]]}'Fraud detection result: [0]
We have reached the end of this tutorial. I hope you enjoyed it and find it useful. We saw how now we have the option to use a pure python framework to do real-time data processing. This is a great advantage as a lot of the data science work out there is in this programming language.