Package: fast-kafka-api
FastKafkaAPIEffortless Kafka integration for your web servicesFastKafkaAPI is a powerful and easy-to-use Python library for building
asynchronous web services that interact with Kafka topics. Built on top
of FastAPI, Starlette, Pydantic, and AIOKafka, FastKafkaAPI simplifies
the process of writing producers and consumers for Kafka topics,
handling all the parsing, networking, and task scheduling automatically.
With FastKafkaAPI, you can quickly prototype and develop
high-performance Kafka-based services with minimal code, making it an
ideal choice for developers looking to streamline their workflow and
accelerate their projects.InstallThis command installs the FastKafkaAPI package from the Python Package
Index (PyPI) using the pip package manager.pipis a command-line tool that allows you to install and manage
Python packages, including FastKafkaAPI. When you run thepip installcommand with the name of a package (in this case, “fast-kafka-api”), pip
will download the package from PyPI, along with any dependencies that
the package requires, and install it on your system.After running this command, you will be able to import and use the
FastKafkaAPI package in your Python code. For example, you might use it
to initialize a FastKafkaAPI application, as shown in the example
bellow, and to use the@consumesand@producesdecorators to define
Kafka consumers and producers in your application.Installing FastKafkaAPI from PyPI usingpipis the recommended way to
install the package, as it makes it easy to manage the package and its
dependencies. If you prefer, you can also install FastKafkaAPI from the
source code by cloning the repository and runningpip install .in the
root directory of the project.pipinstallfast-kafka-apiHow to useHere is an example python script using FastKafkaAPI that takes data from
an input Kafka topic, makes a prediction using a predictive model, and
outputs the prediction to an output Kafka topic.MessagesFastKafkaAPI uses Pydantic to parse input JSON-encoded data into Python
objects, making it easy to work with structured data in your Kafka-based
applications. Pydantic’sBaseModelclass allows you to define messages
using a declarative syntax, making it easy to specify the fields and
types of your messages.This example defines two message classes for use in a FastKafkaAPI
application:InputDataandPrediction.TheInputDataclass is used to represent input data for a predictive
model. It has three fields:user_id,feature_1, andfeature_2. Theuser_idfield is of typeNonNegativeInt, which is a subclass of int
that only allows non-negative integers. Thefeature_1andfeature_2fields are both lists of floating-point numbers and integers,
respectively. These fields are used to represent input features for the
predictive model.ThePredictionclass is used to represent the output of the predictive
model. It has two fields:user_idandscore. Theuser_idfield is
of typeNonNegativeInt, and thescorefield is a floating-point
number. Thescorefield represents the prediction made by the model,
such as the probability of churn in the next 28 days.These message classes will be used to parse and validate incoming data
in Kafka consumers and producers. Using these message classes in
combination with FastKafkaAPI makes it easy to work with structured data
in your Kafka-based applications.fromtypingimportListfrompydanticimportBaseModel,Field,NonNegativeIntclassInputData(BaseModel):user_id:NonNegativeInt=Field(...,example=202020,description="ID of a user")feature_1:List[float]=Field(...,example=[1.2,2.3,4.5,6.7,0.1],description="input feature 1",)feature_2:List[int]=Field(...,example=[2,4,3,1,0],description="input feature 2",)classPrediction(BaseModel):user_id:NonNegativeInt=Field(...,example=202020,description="ID of a user")score:float=Field(...,example=0.4321,description="Prediction score (e.g. the probability of churn in the next 28 days)",ge=0.0,le=1.0,)These message classes will be used to parse and validate incoming data
in a Kafka consumer and to produce a JSON-encoded message in a producer.
Using Pydantic’s BaseModel in combination with FastKafkaAPI makes it
easy to work with structured data in your Kafka-based applications.ApplicationThis example shows how to initialize a FastKafkaAPI application. It
starts by defining two environment variables:KAFKA_HOSTNAMEandKAFKA_PORT, which are used to specify the hostname and port of the
Kafka broker.Next, it defines a dictionary calledkafka_brokers, which contains two
entries: “localhost” and “production”. Each entry specifies the URL,
port, and other details of a Kafka broker. This dictionary is used to
define the available Kafka brokers that can be used in the application.Thekafka_configdictionary specifies the configuration options for
the Kafka broker, such as thebootstrap_serverssetting, which
specifies the hostname and port of the Kafka broker.Finally, the FastKafkaAPI class is initialized with several arguments:title,contact,version,description,kafka_brokers, andkafka_config. These arguments are used to configure various aspects of
the application, such as the title, version, and description of the
application, as well as the available Kafka brokers and the Kafka
configuration options. The resultingFastKafkaAPIobject, which is stored in theappvariable, represents the
initialized FastKafkaAPI application.fromosimportenvironfromfast_kafka_api.applicationimportFastKafkaAPIkafka_server_url=environ["KAFKA_HOSTNAME"]kafka_server_port=environ["KAFKA_PORT"]kafka_brokers={"localhost":{"url":"kafka","description":"local development kafka broker","port":9092,},"production":{"url":"kafka.acme.com","description":"production kafka broker","port":9092,"protocol":"kafka-secure","security":{"type":"plain"},},}kafka_config={"bootstrap_servers":f"{kafka_server_url}:{kafka_server_port}",}app=FastKafkaAPI(title="FastKafkaAPI Example",contact={"name":"airt.ai","url":"https://airt.ai","email":"info@airt.ai"},version="0.0.1",description="A simple example on how to use FastKafkaAPI",kafka_brokers=kafka_brokers,**kafka_config,)Function decoratorsFastKafkaAPI provides convenient function decorators called@consumesand@producesto allow you to delegate the actual processing of data
to user-defined functions. These decorators make it easy to specify the
processing logic for your Kafka consumers and producers, allowing you to
focus on the core business logic of your application without worrying
about the underlying Kafka integration.This example shows how to use the@consumesand@producesdecorators
in a FastKafkaAPI application.The@consumesdecorator is applied to theon_input_datafunction,
which specifies that this function should be called whenever a message
is received on the “input_data” Kafka topic. Theon_input_datafunction takes a single argument,msg, which is expected to be an
instance of theInputDatamessage class.Inside theon_input_datafunction, themodel.predictfunction is
called with thefeature_1andfeature_2fields from themsgargument. This function returns a prediction score, which is then passed
to theto_predictionsfunction along with theuser_idfield from themsgargument.The@producesdecorator is applied to theto_predictionsfunction,
which specifies that this function should produce a message to the
“predictions” Kafka topic whenever it is called. Theto_predictionsfunction takes two arguments:user_idandscore. It creates a newPredictionmessage with these values and then returns it.In summary, this example shows how to use the@consumesand@producesdecorators to specify the processing logic for Kafka
consumers and producers in a FastKafkaAPI application. The@consumesdecorator is applied to functions that should be called when a message
is received on a Kafka topic, and the@producesdecorator is applied
to functions that should produce a message to a Kafka topic. These
decorators make it easy to specify the processing logic for your Kafka
consumers and producers, allowing you to focus on the core business
logic of your application without worrying about the underlying Kafka
integration.@app.consumes(topic="input_data")asyncdefon_input_data(msg:InputData):print(f"msg={msg}")score=awaitmodel.predict(feature_1=msg.feature_1,feature_2=msg.feature_2)awaitto_predictions(user_id=msg.user_id,score=score)@app.produces(topic="predictions")asyncdefto_predictions(user_id:int,score:float)->Prediction:prediction=Prediction(user_id=user_id,score=score)print(f"prediction={prediction}")returnpredictionRunning the serviceThis example shows how to start the FastKafkaAPI service using the
uvicorn library. Theuvicorn.runfunction is called with theappargument (which represents the FastKafkaAPI application) and thehostandportarguments, which specify the hostname and port on which the
service should listen for incoming requests.When the service is started, several log messages are printed to the
console, including information about the application startup, AsyncAPI
specification generation, and consumer loop status.During the lifetime of the service, incoming requests will be processed
by the FastKafkaAPI application and appropriate actions will be taken
based on the defined Kafka consumers and producers. For example, if a
message is received on the “input_data” Kafka topic, theon_input_datafunction will be called to process the message, and if theto_predictionsfunction is called, it will produce a message to the
“predictions” Kafka topic. The service will continue to run until it is
shut down, at which point the application shutdown process will be
initiated and the service will stop.importuvicornuvicorn.run(app._fast_api_app,host="0.0.0.0",port=4000)INFO:     Started server process [21284]
INFO:     Waiting for application startup.

[INFO] fast_kafka_api._components.asyncapi: Old async specifications at '/work/fast-kafka-api/nbs/asyncapi/spec/asyncapi.yml' does not exist.
[INFO] fast_kafka_api._components.asyncapi: New async specifications generated at: 'asyncapi/spec/asyncapi.yml'
[INFO] fast_kafka_api._components.asyncapi: Async docs generated at 'asyncapi/docs'
[INFO] fast_kafka_api._components.asyncapi: Output of '$ npx -y -p @asyncapi/generator ag asyncapi/spec/asyncapi.yml @asyncapi/html-template -o asyncapi/docs --force-write'

Done! ✨
Check out your shiny new generated files at /work/fast-kafka-api/nbs/asyncapi/docs.


[INFO] fast_kafka_api._components.aiokafka_consumer_loop: aiokafka_consumer_loop() starting..
[INFO] fast_kafka_api._components.aiokafka_consumer_loop: aiokafka_consumer_loop(): Consumer created.

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)

[INFO] fast_kafka_api._components.aiokafka_consumer_loop: aiokafka_consumer_loop(): Consumer started.
[INFO] aiokafka.consumer.subscription_state: Updating subscribed topics to: frozenset({'input_data'})
[INFO] aiokafka.consumer.consumer: Subscribed to topic(s): {'input_data'}
[INFO] fast_kafka_api._components.aiokafka_consumer_loop: aiokafka_consumer_loop(): Consumer subscribed.
[INFO] aiokafka.consumer.group_coordinator: Metadata for topic has changed from {} to {'input_data': 1}. 

INFO:     Shutting down
INFO:     Waiting for application shutdown.

[INFO] fast_kafka_api._components.aiokafka_consumer_loop: aiokafka_consumer_loop(): Consumer stopped.
[INFO] fast_kafka_api._components.aiokafka_consumer_loop: aiokafka_consumer_loop() finished.

INFO:     Application shutdown complete.
INFO:     Finished server process [21284]