Deploying Confluent Enterprise on
Microsoft Azure
An Introductory Guide
April 2017

ABSTRACTThis white paper outlines the integration of Confluent Enterprise with Microsoft’s
Azure Cloud Platform. The Confluent Enterprise offer within Azure Marketplace ispresented along with recommended best practices for integrating with otherAzure services.

Table of Contents
Confluent Enterprise ............................................................................................................................... 3
Azure Services ........................................................................................................................................ 4
Topology overview .................................................................................................................................. 5
Download and install the Azure command line tool................................................................................ 6
Connect to the Azure portal .................................................................................................................... 6
Deploy Cluster ......................................................................................................................................... 8
I. Basics ................................................................................................................................................... 8
II. Cluster settings.................................................................................................................................... 9III. Network information ......................................................................................................................... 11
IV. Summary.......................................................................................................................................... 11
V. Buy .................................................................................................................................................... 12
Testing the deployment ......................................................................................................................... 13
Updating the Deployment ..................................................................................................................... 18
Useful Azure Commands ...................................................................................................................... 20
Troubleshooting..................................................................................................................................... 21
Newly configured connector is not available ..................................................................................... 21
Confluent Control Center is unresponsive ........................................................................................ 21© 2016–2017 Confluent, Inc.

| 2

Confluent and Azure
Apache Kafka, created by the founders of Confluent while working at LinkedIn, is fast becoming thede facto technology for moving data across systems in real time. Such frictionless data movement
supports interactive websites, mobile applications, and all manner of Internet-of-Things use cases.
Confluent builds on Kafka with additional tooling to speed developer productivity, improve data
management, and broaden the scope of applications that can benefit from Kafka.
Confluent Enterprise is available as a bring-your-own-license offering in Azure Marketplace. The
integration simplifies deployment of your streaming infrastructure and applications, making it easy toconfigure, deploy, monitor and manage Confluent clusters. Kafka clusters on Azure can support 10s to
1000s of traditional producer/consumer workloads as well as the latest Kafka Connect pipelines.

Confluent Enterprise
The Confluent architecture is summarized in the following diagram:© 2016–2017 Confluent, Inc.

| 3

At the center of Confluent Enterprise is Apache Kafka. Apache Kafka is a modern distributed system formoving data in real time. It allows a single cluster of computers to perform all the data transport
between an organization’s data systems regardless of their number or size. Confluent Enterprise
includes all components from the Apache Kafka Core along with some Apache-licensed open source
additions (client tools, pre-built connectors, and cluster-side services such as Schema Registry and
REST Proxy). The offering is completed with licensed components to support cluster monitoring andmanagement, multi-data center replication, and advanced data balancing capabilities.

Azure Services
With Azure Marketplace, you can provision big data services with a single click. Services can leverage
the same virtual infrastructure to achieve tight integration, or remain loosely coupled though limited
network or storage connectivity. The Azure Marketplace offering for Confluent Enterprise delivers• Automated cluster provisioning, management, and elastic scaling• Complete support for all Confluent Enterprise components, e.g. Kafka, Kafka Connect Workers,
Kafka Streams Apps, Schema Registry, REST Proxy, and Control Center• Point-and-click data pipeline deployment between other Azure services via the Kafka Connect.

© 2016–2017 Confluent, Inc.

| 4

Confluent Enterprise Quick Start on Azure
Initial support for Confluent Enterprise 3.0 was delivered in October 2016, with an upgrade to later
versions of Confluent 3.x April 2017. The services are deployed across a user-defined set of virtual
machine instances that can be sized to meet your specific needs. These installation instructions
assume that you have an active Azure Marketplace subscription and a basic understanding of the
Confluent architecture (the relationship between brokers, workers, and support services {Schema
Registry and REST Proxy}). More information on the Confluent architecture is available here.

Topology overview
For simplicity, the Confluent Enterprise topology supports only three classes of service nodes for the
deployment. Users will specify 3 or more broker nodes and 1 or more worker nodes. Apache Kafka
requires a zookeeper quorum for metadata management; users can specify independent nodes for thezookeeper service, or allow the service to be hosted on the broker nodes. Auxiliary services(Confluent Schema Registry and Confluent REST Proxy) are deployed on the worker nodes.
All standard virtual machine instance types are supported. The broker nodes are automaticallyprovisioned with 4 TB of storage each. All instances will be deployed within a single virtual subnet.
Users can select a pre-existing network within their Azure subscription or create a new one during the
deployment process. By default, the deployed nodes will be assigned public hostnames / IP addresses
and the network ports to access Confluent Enterprise services from outside Azure will be opened forexternal access. The specifics of that access mechanism is discussed later in this document.
The following sections discuss the deployment process in a step-by-step manner.

© 2016–2017 Confluent, Inc.

| 5

Download and install the Azure command line tool
There are occasions when command-line control over your Azure services is useful. For that reason,
you may wish to install the Azure command-line tool by following the instructions found at
http://azure.microsoft.com/en-us/documentation/articles/xplat-cli/#configure . Be sure to authenticate
your client with the Azure portal before proceeding with the rest of the deployment.

Connect to the Azure portal
The Azure portal is available at http://portal.azure.com . After logging in, your browser will show the
top-level dashboard :Click on the Marketplace link to navigate to the services page. Select the “Compute” category and thensearch for the Confluent offerings. Select the desired Confluent Enterprise version and continue withthe deployment.

© 2016–2017 Confluent, Inc.

| 6

NOTE: There can be some confusion in the Services page between the standalone Confluent
VM Images and the complete Azure Topology that will deploy multiple images into a single,
coherent cluster. Be sure to select the Confluent Enterprise solution topology link, not any of
the VM Image links. If the input blades do not match the example below, it’s likely that you
selected the VM Image only.
The solution template allows you to deploy any of the Confluent versions available in Azure.

© 2016–2017 Confluent, Inc.

| 7

Deploy Cluster
The deployment process consists of a series of blades where you will specify the necessary
parameters to configure your cluster. Each field within the blade will have a tool-tip to assist you, aswell as a reasonable default value (when possible)

I. Basics
The first blade consists of the cluster name and system administrator access credentials. You’ll also
specify the Resource Group for the infrastructure and the Azure region for the deployment.

The Cluster name will be used as a seed for the deployed resources (virtual machines, hostnames,
storage accounts, etc.). Within any single resource group, you should not deploy two Confluent
Enterprise clusters of the same name.

© 2016–2017 Confluent, Inc.

| 8

The SysAdmin specifications define a Linux user and authentication type for the server administrator.
That user will have pseudo privileges and will be able to manage the server from an operational
perspective. While you can choose simple password authentication, Microsoft and Confluent
recommend using SSH Public Key authentication and providing the public half of your key-pair for the
deployment.
As with other solution deployments, you must specify the data center location and the resource group
into which the infrastructure will be provisioned. At this time, Azure does not support deploying 3 rd party
solutions such as Confluent Platform into an existing resource group.

II. Cluster settings
The next blade specifies the virtual machine details for the deployment: which machine size you preferand how many hosts for the cluster overall.

© 2016–2017 Confluent, Inc.

| 9

At a minimum, you cluster will have 3 broker nodes and 1 worker node. The zookeeper quorum canbe deployed separately on 1 or 3 nodes, or co-resident with the brokers (when “0” zookeeper nodes are
selected). This last case is perfectly reasonable for development environments, but may not beappropriate for high-volume clusters. For maximum performance and resiliency, Confluent
recommends deploying 3 zookeeper nodes independent from the Kafka brokers.
The password setting in this blade is for the Confluent Enterprise administrator (Linux user: kadmin).
All Confluent Enterprise services are executed as that user within the cluster. Remember that if theAuthentication Type setting from the previous screen is set to SSH Public Key, password
authentication to the kadmin account will be disabled.

© 2016–2017 Confluent, Inc.

| 10

III. Network information
The Confluent Enterprise topology allows you to create a completely new virtual network for the cluster,or select a pre-existing network.

When selecting a pre-existing network, you’ll want to make sure that specified subnet does not restrict
traffic on the ports necessary for the Kafka services. This is highly unlikely, but worth a quick check ofcore Confluent Enterprise ports (2181, 2888, 3888, 8081, 8082, 8083, 9021, and 9092).

IV. Summary
A single-page overview of what you have entered will be displayed, you can return to the previous
blades in the event you need to change any settings before starting the deployment.

© 2016–2017 Confluent, Inc.

| 11

V. Buy
Confirm the final deployment. Note that Confluent Enterprise on Azure is deployed in a bring-yourown-license model. There will be no charges incurred beyond the infrastructure costs. The
enterprise-licensed features will be available in trial mode for 30 days following the deployment.
Provisioning the Azure infrastructure and configuring the Confluent software will take 10 to 20 minutes.
You can track the progress of the deployment in the Azure Portal by monitoring the Resource Groupyou selected for the deployment.

© 2016–2017 Confluent, Inc.

| 12

Testing the deployment
There are several different ways to connect with the running Confluent Enterprise services on Azure.
Whatever system you use for these tests must have proper network connectivity to the resources
hosting the deployment. For simplicity, it is best to utilize a server in the same virtual network as thecluster. Servers outside the virtual network can access the hosts in Azure with proper settings in their
hostname resolution framework. Details on that configuration are available below in Useful Azure
Commands .
The following sections describe some simple options. We’ll run these tests from any of the worker
nodes, as those are guaranteed to have the proper network connectivity. You should connect to theworker node via ssh as user “kadmin”. If you configured the deployment to use sshPublicKey access,the same key you deployed for the System Administrator will work for the kadmin user. Otherwise,
you’ll use the password you defined in the “Cluster settings” blade above. The Confluent software is
installed in /opt/confluent; the kadmin user will have the proper PATH settings to invoke all of thenecessary commands from that location.
Many of the commands require the proper specification of the zookeeper and broker nodes. Theinternal hostnames are of the form <ClusterName>-<group>-<n> (i.e. mycluster-broker-2, mycluster-zk1, or mycluster-worker-1). Recall that the zookeeper quorum may be deployed on the broker nodes …
in which case you’ll use mycluster-broker-1 for the zookeeper host specification.
A. List / create Kafka topics
Kafka topic metadata operations are done via the kafka-topics command$ kafka-topics --zookeeper mycluster-zk-1:2181 --list"connect-configs","connect-offsets","connect-status","_schemas","__consumer_offsets"
In this case, you can see the topics created by the Schema Registry and the Kafka Connect
services. The Confluent Control Center will also create a number of _confluent-controlcenter-*
topics. You can go on to create a new topic with$ kafka-topics --zookeeper mycluster-zk-1:2181 --create \--topic vtest1 --partitions 1 --replication-factor 3
Created topic “vtest1”.
© 2016–2017 Confluent, Inc.

| 13

$ kafka-topics --zookeeper mycluster-zk-1:2181 --list"connect-configs","connect-offsets","connect-status","_schemas","__consumer_offsets",“vtest1”

B. Producing / consuming messages via the REST Proxy interface
The Confluent REST Proxy service is available at all the broker nodes. That allows you to very
simply send or receive messages from a given topic. Each of the broker nodes will host an
instance of the REST Proxy server. This configuration will allow you to post messages to the
newly created vtest1 topic as follows:$ export RPURL=http://mycluster-worker-1:8082$ curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json" \--data '{"records":[{"value":{"foo":"bar"}}]}' RPURL/topics/vtest1{"offsets":[{"partition":0,"offset":0,"error_code":null,"error":null}],"key_schema_id":null,"value_sc
hema_id":null}$ curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json" \--data '{"records":[{"value":{"foo":"baz"}}]}' $RPURL/topics/vtest1{"offsets":[{"partition":0,"offset":1,"error_code":null,"error":null}],"key_schema_id":null,"value_sc
hema_id":null}

Those messages can be read back by creating a temporary consumer (which you should be
careful to delete when you’re finished)© 2016–2017 Confluent, Inc.

| 14

$ curl -X POST -H "Content-Type: application/vnd.kafka.v1+json" \--data '{"name": "ext_consumer_instance","format": "json", "auto.offset.reset": "smallest"}' \$RPURL/consumers/ext_json_consumer{"instance_id":"ext_consumer_instance","base_uri":"http://mycluster-worker-1:8082/
consumers/ext_json_consumer/instances/ext_consumer_instance"}$ curl -X GET -H "Accept: application/vnd.kafka.json.v1+json" \$RPURL/consumers/ext_json_consumer/instances/ext_consumer_instance/topics/vtest1[{"key":null,"value":{"foo":"bar"},"partition":0,"offset":0},{"key":null,"value":{"foo":"baz"},"partition":0,"offset":1}]$ curl -X DELETE $RPURL/consumers/ext_json_consumer/instances/ext_consumer_instance# No content in response

C. Deploying a connector
Your Confluent Enterprise deployment will have 1 or more nodes where the Kafka Connect
worker containers are running. Each node supports a REST interface that can be used toconfigure and run the available connectors. This configuration can be done directly from thecluster nodes or via the Confluent Control Center. We’ll discuss the direct option first.
You can query any worker for which connectors are available with a simple command.
$ export CWURL=http://mycluster-worker-1:8083$ curl $CWURL/connector-plugins[{"class":"io.confluent.connect.hdfs.HdfsSinkConnector"},{"class":"io.confluent.connect.jdbc.JdbcSourceConnector"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector"},© 2016–2017 Confluent, Inc.

| 15

{"class":"io.confluent.connect.hdfs.tools.SchemaSourceConnector"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector"}]You can configure an available connector by uploading the appropriate configuration settings.
$ curl –X PUT -H "Content-Type: application/json" \--data ‘{“name”: “SourceTest1”, “config”: {"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector","topic": "vtest1","file": "/opt/confluent/etc/kafka/connect-distributed.properties"}}’\$CWURL/connectors$ curl $CWURL/connectors["SourceTest1”]$ kafka-console-consumer --new-consumer \--bootstrap-server mycluster-broker-1:9092 \--from-beginning --topic vtest1 \--max-messages 10$ # The first 10 lines of connect-distributed.properties will
$ # be displayed since the SourceConnector was configured to$ # send each line from that file to the “vtest1” topic.

This example was for the simplest of source connectors, just dumping the contents of a file into the
Kafka topic created above. Other connectors will have more complex configurations. Please see
the documentation on Kafka Connectors for more details on building data pipelines in yourConfluent Platform deployment.

© 2016–2017 Confluent, Inc.

| 16

D. Accessing the Control Center
Confluent Control Center is accessed via a web interface. By default, Control Center isdeployed on worker node 1 of your deployment. Identify the public IP address of that node,and use your browser to connect tohttp://<cluster>-<random>-worker-1.<azuredomain>:9021
The actual link will look something like
http://mycluster-gdbxxg64-worker-1.westus.cloudapp.azure.com:9021

From the Control Center, you can deploy instances of the available connectors and monitor the
status of data streaming through the Kafka topics. Details on the Control Center can be foundat http://docs.confluent.io/current/control-center/docs/index.html .

© 2016–2017 Confluent, Inc.

| 17

Updating the Deployment
All Confluent Enterprise services deployed within Azure are readily modifiable. Here are some common
changes that you might wish to make.
A. Change the VM size of a running instance
The Azure portal allows you to redeploy any of the instances in your Confluent topology to adifferent size should the need arise. For example, you may want to expand the memory
capacity of the worker nodes to support a greater number of deployed Connectors. The
screenshot below illustrates how to change the size for the worker node in our sample cluster(easily identified as <cluster>-worker-<n> in the list of Virtual Machines) :When changing the size of your nodes, be sure that each resize operation is complete and theConfluent services on the node are back on line before attempting a resize operation on anothernode.
B. Make additional Kafka Connectors available on the worker nodes
A wide range of Connectors is available for the Confluent deployment. Only the Connectors
supported directly by Confluent are initially deployed into your Azure cluster, but you are free to© 2016–2017 Confluent, Inc.

| 18

install your own connectors or those developed by the Kafka Connect community. To add themappropriately, you’ll need to do the following steps on each of the worker nodes in your cluster:a. Log on to the worker node as user kadminb. Create a directory for the connect jar files: /opt/confluent/share/java/kafka-connectextras. The directory MUST be named kafka-connect-<new-connector> so that theworker tasks will properly load the java classes.
c. Copy the jar file for your connectors into the newly created directoryd. Restart the Kafka Connect worker process with the commandsudo service cp-connect-service restarte. Confirm the availability of the new connector withcurl http://localhost:8083/connector-plugins | jq .
Take care to perform these steps on all the worker nodes in your cluster, as the Kafka Connect
framework will automatically provision Connector tasks on any available worker node.

© 2016–2017 Confluent, Inc.

| 19

Useful Azure Commands
The Azure command line tool (available from https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
can be used to track the infrastructure resources assigned to your Confluent deployment. In
conjunction with other shell tools (including jq), the command can be very powerful. Here are somehelpful examples.
A. List out the public IP addresses and hostnames of a cluster deployed in resource group MyRG
azure network public-ip list --json MyRG | \jq -r '.[] | "\(.ipAddress) \(.dnsSettings.domainNameLabel)"' | \awk '{split($2,a,"-"); print $1" "$2" "a[1]"-"a[3]"-"a[4]}'The ‘awk’ filter at the end of the command has the effect of stripping out the random seed stringfrom the hostnames. That randomness is necessary for the public hostnams to avoid
namespace collisions within cloudapp.azure.com, but unnecessary within the internal network.
The resulting output :13.91.58.24 confsm-zm5awowo-broker-1 confsm-broker-1
13.93.147.47 confsm-zm5awowo-broker-2 confsm-broker-2
13.91.62.4 confsm-zm5awowo-broker-3 confsm-broker-3
13.93.150.33 confsm-zm5awowo-worker-1 confsm-worker-1
13.93.147.183 confsm-gwpnqjw5-zk-1 confsm-zk-1
can be inserted directly in your local /etc/hosts file for easier access to the cluster with thestandard Kafka clients.

© 2016–2017 Confluent, Inc.

| 20

Troubleshooting
The Azure Portal allows you to view the service- and instance-level details for all the components in the
Confluent topology. This section outlines several common approaches to identifying and resolving
configuration issues in your cluster.
Newly configured connector is not available
After updating your deployment to support an additional Connector, you are unable to confirm theavailability of the connector with thecurl http://localhost:8083/connector-plugins | jq .
command. Some community connectors may contain Java classes that conflict with the connectors
distributed as part of Confluent Enterprise. Error messages in the Kafka Connect log file(/opt/confluent/logs/connectDistributed.out) will likely exist in the event of such conflicts. A goodpractice is to deploy the new connector in standalone mode before adding it to the default worker
CLASSPATH.
As discussed earlier, the Connect Worker automatically includes all the Java libraries in the/opt/confluent/share/java/kafka-connect-* directories. This enables you to isolate your connector bysimply renaming those directories and then restarting the Connect Worker service.
Confluent Control Center is unresponsive
The Control Center application is constantly aggregating metrics from all the brokers and topics in yourcluster. On rare occasions, the processing of this data can temporarily disrupt its response to web
page input (mouse clicks and keystrokes). If the situation persists for more than a few minutes, you
may need to increase the CPU and memory capacity of the worker-1 instance.

© 2016–2017 Confluent, Inc.

| 21