Package: esque
esque - an operational Kafka toolIn the Kafka world nothing is easy, butesque(pronouncedesk) is an attempt at it.esqueis a user-centric command line interface for Kafka administration.Why should you care?Some stuff is hard, and that is okay, but listing your kafka topics shouldn't be.While adopting kafka at real.digital we noticed the immense entry barrier it poses to newcomers.
We can't recount how often we wrote Slack messages asking for the script to check the
status of topics or consumer groups. This is partly (but not only) due to a
fragmented and unclear definition of tooling and APIs for kafka.
In a wide array of administration tools,esquedistances itself by striving to provide Kafka Ops for Humans, in a usable and natural way.We feel that the goal ofesqueembodies the principle: “keep easy things easy, and make hard things possible”.Principlesbatteries includedfeature richrobustinsightfulby engineers for engineersFeature OverviewSupport for any type of Kafka deployment >1.2Display Resources (Topics, Consumer Groups, Brokers)Get detailed Overviews of Resources (Topics, Consumer Groups, Brokers)Create/Delete TopicsEdit Topic ConfigurationsEdit Consumer Offset for TopicsSASL/SSL Support out of the boxConsume and Produce to and from Avro and Plaintext Topics (including Avro Schema Resolution from Schema Registry)Context Switch (Easily Switch between pre-defined Clusters)Kafka Ping (Test roundtrip time to your kafka cluster)Command Overview$esque
Usage:esque[OPTIONS]COMMAND[ARGS]...esque-anoperationalkafkatool.IntheKafkaworldnothingiseasy,butesque(pronouncedesk)isanattemptatit.

Options:--versionShowtheversionandexit.-v,--verboseReturnstacktraceonerror.--no-verifySkipallverificationdialogsandanswerthemwithyes.--helpShowthismessageandexit.

Commands:applyApplyasetoftopicconfigurations.configConfiguration-relatedoptions.consumeConsumemessagesfromatopic.createCreateanewinstanceofaresource.ctxListcontextsandswitchbetweenthem.deleteDeletearesource.describeGetdetailedinformationaboutaresource.editEditaresource.getGetaquickoverviewofdifferentresources.ioRunamessagepipeline.pingTesttheconnectiontothekafkacluster.produceProducemessagestoatopic.setSetresourceattributes.urlencodeUrl-encodethegivenvalue.Installation and UsageInstallationesqueis available atpypi.organd can be installed withpip install esque.esquerequires Python 3.8+ to run (Python 3.11 is not yet supported).Installation on Alpine LinuxThere are no wheels for Alpine Linux, soesquerequires a few extra dependencies to build them during installation.apkaddpython3-devpy3-piplibrdkafkalibrdkafka-devg++Installation on Apple SiliconThe installation for Kafka is slightly different for Apple Silicon devices, so simply runningpip install esquemay
result in errors.The fix for this is to first install the librdkafka library withbrew install librdkafka(make note of which version is installed).
Then, add the following to the.zshrcfile with the correct version of librdkafka:exportC_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/X.X.X/include/exportLIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/X.X.X/lib/After starting a new shell (orsource ~/.zshrc), you will be able to installesquenormally with pip.AutocompletionThe autocompletion scripts forbashandzshcan be generated by runningesque config autocomplete.UsageConfig DefinitionWhen startingesquefor the first time the following message will appear:Noconfigprovidedin~/.esque
Shouldasamplefilebecreatedin~/.esque[y/N]:When answering withyesquewill copy over thesample configto~/.esque/esque_config.yaml.
Afterwards you can modify that file to fit your cluster definitions.Alternatively might just provide a config file following the sample config's file in that path.Config Exampleversion:1current_context:localcontexts:# This context corresponds to a local development cluster# created by docker-compose when running esque from the host machine.local:bootstrap_servers:-localhost:9092security_protocol:PLAINTEXTschema_registry:http://localhost:8081default_values:num_partitions:1replication_factor:1Config file for "apply" commandThe config for the apply command has to be a yaml file and
is given with the option -f or --file.In the current version only topic configurations can be
changed and specified.It has to use the same schema, which is used
for the following example:topics:-name:topic_onereplication_factor:3num_partitions:50config:cleanup.policy:compact-name:topic_tworeplication_factor:3num_partitions:50config:cleanup.policy:compactDevelopmentTo setup your development environment, make sure you have at least Python 3.8 &poetryinstalled, then runpoetryinstall
poetryshellPre Commit HooksTo install pre commit hooks run:pipinstallpre-commit
pre-commitinstall
pre-commitinstall-hooksRun testsIntegration Testsesque comes with a docker-compose based kafka stack which you can start up withmake test-suite.You can then run the integration tests against this stack withpytest tests/ --integration --local.Alternatively you can go the fast way and just run the whole stack + integration tests in docker:makeintegration-testUnit TestsIf you only want the unit tests, just run:maketestAlternativesLinkedIn KafkaToolsPyKafka ToolsOfficial Kafka Scriptskafkacatkaf