Package: kashpy
kash.pykash.pyis a Kafka shell based on Python, or, in other words, a Python-based client library for Kafka based onconfluent-kafka-pythonby Magnus Edenhill, which is itself based on the native Kafka client librarylibrdkafkaby the same author.The idea behindkash.pyis to make it as easy as possible to interact with Kafka using Python, without having to know any of the implementation details of the underlyingconfluent-kafka-pythonmodule. To this end, not only are the functions/methods are simpler to use, but also all classes are converted to simple Python types like tuples and dictionaries. As a result, development chores that took numerous of lines of boilerplate code before can be formulated as one-liners, both in interactive mode using the Python REPL, or within convenient scripts.kash.pyhas been built for Kafka users of all kinds:Fordevelopers and devops engineersto view and manipulate Kafka topics using familiar shell syntax (you havels,touch,rm,cp,cat,grep,wcetc.).Fordata scientiststo bridge the gap between batch and stream processing, using functions to upload/download files to/from topics, and even functional abstractions a la Databricks/Apache Spark (there are variousfoldls,flatmaps andmaps for you to explore).kash.pysupportsAvro,ProtobufandJSONSchema, Confluent Cloud, Redpanda, etc...and it will give youKafka superpowersof a kind you have never experienced before. Honestly :)Check out the fullkashpy package documentationif you are interested in seeing the entire functionality ofkash.py.InstallationJust write...pip install kashpy...and off you go.Configurationkash.pymakes use of configuration files suffixed.yamlwhich are being searched for in the folderclusters, starting 1) from the directory in theKASHPY_HOMEenvironment variable, or, if that environment variable is not set, 2) from the current directory.For the YAML-based configuration files sporting environment variable interpolation,kash.pymakes use of the ingenious little libraryPinyfrom Vitaly Samigullin (see also his illustrativeblogabout the genesis of Piny).A barebones configuration file looks like this (including Schema Registry):kafka:
  bootstrap.servers: localhost:9092

schema_registry:
  schema.registry.url: http://localhost:8081You can also set some of the defaults ofkash.pyin thekashsection like this:kash:
  flush.num.messages: 10000
  flush.timeout: -1.0
  retention.ms: -1
  consume.timeout: 1.0
  auto.offset.reset: earliest
  enable.auto.commit: true
  session.timeout.ms: 10000
  progress.num.messages: 1000
  block.num.retries.int: 50
  block.interval: 0.1You can find an in-depth explanation of these settings in thekashpy package documentation, including example configuration files for connecting toConfluent Cloud,Redpandaetc.Kafka Made SimpleFor interactive use, e.g. using your local cluster configured in the fileclusters/local.yaml, just do the following to list the topics:$ python3
>>> from kashpy.kash import *
>>> c = Cluster("local")
>>> c.ls()
['__consumer_offsets', '_schemas']
>>>Or, if you'd like to replicate the topictestholding 1000 messages from a Kafka clusterlocalon your local machine to a Kafka clusterccloudon Confluent Cloud:>>> c_local = Cluster("local")
>>> c_ccloud = Cluster("ccloud")
>>> c.cp(c_local, "test", c_ccloud, "test")
(1000, 1000)
>>>In the following, we go a bit deeper in two short tutorials; the first demonstrating howkash.pyhelps you to fulfill tasks on a single cluster, and the second how to make use of your newly obtained Kafka superpowers across clusters.Tutorial 1 (single cluster)This is the first tutorial, showcasing the single cluster capabilities ofkash.pyin interactive mode.Let's start Python, import kash.py and create aClusterobjectc:$ python3
>>> from kashpy.kash import *
>>> c = Cluster("local")
>>>List the topics on the cluster:>>> c.ls()
['__consumer_offsets', '_schemas']
>>>Create a new topicsnacks:>>> c.touch("snacks")
'snacks'
>>>List the topics on the cluster again:>>> c.ls()
['__consumer_offsets', '_schemas', 'snacks']
>>>Upload the following local filesnacks.txtto the topicsnacks(examples inspired by the great blogKafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schemaby Simon Aubury):{"name": "cookie", "calories": 500.0, "colour": "brown"}
{"name": "cake", "calories": 260.0, "colour": "white"}
{"name": "timtam", "calories": 80.0, "colour": "chocolate"}>>> c.cp("./snacks.txt", "snacks")
(3, 3)
>>>Show the contents of topicsnacks:>>> c.cat("snacks")
{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1678915066323), 'key': None, 'value': '{"name": "cookie", "calories": 500.0, "colour": "brown"}'}
{'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1678915066323), 'key': None, 'value': '{"name": "cake", "calories": 260.0, "colour": "white"}'}
{'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1678915066323), 'key': None, 'value': '{"name": "timtam", "calories": 80.0, "colour": "chocolate"}'}
3
>>>If you'd like to see the output of the values of the messages in an indented fashion, you can tellkash.py1) that you'd like to pretty print the messages and 2) that the values have the typejson:>>> c.cat("snacks", foreach_function=ppretty, value_type="json")
{
  "headers": null,
  "partition": 0,
  "offset": 0,
  "timestamp": [
    1,
    1678915066323
  ],
  "key": null,
  "value": {
    "name": "cookie",
    "calories": 500.0,
    "colour": "brown"
  }
}
{
  "headers": null,
  "partition": 0,
  "offset": 1,
  "timestamp": [
    1,
    1678915066323
  ],
  "key": null,
  "value": {
    "name": "cake",
    "calories": 260.0,
    "colour": "white"
  }
}
{
  "headers": null,
  "partition": 0,
  "offset": 2,
  "timestamp": [
    1,
    1678915066323
  ],
  "key": null,
  "value": {
    "name": "timtam",
    "calories": 80.0,
    "colour": "chocolate"
  }
}
3
>>>Count the number of messages, words and bytes of the topicsnacks:>>> c.wc("snacks")
(3, 18, 169)
>>>Find those messages whose values matches the regular expression.*cake.*:>>> c.grep("snacks", ".*cake.*")
Found matching message on partition 0, offset 1.
([{'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cake", "calories": 260.0, "colour": "white"}'}], 1, 3)
>>>Filter the messages to only keep those where thecolourisbrown:>>> c.filter("snacks", lambda x: x["value"]["colour"] == "brown", value_type="json")
([{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1666090118747), 'key': None, 'value': {'name': 'cookie', 'calories': 500.0, 'colour': 'brown'}}], 3)
>>>Create a new topicsnacks_protobuf:>>> c.touch("snacks_protobuf")
'snacks_protobuf'
>>>Copy the topicsnacksonto another topicsnacks_protobufusing Protobuf (and storing the schema in the Schema Registry):>>> c.cp("snacks", "snacks_protobuf", target_value_type="protobuf", target_value_schema='message Snack { required string name = 1; required float calories = 2; optional string colour = 3; }')
(3, 3)
>>>Show the contents of topicsnacks_protobuf(showing the values directly asbytes):>>> c.cat("snacks_protobuf", value_type="bytes")
{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': b'\x00\x00\x00\x00\x03\x00\n\x06cookie\x15\x00\x00\xfaC\x1a\x05brown'}
{'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': b'\x00\x00\x00\x00\x03\x00\n\x04cake\x15\x00\x00\x82C\x1a\x05white'}
{'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': b'\x00\x00\x00\x00\x03\x00\n\x06timtam\x15\x00\x00\xa0B\x1a\tchocolate'}
>>>Show the contents of the topicsnacks_protobufagain (decoding the values using Protobuf and the Schema Registry):>>> c.cat("snacks_protobuf", value_type="protobuf")
{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cookie', 'calories': 500.0, 'colour': 'brown'}}
{'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cake', 'calories': 260.0, 'colour': 'white'}}
{'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'timtam', 'calories': 80.0, 'colour': 'chocolate'}}
3
>>>Again,kash.pycan give you a prettified indented output:>>> c.cat("snacks_protobuf", foreach_function=ppretty, value_type="protobuf")
{
  "headers": null,
  "partition": 0,
  "offset": 0,
  "timestamp": [
    1,
    1678915066323
  ],
  "key": null,
  "value": {
    "name": "cookie",
    "calories": 500.0,
    "colour": "brown"
  }
}
{
  "headers": null,
  "partition": 0,
  "offset": 1,
  "timestamp": [
    1,
    1678915066323
  ],
  "key": null,
  "value": {
    "name": "cake",
    "calories": 260.0,
    "colour": "white"
  }
}
{
  "headers": null,
  "partition": 0,
  "offset": 2,
  "timestamp": [
    1,
    1678915066323
  ],
  "key": null,
  "value": {
    "name": "timtam",
    "calories": 80.0,
    "colour": "chocolate"
  }
}
3
>>>Get a diff of the two topicssnacksandsnacks_protobuf, comparing the dictionaries obtained by converting the string payload insnacksto Python dictionaries, and the Protobuf payload insnacks_protobufto Python dictionaries as well:>>> c.diff("snacks", "snacks_protobuf", value_type1="json", value_type2="protobuf")
([], 3, 3)
>>>Now we are getting functional - using afoldloperation to sum up the calories of the messages insnacks:>>> c.foldl("snacks", lambda acc, x: acc + x["value"]["calories"], 0, value_type="json")
(840.0, 3)
>>>We can also use aflatmapoperation to get a list of all messages insnackswhere each message is duplicated:>>> c.flatmap("snacks", lambda x: [x, x])
([{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cookie", "calories": 500.0, "colour": "brown"}'}, {'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cookie", "calories": 500.0, "colour": "brown"}'}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cake", "calories": 260.0, "colour": "white"}'}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cake", "calories": 260.0, "colour": "white"}'}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "timtam", "calories": 80.0, "colour": "chocolate"}'}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "timtam", "calories": 80.0, "colour": "chocolate"}'}], 3)
>>>Next, we use amapoperation to add the suffixishto all the colours:>>> def map_function(x):
...   x["value"]["colour"] += "ish"
...   return x
... 
>>> c.map("snacks", map_function, value_type="json")
([{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cookie', 'calories': 500.0, 'colour': 'brownish'}}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cake', 'calories': 260.0, 'colour': 'whiteish'}}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'timtam', 'calories': 80.0, 'colour': 'chocolateish'}}], 3)
>>>And last, but not least, we copy the topicsnacksback to a local filesnacks1.txtwhile both duplicating the messages and adding the suffixishto all the colours:>>> def flatmap_function(x):
...   x["value"]["colour"] += "ish"
...   return [x, x]
... 
>>> c.flatmap_to_file("snacks", "./snacks1.txt", flatmap_function, value_type="json")
(3, 6)
>>>The resulting filesnacks1.txtlooks like this:{"name": "cookie", "calories": 500.0, "colour": "brownish"}
{"name": "cookie", "calories": 500.0, "colour": "brownish"}
{"name": "cake", "calories": 260.0, "colour": "whiteish"}
{"name": "cake", "calories": 260.0, "colour": "whiteish"}
{"name": "timtam", "calories": 80.0, "colour": "chocolateish"}
{"name": "timtam", "calories": 80.0, "colour": "chocolateish"}Tutorial 2 (cross-cluster)This is the second tutorial, showcasing the cross-cluster capabilities ofkash.pyin interactive mode.First, let's start Python, import kash.py and let's see what clusters we have defined in ourclustersdirectory:$ python3
>>> from kashpy.kash import *
>>> clusters()
['ccloud', 'local', 'redpanda']Next, we create aClusterobjectc1:>>> c1 = Cluster("local")
>>>Create a new topicsnacks1on cluster c1:>>> c1.touch("snacks1")
'snacks1'
>>>Upload the following local filesnacks.txtto the topicsnacks1on cluster c1:{"name": "cookie", "calories": 500.0, "colour": "brown"}
{"name": "cake", "calories": 260.0, "colour": "white"}
{"name": "timtam", "calories": 80.0, "colour": "chocolate"}>>> c1.cp("./snacks.txt", "snacks1")
(3, 3)Create a newClusterobjectc2:>>> c2 = Cluster("ccloud")
>>>Create a new topicsnacks2onc2:>>> c2.touch("snacks2")
'snacks2'
>>>Copy the topicsnacks1from clusterc1to topicsnacks2on clusterc2:>>> cp(c1, "snacks1", c2, "snacks2")
(3, 3)
>>>Get the diff from topicsnacks1on clusterc1to a topicsnacks2on clusterc2:>>> diff(c1, "snacks1", c2, "snacks2")
([], 3, 3)
>>>Now let's venture into the cross-cluster functional programming domain... let's copy topicsnacks1on clusterc1to a topicsnacks2_duplicateon clusterc2, while duplicating each message using aflatmapoperation:>>> flatmap(c1, "snacks1", c2, "snacks2_duplicate", lambda x: [x, x])
(3, 6)
>>>Next, we use a cross-clustermapoperation to add the suffixishto all the colours and produce the resulting messages to topicsnacks2_ishon clusterc2:>>> def map_function(x):
...   x["value"]["colour"] += "ish"
...   return x
... 
>>> map(c1, "snacks1", c2, "snacks2_ish", map_function, source_value_type="json")
(3, 3)
>>>Now for the most advanced operation of the tutorials: Azipof topicsnacks1on clusterc1and topicsnacks2_ishon clusterc2followed by afoldlaccumulating pairs of those messages from both topics where the colour of the message from topicsnacks2_ishends witheish:>>> def zip_foldl_function(acc, x1, x2):
...   return acc + [(x1, x2)] if x2["value"]["colour"].endswith("eish") else acc
... 
>>> zip_foldl(c1, "snacks1", c2, "snacks2_ish", zip_foldl_function, [], value_type2="json")
([({'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': b'{"name": "cake", "calories": 260.0, "colour": "white"}'}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cake', 'calories': 260.0, 'colour': 'whiteish'}}), ({'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': b'{"name": "timtam", "calories": 80.0, "colour": "chocolate"}'}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'timtam', 'calories': 80.0, 'colour': 'chocolateish'}})], 3, 3)
>>>kash.py and Functional ProgrammingAs already hinted at in the tutorials above, the more advanced functionality ofkash.pyrests on aFunctional Programming(FP) backbone. If you intend to usekash.pyto ease your everyday tasks, you don't have to be aware of that - the bash-like abstractions on top of the functional backbone should already help you out a lot. However, if you are familiar with FP or you would like to become familiar with it, this functional backbone ofkash.pycan actually let you wield even strongerKafka superpowers.Single ClusterKafka Topic to Python DatatypeWe start with functional backbone of those functions which consume a Kafka topic and return a Python datatype. Those functions are based on thefoldl("fold left") function.foldlconsumes messages from a topic and then, for each message, calls a function that takes this message and an accumulator of any type, and returns the updated accumulator.foldlcan be used e.g. to aggregate information from topics, such as the sum of calories of the messages of the topicsnacksin the following example:>>> c.foldl("snacks", lambda acc, x: acc + x["value"]["calories"], 0, value_type="json")
(840.0, 3)
>>>foldlis the basis forflatmapwhich also consumes messages from a topic, but then calls a simpler function that takes just this message and returns a list of anything (including the empty list).flatmapcan be used e.g. to duplicate the messages of a topic, as in the following example:>>> c.flatmap("snacks", lambda x: [x, x])
([{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cookie", "calories": 500.0, "colour": "brown"}'}, {'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cookie", "calories": 500.0, "colour": "brown"}'}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cake", "calories": 260.0, "colour": "white"}'}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "cake", "calories": 260.0, "colour": "white"}'}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "timtam", "calories": 80.0, "colour": "chocolate"}'}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': '{"name": "timtam", "calories": 80.0, "colour": "chocolate"}'}], 3)
>>>flatmap, in turn, is the basis formapwhich again consumes messages from a topic, but then calls a function that takes just this message and returns anything.mapcan be used e.g. to modify the messages of a topic, as in the following example:>>> def map_function(x):
...   x["value"]["colour"] += "ish"
...   return x
... 
>>> c.map("snacks", map_function, value_type="json")
([{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cookie', 'calories': 500.0, 'colour': 'brownish'}}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cake', 'calories': 260.0, 'colour': 'whiteish'}}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'timtam', 'calories': 80.0, 'colour': 'chocolateish'}}], 3)
>>>Filter the messages to only keep those where thecolourisbrown:>>> c.filter("snacks", lambda x: x["value"]["colour"] == "brown", value_type="json")
([{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1666090118747), 'key': None, 'value': {'name': 'cookie', 'calories': 500.0, 'colour': 'brown'}}], 3)
>>>Here is an overview of the relations between those functions ofkash.pywhich consume messages from Kafka topics and return a Python datatype (or nothing):foldlflatmap,cat,foreach,wcmap,filter,grep,grep_funAs explained above,foldlis the basis of it all,flatmapis based onfoldlandmapis based onflatmap. In addition,cat,foreachandwcare also based onfoldl, andfilterandgrepandgrep_funare based onflatmap.Kafka Topic to Local FileWe turn to those functions consuming messages from a Kafka topic and writing them to a local file, transforming or even removing messages. Here, the basis function isflatmap_to_file. Here is an example where the flatmap function is the identity function. Hence, the messages from the topicsnacksare written to the local file./snacks.txt:>>> c.flatmap_to_file("snacks", "./snacks.txt", lambda x: [x])
(3, 3)
>>>map_to_fileis based onflatmap_to_fileand can be used to transform each message before it is written to the local file:>>> def map_function(x):
...   x["value"]["colour"] += "ish"
...   return x
... 
>>> c.map_to_file("snacks", "./snacks.txt", map_function)
(3, 3)
>>>filter_to_fileis also based onflatmap_to_fileand writes only those messages to the local file which fulfil a condition, for example:>>> def filter_function(message_dict):
...   return "cake" in message_dict["value"]
... 
>>> c.filter_to_file("snacks", "./snacks.txt", filter_function)
(3, 1)Here is an overview of the relations between those functions ofkash.pywhich consume messages from Kafka topics and write them to a local file:flatmap_to_filemap_to_file,filter_to_file,cp/downloadAgain, as explained above,map_to_fileandfilter_to_fileare based onflatmap_to_file, as well ascp(with the first not being a file and the second argument being a file marked by it containing a slash/) anddownload.Local File to Kafka TopicThe last set of functional functions are those reading lines/messages from a local file and producing them to a Kafka topic. Here, the basis function isflatmap_from_file. Here is an example where the flatmap function is the identity function. Hence, the messages from the local file./snacks.txtare produced into the topicsnacks:>>> c.flatmap_from_file("./snacks.txt", "snacks", lambda x: [x])
(3, 3)
>>>map_from_fileis based onflatmap_from_fileand can be used to transform each line/message before it is written to the local file:>>> c.map_to_file("snacks", "./snacks.txt", lambda x: x)
(3, 3)
>>>filter_from_fileis also based onflatmap_from_fileand produces only those messages to the Kafka topic which fulfil a condition. In the example below we produce only those messages which have the stringcakein their value:>>> def filter_function(xy):
...   x, y = xy
...   return "cake" in y
... 
>>> c.filter_from_file("./snacks.txt", "snacks", filter_function)
(3, 1)
>>>Here is an overview of the relations between those functions ofkash.pywhich read lines/messages from a file and produce them to a Kafka topic:flatmap_from_filemap_from_file,filter_from_file,cp/downloadAgain, as explained above,map_from_fileandfilter_from_fileare based onflatmap_from_file, as well ascp(with the first argument being a file marked by it containing a slash/and the second not a file) andupload.There is also a functionfoldl_from_file. Here is an example where we use it to sum up the calories of the lines/messages from the file./snacks.txt:>> foldl_from_file("./snacks.txt", lambda acc, x: acc + json.loads(x)["calories"], 0)
(840.0, 3)
>>>Cross-ClusterKafka Topic to Kafka Topickash.pyalso offers functional functions to use across clusters, based onflatmap. In the example below, we useflatmapto copy the topicsnacks1on clusterc1to a topicsnacks2_duplicateon clusterc2, while duplicating each message using aflatmapoperation:>>> flatmap(c1, "snacks1", c2, "snacks2_duplicate", lambda x: [x, x])
(3, 6)
>>>mapis based onflatmap. In the following example, we use a cross-clustermapto add the suffixishto all the colours and produce the resulting messages to topicsnacks2_ishon clusterc2:>>> def map_function(x):
...   x["value"]["colour"] += "ish"
...   return x
... 
>>> map(c1, "snacks1", c2, "snacks2_ish", map_function, source_value_type="json")
(3, 3)
>>>The functionfilteris also based onflatmapand, in the example below, is used to only keep those messages where thecolourisbrown:>>> c.filter("snacks", lambda x: x["value"]["colour"] == "brown", value_type="json")
([{'headers': None, 'partition': 0, 'offset': 0, 'timestamp': (1, 1666090118747), 'key': None, 'value': {'name': 'cookie', 'calories': 500.0, 'colour': 'brown'}}], 3)
>>>Here is an overview of the relations between those cross-cluster functions ofkash.pywhich consume messages from a topic on one cluster and produce them in another (or equally named) topic on another (or the same) cluster:flatmapmap,filter,cpConsuming Two Topics At the Same TimeThediffanddiff_funfunctions are based on the functional abstractionzip_foldl. As an example,
we show thezipof topicsnacks1on clusterc1and topicsnacks2_ishon clusterc2followed by afoldlaccumulating pairs of those messages from both topics where the colour of the message from topicsnacks2_ishends witheish:>>> def zip_foldl_function(acc, x1, x2):
...   return acc + [(x1, x2)] if x2["value"]["colour"].endswith("eish") else acc
... 
>>> zip_foldl(c1, "snacks1", c2, "snacks2_ish", zip_foldl_function, [], value_type2="json")
([({'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': b'{"name": "cake", "calories": 260.0, "colour": "white"}'}, {'headers': None, 'partition': 0, 'offset': 1, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'cake', 'calories': 260.0, 'colour': 'whiteish'}}), ({'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': b'{"name": "timtam", "calories": 80.0, "colour": "chocolate"}'}, {'headers': None, 'partition': 0, 'offset': 2, 'timestamp': (1, 1664989815680), 'key': None, 'value': {'name': 'timtam', 'calories': 80.0, 'colour': 'chocolateish'}})], 3, 3)
>>>Here is the overview of the relations between those functions ofkash.pywhich consume messages from two topics (on the same cluster or on different clusters):zip_foldldiff/diff_fun