Package: edx-event-bus-kafka
edX Event Bus - KafkaKafka implementation for Open edX event bus.OverviewThis package implements an event bus for Open edX using Kafka.The event bus acts as a broker between services publishing events and other services that consume these events.
Implementing the event bus will allow for asynchronous messaging across services which greatly improves efficiency as we don’t have to wait for scheduled batch synchronizations.
Additionally, since the services all speak to the event bus, they are independent of one another and can still function if one service crashes.This package contains both the publishing code, which processes events into messages to send to the broker, and the consumer code,
which polls the broker using awhile Trueloop in order to turn messages back into event data to be emitted.
The actual Kafka host will be configurable.The goal for event-bus-kafka is to eventually have a flexible event bus that can be easily brought into other apps and repositories to produce and consume arbitrary topics.
Ideally, the event bus itself will also be an abstraction behind which platform maintainers can use non-Kafka implementations (Redis, Pulsar, etc.).
The documentation/ADRs may also be moved to more appropriate places as the process matures.The repository works together with the openedx/openedx-events repository to make the fully functional event bus.DocumentationTo use this implementation of the Event Bus with openedx-events, you’ll need to ensure that you include the dependencyconfluent_kafka[avro,schema-registry](seeADR 5for an explanation) and set the following Django settings:EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS: ...
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL: ...
EVENT_BUS_TOPIC_PREFIX: ...

# On the producing side:
EVENT_BUS_PRODUCER: edx_event_bus_kafka.create_producer

# On the consuming side:
EVENT_BUS_CONSUMER: edx_event_bus_kafka.KafkaEventConsumerOptional settings that are worth considering:EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMITFor manual testing, seedocs/how_tos/manual_testing.rst.Django management commands:If you need to test event production, use theproduce_eventmanagement commandTo actually run the consumer, use openedx-events’consume_eventsmanagement commandOEP-52 documentation:https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0052-arch-event-bus-architecture.html(TODO:Set up documentation)Development WorkflowOne Time Setup# Clone the repository
git clone git@github.com:openedx/event-bus-kafka.git
cd event-bus-kafka

# Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it
mkvirtualenv -p python3.8 event-bus-kafkaEvery time you develop something in this repo# Activate the virtualenv
workon event-bus-kafka

# Grab the latest code
git checkout main
git pull

# Install/update the dev requirements
make requirements

# Run the tests and quality checks (to verify the status before you make any changes)
make validate

# Make a new branch for your changes
git checkout -b <your_github_username>/<short_description>

# Using your favorite editor, edit the code to make your change.
vim …

# Run your new tests
pytest ./path/to/new/tests

# Run all the tests and quality checks
make validate

# Commit all your changes
git commit …
git push

# Open a PR and ask for review.InstallationThis library is not intended to be a direct dependency of any service. Instead, it will likely be installed by some private mechanism. Unless the platform gains new processes for installing private dependencies, upgrades will need to be manually managed via a pin.LicenseThe code in this repository is licensed under the AGPL 3.0 unless
otherwise noted.Please seeLICENSE.txtfor details.How To ContributeContributions are very welcome.
Please readHow To Contributefor details.The pull request description template should be automatically applied if you are creating a pull request from GitHub. Otherwise you
can find it atPULL_REQUEST_TEMPLATE.md.The issue report template should be automatically applied if you are creating an issue on GitHub as well. Otherwise you
can find it atISSUE_TEMPLATE.md.Reporting Security IssuesPlease do not report security issues in public. Please emailsecurity@openedx.org.Getting HelpIf you’re having trouble, we have discussion forums athttps://discuss.openedx.orgwhere you can connect with others in the community.Our real-time conversations are on Slack. You can request aSlack invitation, then join ourcommunity Slack workspace.For more information about these options, see theGetting Helppage.Change LogUnreleased[5.6.0] - 2024-01-25ChangedAdded client.id to base configuration.[5.5.0] - 2023-09-21ChangedReset edx-django-utils RequestCache before handling each event[5.4.0] - 2023-08-28ChangedChanged ordering of certain context assignments in producer code.Adds custom exceptions for producing and consuming errors.[5.3.1] - 2023-08-10FixedConstrain openedx-events version to at least 8.5.0 (for EventsMetadata serialization support)[5.3.0] - 2023-08-08ChangedAdded event_metadata_as_dict to ProducingContext for easier replay from logs[5.2.0] - 2023-08-03ChangedAdded event_data_as_dict to ProducingContext for easier replay from logs[5.1.0] - 2023-05-17ChangedReconfigured serializers to use topic_record_name_strategy, allowing multiple event types per topic[5.0.0] - 2023-05-17ChangedBREAKING CHANGE: Removed deprecatedsignalargument from consumer initialization[4.0.1] - 2023-05-10FixedAddedsignalback as an argument when creating a consumer for compatibility with the openedx-events API[4.0.0] - 2023-05-10ChangedImplement openedx-events Event Bus Consumer API.BREAKING CHANGE: Removeconsume_eventsmanagement command as this command will be provided by openedx_events. To replay events using the
openedx-events version of the management command, pass--extra'{"offset_time":"2023-01-08T06:46:22"}'instead of-o2023-01-08T06:46:22.BREAKING Change:offset_timestampargument has been removed fromconsume_indefinitelyandreset_offsets_and_sleep_indefinitelymethods.
It is now added as an optional argument namedoffset_timetoKafkaEventConsumerconstructor.[3.10.0] - 2023-05-05ChangedSwitch fromedx-sphinx-themetosphinx-book-themesince the former is
deprecatedRefactored consumer to manually deserialize messages instead of using DeserializingConsumerMake signal argument optional in consumer command (take signal from message headers)[3.9.6] - 2023-02-24AddedAdd function tracing to the event consumption workflow for better monitoring.[3.9.5] - 2023-02-24Fixedconsume_eventscommand now loads all public signals so that the consumer can load signals byevent_typeeven if their modules were not already imported[3.9.4] - 2023-02-16Fixedproduce_event management command fixed to pass required events_metadata parameter[3.9.3] - 2023-02-10FixedIncludemessage_idin audit log when message is produced (wasNone)[3.9.2] - 2023-02-08FixedAdded documentation to all Django settings used in consumer and producer[3.9.1] - 2023-02-07There was no version 3.9.0, due to a release issue. (Ignore anyv3.9.0tag.)AddedAdded logging of successful message production, consumption, and processing (not just errors)[3.8.1] - 2023-02-03ChangedRequire and use openedx-events 5.0.0, which has a breaking API change that affects event bus consumers.[3.8.0] - 2023-01-31AddedProducer now passes all metadata fields as headersConsumer emits events with the original metadata information (from the producer)[3.7.1] - 2023-01-31FixedConsumer management command exits with useful error message if confluent-kafka library not available.[3.7.0] - 2023-01-30ChangedAddedreset_offsets_and_sleep_indefinitelymethod to consumer; relying code should switch to calling this when an offset timestamp is set.Deprecated theoffset_timestampparameter on the consumer’sconsume_indefinitelymethod, sincereset_offsets_and_sleep_indefinitelyshould be used instead.[3.6.3] - 2023-01-27FixedIn the consumer loop, open new database connection if the old one has closed before processing messages[3.6.2] - 2023-01-26FixedReset mode now commits the correct offsets to Kafka.[3.6.1] - 2023-01-20FixedAdded a poll call to force resets to be processed during replay/offset-reset mode.[3.6.0] - 2023-01-06ChangedConsumers do not consume events after resetting offsets.[3.5.1] - 2023-01-06FixedConvert timestamps to millisecond offsets as expected by the Kafka API when replaying by timestamp in the consumer[3.5.0] - 2023-01-05AddedNew settingEVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMITwill terminate the consumer if too many errors occur in a row, supporting an automated version of “have you tried turning it off and on again” (as long as consumer will automatically be restarted e.g. by Kubernetes).[3.4.1] - 2022-12-20FixedFixed bugs in the event replay/offset handling code for consumers.[3.4.0] - 2022-12-16ChangedKill infinite consumer loop when we see a fatal KafkaError, as recommended in the documentation. Seehttps://github.com/confluentinc/librdkafka/blob/e0b9e92a0b492b5b1a6f1bcf08744928d45bf396/INTRODUCTION.md#fatal-consumer-errors.[3.3.0] - 2022-12-15ChangedAdded/removed some custom attributes used for monitoring. Search for custom_attribute_name annotations for details.[3.2.0] - 2022-12-14ChangedAdd timestamp parameter for consumer, allowing the starting offset for consuming to be overridden from the default.[3.1.0] - 2022-12-07AddedA variety of custom attributes are now set for monitoring purposes. Search for custom_attribute_name annotations for details.ChangedError recording to the log will now include message details when the message is available on the error.[3.0.0] - 2022-12-02ChangedBREAKING CHANGE: Make event_metadata parameter required[2.1.0] - 2022-12-01ChangedSet CloudEvent headers on events using an optional event_metadata parameter[2.0.0] - 2022-11-28ChangedImplement openedx-events Event Bus Producer APIBREAKING CHANGE: Remove caching fromget_producerand rename tocreate_producer, as we now rely on the wrapper in openedx-events to cache that callUpgrading library from 1.x:Replace calls toedx_event_bus_kafka.get_producerwithopenedx_events.event_bus.get_producerAdd Django settingEVENT_BUS_PRODUCER = "edx_event_bus_kafka.create_producer"These breaking changes are only relevant for the producing side. (This should only include the CMS at the moment.)[1.10.0] - 2022-11-21ChangedImprove receiver error log message – mention that stack traces are elsewhere in log[1.9.0] - 2022-11-15ChangedLog and record receiver errors the same way as other errors (with offset, partition, etc.)[1.8.1] - 2022-11-10ChangedCommit consumer offset asynchronously[1.8.0] - 2022-11-09AddedConsumer logs a warning for receivers that fail with an exception[1.7.0] - 2022-11-04ChangedManually manage commits instead of using auto-commit on the consumerCatch Exception instead of BaseException on both producer and consumer[1.6.0] - 2022-11-04ChangedEnhanced error logging in consumer, including telemetry for exceptionsConsumer loop will no longer exit when an error is encountered[1.5.0] - 2022-11-01ChangedLog full event data on all producer errors[1.4.3] - 2022-10-31FixedUpgrade openedx-events and fastavro to bring in a fix for schema creation[1.4.2] - 2022-10-31FixedRemoved proof-of-concept code that logged user-login events[1.4.1] - 2022-10-28FixedCorrect and clarify management command help strings (some copy-paste errors)Update TODO comments[1.4.0] - 2022-10-21ChangedRemove override of auto.offset.reset on consumer (which will default to “latest”). New consumer groups will consume only messages that are sent after the group was initialized.Remove redundant lookup of signal in consumer loop (should not have any effect)Explicitly encode message header values as UTF-8 (no change in behavior)[1.3.0] - 2022-10-20ChangedUpgrade openedx-events. When AvroSignalSerializer gets event schemas, it will get whatever is currently defined in openedx-events, so this will update the COURSE_CATALOG_EVENT_CHANGED schema (droppingeffortfield)[1.2.0] - 2022-10-13ChangedEVENT_BUS_KAFKA_CONSUMERS_ENABLEDnow defaults to True instead of FalseRemoved manual monitoring since New Relic tracks these now.[1.1.0] - 2022-10-06ChangedAdded monitoring for consumption tasks.[1.0.0] - 2022-10-03ChangedFixed bug in schema registry that was sending schemas to the wrong topicBump version to 1.x to acknowledge that this is in use in production[0.7.0] - 2022-09-08ChangedBreaking changesEventProducerKafkais nowKafkaEventProducerKafkaEventConsumer is now part of the public API[0.6.2] - 2022-09-08AddedTopic names can be autoprefixed by settingEVENT_BUS_TOPIC_PREFIX[0.6.1] - 2022-09-06AddedProducer now polls on an interval, improving callback reliability. Configurable withEVENT_BUS_KAFKA_POLL_INTERVAL_SEC.[0.6.0] - 2022-09-01ChangedBreaking change: Public API is now defined inedx_event_bus_kafkapackage andedx_event_bus_kafka.management.commandspackage; all other modules should be considered unstable and not for external use.[0.5.1] - 2022-08-31FixedVarious lint issues (and missing__init__.pyfiles.)[0.5.0] - 2022-08-31ChangedBreaking changesin the producer module, refactored to expose a better API:Rather thansend_to_event_bus(...), relying code should now callget_producer().send(...).Thesynckwarg is gone; to flush and sync messages before shutdown, callget_producer().prepare_for_shutdown()instead.Clarify that config module is for internal use only.Implementation changes: Only a single Producer is created, and is used for all signals.[0.4.4] - 2022-08-26FixedFixed bug in test module for when confluent-kafka isn’t present[0.4.3] - 2022-08-24FixedNever evict producers from cache. There wasn’t a real risk of this, but now we can rely on them being long-lived. Addresses remainder ofhttps://github.com/openedx/event-bus-kafka/issues/16.[0.4.2] - 2022-08-24FixedProperly load auth settings for producer/consumer. (Auth settings were ignored since 0.3.1.)[0.4.1] - 2022-08-18ChangedRemove confluent-kafka as a formal dependency of the repository.Note: This library will not work without confluent-kafka.Add an ADR to explain why this work was done.[0.4.0] - 2022-08-15ChangedRename settings to have consistent prefix.KAFKA_CONSUMERS_ENABLEDbecomesEVENT_BUS_KAFKA_CONSUMERS_ENABLEDCONSUMER_POLL_TIMEOUTbecomesEVENT_BUS_KAFKA_CONSUMER_POLL_TIMEOUTUpdates to documentation and tests for various settings previously renamed[0.3.1] - 2022-08-11ChangedRefactored consumer to use common configuration.[0.3.0] - 2022-08-10ChangedMoved configuration onto separate file.Updated configuration settings to have EVENT_BUS_KAFKA prefix.[0.2.0] - 2022-08-09FixedCache producers so that they don’t lose data.[0.1.0] - 2022-06-16AddedFirst release on PyPI.