Package: edx-event-bus-redis
edx_event_bus_redisPurposeRedis Streams implementation for the Open edX event bus.OverviewThis package implements an event bus for Open EdX using Redis streams.The event bus acts as a broker between services publishing events and other services that consume these events.This package contains both the publishing code, which processes events into
messages to send to the stream, and the consumer code, which polls the stream
using awhile Trueloop in order to turn messages back into django signal to
be emitted. This django signal contains event data which can be consumed by the
host application which does the actual event handling.
The actual Redis host is configurable.The repository works together with the openedx/openedx-events repository to make the fully functional event bus.DocumentationTo use this implementation of the Event Bus with openedx-events, you’ll need to ensure that below the following Django settings are set:# redis connection url
# https://redis.readthedocs.io/en/stable/examples/ssl_connection_examples.html#Connecting-to-a-Redis-instance-via-a-URL-string
EVENT_BUS_REDIS_CONNECTION_URL: redis://:password@localhost:6379/
EVENT_BUS_TOPIC_PREFIX: dev

# Required, on the producing side only:
# https://github.com/openedx/openedx-events/blob/06635f3642cee4020d6787df68bba694bd1233fe/openedx_events/event_bus/__init__.py#L105-L112
# This will load a producer class which can send events to redis streams.
EVENT_BUS_PRODUCER: edx_event_bus_redis.create_producer

# Required, on the consumer side only:
# https://github.com/openedx/openedx-events/blob/06635f3642cee4020d6787df68bba694bd1233fe/openedx_events/event_bus/__init__.py#L150-L157
# This will load a consumer class which can consume events from redis streams.
EVENT_BUS_CONSUMER: edx_event_bus_redis.RedisEventConsumerOptional settings that are worth considering:# If the consumer encounters this many consecutive errors, exit with an error. This is intended to be used in a context where a management system (such as Kubernetes) will relaunch the consumer automatically.
EVENT_BUS_REDIS_CONSUMER_CONSECUTIVE_ERRORS_LIMIT (defaults to None)

# How long the consumer should wait for new entries in a stream.
# As we are running the consumer in a while True loop, changing this setting doesn't make much difference
# expect for changing number of monitoring messages while waiting for new events.
# https://redis.io/commands/xread/#blocking-for-data
EVENT_BUS_REDIS_CONSUMER_POLL_TIMEOUT (defaults to 60 seconds)

# Limits stream size to approximately this number
EVENT_BUS_REDIS_STREAM_MAX_LEN (defaults to 10_000)For manual local testing, seeTesting locallysection below.Getting StartedDevelopingOne Time Setup# Clone the repository
git clone git@github.com:openedx/event-bus-redis.git
cd event-bus-redis

# Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it
mkvirtualenv -p python3.8 event-bus-redisEvery time you develop something in this repo# Activate the virtualenv
workon event-bus-redis

# Grab the latest code
git checkout main
git pull

# Install/update the dev requirements
make requirements

# Run the tests and quality checks (to verify the status before you make any changes)
make validate

# Make a new branch for your changes
git checkout -b <your_github_username>/<short_description>

# Using your favorite editor, edit the code to make your change.
vim ...

# Run your new tests
pytest ./path/to/new/tests

# Run all the tests and quality checks
make validate

# Commit all your changes
git commit ...
git push

# Open a PR and ask for review.Testing locallyPlease execute below commands in virtual environment to avoid messing with
your main python installation.Install all dependencies usingmake requirementsRunmakeredis-upin current directory.Runmake consume_test_eventto start running a single consumer ormake multiple_consumer_test_eventto run two consumers with different consumer names.Runmake produce_test_eventin a separate terminal to produce a fake event, the consumer should log this event.You can also add a fake handler to test emitted signal via consumer. Add below code snippet toedx_event_bus_redis/internal/consumer.py.fromdjango.dispatchimportreceiverfromopenedx_events.content_authoring.signalsimportXBLOCK_DELETED@receiver(XBLOCK_DELETED)defdeleted_handler(sender,signal,**kwargs):print(f"""=======================================  signal:{signal}""")print(f"""=======================================  kwargs:{kwargs}""")DeployingAfter setting up required configuration, events are produced using theopenedx_events.get_producer().send()method which needs to be called from
the producing side. For more information, visit thislink.To consume events, openedx_events provides a management command calledconsume_eventswhich can be called like so:# consume events from topic xblock-statuspythonmanage.pyconsume_events--topicxblock-status--group_idtest_group--extra'{"consumer_name": "test_group.c1"}'# replay events from specific redis msg idpythonmanage.pyconsume_events--topicxblock-deleted--group_idtest_group--extra'{"consumer_name": "test_group.c1", "last_read_msg_id": "1679676448892-0"}'# process all messages that were not read by this consumer group.pythonmanage.pyconsume_events-tuser-login-guser-activity-service--extra'{"check_backlog": true, "consumer_name": "c1"}'# claim messages pending for more than 30 minutes (1,800,000 milliseconds) from other consumers in the group.pythonmanage.pyconsume_events-tuser-login-guser-activity-service--extra'{"claim_msgs_older_than": 1800000, "consumer_name": "c1"}'Note that theconsumer_namein--extraargument is required for redis
event bus as this name uniquely identifies the consumer in a group and helps
with tracking processed and pending messages.If required, you can also replay events i.e. process messages from a specific
point in history.# replay events from specific redis msg idpythonmanage.pyconsume_events--signalorg.openedx.content_authoring.xblock.deleted.v1--topicxblock-deleted--group_idtest_group--extra'{"consumer_name": "c1", "last_read_msg_id": "1684306039300-0"}'The redis message id can be found from the producer logs in the host application, example:Message delivered to Redis event bus: topic=dev-xblock-deleted, message_id=ab289110-f47e-11ed-bd90-1c83413013cb, signal=<OpenEdxPublicSignal: org.openedx.content_authoring.xblock.deleted.v1>, redis_msg_id=b'1684306039300-0'Getting HelpDocumentationPLACEHOLDER: Start by going throughthe documentation.  If you need more help see below.(TODO:Set up documentation)More HelpIf you’re having trouble, we have discussion forums athttps://discuss.openedx.orgwhere you can connect with others in the
community.Our real-time conversations are on Slack. You can request aSlack
invitation, then join ourcommunity Slack workspace.For anything non-trivial, the best path is to open an issue in this
repository with as many details about the issue you are facing as you
can provide.https://github.com/openedx/event-bus-redis/issuesFor more information about these options, see theGetting Helppage.LicenseThe code in this repository is licensed under the AGPL 3.0 unless
otherwise noted.Please seeLICENSE.txtfor details.ContributingContributions are very welcome.
Please readHow To Contributefor details.This project is currently accepting all types of contributions, bug fixes,
security fixes, maintenance work, or new features.  However, please make sure
to have a discussion about your new feature idea with the maintainers prior to
beginning development to maximize the chances of your change being accepted.
You can start a conversation by creating a new issue on this repo summarizing
your idea.The Open edX Code of ConductAll community members are expected to follow theOpen edX Code of Conduct.PeopleThe assigned maintainers for this component and other project details may be
found inBackstage. Backstage pulls this data from thecatalog-info.yamlfile in this repo.Reporting Security IssuesPlease do not report security issues in public. Please emailsecurity@openedx.org.Change LogUnreleased[0.3.3] - 2023-10-13AddedUse utility from openedx_events to reset application state before processing
event[0.3.2] - 2023-09-01AddedAdds custom exceptions for producing and consuming errors.[0.3.1] - 2023-05-24AddedOption to limit length of stream.ChangedUpdated README.[0.3.0] - 2023-05-23ChangedBREAKING CHANGE: Removed deprecatedsignalargument from consumer.[0.2.1] - 2023-05-12ChangedDeprecatedsignalargument in consumer (made optional in preparation for removal)[0.1.1] - 2023-05-12AddedOption to claim messages from other consumers based on idle time.ChangedSettingcheck_backlogwill read messages that were not read by this consumer group.[0.1.0] - 2023-05-04AddedFirst release on PyPI.Redis streams consumer and producer implemented.