Issue 1:
Apache Airflow version

2.5.0

What happened

I am trying to set up Airflow Celery to use sentinel. However, we have multiple sentinel instances up and running. So referring to the examples on the internet, I tried to pass multiple sentinel URL as below in ENV VAR.

AIRFLOW__CELERY__BROKER_URL='sentinel://redis_sentinel1:26379;sentinel://redis_sentinel2:26379' (tried with/without quote)

After executing, the worker keeps throwing errors as below:

Traceback (most recent call last):

File "", line 1, in

File "/usr/local/lib/python3.8/urllib/parse.py", line 177, in port

raise ValueError(message) from None

ValueError: Port could not be cast to integer value as '26379;sentinel:'

From the error, it looks like the parser is NOT separating sentinel URLs with a semicolon separator. Is there any workaround for it?

What you think should happen instead

Using a semicolon separator, sentinel URLs should be separated, and the Redis connection pool should make use of all the sentinel URLs.

How to reproduce

Try to pass multiple sentinel URLs to AIRFLOW__CELERY__BROKER_URL environment variables and then try to bring up the airflow workers

Operating System

ubuntu

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Apache Airflow version

2.4.3

What happened

When configuring Airflow/Celery to use Redis Sentinel as a broker, the following pops up:

airflow.exceptions.AirflowException: The broker you configured does not support SSL_ACTIVE to be True. Please use RabbitMQ or Redis if you would like to use SSL for broker.

What you think should happen instead

Celery has supported TLS on Redis Sentinel for a while now.

It looks like this piece of code explicitly prohibits from passing a valid Redis Sentinel TLS configuration through to Celery. (Since Sentinel broker URL's are prefixed with sentinel:// instead of redis://.)

How to reproduce

This problem can be reproduced by deploying Airflow using Docker with the following environment variables:

AIRFLOW__CELERY__BROKER_URL=sentinel://sentinel1:26379;sentinel://sentinel2:26379;sentinel://sentinel3:26379
AIRFLOW__CELERY__SSL_ACTIVE=true
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__MASTER_NAME='some-master-name'
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__PASSWORD='some-password'
AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG

Note that I'm not 100% certain of the syntax for the password environment var. I can't get to the point of testing this because without TLS connections to our internal brokers are denied (because they require TLS), and with TLS it doesn't attempt a connection because of the earlier linked code.

I've verified with the reference redis-cli that the settings we use for master-name does result in a valid response and the Sentinel set-up works as expected.

Operating System

Docker (apache/airflow:2.4.3-python3.10)

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

Deployed using Nomad.

Anything else

This is my first issue with this open source project. Please let me know if there's more relevant information I can provide to follow through on this issue.

I will try to make some time available soon to see if a simple code change in the earlier mentioned file would work, but as this is my first issue here I would still have to set-up a full development environment.

Are you willing to submit PR?

Yes I am willing to submit a PR!

If this is indeed a simple fix I'd be willing to look into making a PR. I would like some feedback on the problem first though if possible!

Code of Conduct

I agree to follow this project's Code of Conduct