Issue 1:
Why Mypy re-enable

For a few weeks MyPy checks have been disabled after the switch to Python 3.7 (per #19317).

We should, however, re-enable it back as it is very useful in catching a number of mistakes.

How does it work

We 've re-added the mypy pre-commit now - with mypy bumped to 0.910. This version detects far more errors and we should fix them all before we switch the CI check back.

mypy will be running for incremental changes in pre-commit, same as before. This will enable incremental fixes of the code changed by committers who use pre-commits locally

mypy on CI runs in non-failing mode. When the main pre-commit check is run, mypy is disabled, but then it is run as a separate step (which does not fail but will show the result of running mypy on all our code). This will enable us to track the progress of fixes

Can I help with the effort, you ask?

We started concerted effort now and incrementally fix all the mypy incompatibilities - ideally package/by/package to avoid huge code reviews. We'd really appreciate a number of people to contribute, so that we can re-enable mypy back fully and quickly :).

How can I help?

What you need is:

checkout main

./breeeze build-image

pip install pre-commit

pre-commit install

This will enable automated checks for when you do a regular contribution. When you make your change, any MyPy issues will be reporteed and you need to fix them all to commit. You can also commit with --no-verify flag to skip that, bu, well, if you can improve airlfow a little - why not?

How can I help more ?

You can add PRs that are fixing whole packages, without contributing features or bugfixes. Please refer to this issue #19891 and ideally comment below in the issue that you want to take care of a package (to avoid duplicate work).

An easy way to run MyPy check for package can be done either from the host:

find DIRECTORY -name "*.py" | xargs pre-commit run mypy --files

or from ./breeze shell:

mypy --namespace-packages DIRECTORY

Current list of mypy PRs:

https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+label%3Amypy

Remaining packages

Here is the list of remaining packages to be "mypy compliant" generated with:

pre-commit run mypy --all-files 2>&1 | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | grep "error:" | sort | awk 'FS=":" { print $1 }' | xargs dirname | sort | uniq -c | xargs -n 2 printf "* [ ] (%4d) %s\n"

( 1) airflow/api/common/experimental

( 1) airflow/contrib/sensors

( 1) airflow/example_dags

( 1) airflow/jobs

( 4) airflow/models

( 1) airflow/providers/microsoft/winrm/hooks

( 1) airflow/providers/ssh/hooks

( 1) tests/providers/amazon/aws/hooks

( 1) tests/providers/google/cloud/hooks

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Issue 2:
Description

Allow DAG to accept list of cron expression and schedule the dag in correlation to all of them.

Similar to how it can be done in cron job

Use case / motivation

Some scheduling like: every 10 min between 16:30 to 18:10 can not be obtained with single cron expression. The idea is that DAG will have the ability to be set according to more than 1 cron but without duplicating the DAG code or the DAG entry in the UI

Even simple scheduling which is common for ETL : bi-weekly can not be done with single cron expression: https://serverfault.com/questions/404398/how-to-schedule-a-biweekly-cronjob