Issue 1:
Apache Airflow version

2.2.3

What happened

When a task instance is in a failed state (or any other non running state), and then it's marked as successful through the UI, the on_success_callback set on the task instance is not run.

What you think should happen instead

The on_success_callback should run when marking a task instance successful from the UI regardless of its state.

How to reproduce

Create a DAG

Add a task with an on_success_callback that throws an exception (so it fails)

Run the DAG in the UI

Wait for the task instance to fail for that DAG run

Mark the task instance successful in the UI for that DAG run

Operating System

Debian 10 (Buster)

Versions of Apache Airflow Providers

apache-airflow-providers-databricks==2.5.0

This is the main provider we care about at the moment.

Deployment

Other Docker-based deployment

Anything else

From my observations, it seems as though the on_success_callback is run when a task instance is in a RUNNING state, and then is marked successful from the UI, but not when the task instance is in any other state. From some brief digging this behaviour appears to be related to the heartbeat_callback based on this old bug report but I can't see any equivalent logic for when the task isn't running.

For context - we are using this behaviour to track issues occurring in Airflow with a 3rd party tool where marking a task instance as successful is effectively an "ignore", and our on_success_callback lets us register that ignore to another tool that tracks alerting.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Apache Airflow version

2.1.1

Operating System

Ubuntu 20.04.3 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==2.0.0

apache-airflow-providers-celery==2.0.0

apache-airflow-providers-cncf-kubernetes==2.0.0

apache-airflow-providers-databricks==2.0.0

apache-airflow-providers-ftp==2.0.0

apache-airflow-providers-imap==2.0.0

apache-airflow-providers-mysql==2.0.0

apache-airflow-providers-postgres==2.0.0

apache-airflow-providers-redis==2.0.0

apache-airflow-providers-sqlite==2.0.0

apache-airflow-providers-ssh==2.0.0

Deployment

Virtualenv installation

Deployment details

No response

What happened

'on_success_callback' and 'on_failure_callback' functions were not called when manually marking success or failure on a completed dag run.

What you expected to happen

'on_success_callback' and 'on_failure_callback' functions should be called whenever marking success or failure on a dag run even after it completes.

How to reproduce

Provide callback function as 'on_success_callback' argument when initializing DAG operator. Let one dag run complete. Check if 'on_success_callback' function is called. Then mark the dag run as failed. Then mark the dag run as succeeded. The 'on_success_callback' function is not called.

Anything else

The callback functions are called when a dagrun is automatically marked success/failure by scheduler but aren't called when manually setting them after run completion. I have read earlier posts on stack overflow that this was not intended behaviour in Airflow 1 but could not find similar posts for Airflow 2 if this is a bug and how to fix it.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct