Repository: apache/airflow

Issue:
Add ability to override waiter delay in EcsRunTaskOperator
Add ability to override waiter delay in `EcsRunTaskOperator`. The waiter might hit some throttling limit. Adding the capability to the user to override default waiter config in order to go around that limit.



<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.

Feel free to ping committers for the review!

In case of an existing issue, reference it using one of the following:

closes: #ISSUE
related: #ISSUE

How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->

---
**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).



Discussion:

Resolving patch:
diff --git a/airflow/providers/amazon/aws/operators/ecs.py b/airflow/providers/amazon/aws/operators/ecs.py
--- a/airflow/providers/amazon/aws/operators/ecs.py
+++ b/airflow/providers/amazon/aws/operators/ecs.py
@@ -18,7 +18,6 @@
 from __future__ import annotations
 
 import re
-import sys
 import warnings
 from datetime import timedelta
 from functools import cached_property
@@ -476,7 +475,9 @@ def __init__(
         number_logs_exception: int = 10,
         wait_for_completion: bool = True,
         waiter_delay: int = 6,
-        waiter_max_attempts: int = 100,
+        waiter_max_attempts: int = 1000000 * 365 * 24 * 60 * 10,
+        # Set the default waiter duration to 1M years (attempts*delay)
+        # Airflow execution_timeout handles task timeout
         deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
         **kwargs,
     ):
@@ -665,7 +666,6 @@ def _wait_for_task_ended(self) -> None:
             return
 
         waiter = self.client.get_waiter("tasks_stopped")
-        waiter.config.max_attempts = sys.maxsize  # timeout is managed by airflow
         waiter.wait(
             cluster=self.cluster,
             tasks=[self.arn],


Test patch:
diff --git a/tests/providers/amazon/aws/operators/test_ecs.py b/tests/providers/amazon/aws/operators/test_ecs.py
--- a/tests/providers/amazon/aws/operators/test_ecs.py
+++ b/tests/providers/amazon/aws/operators/test_ecs.py
@@ -17,7 +17,6 @@
 # under the License.
 from __future__ import annotations
 
-import sys
 from copy import deepcopy
 from unittest import mock
 from unittest.mock import MagicMock, PropertyMock
@@ -348,9 +347,8 @@ def test_wait_end_tasks(self, client_mock):
         self.ecs._wait_for_task_ended()
         client_mock.get_waiter.assert_called_once_with("tasks_stopped")
         client_mock.get_waiter.return_value.wait.assert_called_once_with(
-            cluster="c", tasks=["arn"], WaiterConfig={"Delay": 6, "MaxAttempts": 100}
+            cluster="c", tasks=["arn"], WaiterConfig={"Delay": 6, "MaxAttempts": 1000000 * 365 * 24 * 60 * 10}
         )
-        assert sys.maxsize == client_mock.get_waiter.return_value.config.max_attempts
 
     @mock.patch.object(EcsBaseOperator, "client")
     def test_check_success_tasks_raises_failed_to_start(self, client_mock):