Repository: pyinfra-dev/pyinfra

Issue:
Add retries, retry_delay, and retry_until kwargs for operation retries (Ansible-style behavior)
Today pyinfra executes each operation only once per host. If an operation depends on eventual consistency (like a service becoming available), users have to write manual retry logic. Other tools like Ansible provide retries as a built-in feature.

I’d like to propose adding three kwargs to all pyinfra operations:
_retries (default: 0)
_retry_delay (default: 5 seconds)
_retry_until (optional callable receiving state, host)

Behavior:
If _retry_until is defined → pyinfra retries until it returns True or retries are exhausted.
If _retry_until is not defined → pyinfra retries based on operation failure, up to retries times.
If retries is 0 → existing behavior remains.

I’m happy to open a PR following the contributing guide, modifying operation.py, with tests and documentation included.



Discussion:

Patch:
diff --git a/docs/cli.md b/docs/cli.md
index 691776c35..b70786000 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -43,6 +43,21 @@ By default pyinfra only prints high level information (this host connected, this
 + `-vv`: as above plus print shell input to the remote host
 + `-vvv` as above plus print shell output from the remote host
 
+### Retry Options
+
+pyinfra supports automatic retry of failed operations via CLI options:
+
++ `--retry N`: Retry failed operations up to N times (default: 0)
++ `--retry-delay N`: Wait N seconds between retry attempts (default: 5)
+
+```sh
+# Retry failed operations up to 3 times with default 5 second delay
+pyinfra inventory.py deploy.py --retry 3
+
+# Retry with custom delay
+pyinfra inventory.py deploy.py --retry 2 --retry-delay 10
+```
+
 
 ## Inventory
 
diff --git a/docs/faq.rst b/docs/faq.rst
index 34842f01e..b8f15d0b0 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -49,3 +49,32 @@ Use the LINK ``files.file``, ``files.directory`` or ``files.link`` operations to
         group="pyinfra",
         mode=644,
     )
+
+How do I handle unreliable operations or network issues?
+--------------------------------------------------------
+
+Use the `retry behavior arguments <arguments.html#retry-behavior>`_ to automatically retry failed operations. This is especially useful for network operations or services that may be temporarily unavailable:
+
+.. code:: python
+
+    # Retry a network operation up to 3 times
+    server.shell(
+        name="Download file with retries",
+        commands=["wget https://example.com/file.zip"],
+        _retries=3,
+        _retry_delay=5,  # wait 5 seconds between retries
+    )
+
+    # Use custom retry logic for specific error conditions
+    def should_retry_download(output_data):
+        # Retry only on temporary network errors, not permanent failures
+        stderr_text = " ".join(output_data["stderr_lines"]).lower()
+        temporary_errors = ["timeout", "connection refused", "temporary failure"]
+        return any(error in stderr_text for error in temporary_errors)
+
+    server.shell(
+        name="Download with smart retry logic",
+        commands=["wget https://example.com/large-file.zip"],
+        _retries=3,
+        _retry_until=should_retry_download,
+    )
diff --git a/docs/using-operations.rst b/docs/using-operations.rst
index fa6ecd4b4..66aab3e0e 100644
--- a/docs/using-operations.rst
+++ b/docs/using-operations.rst
@@ -49,6 +49,45 @@ Global arguments are covered in detail here: :doc:`arguments`. There is a set of
         _sudo_user="pyinfra",
     )
 
+Retry Functionality
+-------------------
+
+Operations can be configured to retry automatically on failure using retry arguments:
+
+.. code:: python
+
+    from pyinfra.operations import server
+
+    # Retry a flaky command up to 3 times with default 5 second delay
+    server.shell(
+        name="Download file with retries",
+        commands=["curl -o /tmp/file.tar.gz https://example.com/file.tar.gz"],
+        _retries=3,
+    )
+
+    # Retry with custom delay between attempts
+    server.shell(
+        name="Check service status with retries",
+        commands=["systemctl is-active myservice"],
+        _retries=2,
+        _retry_delay=10,  # 10 second delay between retries
+    )
+
+    # Use custom retry condition to control when to retry
+    def retry_on_network_error(output_data):
+        # Retry if stderr contains network-related errors
+        for line in output_data["stderr_lines"]:
+            if any(keyword in line.lower() for keyword in ["network", "timeout", "connection"]):
+                return True
+        return False
+
+    server.shell(
+        name="Network operation with conditional retry",
+        commands=["wget https://example.com/large-file.zip"],
+        _retries=5,
+        _retry_until=retry_on_network_error,
+    )
+
 
 The ``host`` Object
 -------------------
diff --git a/pyinfra/api/arguments.py b/pyinfra/api/arguments.py
index ca217fca9..a49401774 100644
--- a/pyinfra/api/arguments.py
+++ b/pyinfra/api/arguments.py
@@ -72,6 +72,11 @@ class ConnectorArguments(TypedDict, total=False):
     _get_pty: bool
     _stdin: Union[str, Iterable[str]]
 
+    # Retry arguments
+    _retries: int
+    _retry_delay: Union[int, float]
+    _retry_until: Optional[Callable[[dict], bool]]
+
 
 def generate_env(config: "Config", value: dict) -> dict:
     env = config.ENV.copy()
@@ -232,11 +237,28 @@ def all_global_arguments() -> List[tuple[str, Type]]:
     return list(get_type_hints(AllArguments).items())
 
 
+# Create a dictionary for retry arguments
+retry_argument_meta: dict[str, ArgumentMeta] = {
+    "_retries": ArgumentMeta(
+        "Number of times to retry failed operations.",
+        default=lambda config: config.RETRY,
+    ),
+    "_retry_delay": ArgumentMeta(
+        "Delay in seconds between retry attempts.",
+        default=lambda config: config.RETRY_DELAY,
+    ),
+    "_retry_until": ArgumentMeta(
+        "Callable taking output data that returns True to continue retrying.",
+        default=lambda config: None,
+    ),
+}
+
 all_argument_meta: dict[str, ArgumentMeta] = {
     **auth_argument_meta,
     **shell_argument_meta,
     **meta_argument_meta,
     **execution_argument_meta,
+    **retry_argument_meta,  # Add retry arguments
 }
 
 EXECUTION_KWARG_KEYS = list(ExecutionArguments.__annotations__.keys())
@@ -286,6 +308,45 @@ def all_global_arguments() -> List[tuple[str, Type]]:
     ),
     "Operation meta & callbacks": (meta_argument_meta, "", ""),
     "Execution strategy": (execution_argument_meta, "", ""),
+    "Retry behavior": (
+        retry_argument_meta,
+        """
+        Retry arguments allow you to automatically retry operations that fail. You can specify
+        how many times to retry, the delay between retries, and optionally a condition
+        function to determine when to stop retrying.
+        """,
+        """
+        .. code:: python
+
+            # Retry a command up to 3 times with the default 5 second delay
+            server.shell(
+                name="Run flaky command with retries",
+                commands=["flaky_command"],
+                _retries=3,
+            )
+            # Retry with a custom delay
+            server.shell(
+                name="Run flaky command with custom delay",
+                commands=["flaky_command"],
+                _retries=2,
+                _retry_delay=10,  # 10 second delay between retries
+            )
+            # Retry with a custom condition
+            def retry_on_specific_error(output_data):
+                # Retry if stderr contains "temporary failure"
+                for line in output_data["stderr_lines"]:
+                    if "temporary failure" in line.lower():
+                        return True
+                return False
+
+            server.shell(
+                name="Run command with conditional retry",
+                commands=["flaky_command"],
+                _retries=5,
+                _retry_until=retry_on_specific_error,
+            )
+        """,
+    ),
 }
 
 
diff --git a/pyinfra/api/config.py b/pyinfra/api/config.py
index 3a8a4f6a6..8c66f2645 100644
--- a/pyinfra/api/config.py
+++ b/pyinfra/api/config.py
@@ -53,6 +53,10 @@ class ConfigDefaults:
     IGNORE_ERRORS: bool = False
     # Shell to use to execute commands
     SHELL: str = "sh"
+    # Number of times to retry failed operations
+    RETRY: int = 0
+    # Delay in seconds between retry attempts
+    RETRY_DELAY: int = 5
 
 
 config_defaults = {key: value for key, value in ConfigDefaults.__dict__.items() if key.isupper()}
diff --git a/pyinfra/api/operation.py b/pyinfra/api/operation.py
index 6eabae425..2911f63e5 100644
--- a/pyinfra/api/operation.py
+++ b/pyinfra/api/operation.py
@@ -47,6 +47,9 @@ class OperationMeta:
     _commands: Optional[list[Any]] = None
     _maybe_is_change: Optional[bool] = None
     _success: Optional[bool] = None
+    _retry_attempts: int = 0
+    _max_retries: int = 0
+    _retry_succeeded: Optional[bool] = None
 
     def __init__(self, hash, is_change: Optional[bool]):
         self._hash = hash
@@ -59,9 +62,17 @@ def __repr__(self) -> str:
         """
 
         if self._commands is not None:
+            retry_info = ""
+            if self._retry_attempts > 0:
+                retry_result = "succeeded" if self._retry_succeeded else "failed"
+                retry_info = (
+                    f", retries={self._retry_attempts}/{self._max_retries} ({retry_result})"
+                )
+
             return (
                 "OperationMeta(executed=True, "
-                f"success={self.did_succeed()}, hash={self._hash}, commands={len(self._commands)})"
+                f"success={self.did_succeed()}, hash={self._hash}, "
+                f"commands={len(self._commands)}{retry_info})"
             )
         return (
             "OperationMeta(executed=False, "
@@ -74,12 +85,20 @@ def set_complete(
         success: bool,
         commands: list[Any],
         combined_output: "CommandOutput",
+        retry_attempts: int = 0,
+        max_retries: int = 0,
     ) -> None:
         if self.is_complete():
             raise RuntimeError("Cannot complete an already complete operation")
         self._success = success
         self._commands = commands
         self._combined_output = combined_output
+        self._retry_attempts = retry_attempts
+        self._max_retries = max_retries
+
+        # Determine if operation succeeded after retries
+        if retry_attempts > 0:
+            self._retry_succeeded = success
 
     def is_complete(self) -> bool:
         return self._success is not None
@@ -150,6 +169,40 @@ def stdout(self) -> str:
     def stderr(self) -> str:
         return "\n".join(self.stderr_lines)
 
+    @property
+    def retry_attempts(self) -> int:
+        return self._retry_attempts
+
+    @property
+    def max_retries(self) -> int:
+        return self._max_retries
+
+    @property
+    def was_retried(self) -> bool:
+        """
+        Returns whether this operation was retried at least once.
+        """
+        return self._retry_attempts > 0
+
+    @property
+    def retry_succeeded(self) -> Optional[bool]:
+        """
+        Returns whether this operation succeeded after retries.
+        Returns None if the operation was not retried.
+        """
+        return self._retry_succeeded
+
+    def get_retry_info(self) -> dict[str, Any]:
+        """
+        Returns a dictionary with all retry-related information.
+        """
+        return {
+            "retry_attempts": self._retry_attempts,
+            "max_retries": self._max_retries,
+            "was_retried": self.was_retried,
+            "retry_succeeded": self._retry_succeeded,
+        }
+
 
 def add_op(state: State, op_func, *args, **kwargs):
     """
diff --git a/pyinfra/api/operations.py b/pyinfra/api/operations.py
index 706e1b24f..415bcb05b 100644
--- a/pyinfra/api/operations.py
+++ b/pyinfra/api/operations.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import time
 import traceback
 from itertools import product
 from socket import error as socket_error, timeout as timeout_error
@@ -66,6 +67,11 @@ def _run_host_op(state: "State", host: "Host", op_hash: str) -> Optional[bool]:
     continue_on_error = global_arguments["_continue_on_error"]
     timeout = global_arguments.get("_timeout", 0)
 
+    # Extract retry arguments
+    retries = global_arguments.get("_retries", 0)
+    retry_delay = global_arguments.get("_retry_delay", 5)
+    retry_until = global_arguments.get("_retry_until", None)
+
     executor_kwarg_keys = CONNECTOR_ARGUMENT_KEYS
     # See: https://github.com/python/mypy/issues/10371
     base_connector_arguments: ConnectorArguments = cast(
@@ -73,67 +79,114 @@ def _run_host_op(state: "State", host: "Host", op_hash: str) -> Optional[bool]:
         {key: global_arguments[key] for key in executor_kwarg_keys if key in global_arguments},  # type: ignore[literal-required] # noqa
     )
 
+    retry_attempt = 0
     did_error = False
     executed_commands = 0
-    commands = []
+    commands: list[PyinfraCommand] = []
     all_output_lines: list[OutputLine] = []
 
-    for command in op_data.command_generator():
-        commands.append(command)
-
-        status = False
-
-        connector_arguments = base_connector_arguments.copy()
-        connector_arguments.update(command.connector_arguments)
-
-        if not isinstance(command, PyinfraCommand):
-            raise TypeError("{0} is an invalid pyinfra command!".format(command))
-
-        if isinstance(command, FunctionCommand):
-            try:
-                status = command.execute(state, host, connector_arguments)
-            except Exception as e:
-                # Custom functions could do anything, so expect anything!
-                logger.warning(traceback.format_exc())
-                host.log_styled(
-                    f"Unexpected error in Python callback: {format_exception(e)}",
-                    fg="red",
-                    log_func=logger.warning,
-                )
-
-        elif isinstance(command, StringCommand):
-            output_lines = CommandOutput([])
-            try:
-                status, output_lines = command.execute(
-                    state,
-                    host,
-                    connector_arguments,
-                )
-            except (timeout_error, socket_error, SSHException) as e:
-                log_host_command_error(host, e, timeout=timeout)
-            all_output_lines.extend(output_lines)
-            # If we failed and have not already printed the stderr, print it
-            if status is False and not state.print_output:
-                print_host_combined_output(host, output_lines)
-
-        else:
-            try:
-                status = command.execute(state, host, connector_arguments)
-            except (timeout_error, socket_error, SSHException, IOError) as e:
-                log_host_command_error(host, e, timeout=timeout)
-
-        # Break the loop to trigger a failure
-        if status is False:
-            did_error = True
-            if continue_on_error is True:
-                continue
-            break
+    # Retry loop
+    while retry_attempt <= retries:
+        did_error = False
+        executed_commands = 0
+        commands = []
+        all_output_lines = []
+
+        for command in op_data.command_generator():
+            commands.append(command)
+            status = False
+            connector_arguments = base_connector_arguments.copy()
+            connector_arguments.update(command.connector_arguments)
+
+            if not isinstance(command, PyinfraCommand):
+                raise TypeError("{0} is an invalid pyinfra command!".format(command))
+
+            if isinstance(command, FunctionCommand):
+                try:
+                    status = command.execute(state, host, connector_arguments)
+                except Exception as e:
+                    # Custom functions could do anything, so expect anything!
+                    logger.warning(traceback.format_exc())
+                    host.log_styled(
+                        f"Unexpected error in Python callback: {format_exception(e)}",
+                        fg="red",
+                        log_func=logger.warning,
+                    )
+
+            elif isinstance(command, StringCommand):
+                output_lines = CommandOutput([])
+                try:
+                    status, output_lines = command.execute(
+                        state,
+                        host,
+                        connector_arguments,
+                    )
+                except (timeout_error, socket_error, SSHException) as e:
+                    log_host_command_error(host, e, timeout=timeout)
+                all_output_lines.extend(output_lines)
+                # If we failed and have not already printed the stderr, print it
+                if status is False and not state.print_output:
+                    print_host_combined_output(host, output_lines)
+
+            else:
+                try:
+                    status = command.execute(state, host, connector_arguments)
+                except (timeout_error, socket_error, SSHException, IOError) as e:
+                    log_host_command_error(host, e, timeout=timeout)
+
+            # Break the loop to trigger a failure
+            if status is False:
+                did_error = True
+                if continue_on_error is True:
+                    continue
+                break
+
+            executed_commands += 1
+
+        # Check if we should retry
+        should_retry = False
+        if retry_attempt < retries:
+            # Retry on error
+            if did_error:
+                should_retry = True
+            # Retry on condition if no error
+            elif retry_until and not did_error:
+                try:
+                    output_data = {
+                        "stdout_lines": [
+                            line.line for line in all_output_lines if line.buffer_name == "stdout"
+                        ],
+                        "stderr_lines": [
+                            line.line for line in all_output_lines if line.buffer_name == "stderr"
+                        ],
+                        "commands": [str(command) for command in commands],
+                        "executed_commands": executed_commands,
+                        "host": host.name,
+                        "operation": ", ".join(state.get_op_meta(op_hash).names) or "Operation",
+                    }
+                    should_retry = retry_until(output_data)
+                except Exception as e:
+                    host.log_styled(
+                        f"Error in retry_until function: {format_exception(e)}",
+                        fg="red",
+                        log_func=logger.warning,
+                    )
+
+        if should_retry:
+            retry_attempt += 1
+            state.trigger_callbacks("operation_host_retry", host, op_hash, retry_attempt, retries)
+            op_name = ", ".join(state.get_op_meta(op_hash).names) or "Operation"
+            host.log_styled(
+                f"Retrying {op_name} (attempt {retry_attempt}/{retries}) after {retry_delay}s...",
+                fg="yellow",
+                log_func=logger.info,
+            )
+            time.sleep(retry_delay)
+            continue
 
-        executed_commands += 1
+        break
 
     # Handle results
-    #
-
     op_success = return_status = not did_error
     host_results = state.get_results_for_host(host)
 
@@ -142,10 +195,13 @@ def _run_host_op(state: "State", host: "Host", op_hash: str) -> Optional[bool]:
         host_results.success_ops += 1
 
         _status_log = "Success" if executed_commands > 0 else "No changes"
+        if retry_attempt > 0:
+            _status_log = f"{_status_log} on retry {retry_attempt}"
+
         _click_log_status = click.style(_status_log, "green")
         logger.info("{0}{1}".format(host.print_prefix, _click_log_status))
 
-        state.trigger_callbacks("operation_host_success", host, op_hash)
+        state.trigger_callbacks("operation_host_success", host, op_hash, retry_attempt)
     else:
         if ignore_errors:
             host_results.ignored_error_ops += 1
@@ -156,6 +212,11 @@ def _run_host_op(state: "State", host: "Host", op_hash: str) -> Optional[bool]:
             host_results.partial_ops += 1
 
         _command_description = f"executed {executed_commands} commands"
+        if retry_attempt > 0:
+            _command_description = (
+                f"{_command_description} (failed after {retry_attempt}/{retries} retries)"
+            )
+
         log_error_or_warning(host, ignore_errors, _command_description, continue_on_error)
 
         # Ignored, op "completes" w/ ignored error
@@ -164,12 +225,14 @@ def _run_host_op(state: "State", host: "Host", op_hash: str) -> Optional[bool]:
             return_status = True
 
         # Unignored error -> False
-        state.trigger_callbacks("operation_host_error", host, op_hash)
+        state.trigger_callbacks("operation_host_error", host, op_hash, retry_attempt, retries)
 
     op_data.operation_meta.set_complete(
         op_success,
         commands,
         CommandOutput(all_output_lines),
+        retry_attempts=retry_attempt,
+        max_retries=retries,
     )
 
     return return_status
diff --git a/pyinfra/api/state.py b/pyinfra/api/state.py
index ab4af5bdb..7937c09fb 100644
--- a/pyinfra/api/state.py
+++ b/pyinfra/api/state.py
@@ -70,11 +70,19 @@ def operation_host_start(state: "State", host: "Host", op_hash):
         pass
 
     @staticmethod
-    def operation_host_success(state: "State", host: "Host", op_hash):
+    def operation_host_success(state: "State", host: "Host", op_hash, retry_count: int = 0):
         pass
 
     @staticmethod
-    def operation_host_error(state: "State", host: "Host", op_hash):
+    def operation_host_error(
+        state: "State", host: "Host", op_hash, retry_count: int = 0, max_retries: int = 0
+    ):
+        pass
+
+    @staticmethod
+    def operation_host_retry(
+        state: "State", host: "Host", op_hash, retry_num: int, max_retries: int
+    ):
         pass
 
     @staticmethod
diff --git a/pyinfra/connectors/util.py b/pyinfra/connectors/util.py
index 3708e1499..abe46f7e9 100644
--- a/pyinfra/connectors/util.py
+++ b/pyinfra/connectors/util.py
@@ -314,6 +314,10 @@ def make_unix_command(
     # Doas config
     _doas=False,
     _doas_user=None,
+    # Retry config (ignored in command generation but passed through)
+    _retries=0,
+    _retry_delay=0,
+    _retry_until=None,
 ) -> StringCommand:
     """
     Builds a shell command with various kwargs.
diff --git a/pyinfra_cli/main.py b/pyinfra_cli/main.py
index 98cdb615e..f81f6ffad 100644
--- a/pyinfra_cli/main.py
+++ b/pyinfra_cli/main.py
@@ -132,6 +132,18 @@ def _print_support(ctx, param, value):
     default=False,
     help="Run operations in serial, host by host.",
 )
+@click.option(
+    "--retry",
+    type=int,
+    default=0,
+    help="Number of times to retry failed operations.",
+)
+@click.option(
+    "--retry-delay",
+    type=int,
+    default=5,
+    help="Delay in seconds between retry attempts.",
+)
 # SSH connector args
 # TODO: remove the non-ssh-prefixed variants
 @click.option("--ssh-user", "--user", "ssh_user", help="SSH user to connect as.")
@@ -271,6 +283,8 @@ def _main(
     limit: Iterable,
     no_wait: bool,
     serial: bool,
+    retry: int,
+    retry_delay: int,
     debug: bool,
     debug_all: bool,
     debug_facts: bool,
@@ -310,6 +324,8 @@ def _main(
         shell_executable,
         fail_percent,
         yes,
+        retry,
+        retry_delay,
     )
     override_data = _set_override_data(
         data,
@@ -549,6 +565,8 @@ def _set_config(
     shell_executable,
     fail_percent,
     yes,
+    retry,
+    retry_delay,
 ):
     logger.info("--> Loading config...")
 
@@ -583,6 +601,12 @@ def _set_config(
     if fail_percent is not None:
         config.FAIL_PERCENT = fail_percent
 
+    if retry is not None:
+        config.RETRY = retry
+
+    if retry_delay is not None:
+        config.RETRY_DELAY = retry_delay
+
     return config
 
 
@@ -709,10 +733,13 @@ def _run_fact_operations(state, config, operations):
 
 def _prepare_exec_operations(state, config, operations):
     state.print_output = True
+    # Pass the retry settings from config to the shell operation
     load_func(
         state,
         server.shell,
         " ".join(operations),
+        _retries=config.RETRY,
+        _retry_delay=config.RETRY_DELAY,
     )
     return state
 


Test patch:
diff --git a/tests/test_api/test_api_operations.py b/tests/test_api/test_api_operations.py
index 3bee3dc74..f4a7542ba 100644
--- a/tests/test_api/test_api_operations.py
+++ b/tests/test_api/test_api_operations.py
@@ -19,6 +19,7 @@
 from pyinfra.api.operation import OperationMeta, add_op
 from pyinfra.api.operations import run_ops
 from pyinfra.api.state import StateOperationMeta
+from pyinfra.connectors.util import CommandOutput, OutputLine
 from pyinfra.context import ctx_host, ctx_state
 from pyinfra.operations import files, python, server
 
@@ -576,4 +577,351 @@ def add_another_op():
         assert op_order[1] == second_op_hash
 
 
+class TestOperationRetry(PatchSSHTestCase):
+    """
+    Tests for the retry functionality in operations.
+    """
+
+    @patch("pyinfra.connectors.ssh.SSHConnector.run_shell_command")
+    def test_basic_retry_behavior(self, fake_run_command):
+        """
+        Test that operations retry the correct number of times on failure.
+        """
+        # Create inventory with just one host to simplify testing
+        inventory = make_inventory(hosts=("somehost",))
+        state = State(inventory, Config())
+        connect_all(state)
+
+        # Add operation with retry settings
+        add_op(
+            state,
+            server.shell,
+            'echo "testing retries"',
+            _retries=2,
+            _retry_delay=0.1,  # Use small delay for tests
+        )
+
+        # Track how many times run_shell_command was called
+        call_count = 0
+
+        # First call fails, second succeeds
+        def side_effect(*args, **kwargs):
+            nonlocal call_count
+            call_count += 1
+            if call_count == 1:
+                # First call fails
+                fake_channel = FakeChannel(1)
+                return (False, FakeBuffer("", fake_channel))
+            else:
+                # Second call succeeds
+                fake_channel = FakeChannel(0)
+                return (True, FakeBuffer("success", fake_channel))
+
+        fake_run_command.side_effect = side_effect
+
+        # Run the operation
+        run_ops(state)
+
+        # Check that run_shell_command was called twice (original + 1 retry)
+        self.assertEqual(call_count, 2)
+
+        # Verify results
+        somehost = inventory.get_host("somehost")
+
+        # Operation should be successful (because the retry succeeded)
+        self.assertEqual(state.results[somehost].success_ops, 1)
+        self.assertEqual(state.results[somehost].error_ops, 0)
+
+        # Get the operation hash
+        op_hash = state.get_op_order()[0]
+
+        # Check retry info in OperationMeta
+        op_meta = state.ops[somehost][op_hash].operation_meta
+        self.assertEqual(op_meta.retry_attempts, 1)
+        self.assertEqual(op_meta.max_retries, 2)
+        self.assertTrue(op_meta.was_retried)
+        self.assertTrue(op_meta.retry_succeeded)
+
+    @patch("pyinfra.connectors.ssh.SSHConnector.run_shell_command")
+    def test_retry_max_attempts_failure(self, fake_run_command):
+        """
+        Test that operations stop retrying after max attempts and report failure.
+        """
+        inventory = make_inventory(hosts=("somehost",))
+        state = State(inventory, Config())
+        connect_all(state)
+
+        # Add operation with retry settings
+        add_op(
+            state,
+            server.shell,
+            'echo "testing max retries"',
+            _retries=2,
+            _retry_delay=0.1,
+        )
+
+        # Make all attempts fail
+        fake_channel = FakeChannel(1)
+        fake_run_command.return_value = (False, FakeBuffer("", fake_channel))
+
+        # This should fail after all retries
+        with self.assertRaises(PyinfraError) as e:
+            run_ops(state)
+
+        self.assertEqual(e.exception.args[0], "No hosts remaining!")
+
+        # Check that run_shell_command was called the right number of times (1 original + 2 retries)
+        self.assertEqual(fake_run_command.call_count, 3)
+
+        somehost = inventory.get_host("somehost")
+
+        # Operation should be marked as error
+        self.assertEqual(state.results[somehost].success_ops, 0)
+        self.assertEqual(state.results[somehost].error_ops, 1)
+
+        # Get the operation hash
+        op_hash = state.get_op_order()[0]
+
+        # Check retry info
+        op_meta = state.ops[somehost][op_hash].operation_meta
+        self.assertEqual(op_meta.retry_attempts, 2)
+        self.assertEqual(op_meta.max_retries, 2)
+        self.assertTrue(op_meta.was_retried)
+        self.assertFalse(op_meta.retry_succeeded)
+
+    @patch("pyinfra.connectors.ssh.SSHConnector.run_shell_command")
+    @patch("time.sleep")
+    def test_retry_until_condition(self, fake_sleep, fake_run_command):
+        """
+        Test that operations retry based on the retry_until callable condition.
+        """
+        # Setup inventory and state using the utility function
+        inventory = make_inventory(hosts=("somehost",))
+        state = State(inventory, Config())
+        connect_all(state)
+
+        # Create a counter to track retry_until calls
+        call_counter = [0]
+
+        # Create a retry_until function that returns True (retry) for first two calls
+        def retry_until_func(output_data):
+            call_counter[0] += 1
+            return call_counter[0] < 3  # Retry twice, then stop
+
+        # Add operation with retry_until
+        add_op(
+            state,
+            server.shell,
+            'echo "test retry_until"',
+            _retries=3,
+            _retry_delay=0.1,
+            _retry_until=retry_until_func,
+        )
+
+        # Set up fake command execution - always succeed but with proper output format
+        # Use the existing FakeBuffer/FakeChannel from test utils
+
+        # First two calls trigger retry_until, third doesn't
+        def command_side_effect(*args, **kwargs):
+            # Create proper CommandOutput for the retry_until function to process
+            lines = [OutputLine("stdout", "test output"), OutputLine("stderr", "no errors")]
+            return True, CommandOutput(lines)
+
+        fake_run_command.side_effect = command_side_effect
+
+        # Run the operations
+        run_ops(state)
+
+        # The command should be called 3 times total (initial + 2 retries)
+        self.assertEqual(fake_run_command.call_count, 3)
+
+        # The retry_until function should be called 3 times
+        self.assertEqual(call_counter[0], 3)
+
+        # Get the operation metadata to check retry info
+        somehost = inventory.get_host("somehost")
+        op_hash = state.get_op_order()[0]
+        op_meta = state.ops[somehost][op_hash].operation_meta
+
+        # Check retry metadata
+        self.assertEqual(op_meta.retry_attempts, 2)
+        self.assertEqual(op_meta.max_retries, 3)
+        self.assertTrue(op_meta.was_retried)
+        self.assertTrue(op_meta.retry_succeeded)
+
+    @patch("pyinfra.connectors.ssh.SSHConnector.run_shell_command")
+    @patch("time.sleep")
+    def test_retry_delay(self, fake_sleep, fake_run_command):
+        """
+        Test that retry delay is properly applied between attempts.
+        """
+        inventory = make_inventory(hosts=("somehost",))
+        state = State(inventory, Config())
+        connect_all(state)
+
+        retry_delay = 5
+
+        # Add operation with retry settings
+        add_op(
+            state,
+            server.shell,
+            'echo "testing retry delay"',
+            _retries=2,
+            _retry_delay=retry_delay,
+        )
+
+        # Make first call fail, second succeed
+        call_count = 0
+
+        def side_effect(*args, **kwargs):
+            nonlocal call_count
+            call_count += 1
+            if call_count == 1:
+                fake_channel = FakeChannel(1)
+                return (False, FakeBuffer("", fake_channel))
+            else:
+                fake_channel = FakeChannel(0)
+                return (True, FakeBuffer("", fake_channel))
+
+        fake_run_command.side_effect = side_effect
+
+        # Run the operation
+        run_ops(state)
+
+        # Check that sleep was called with the correct delay
+        fake_sleep.assert_called_once_with(retry_delay)
+
+    @patch("pyinfra.connectors.ssh.SSHConnector.run_shell_command")
+    @patch("time.sleep")
+    def test_retry_until_with_error_handling(self, fake_sleep, fake_run_command):
+        """
+        Test that operations handle errors in retry_until functions gracefully.
+        """
+        inventory = make_inventory(hosts=("somehost",))
+        state = State(inventory, Config())
+        connect_all(state)
+
+        # Create a retry_until function that raises an exception
+        def failing_retry_until_func(output_data):
+            raise ValueError("Test error in retry_until function")
+
+        # Add operation with failing retry_until
+        add_op(
+            state,
+            server.shell,
+            'echo "test failing retry_until"',
+            _retries=2,
+            _retry_delay=0.1,
+            _retry_until=failing_retry_until_func,
+        )
+
+        # Set up fake command execution
+
+        def command_side_effect(*args, **kwargs):
+            lines = [OutputLine("stdout", "test output"), OutputLine("stderr", "no errors")]
+            return True, CommandOutput(lines)
+
+        fake_run_command.side_effect = command_side_effect
+
+        # Run the operations - should succeed despite retry_until error
+        run_ops(state)
+
+        # The command should be called only once (no retries due to error)
+        self.assertEqual(fake_run_command.call_count, 1)
+
+        # Verify operation completed successfully
+        somehost = inventory.get_host("somehost")
+        self.assertEqual(state.results[somehost].success_ops, 1)
+        self.assertEqual(state.results[somehost].error_ops, 0)
+
+    @patch("pyinfra.connectors.ssh.SSHConnector.run_shell_command")
+    @patch("time.sleep")
+    def test_retry_until_with_complex_output_parsing(self, fake_sleep, fake_run_command):
+        """
+        Test retry_until with complex output parsing scenarios.
+        """
+        inventory = make_inventory(hosts=("somehost",))
+        state = State(inventory, Config())
+        connect_all(state)
+
+        # Track what output we've seen
+        outputs_seen = []
+
+        def complex_retry_until_func(output_data):
+            # Store the output data for verification
+            outputs_seen.append(output_data)
+
+            # Check for specific patterns in stdout
+            stdout_text = " ".join(output_data["stdout_lines"])
+
+            # Continue retrying until we see "READY" in stdout
+            return "READY" not in stdout_text
+
+        # Add operation with complex retry_until
+        add_op(
+            state,
+            server.shell,
+            'echo "service status check"',
+            _retries=3,
+            _retry_delay=0.1,
+            _retry_until=complex_retry_until_func,
+        )
+
+        # Set up fake command execution with changing output
+
+        call_count = 0
+
+        def command_side_effect(*args, **kwargs):
+            nonlocal call_count
+            call_count += 1
+
+            if call_count == 1:
+                lines = [
+                    OutputLine("stdout", "Service starting..."),
+                    OutputLine("stderr", "Loading config"),
+                ]
+            elif call_count == 2:
+                lines = [
+                    OutputLine("stdout", "Service initializing..."),
+                    OutputLine("stderr", "Connecting to database"),
+                ]
+            else:  # call_count == 3
+                lines = [
+                    OutputLine("stdout", "Service READY"),
+                    OutputLine("stderr", "All systems operational"),
+                ]
+
+            return True, CommandOutput(lines)
+
+        fake_run_command.side_effect = command_side_effect
+
+        # Run the operations
+        run_ops(state)
+
+        # The command should be called 3 times
+        self.assertEqual(fake_run_command.call_count, 3)
+
+        # Verify retry_until was called 3 times with correct data
+        self.assertEqual(len(outputs_seen), 3)
+
+        # Check the output data structure
+        for output_data in outputs_seen:
+            self.assertIn("stdout_lines", output_data)
+            self.assertIn("stderr_lines", output_data)
+            self.assertIn("commands", output_data)
+            self.assertIn("executed_commands", output_data)
+            self.assertIn("host", output_data)
+            self.assertIn("operation", output_data)
+
+        # Verify operation metadata
+        somehost = inventory.get_host("somehost")
+        op_hash = state.get_op_order()[0]
+        op_meta = state.ops[somehost][op_hash].operation_meta
+
+        self.assertEqual(op_meta.retry_attempts, 2)
+        self.assertEqual(op_meta.max_retries, 3)
+        self.assertTrue(op_meta.was_retried)
+        self.assertTrue(op_meta.retry_succeeded)
+
+
 this_filename = path.join("tests", "test_api", "test_api_operations.py")
diff --git a/tests/test_cli/test_cli.py b/tests/test_cli/test_cli.py
index c2677864f..fb1e12596 100644
--- a/tests/test_cli/test_cli.py
+++ b/tests/test_cli/test_cli.py
@@ -188,5 +188,7 @@ def test_deploy_operation_direct(self):
                 debug_all=False,
                 debug_operations=False,
                 config_filename="config.py",
+                retry=0,
+                retry_delay=5,
             )
             assert e.args == (0,)