Repository: apache/airflow

Issue:
Jobs in Azure Containers restart infinitely if logger crashes, despite retries being set to off.
### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

We are using Airflow 2.5.2 to deploy python scripts in Azure containers.

In cases where the logger breaks (in our case because someone used tqdm for progress bars which are known to break it), Airflow failing to find the log keeps re-provisioning the container and restarting the job infinitely, even if all is set to not retry in Airflow. This incurs costs on API calls for us and thus is an impactful problem.

The issue could be because of the handling in _monitor_logging() in [Azure cointainer_instances.py line 298](https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/operators/container_instances.py#L298) where it changes the state to provisioning, but then doesn't do anything with it when it continues to fail to get instance_view. Maybe some form of check like if state=="Provisioning" and last_state=="Running": return 1 if retries are off could help handle it? 

Any insight would be appreciated. I am happy to help write a fix, if you can help me understand this flow a bit better.

### What you think should happen instead

The job should fail/exit code 1 instead of reprovisioning/retrying.

### How to reproduce

Run an airflow job in which a script is run in an Azure container, which employs tqdm progress bars, or otherwise overwhelms the logger and makes it fail.

### Operating System

Ubuntu 20.04

### Versions of Apache Airflow Providers

apache-airflow-providers-common-sql==1.3.4
apache-airflow-providers-ftp==2.1.1
apache-airflow-providers-http==2.1.1
apache-airflow-providers-imap==2.2.2
apache-airflow-providers-microsoft-azure==3.7.2
apache-airflow-providers-postgres==4.0.1
apache-airflow-providers-sqlite==2.1.2
apache-airflow-providers-ssh==3.1.0

### Deployment

Virtualenv installation

### Deployment details

Airflow running on a VM hosted in Azure.

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
A snippet of the log when this issue happens:

```console
[2023-09-21, 13:26:28 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:29 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:31 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:32 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:33 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:35 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:36 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:37 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:38 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:40 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:41 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:42 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:43 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:45 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:46 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:47 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:26:48 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:27:10 UTC] {container_instances.py:344} ERROR - Exception while getting container groups
Traceback (most recent call last):
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 316, in _monitor_logging
    logs = self._ci_hook.get_logs(resource_group, name)
  File "/home/azureuser/pyenv-airflow/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 119, in get_logs
    return logs.content.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
[2023-09-21, 13:27:11 UTC] {container_instances.py:311} INFO - Container group state changed to Waiting
[2023-09-21, 13:27:13 UTC] {container_instances.py:311} INFO - Container group state changed to Running
```
Here's the code that's run inside the Azure Container instance that can replicate this issue.

```
import re
import pandas as pd
import phonenumbers
import tldextract
from sentence_transformers import SentenceTransformer
from urlextract import URLExtract
import logging
from tqdm.auto import tqdm

tqdm.pandas()

df = pd.DataFrame()
datalist = [str(i) for i in range(int(10000))]
df["data"] = datalist

df[
    [
        "possible_uk_numbers",
        "possible_foreign_numbers",
        "definite_uk_numbers",
        "definite_foreign_numbers",
    ]
] = df.data.progress_apply(func1)
df[["uk_urls", "foreign_urls"]] = df.data.progress_apply(func2)
```
To note, this seems to occur specifically when there is more than 1 progress bar. 

Resolving patch:
diff --git a/airflow/providers/microsoft/azure/hooks/container_instance.py b/airflow/providers/microsoft/azure/hooks/container_instance.py
--- a/airflow/providers/microsoft/azure/hooks/container_instance.py
+++ b/airflow/providers/microsoft/azure/hooks/container_instance.py
@@ -177,6 +177,8 @@ def get_logs(self, resource_group: str, name: str, tail: int = 1000) -> list:
         :return: A list of log messages
         """
         logs = self.connection.containers.list_logs(resource_group, name, name, tail=tail)
+        if logs.content is None:
+            return [None]
         return logs.content.splitlines(True)
 
     def delete(self, resource_group: str, name: str) -> None:
diff --git a/airflow/providers/microsoft/azure/operators/container_instances.py b/airflow/providers/microsoft/azure/operators/container_instances.py
--- a/airflow/providers/microsoft/azure/operators/container_instances.py
+++ b/airflow/providers/microsoft/azure/operators/container_instances.py
@@ -319,6 +319,9 @@ def _monitor_logging(self, resource_group: str, name: str) -> int:
                 if state in ["Running", "Terminated", "Succeeded"]:
                     try:
                         logs = self._ci_hook.get_logs(resource_group, name)
+                        if logs and logs[0] is None:
+                            self.log.error("Container log is broken, marking as failed.")
+                            return 1
                         last_line_logged = self._log_last(logs, last_line_logged)
                     except CloudError:
                         self.log.exception(


Test patch: