internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: apache/airflow

Issue:
[Smart sensor] Runtime error: dictionary changed size during iteration
<!--

Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.

-->


**What happened**:

<!-- (please include exact error messages if you can) -->
Smart Sensor TI crashes with a Runtime error. Here's the logs:
```
RuntimeError: dictionary changed size during iteration
  File "airflow/sentry.py", line 159, in wrapper
    return func(task_instance, *args, session=session, **kwargs)
  File "airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "airflow/sensors/smart_sensor.py", line 736, in execute
    self.flush_cached_sensor_poke_results()
  File "airflow/sensors/smart_sensor.py", line 681, in flush_cached_sensor_poke_results
    for ti_key, sensor_exception in self.cached_sensor_exceptions.items():
```


**What you expected to happen**:

<!-- What do you think went wrong? -->
Smart sensor should always execute without any runtime error.

**How to reproduce it**:
I haven't been able to reproduce it consistently since it sometimes works and sometimes errors.

**Anything else we need to know**:
It's a really noisy error in Sentry. In just 4 days, 3.8k events were reported in Sentry.
<!--

How often does this problem occur? Once? Every time etc?

Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>

-->



Discussion:

Resolving patch:
diff --git a/airflow/sensors/smart_sensor.py b/airflow/sensors/smart_sensor.py
--- a/airflow/sensors/smart_sensor.py
+++ b/airflow/sensors/smart_sensor.py
@@ -446,6 +446,7 @@ def mark_state(ti, sensor_instance):
         TI = TaskInstance
 
         count_marked = 0
+        query_result = []
         try:
             query_result = (
                 session.query(TI, SI)


Test patch:
Repository: apache/airflow

Issue:
Bump supported mysqlclient to <1.5

**Description**

version 1.4.X introduced in Jan 2019 

we should support it if we can.

**Use case / motivation**

pin of <1.4 was done in https://github.com/apache/airflow/pull/4558 due to lack of Python 2 compatibility. Since Master doesn't support Python 2 anymore there is no need for that restriction

**Related Issues**

Moved from https://issues.apache.org/jira/browse/AIRFLOW-4810

I tried to fix it in https://github.com/apache/airflow/pull/5430 but didn't get help with the tests so if any one wants to pick it up be my guest.


Discussion:
I will take a look later this week. I have plans to review al the requirements and remove some conflicting ones. 
@potiuk Are you still reviewing all requirements? Happy to work on this if not :)
I'm currently looking into this.

@potiuk are you working on this or it's open for anyone to pick?
Free to pick. I have forgotten about this one completely
I’ll probably find some time to create a new PR this week. The last one seems to have got pretty close but unfortunately went stale.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -365,7 +365,7 @@ def get_sphinx_theme_version() -> str:
 ]
 mysql = [
     'mysql-connector-python>=8.0.11, <=8.0.22',
-    'mysqlclient>=1.3.6,<1.4',
+    'mysqlclient>=1.3.6,<3',
 ]
 neo4j = ['neo4j>=4.2.1']
 odbc = [


Test patch:
Repository: apache/airflow

Issue:
Remove 'user_id', 'role_id' from User and Role in OpenAPI schema 
Would be good to remove the 'id' of both User and Role schemas from what is dumped in REST API endpoints. ID of User and Role table are sensitive data that would be fine to hide from the endpoints



Discussion:
can you help me out from where i can start with? @ephraimbuddy 
Hi @rijusougata13, you can start by setting up the development environment. Here are instructions on how to contribute https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst. 
The codes for the REST API are located at `airflow/api_connexion.`
Let me know if I should assign this to you.

Resolving patch:
diff --git a/airflow/api_connexion/schemas/user_schema.py b/airflow/api_connexion/schemas/user_schema.py
--- a/airflow/api_connexion/schemas/user_schema.py
+++ b/airflow/api_connexion/schemas/user_schema.py
@@ -33,7 +33,6 @@ class Meta:
         model = User
         dateformat = "iso"
 
-    user_id = auto_field('id', dump_only=True)
     first_name = auto_field()
     last_name = auto_field()
     username = auto_field()


Test patch:
diff --git a/tests/api_connexion/endpoints/test_user_endpoint.py b/tests/api_connexion/endpoints/test_user_endpoint.py
--- a/tests/api_connexion/endpoints/test_user_endpoint.py
+++ b/tests/api_connexion/endpoints/test_user_endpoint.py
@@ -98,7 +98,6 @@ def test_should_respond_200(self):
             'last_name': 'test1',
             'login_count': None,
             'roles': [],
-            'user_id': users[0].id,
             'username': 'TEST_USER1',
         }
 
diff --git a/tests/api_connexion/schemas/test_user_schema.py b/tests/api_connexion/schemas/test_user_schema.py
--- a/tests/api_connexion/schemas/test_user_schema.py
+++ b/tests/api_connexion/schemas/test_user_schema.py
@@ -71,12 +71,11 @@ def test_serialize(self):
         self.session.commit()
         user = self.session.query(User).filter(User.email == TEST_EMAIL).first()
         deserialized_user = user_collection_item_schema.dump(user)
-        # No password in dump
+        # No user_id and password in dump
         assert deserialized_user == {
             'created_on': DEFAULT_TIME,
             'email': 'test@example.org',
             'changed_on': DEFAULT_TIME,
-            'user_id': user.id,
             'active': None,
             'last_login': None,
             'last_name': 'Bar',
@@ -103,13 +102,12 @@ def test_serialize(self):
         self.session.commit()
         user = self.session.query(User).filter(User.email == TEST_EMAIL).first()
         deserialized_user = user_schema.dump(user)
-        # No password in dump
+        # No user_id and password in dump
         assert deserialized_user == {
             'roles': [],
             'created_on': DEFAULT_TIME,
             'email': 'test@example.org',
             'changed_on': DEFAULT_TIME,
-            'user_id': user.id,
             'active': None,
             'last_login': None,
             'last_name': 'Bar',
Repository: apache/airflow

Issue:
KubernetesPodOperator pod_template_file content doesn't support jinja airflow template variables
KubernetesPodOperator  pod_template_file content doesn't support jinja airflow template variables. pod_template_file  is part of templated_fields list. 
https://github.com/apache/airflow/blob/master/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L165

template_fields: Iterable[str] = (        'image',        'cmds',        'arguments',        'env_vars',        'labels',        'config_file',        'pod_template_file',    )

But pod_template_file content is not supporting template variables. pod_template_file can be implemented the way SparkKubernetesOperator implemented using template_ext

https://github.com/apache/airflow/blob/master/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L46
 template_ext = ('yaml', 'yml', 'json')


Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

I will put up a PR for this @kaxil 
@Dr-Denzy  can you share PR?

Resolving patch:
diff --git a/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py b/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
--- a/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
+++ b/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
@@ -172,6 +172,8 @@ class KubernetesPodOperator(BaseOperator):  # pylint: disable=too-many-instance-
         'pod_template_file',
     )
 
+    template_ext = ('yaml', 'yml', 'json')
+
     # fmt: off
     def __init__(  # pylint: disable=too-many-arguments,too-many-locals
         # fmt: on


Test patch:
Repository: apache/airflow

Issue:
Consider and add common sensitive names
**Description** 

Since sensitive informations in the connection object (specifically the extras field) are now being masked based on sensitive key names, we should consider adding some common sensitive key names.

`private_key` from [ssh connection](https://airflow.apache.org/docs/apache-airflow-providers-ssh/stable/connections/ssh.html) is an examples.

**Use case / motivation**

Extras field used to be blocked out entirely before the sensitive value masking feature (#15599).

[Before in 2.0.2](https://github.com/apache/airflow/blob/2.0.2/airflow/hooks/base.py#L78
) and [after in 2.1.0](https://github.com/apache/airflow/blob/2.1.0/airflow/hooks/base.py#L78
).

Extras field containing sensitive information now shown unless the key contains sensitive names.

**Are you willing to submit a PR?** 

@ashb has expressed interest in adding this.



Discussion:

Resolving patch:
diff --git a/airflow/utils/log/secrets_masker.py b/airflow/utils/log/secrets_masker.py
--- a/airflow/utils/log/secrets_masker.py
+++ b/airflow/utils/log/secrets_masker.py
@@ -34,13 +34,15 @@
 
 DEFAULT_SENSITIVE_FIELDS = frozenset(
     {
-        'password',
-        'secret',
-        'passwd',
-        'authorization',
+        'access_token',
         'api_key',
         'apikey',
-        'access_token',
+        'authorization',
+        'passphrase',
+        'passwd',
+        'password',
+        'private_key',
+        'secret',
     }
 )
 """Names of fields (Connection extra, Variable key name etc.) that are deemed sensitive"""


Test patch:
Repository: apache/airflow

Issue:
Upgrade `importlib-resources` version
### Description

The version for `importlib-resources` constraint sets it to be [v1.5.0](https://github.com/python/importlib_resources/tree/v1.5.0) which is over a year old. For compatibility sake (for instance with something like Datapane) I would suggest upgrading it. 

### Use case/motivation

Upgrade a an old dependency to keep code up to date.

### Related issues

Not that I am aware of, maybe somewhat #12120, or #15991.

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Feel free. It is configured in setup.cfg - you need to change it and see what happens when all tests run. If all tests pass, we will merged. Remember that importlib_resources are backport of the 3.9 native library and we need to make sure we are compatible with 3.6 - 3.9.
> Feel free. It is configured in setup.cfg - you need to change it and see what happens when all tests run. If all tests pass, we will merged. Remember that importlib_resources are backport of the 3.9 native library and we need to make sure we are compatible with 3.6 - 3.9.

Got it! Will do. 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -512,7 +512,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
     'freezegun',
     'github3.py',
     'gitpython',
-    'importlib-resources~=1.4',
+    'importlib-resources~=5.0',
     'ipdb',
     'jira',
     'jsondiff',


Test patch:
Repository: apache/airflow

Issue:
Deferred TI's `next_method` and `next_kwargs` not cleared on retries
### Apache Airflow version

main (development)

### Operating System

macOS 11.5.2

### Versions of Apache Airflow Providers

_No response_

### Deployment

Virtualenv installation

### Deployment details

_No response_

### What happened

If your first try fails and you have retries enabled, the subsequent tries skip right to the final `next_method(**next_kwargs)` instead of starting with `execute` again.

### What you expected to happen

Like we reset things like start date, we should wipe `next_method` and `next_kwargs` so we can retry the task from the beginning.

### How to reproduce

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:

Resolving patch:
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -1693,6 +1693,11 @@ def handle_failure(
             # Log failure duration
             session.add(TaskFail(task, self.execution_date, self.start_date, self.end_date))
 
+        # Ensure we unset next_method and next_kwargs to ensure that any
+        # retries don't re-use them.
+        self.next_method = None
+        self.next_kwargs = None
+
         # Set state correctly and figure out how to log it and decide whether
         # to email
 


Test patch:
diff --git a/tests/models/test_taskinstance.py b/tests/models/test_taskinstance.py
--- a/tests/models/test_taskinstance.py
+++ b/tests/models/test_taskinstance.py
@@ -459,6 +459,36 @@ def task_function(ti):
         ti.refresh_from_db()
         assert ti.state == State.UP_FOR_RETRY
 
+    def test_task_retry_wipes_next_fields(self, session, dag_maker):
+        """
+        Test that ensures that tasks wipe their next_method and next_kwargs
+        fields when they are queued for retry after a failure.
+        """
+
+        with dag_maker('test_mark_failure_2'):
+            task = BashOperator(
+                task_id='test_retry_handling_op',
+                bash_command='exit 1',
+                retries=1,
+                retry_delay=datetime.timedelta(seconds=2),
+            )
+
+        dr = dag_maker.create_dagrun()
+        ti = dr.task_instances[0]
+        ti.next_method = "execute"
+        ti.next_kwargs = {}
+        session.merge(ti)
+        session.commit()
+
+        ti.task = task
+        with pytest.raises(AirflowException):
+            ti.run()
+        ti.refresh_from_db()
+
+        assert ti.next_method is None
+        assert ti.next_kwargs is None
+        assert ti.state == State.UP_FOR_RETRY
+
     def test_retry_delay(self, dag_maker):
         """
         Test that retry delays are respected
Repository: apache/airflow

Issue:
Can't open task log from Gantt view
### Apache Airflow version

2.2.1

### Operating System

Linux 5.4.149-73.259.amzn2.x86_64

### Versions of Apache Airflow Providers

default for 2.2.1

### Deployment

Other 3rd-party Helm chart

### Deployment details

aws eks using own-developed helm chart

### What happened

When trying to open log from gantt view - receiving an exception
```
  File "/home/airflow/.local/lib/python3.9/site-packages/pendulum/parsing/__init__.py", line 177, in _parse_common
    return date(year, month, day)
ValueError: year 0 is out of range
```

due to incorrect query parameter push: no value for `execution_date` pushed
```
/log?dag_id=report_generator_daily&task_id=check_quints_earnings&execution_date=
```

### What you expected to happen

Logs should be available

### How to reproduce

Open dag's `gantt` chart
click on task ribbon
click on `log`
observe an error

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
You're navigating to logs via this modal, right? If so, does the execution exist in the modal header like the screenshot below?

<img width="588" alt="Screen Shot 2021-11-16 at 11 08 49 AM" src="https://user-images.githubusercontent.com/4600967/142032119-0db919fd-ac31-41ed-96c6-373b37982950.png">


Resolving patch:
diff --git a/airflow/www/views.py b/airflow/www/views.py
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2889,7 +2889,7 @@ def gantt(self, session=None):
             task_dict['end_date'] = task_dict['end_date'] or timezone.utcnow()
             task_dict['extraLinks'] = dag.get_task(ti.task_id).extra_links
             task_dict['try_number'] = try_count
-            task_dict['execution_date'] = dttm
+            task_dict['execution_date'] = dttm.isoformat()
             tasks.append(task_dict)
 
         tf_count = 0
@@ -2911,7 +2911,7 @@ def gantt(self, session=None):
             task_dict['operator'] = task.task_type
             task_dict['try_number'] = try_count
             task_dict['extraLinks'] = task.extra_links
-            task_dict['execution_date'] = dttm
+            task_dict['execution_date'] = dttm.isoformat()
             tasks.append(task_dict)
 
         task_names = [ti.task_id for ti in tis]


Test patch:
Repository: apache/airflow

Issue:
Allow for Markup in UI page title
### Description

A custom page title can be set on the UI with the  `instance_name` variable in `airflow.cfg`. It would be nice to have the option to include Markup text in that variable for further customization of the title, similar to how dashboard alerts introduced in #18284 allow for `html=True`.

### Use case/motivation

It would be useful to be able to use formatting like underline, bold, color, etc, in the UI title. For example, color-coded environments to minimize chances of a developer triggering a DAG in the wrong environment:

![title](https://user-images.githubusercontent.com/50751110/149528236-403473ed-eba5-4102-a804-0fa78fca8856.JPG)


### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:

Resolving patch:
diff --git a/airflow/www/views.py b/airflow/www/views.py
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -770,6 +770,9 @@ def index(self):
         state_color_mapping["null"] = state_color_mapping.pop(None)
 
         page_title = conf.get(section="webserver", key="instance_name", fallback="DAGs")
+        page_title_has_markup = conf.getboolean(
+            section="webserver", key="instance_name_has_markup", fallback=False
+        )
 
         dashboard_alerts = [
             fm for fm in settings.DASHBOARD_UIALERTS if fm.should_show(current_app.appbuilder.sm)
@@ -815,7 +818,7 @@ def _iter_parsed_moved_data_table_names():
             migration_moved_data_alerts=sorted(set(_iter_parsed_moved_data_table_names())),
             current_page=current_page,
             search_query=arg_search_query if arg_search_query else '',
-            page_title=page_title,
+            page_title=Markup(page_title) if page_title_has_markup else page_title,
             page_size=dags_per_page,
             num_of_pages=num_of_pages,
             num_dag_from=min(start + 1, num_of_all_dags),


Test patch:
diff --git a/tests/www/views/test_views_base.py b/tests/www/views/test_views_base.py
--- a/tests/www/views/test_views_base.py
+++ b/tests/www/views/test_views_base.py
@@ -390,3 +390,14 @@ def test_page_instance_name_xss_prevention(admin_client):
         escaped_xss_string = "&lt;script&gt;alert(&#39;Give me your credit card number&#39;)&lt;/script&gt;"
         check_content_in_response(escaped_xss_string, resp)
         check_content_not_in_response(xss_string, resp)
+
+
+@conf_vars(
+    {
+        ("webserver", "instance_name"): "<b>Bold Site Title Test</b>",
+        ("webserver", "instance_name_has_markup"): "True",
+    }
+)
+def test_page_instance_name_with_markup(admin_client):
+    resp = admin_client.get('home', follow_redirects=True)
+    check_content_in_response('<b>Bold Site Title Test</b>', resp)
Repository: apache/airflow

Issue:
Consider depending on `azure-keyvault-secrets` instead of `azure-keyvault` metapackage
### Description

It appears that the `microsoft-azure` provider only depends on `azure-keyvault-secrets`:
https://github.com/apache/airflow/blob/388723950de9ca519108e0a8f6818f0fc0dd91d4/airflow/providers/microsoft/azure/secrets/key_vault.py#L24
and not the other 2 packages in the `azure-keyvault` metapackage.

### Use case/motivation

I am the maintainer of the `apache-airflow-providers-*` packages on `conda-forge` and I'm running into small issues with the way `azure-keyvault` is maintained as a metapackage on `conda-forge`.  I think depending on `azure-keyvault-secrets` explicitly would solve my problem and also provide better clarity for the `microsoft-azure` provider in general.

### Related issues

https://github.com/conda-forge/azure-keyvault-feedstock/issues/6
https://github.com/conda-forge/apache-airflow-providers-microsoft-azure-feedstock/pull/13

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
@xylar Assigned!

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -218,7 +218,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
     'azure-cosmos>=4.0.0',
     'azure-datalake-store>=0.0.45',
     'azure-identity>=1.3.1',
-    'azure-keyvault>=4.1.0',
+    'azure-keyvault-secrets>=4.1.0,<5.0',
     'azure-kusto-data>=0.0.43,<0.1',
     # Azure integration uses old librarires and the limits below reflect that
     # TODO: upgrade to newer versions of all the below libraries


Test patch:
diff --git a/docker_tests/test_prod_image.py b/docker_tests/test_prod_image.py
--- a/docker_tests/test_prod_image.py
+++ b/docker_tests/test_prod_image.py
@@ -106,7 +106,7 @@ def test_pip_dependencies_conflict(self):
             'azure.cosmos',
             'azure.datalake.store',
             'azure.identity',
-            'azure.keyvault',
+            'azure.keyvault.secrets',
             'azure.kusto.data',
             'azure.mgmt.containerinstance',
             'azure.mgmt.datalake.store',