internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: apache/airflow

Issue:
Make sure that local installation works with installing airflow + all providers.
We have to double-check that providers work and can be tested locally when installing airflow in a local virtualenv.

After we've changed to separate providers and core package, pip install . will not install those packages, but with local -e it will likely work as providers packages are in the same folder. 

We still need to check how local virtualenv work when we install without -e though.

Most likely solution will be dynamic selection of packages in setup.py based on the fact whether you are installing airflow from packages or sources. 

This is connected with #11464 



Discussion:
@ashb  I also added a description to that issue - if you want, you can work on this one if you would like to. I am working on the Docker + extras ones though. 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -743,6 +743,10 @@ def is_package_excluded(package: str, exclusion_list: List[str]):
 
 def do_setup():
     """Perform the Airflow package setup."""
+    install_providers_from_sources = os.getenv('INSTALL_PROVIDERS_FROM_SOURCES')
+    exclude_patterns = \
+        [] if install_providers_from_sources and install_providers_from_sources == 'true' \
+        else ['airflow.providers', 'airflow.providers.*']
     write_version()
     setup(
         name='apache-airflow',
@@ -753,7 +757,7 @@ def do_setup():
         version=version,
         packages=find_packages(
             include=['airflow*'],
-            exclude=['airflow.providers', 'airflow.providers.*']),
+            exclude=exclude_patterns),
         package_data={
             'airflow': ['py.typed'],
             '': ['airflow/alembic.ini', "airflow/git_version", "*.ipynb",


Test patch:
Repository: apache/airflow

Issue:
Provider dependencies do not work well with alphas.
The dependencies in Alphas are currently >= 2.0.0 and should be >=2.0.0a0 in order to work with Alphas in cases which are not PEP 440-compliant.

According to https://www.python.org/dev/peps/pep-0440/, >= 2.0.0 should also work with alpha/beta releases (a1/a2) but in some cases it does not (https://apache-airflow.slack.com/archives/C0146STM600/p1602774750041800) 

Changing to ">=2.0.0a0" should help.


Discussion:

Resolving patch:
diff --git a/provider_packages/setup_provider_packages.py b/provider_packages/setup_provider_packages.py
--- a/provider_packages/setup_provider_packages.py
+++ b/provider_packages/setup_provider_packages.py
@@ -279,7 +279,7 @@ def do_setup_package_providers(provider_package_id: str,
         airflow_dependency = 'apache-airflow~=1.10' if provider_package_id != 'cncf.kubernetes' \
             else 'apache-airflow>=1.10.12, <2.0.0'
     else:
-        airflow_dependency = 'apache-airflow>=2.0.0'
+        airflow_dependency = 'apache-airflow>=2.0.0a0'
 
     install_requires = [
         airflow_dependency


Test patch:
Repository: apache/airflow

Issue:
Exception logging success function instead of failure
https://github.com/apache/airflow/blob/fdd9b6f65b608c516b8a062b058972d9a45ec9e3/airflow/sensors/sql_sensor.py#L97




Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Can you share an example of a DAG that causes problems?
@mik-laj @turbaszek If this issue is not taken yet, I'd be interested in taking this on :)
I assigned you to this ticket. 🐈

Resolving patch:
diff --git a/airflow/sensors/sql_sensor.py b/airflow/sensors/sql_sensor.py
--- a/airflow/sensors/sql_sensor.py
+++ b/airflow/sensors/sql_sensor.py
@@ -109,7 +109,7 @@ def poke(self, context):
                 if self.failure(first_cell):
                     raise AirflowException(f"Failure criteria met. self.failure({first_cell}) returned True")
             else:
-                raise AirflowException(f"self.failure is present, but not callable -> {self.success}")
+                raise AirflowException(f"self.failure is present, but not callable -> {self.failure}")
         if self.success is not None:
             if callable(self.success):
                 return self.success(first_cell)


Test patch:
diff --git a/tests/sensors/test_sql_sensor.py b/tests/sensors/test_sql_sensor.py
--- a/tests/sensors/test_sql_sensor.py
+++ b/tests/sensors/test_sql_sensor.py
@@ -217,7 +217,9 @@ def test_sql_sensor_postgres_poke_invalid_failure(self, mock_hook):
         mock_get_records = mock_hook.get_connection.return_value.get_hook.return_value.get_records
 
         mock_get_records.return_value = [[1]]
-        self.assertRaises(AirflowException, op.poke, None)
+        with self.assertRaises(AirflowException) as e:
+            op.poke(None)
+        self.assertEqual("self.failure is present, but not callable -> [1]", str(e.exception))
 
     @mock.patch('airflow.sensors.sql_sensor.BaseHook')
     def test_sql_sensor_postgres_poke_invalid_success(self, mock_hook):
@@ -232,7 +234,9 @@ def test_sql_sensor_postgres_poke_invalid_success(self, mock_hook):
         mock_get_records = mock_hook.get_connection.return_value.get_hook.return_value.get_records
 
         mock_get_records.return_value = [[1]]
-        self.assertRaises(AirflowException, op.poke, None)
+        with self.assertRaises(AirflowException) as e:
+            op.poke(None)
+        self.assertEqual("self.success is present, but not callable -> [1]", str(e.exception))
 
     @unittest.skipIf(
         'AIRFLOW_RUNALL_TESTS' not in os.environ, "Skipped because AIRFLOW_RUNALL_TESTS is not set"
Repository: apache/airflow

Issue:
Production image has only amazon and google providers installed
When "production" image is prepared for, only amazon and google providers are installed from sources.

**Apache Airflow version**:

master

**What you expected to happen**:

All providers should be installed

**How to reproduce it**:

```
./breeze --production-image --python 3.6
```

Then:

```
./breeze --production-image --python 3.6 shell bash
```

then

```
ls -la ~/.local/lib/python3.6/site-packages/airflow/providers/

amazon
google

```


UPDATE:

They are not even installed: 

```
.
./amazon
./amazon/aws
./amazon/aws/hooks
./amazon/aws/hooks/batch_waiters.json
./google
./google/cloud
./google/cloud/example_dags
./google/cloud/example_dags/example_bigquery_query.sql
./google/cloud/example_dags/example_cloud_build.yaml
./google/cloud/example_dags/example_spanner.sql
```



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -27,7 +27,7 @@
 from textwrap import wrap
 from typing import Dict, Iterable, List
 
-from setuptools import Command, find_packages, setup
+from setuptools import Command, find_namespace_packages, find_packages, setup
 
 logger = logging.getLogger(__name__)
 
@@ -771,6 +771,11 @@ def do_setup():
         else ['airflow.providers', 'airflow.providers.*']
     )
     write_version()
+    packages_to_install = (
+        find_namespace_packages(include=['airflow*'], exclude=exclude_patterns)
+        if install_providers_from_sources
+        else find_packages(include=['airflow*'], exclude=exclude_patterns)
+    )
     setup(
         name='apache-airflow',
         description='Programmatically author, schedule and monitor data pipelines',
@@ -778,7 +783,7 @@ def do_setup():
         long_description_content_type='text/markdown',
         license='Apache License 2.0',
         version=version,
-        packages=find_packages(include=['airflow*'], exclude=exclude_patterns),
+        packages=packages_to_install,
         package_data={
             'airflow': ['py.typed'],
             '': [


Test patch:
Repository: apache/airflow

Issue:
Provider packages don't include datafiles
The amazon and google providers don't include necessary datafiles in them.

They were previously included in the sdist via MANIFEST.in (see https://github.com/apache/airflow/pull/12196) and in the bdist via include_package_data from the top level setup.py

Both of these are currently missing.

I've put this against 2.0.0-beta1, it _could_ be changed separately as providers are separate releases.


Discussion:

Resolving patch:
diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
--- a/dev/provider_packages/prepare_provider_packages.py
+++ b/dev/provider_packages/prepare_provider_packages.py
@@ -1220,6 +1220,7 @@ def update_generated_files_for_package(
     if update_setup:
         prepare_setup_py_file(context)
         prepare_setup_cfg_file(context)
+        prepare_manifest_in_file(context)
 
     bad = bad + sum([len(entity_summary.wrong_entities) for entity_summary in entity_summaries.values()])
     if bad != 0:
@@ -1318,6 +1319,19 @@ def prepare_setup_cfg_file(context):
         setup_file.write(setup_content)
 
 
+def prepare_manifest_in_file(context):
+    target = os.path.abspath(os.path.join(get_target_folder(), "MANIFEST.in"))
+    content = render_template(
+        template_name="MANIFEST",
+        context=context,
+        extension='.in',
+        autoescape=False,
+        keep_trailing_newline=True,
+    )
+    with open(target, "wt") as fh:
+        fh.write(content)
+
+
 def update_release_notes_for_packages(
     provider_ids: List[str], release_version: str, version_suffix: str, backport_packages: bool
 ):


Test patch:
Repository: apache/airflow

Issue:
[ldap] section in configuration is not applicable anymore in 2.0
**Apache Airflow version**: 2.0.0b* / master


**What happened**:

`[ldap]` section in `airflow.cfg` is not applicable anymore in 2.0 and `master`, because the LDAP authentication (for webserver and API) is handled by FAB, and the configuration for this is handled by `webserver_config.py` file.

![airflow_–_default_airflow_cfg](https://user-images.githubusercontent.com/11539188/99289209-ef210700-283c-11eb-948e-16811ba09293.png)


**What you expected to happen**:

The `[ldap]` section should be removed from `airflow/config_templates/default_airflow.cfg` and `airflow/config_templates/config.yml` (and some other applicable files).

Otherwise leaving this section there will be a big confusion for users.


Discussion:
@ashb @kaxil @turbaszek @mik-laj may you please help confirm if you agree or if I'm wrong?

If my understanding above is correct I will proceed to prepare the PR. Thanks.
Correct, this is for non-FAB UI :boom:
> Correct, this is for non-FAB UI :boom:

Good. Will prepare a PR tonight

Resolving patch:
diff --git a/airflow/configuration.py b/airflow/configuration.py
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -129,7 +129,6 @@ class AirflowConfigParser(ConfigParser):  # pylint: disable=too-many-ancestors
         ('celery', 'result_backend'),
         ('atlas', 'password'),
         ('smtp', 'smtp_password'),
-        ('ldap', 'bind_password'),
         ('kubernetes', 'git_password'),
     }
 


Test patch:
diff --git a/tests/core/test_config_templates.py b/tests/core/test_config_templates.py
--- a/tests/core/test_config_templates.py
+++ b/tests/core/test_config_templates.py
@@ -45,7 +45,6 @@
     'celery_broker_transport_options',
     'dask',
     'scheduler',
-    'ldap',
     'kerberos',
     'github_enterprise',
     'admin',
Repository: apache/airflow

Issue:
Add detailed documentation for scheduler "tuning" parameters
https://github.com/apache/airflow/pull/12139 added some new params to the scheduler -- we have "cursory" docs for them in the comments in the config file, but we should have a more details docs in the scheduler.rst page.


Discussion:
If possible, I would change the option name from `clean_tis_without_dagrun ` to `clean_tis_without_dagrun_interval ` The other configuration options have this suffix.

Hey @ashb @mik-laj - something for 2.0.0rc1? or will you change the milestone?

Resolving patch:
diff --git a/airflow/jobs/scheduler_job.py b/airflow/jobs/scheduler_job.py
--- a/airflow/jobs/scheduler_job.py
+++ b/airflow/jobs/scheduler_job.py
@@ -1359,7 +1359,7 @@ def repeat(*args, **kwargs):
         )
 
         call_regular_interval(
-            conf.getfloat('scheduler', 'clean_tis_without_dagrun', fallback=15.0),
+            conf.getfloat('scheduler', 'clean_tis_without_dagrun_interval', fallback=15.0),
             self._clean_tis_without_dagrun,
         )
 


Test patch:
Repository: apache/airflow

Issue:
Remove or limit table of content at the main Airflow doc page
Table of content here : http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/index.html at the main page of "apache-airflow" documentation is huge and useless (especially that we have it in directory on the left). 


We should remove it or limit heavily (to 1st level only)


![Screenshot from 2020-11-22 20-34-28](https://user-images.githubusercontent.com/595491/99915147-27738a00-2d02-11eb-8e6b-1e2717180f26.png)





Discussion:
cc: @mik-laj 
It seems to me that it cannot be removed completely, but it will probably be possible to limit the levels of the depth quite simply.

I don't have plans to do it any time soon. I will add the label "good-first-issues" label.Maybe another person will be willing to take care of it.
Cool!
@mik-laj You reverted this change so it's a problem again.
> We should remove it or **limit heavily (to 1st level only)**

In my opinion, this ticket is already resolved.

http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/index.html

What else should we do here?

Resolving patch:
diff --git a/airflow/serialization/serialized_objects.py b/airflow/serialization/serialized_objects.py
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -67,6 +67,9 @@
 _OPERATOR_EXTRA_LINKS: Set[str] = {
     "airflow.operators.trigger_dagrun.TriggerDagRunLink",
     "airflow.sensors.external_task.ExternalTaskSensorLink",
+    # Deprecated names, so that existing serialized dags load straight away.
+    "airflow.operators.dagrun_operator.TriggerDagRunLink",
+    "airflow.sensors.external_task_sensor.ExternalTaskSensorLink",
 }
 
 
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -245,8 +245,10 @@ def _get_rst_filepath_from_path(filepath: str):
 html_show_copyright = False
 
 # Theme configuration
-sphinx_airflow_theme_navbar_links = [{'href': '/index.html', 'text': 'Documentation'}]
-sphinx_airflow_theme_hide_website_buttons = True
+html_theme_options = {
+    'hide_website_buttons': True,
+}
+
 # A dictionary of values to pass into the template engine’s context for all pages.
 html_context = {
     # Google Analytics ID.
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -154,8 +154,8 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
 
 
 _SPHINX_AIRFLOW_THEME_URL = (
-    "https://github.com/apache/airflow-site/releases/download/v0.0.1/"
-    "sphinx_airflow_theme-0.0.1-py3-none-any.whl"
+    "https://github.com/apache/airflow-site/releases/download/v0.0.2/"
+    "sphinx_airflow_theme-0.0.2-py3-none-any.whl"
 )
 
 # 'Start dependencies group' and 'Start dependencies group' are mark for ./scripts/ci/check_order_setup.py


Test patch:
Repository: apache/airflow

Issue:
Version Incompatibility in 1.10.14 resulting from snowflake workaround
**Apache Airflow version**: 1.10.14
**Environment**: Python 3.8
- **Install tools**: pipenv

**What happened**:

So I appear to have stumbled across a version incompatibility for `airflow-1.10.14`
apache-airflow[aws] was modified with this [pr](https://github.com/apache/airflow/commit/3f438461498b2f6c13671fed8f70a6a12a51f418#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7R185)  to restrict boto to boto3~=1.10,<1.11 with the comment for snowflake
When attempting to install the latest amazon-provider backport apache-airflow-backport-providers-amazon==2020.11.23->-r I’m faced with an incompatible version in resolved dependancies as the backport requires boto3<2.0.0,>=1.12.0

attempting to install `aws` extra and amazon-provide backport results in a version incompatibility error.

```
There are incompatible versions in the resolved dependencies:
  boto3<1.11,~=1.10 (from apache-airflow[aws]==1.10.14->-r /var/folders/dw/8s4dltks7bg5ws8kht1wfbqw0000gp/T/pipenvtfsk4ltbrequirements/pipenv-ukucfwxw-constraints.txt (line 2))
  boto3<2.0.0,>=1.12.0 (from apache-airflow-backport-providers-amazon==2020.11.23->-r /var/folders/dw/8s4dltks7bg5ws8kht1wfbqw0000gp/T/pipenvtfsk4ltbrequirements/pipenv-ukucfwxw-constraints.txt (line 3))
```

**What you expected to happen**:

Airflow has compatible versions.

**How to reproduce it**:

<details><summary> 1. Create Pipfile with the following contents</summary> 

```
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
apache-airflow = {extras = ["aws"],version = "*"}
# https://github.com/apache/airflow/blob/master/README.md#backport-packages
apache-airflow-backport-providers-amazon = "*"

[requires]
python_version = "3.8"
```
</details>
2. run `pipenv lock`




**Anything else we need to know**:

Suggested solution: move all snowflake version modifications out of related airflow extras and directly into the snowflake extra.
Slack Thread: https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1608057981159400



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -182,7 +182,7 @@ def write_version(filename=os.path.join(*[my_dir, "airflow", "git_version"])):
     'atlasclient>=0.1.2',
 ]
 aws = [
-    'boto3~=1.10,<1.11',  # required by snowflake
+    'boto3~=1.10',
 ]
 azure_blob_storage = [
     'azure-storage>=0.34.0, <0.37.0',
@@ -390,6 +390,7 @@ def write_version(filename=os.path.join(*[my_dir, "airflow", "git_version"])):
     'websocket-client<0.55.0'
 ]
 snowflake = [
+    'boto3<1.11',
     'snowflake-connector-python>=1.5.2',
     'snowflake-sqlalchemy>=1.1.0',
 ]


Test patch:
Repository: apache/airflow

Issue:
The "ldap" extra misses libldap dependency
The 'ldap' provider misses 'ldap' extra dep (which adds ldap3 pip dependency).


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -317,6 +317,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
 kylin = ['kylinpy>=2.6']
 ldap = [
     'ldap3>=2.5.1',
+    'python-ldap',
 ]
 mongo = [
     'dnspython>=1.13.0,<2.0.0',


Test patch:
Repository: apache/airflow

Issue:
Add 'mongo_collection' to template_fields in MongoToS3Operator
<!--

Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

**Description**

<!-- A short description of your feature -->

Make `MongoToS3Operator` `mongo_collection` parameter templated.

**Use case / motivation**

<!-- What do you want to happen?

Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.

-->

This would allow for passing a templated mongo collection from other tasks, such as a mongo collection used as data destination by using `S3Hook`. For instance, we could use templated mongo collection to write data for different dates in different collections by using: `mycollection.{{ ds_nodash }}`.

**Are you willing to submit a PR?**

<!--- We accept contributions! -->

Yes.

**Related Issues**

<!-- Is there currently another issue associated with this? -->

N/A



Discussion:

Resolving patch:
diff --git a/airflow/providers/amazon/aws/transfers/mongo_to_s3.py b/airflow/providers/amazon/aws/transfers/mongo_to_s3.py
--- a/airflow/providers/amazon/aws/transfers/mongo_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/mongo_to_s3.py
@@ -38,7 +38,7 @@ class MongoToS3Operator(BaseOperator):
                 to perform transformations unique to those operators needs
     """
 
-    template_fields = ['s3_key', 'mongo_query']
+    template_fields = ['s3_key', 'mongo_query', 'mongo_collection']
     # pylint: disable=too-many-instance-attributes
 
     @apply_defaults


Test patch:
diff --git a/tests/providers/amazon/aws/transfers/test_mongo_to_s3.py b/tests/providers/amazon/aws/transfers/test_mongo_to_s3.py
--- a/tests/providers/amazon/aws/transfers/test_mongo_to_s3.py
+++ b/tests/providers/amazon/aws/transfers/test_mongo_to_s3.py
@@ -65,7 +65,7 @@ def test_init(self):
         self.assertEqual(self.mock_operator.s3_key, S3_KEY)
 
     def test_template_field_overrides(self):
-        self.assertEqual(self.mock_operator.template_fields, ['s3_key', 'mongo_query'])
+        self.assertEqual(self.mock_operator.template_fields, ['s3_key', 'mongo_query', 'mongo_collection'])
 
     def test_render_template(self):
         ti = TaskInstance(self.mock_operator, DEFAULT_DATE)
Repository: apache/airflow

Issue:
Update slackapiclient / slack_sdk to v3
Hello,

Slack has released updates to its library and we can start using it for it. 

We especially like one change.

> slack_sdk has no required dependencies. This means aiohttp is no longer automatically resolved.

I've looked through the documentation and it doesn't look like a difficult task, but I think it's still worth testing.
More info: https://slack.dev/python-slack-sdk/v3-migration/index.html#from-slackclient-2-x

Best regards,
Kamil Breguła


Discussion:
Can I take this up?
@jyotidhiman0610 I assigned you to this ticket ;-) 

Resolving patch:
diff --git a/airflow/providers/slack/hooks/slack.py b/airflow/providers/slack/hooks/slack.py
--- a/airflow/providers/slack/hooks/slack.py
+++ b/airflow/providers/slack/hooks/slack.py
@@ -18,7 +18,7 @@
 """Hook for Slack"""
 from typing import Any, Optional
 
-from slack import WebClient
+from slack_sdk import WebClient
 
 from airflow.exceptions import AirflowException
 from airflow.hooks.base import BaseHook
@@ -41,7 +41,7 @@ class SlackHook(BaseHook):  # noqa
         slack_hook.call("chat.postMessage", json={"channel": "#random", "text": "Hello world!"})
 
         # Call method from Slack SDK (you have to handle errors yourself)
-        #  For more details check https://slack.dev/python-slackclient/basic_usage.html#sending-a-message
+        #  For more details check https://slack.dev/python-slack-sdk/web/index.html#messaging
         slack_hook.client.chat_postMessage(channel="#random", text="Hello world!")
 
     :param token: Slack API token
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -370,7 +370,7 @@ def _load_config():
     'qds_sdk',
     'redis',
     'simple_salesforce',
-    'slackclient',
+    'slack_sdk',
     'smbclient',
     'snowflake',
     'sshtunnel',
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -417,7 +417,7 @@ def get_sphinx_theme_version() -> str:
 ]
 singularity = ['spython>=0.0.56']
 slack = [
-    'slackclient>=2.0.0,<3.0.0',
+    'slack_sdk>=3.0.0,<4.0.0',
 ]
 snowflake = [
     # The `azure` provider uses legacy `azure-storage` library, where `snowflake` uses the


Test patch:
diff --git a/tests/providers/slack/hooks/test_slack.py b/tests/providers/slack/hooks/test_slack.py
--- a/tests/providers/slack/hooks/test_slack.py
+++ b/tests/providers/slack/hooks/test_slack.py
@@ -20,7 +20,7 @@
 from unittest import mock
 
 import pytest
-from slack.errors import SlackApiError
+from slack_sdk.errors import SlackApiError
 
 from airflow.exceptions import AirflowException
 from airflow.providers.slack.hooks.slack import SlackHook