internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: apache/airflow

Issue:
ExternalTaskSensor logs a poke message that lists the current execution_date, it should show the execution_date it is poking for.
The log for ExternalTaskSensor shows a poke for "dag.task on context[executino_date]" which is the execution_date of the running task.  But it would be more helpful to know what execution date it is actually looking for.



Discussion:

Resolving patch:
diff --git a/airflow/operators/sensors.py b/airflow/operators/sensors.py
--- a/airflow/operators/sensors.py
+++ b/airflow/operators/sensors.py
@@ -177,18 +177,18 @@ def __init__(
         self.external_task_id = external_task_id
 
     def poke(self, context):
+        if self.execution_delta:
+            dttm = context['execution_date'] - self.execution_delta
+        else:
+            dttm = context['execution_date']
+
         logging.info(
             'Poking for '
             '{self.external_dag_id}.'
             '{self.external_task_id} on '
-            '{context[execution_date]} ... '.format(**locals()))
+            '{dttm} ... '.format(**locals()))
         TI = TaskInstance
 
-        if self.execution_delta:
-            dttm = context['execution_date'] - self.execution_delta
-        else:
-            dttm = context['execution_date']
-
         session = settings.Session()
         count = session.query(TI).filter(
             TI.dag_id == self.external_dag_id,


Test patch:
Repository: apache/airflow

Issue:
None configuration variables being converted to 'None'
Unspecified parameters in airflow.cfg are being returned as 'None' instead of None.

For example, try creating an airflow.cfg without a core > plugins_folder configuration and then fire up python

```
>>> from airflow import configuration
>>> configuration.get('core', 'airflow_home')
'/data/airflow'
>>> configuration.get('core', 'plugins_folder')
'None'
```



Discussion:

Resolving patch:
diff --git a/airflow/configuration.py b/airflow/configuration.py
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -34,6 +34,8 @@ def expand_env_var(env_var):
     `expandvars` and `expanduser` until interpolation stops having
     any effect.
     """
+    if not env_var:
+        return env_var
     while True:
         interpolated = os.path.expanduser(os.path.expandvars(str(env_var)))
         if interpolated == env_var:


Test patch:
Repository: apache/airflow

Issue:
Multibyte Upload Behavior of S3Hook.load_file Causing Issues
When using the `S3Hook.load_file`, if `multipart_bytes` is set, uploads will fail if the file is smaller than `multipart_bytes`. The acute cause can be found [here](https://github.com/airbnb/airflow/blob/master/airflow/hooks/S3_hook.py#L316).

What happens is when the method calculates the number of chunks to use, it gets zero. The range iterator thus returns an empty set and no parts get uploaded. When the `complete_upload` method of the multipart uploader gets called, the whole thing crashes.

We've created a plugin to copy files from HDFS to S3 and it calls an `S3Hook`. We want to use multipart uploads if the file is larger than `multipart_bytes`. It would be nice if the hook could make that distinction and automatically choose whether to use multipart upload or not. At the very least, it should definitely not fail.

The easy way to fix this would be to stat the file to get its size, and then branch to either a multipart upload or a regular upload. Would be happy to submit a PR for the fix if that sounds reasonable.



Discussion:
Sounds very reasonable. I would be happy to review the PR.

I'm on it


Resolving patch:
diff --git a/airflow/hooks/S3_hook.py b/airflow/hooks/S3_hook.py
--- a/airflow/hooks/S3_hook.py
+++ b/airflow/hooks/S3_hook.py
@@ -283,7 +283,7 @@ def load_file(
             key,
             bucket_name=None,
             replace=False,
-            multipart_bytes=None):
+            multipart_bytes=5 * (1024 ** 3)):
         """
         Loads a local file to S3
 
@@ -298,8 +298,10 @@ def load_file(
             error will be raised.
         :type replace: bool
         :param multipart_bytes: If provided, the file is uploaded in parts of
-            this size (minimum 5242880). If None, the whole file is uploaded at
-            once.
+            this size (minimum 5242880). The default value is 5GB, since S3
+            cannot accept non-multipart uploads for files larger than 5GB. If
+            the file is smaller than the specified limit, the option will be
+            ignored.
         :type multipart_bytes: int
         """
         if not bucket_name:
@@ -309,9 +311,11 @@ def load_file(
         if not replace and key_obj:
             raise ValueError("The key {key} already exists.".format(
                 **locals()))
-        if multipart_bytes:
+
+        key_size = os.path.getsize(filename)
+        if multipart_bytes and key_size >= multipart_bytes:
+            # multipart upload
             from filechunkio import FileChunkIO
-            key_size = os.path.getsize(filename)
             mp = bucket.initiate_multipart_upload(key_name=key)
             total_chunks = int(math.ceil(key_size / multipart_bytes))
             sent_bytes = 0
@@ -329,6 +333,7 @@ def load_file(
                 raise
             mp.complete_upload()
         else:
+            # regular upload
             if not key_obj:
                 key_obj = bucket.new_key(key_name=key)
             key_size = key_obj.set_contents_from_filename(filename,


Test patch:
Repository: apache/airflow

Issue:
Add ssl extra arguments to postgres hook
This enables one to specify items such as ssl certificate paths in the extra field of a postgres connection object. Rather than simply pass-through everything from the extra field, I followed the existing code pattern of selecting for specific arguments, as demonstrated in other hooks. I tested this in Python 2.7.6 and Python 2.7.10.



Discussion:
@jthomas123, thanks for the PR, but it looks like it isn't python 3 compatible, let me add a comment in the code

@jthomas123 do you still want to update this PR or should I close this?


Resolving patch:
diff --git a/airflow/hooks/postgres_hook.py b/airflow/hooks/postgres_hook.py
--- a/airflow/hooks/postgres_hook.py
+++ b/airflow/hooks/postgres_hook.py
@@ -6,6 +6,8 @@
 class PostgresHook(DbApiHook):
     '''
     Interact with Postgres.
+    You can specify ssl parameters in the extra field of your connection
+    as ``{"sslmode": "require", "sslcert": "/path/to/cert.pem", etc}``.
     '''
     conn_name_attr = 'postgres_conn_id'
     default_conn_name = 'postgres_default'
@@ -13,9 +15,14 @@ class PostgresHook(DbApiHook):
 
     def get_conn(self):
         conn = self.get_connection(self.postgres_conn_id)
-        return psycopg2.connect(
+        conn_args = dict(
             host=conn.host,
             user=conn.login,
             password=conn.password,
             dbname=conn.schema,
             port=conn.port)
+        # check for ssl parameters in conn.extra
+        for arg_name, arg_val in conn.extra_dejson.items():
+            if arg_name in ['sslmode', 'sslcert', 'sslkey', 'sslrootcert', 'sslcrl']:
+                conn_args[arg_name] = arg_val
+        return psycopg2.connect(**conn_args)


Test patch:
Repository: apache/airflow

Issue:
Warning e-mails are sent without the Date header
The warning e-mails don't have the "Date" header.



Discussion:

Resolving patch:
diff --git a/airflow/utils.py b/airflow/utils.py
--- a/airflow/utils.py
+++ b/airflow/utils.py
@@ -12,6 +12,7 @@
 from email.mime.text import MIMEText
 from email.mime.multipart import MIMEMultipart
 from email.mime.application import MIMEApplication
+from email.Utils import formatdate
 import errno
 from functools import wraps
 import imp
@@ -529,6 +530,7 @@ def send_email_smtp(to, subject, html_content, files=None, dryrun=False):
     msg['Subject'] = subject
     msg['From'] = SMTP_MAIL_FROM
     msg['To'] = ", ".join(to)
+    msg["Date"] = formatdate(localtime=True)
     mime_text = MIMEText(html_content, 'html')
     msg.attach(mime_text)
 


Test patch:
Repository: apache/airflow

Issue:
DAG with schedule interval '@once' cannot be scheduled
Dear Airflow Maintainers,

Before I tell you about my issue, let me describe my environment:
# Environment
- Version of Airflow (e.g. a release version, running your own fork, running off master -- provide a git log snippet) : **1.7.0**
- Example code to reproduce the bug (as a code snippet in markdown) **example_xcom from stock examples**
- Stack trace if applicable:

```
[2016-03-30 15:36:49,858] {models.py:204} INFO - Importing /usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_xcom.py
[2016-03-30 15:36:49,863] {models.py:296} INFO - Loaded DAG <DAG: example_xcom>
[2016-03-30 15:36:49,874] {jobs.py:642} ERROR - list index out of range
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 638, in _execute
    self.schedule_dag(dag)
  File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 397, in schedule_dag
    next_run_date = dag.date_range(latest_run, -5)[0]
IndexError: list index out of range
```
- Operating System: (Windows Version or `$ uname -a`) : **Ubuntu 14.04**
- Python Version: `$ python --version` **2.7.6**

Now that you know a little about me, let me tell you about the issue I am having:
# Description of Issue
- What did you expect to happen? **For the DAG to be scheduled**
- What happened instead? **Error shown above**
- Here is how you can reproduce this issue on your machine:
## Reproduction Steps
1. Make sure the stock example DAG `example_xcom` is unpaused
2. If there is no example_xcom example, create any DAG with a schedule_interval of `@once`
3. Run the scheduler



Discussion:

Resolving patch:
diff --git a/airflow/jobs.py b/airflow/jobs.py
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -401,7 +401,9 @@ def schedule_dag(self, dag):
                             DagRun.run_id.like(DagRun.ID_PREFIX+'%')))
             last_scheduled_run = qry.scalar()
             next_run_date = None
-            if not last_scheduled_run:
+            if dag.schedule_interval == '@once' and not last_scheduled_run:
+                next_run_date = datetime.now()
+            elif not last_scheduled_run:
                 # First run
                 TI = models.TaskInstance
                 latest_run = (
@@ -417,8 +419,6 @@ def schedule_dag(self, dag):
                     next_run_date = min([t.start_date for t in dag.tasks])
             elif dag.schedule_interval != '@once':
                 next_run_date = dag.following_schedule(last_scheduled_run)
-            elif dag.schedule_interval == '@once' and not last_scheduled_run:
-                next_run_date = datetime.now()
 
             # this structure is necessary to avoid a TypeError from concatenating
             # NoneType


Test patch:
Repository: apache/airflow

Issue:
Inconsistent 'owner' field in examples
Dear Airflow Maintainers,
### Environment
- Version of Airflow (e.g. a release version, running your own fork, running off master -- provide a git log snippet): **1.7.0**
- Screen shots of your DAG's graph and tree views:
  ![owner field](http://i.imgur.com/9U8rzil.png)
- Operating System: (Windows Version or `$ uname -a`) **Ubuntu 14.04**
- Python Version: `$ python --version` **2.7**
### Description of Issue
- What did you expect to happen? **All of the examples have a consistent owner, probably 'airflow'**
- What happened instead? **[Some](https://github.com/airbnb/airflow/blob/master/airflow/example_dags/example_python_operator.py) examples have `airflow`, [some](https://github.com/airbnb/airflow/blob/master/airflow/example_dags/example_passing_params_via_test_command.py) have `me`**
### Reproduction Steps
1. install airflow 1.7.0 via pip
2. start the webserver
3. look at the web UI, probably http://localhost:8080

**Inconsistent hard-coding in the examples will likely lead to confusion for new users.**



Discussion:

Resolving patch:
diff --git a/airflow/example_dags/example_trigger_controller_dag.py b/airflow/example_dags/example_trigger_controller_dag.py
--- a/airflow/example_dags/example_trigger_controller_dag.py
+++ b/airflow/example_dags/example_trigger_controller_dag.py
@@ -37,7 +37,7 @@ def conditionally_trigger(context, dag_run_obj):
 
 # Define the DAG
 dag = DAG(dag_id='example_trigger_controller_dag',
-          default_args={"owner": "me",
+          default_args={"owner": "airflow",
                         "start_date": datetime.now()},
           schedule_interval='@once')
 


Test patch:
diff --git a/airflow/example_dags/example_passing_params_via_test_command.py b/airflow/example_dags/example_passing_params_via_test_command.py
--- a/airflow/example_dags/example_passing_params_via_test_command.py
+++ b/airflow/example_dags/example_passing_params_via_test_command.py
@@ -19,7 +19,7 @@
 from airflow.operators import BashOperator, PythonOperator
 
 dag = DAG("example_passing_params_via_test_command",
-          default_args={"owner" : "me",
+          default_args={"owner" : "airflow",
                         "start_date":datetime.now()},
           schedule_interval='*/1 * * * *',
           dagrun_timeout=timedelta(minutes=4)
Repository: apache/airflow

Issue:
Replace flask_oauthlib with Authlib

**Description**

flask_oauthlib has been deprecated in favour of Authlib. It would be good if airflow starts using Authlib

**Use case / motivation**

FlaskAppBuilder is now using Authlib. 
Since FlaskAppBuilder is deeply integrated into Airflow, it will be good to also have this Authlib. Flask-oauthlib documentation recommends Authlib

**Related Issues**



Discussion:
Flask OAuthlib has not been updated since May, 2018 (https://pypi.org/project/Flask-OAuthlib/) and depends on OAuthlib < 3.0.0 (also from May, 2018).
We want to start using Authlib. Can you help @scoiatael?

Resolving patch:
diff --git a/airflow/config_templates/default_webserver_config.py b/airflow/config_templates/default_webserver_config.py
--- a/airflow/config_templates/default_webserver_config.py
+++ b/airflow/config_templates/default_webserver_config.py
@@ -70,15 +70,15 @@
 #     'token_key':'access_token',
 #     'icon':'fa-google',
 #         'remote_app': {
-#             'base_url':'https://www.googleapis.com/oauth2/v2/',
-#             'request_token_params':{
+#             'api_base_url':'https://www.googleapis.com/oauth2/v2/',
+#             'client_kwargs':{
 #                 'scope': 'email profile'
 #             },
 #             'access_token_url':'https://accounts.google.com/o/oauth2/token',
 #             'authorize_url':'https://accounts.google.com/o/oauth2/auth',
 #             'request_token_url': None,
-#             'consumer_key': CONSUMER_KEY,
-#             'consumer_secret': SECRET_KEY,
+#             'client_id': GOOGLE_KEY,
+#             'client_secret': GOOGLE_SECRET_KEY,
 #         }
 # }]
 


Test patch:
Repository: apache/airflow

Issue:
Limit all google-cloud api to <2.0.0 as it introduces breaking changes
**Description**

Google Cloud APIs introduced breaking changes in 2.0.0 (https://github.com/googleapis/python-container/blob/master/UPGRADING.md) and they already caused a number of changes. We should (for now - before we migrate to 2.0+ ) limit all our google-cloud deps to <2.0.0

**Use case / motivation**

Make sure that further releases of google cloud APIs are not breaking our builds.




Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -260,31 +260,31 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
 google = [
     'PyOpenSSL',
     'google-ads>=4.0.0',
-    'google-api-python-client>=1.6.0, <2.0.0dev',
-    'google-auth>=1.0.0, <2.0.0dev',
+    'google-api-python-client>=1.6.0,<2.0.0',
+    'google-auth>=1.0.0,<2.0.0',
     'google-auth-httplib2>=0.0.1',
-    'google-cloud-automl>=0.4.0',
-    'google-cloud-bigquery-datatransfer>=0.4.0',
-    'google-cloud-bigtable>=1.0.0',
-    'google-cloud-container>=0.1.1,<2.0',
-    'google-cloud-datacatalog>=0.5.0,<0.8',
+    'google-cloud-automl>=0.4.0,<2.0.0',
+    'google-cloud-bigquery-datatransfer>=0.4.0,<2.0.0',
+    'google-cloud-bigtable>=1.0.0,<2.0.0',
+    'google-cloud-container>=0.1.1,<2.0.0',
+    'google-cloud-datacatalog>=0.5.0, <0.8',  # TODO: we should migrate to 1.0 likely and add <2.0.0 then
     'google-cloud-dataproc>=1.0.1,<2.0.0',
-    'google-cloud-dlp>=0.11.0',
+    'google-cloud-dlp>=0.11.0,<2.0.0',
     'google-cloud-kms>=1.2.1,<2.0.0',
-    'google-cloud-language>=1.1.1',
-    'google-cloud-logging>=1.14.0',
-    'google-cloud-monitoring>=0.34.0',
-    'google-cloud-pubsub>=1.0.0',
-    'google-cloud-redis>=0.3.0',
-    'google-cloud-secret-manager>=0.2.0',
-    'google-cloud-spanner>=1.10.0',
-    'google-cloud-speech>=0.36.3',
-    'google-cloud-storage>=1.16',
-    'google-cloud-tasks>=1.2.1',
-    'google-cloud-texttospeech>=0.4.0,<2',
-    'google-cloud-translate>=1.5.0',
-    'google-cloud-videointelligence>=1.7.0',
-    'google-cloud-vision>=0.35.2',
+    'google-cloud-language>=1.1.1,<2.0.0',
+    'google-cloud-logging>=1.14.0,<2.0.0',
+    'google-cloud-monitoring>=0.34.0,<2.0.0',
+    'google-cloud-pubsub>=1.0.0,<2.0.0',
+    'google-cloud-redis>=0.3.0,<2.0.0',
+    'google-cloud-secret-manager>=0.2.0,<2.0.0',
+    'google-cloud-spanner>=1.10.0,<2.0.0',
+    'google-cloud-speech>=0.36.3,<2.0.0',
+    'google-cloud-storage>=1.16,<2.0.0',
+    'google-cloud-tasks>=1.2.1,<2.0.0',
+    'google-cloud-texttospeech>=0.4.0,<2.0.0',
+    'google-cloud-translate>=1.5.0,<2.0.0',
+    'google-cloud-videointelligence>=1.7.0,<2.0.0',
+    'google-cloud-vision>=0.35.2,<2.0.0',
     'grpcio-gcp>=0.2.2',
     'pandas-gbq',
 ]


Test patch:
Repository: apache/airflow

Issue:
Graph View shows other relations than in DAG
**Apache Airflow version**:

master

**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):

**Environment**: breeze

**What happened**:

This DAG
```python
from airflow import models
from airflow.operators.dummy_operator import DummyOperator
from airflow.utils.dates import days_ago

with models.DAG("test", start_date=days_ago(1), schedule_interval=None,) as dag:
    t1 = DummyOperator(task_id="t1")
    t2 = DummyOperator(task_id="t2")
    t1 >> t2
```

is rendering like that:

<img width="1374" alt="Screenshot 2020-08-27 at 19 59 41" src="https://user-images.githubusercontent.com/9528307/91478403-11d7fb00-e8a0-11ea-91d0-d7d578bcb5a2.png">

**What you expected to happen**:

I expect to see same relations as defined in DAG file

**How to reproduce it**:

Render the example DAG from above.


**Anything else we need to know**:

I'm surprised by this bug 👀 



Discussion:
Seems that the bug was introduced in https://github.com/apache/airflow/commit/8fdcc5760a104b0e4e5558191e0269bf873f8768#diff-e3fba382e484c6174a6fbef925b2e575

Resolving patch:
diff --git a/airflow/www/views.py b/airflow/www/views.py
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -1625,7 +1625,7 @@ def graph(self, session=None):
         def get_downstream(task):
             for downstream_task in task.downstream_list:
                 edge = {
-                    'source_id': downstream_task.task_id,
+                    'source_id': task.task_id,
                     'target_id': downstream_task.task_id,
                 }
                 if edge not in edges:


Test patch:
Repository: apache/airflow

Issue:
Start Date in main DAG view shows redundant information
This is shows:
![Screen Shot 2020-08-16 at 11 30 36](https://user-images.githubusercontent.com/63792998/90334314-81630600-dfd5-11ea-831d-c5ce695b6013.png)

For all DAGs it shows the same YYYY-MM-DD HH:MM as we see on the main view. the (i) tooltip simply doesn't give any more information.

As far as I remember it wasn't like this before.
It used to show the timestamp which the dag started to run.

So in the above image the main DAG was:
2020-07-19 07:00 but the (i) was 2020-07-19 **08:00** 
as 08:00 is the end of the interval of hourly dag so the DAG started to run on 08:00

I don't know why this logic was changed on the UI but if that's stays like that there is no need for the tooltip.


Discussion:

Resolving patch:
diff --git a/airflow/www/views.py b/airflow/www/views.py
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -726,7 +726,9 @@ def last_dagruns(self, session=None):
             return wwwutils.json_response({})
 
         query = session.query(
-            DagRun.dag_id, sqla.func.max(DagRun.execution_date).label('last_run')
+            DagRun.dag_id,
+            sqla.func.max(DagRun.execution_date).label('execution_date'),
+            sqla.func.max(DagRun.start_date).label('start_date'),
         ).group_by(DagRun.dag_id)
 
         # Filter to only ask for accessible and selected dags
@@ -735,7 +737,8 @@ def last_dagruns(self, session=None):
         resp = {
             r.dag_id.replace('.', '__dot__'): {
                 'dag_id': r.dag_id,
-                'last_run': r.last_run.isoformat(),
+                'execution_date': r.execution_date.isoformat(),
+                'start_date': r.start_date.isoformat(),
             } for r in query
         }
         return wwwutils.json_response(resp)


Test patch:
Repository: apache/airflow

Issue:
Replace Airflow Slack Invite old link to short link
Follow up to https://github.com/apache/airflow/pull/10034

https://apache-airflow-slack.herokuapp.com/ to https://s.apache.org/airflow-slack/

<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.

Feel free to ping committers for the review!

In case of existing issue, reference it using one of the following:

closes: #ISSUE
related: #ISSUE

How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->

---
**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).



Discussion:

Resolving patch:
diff --git a/docs/build_docs.py b/docs/build_docs.py
--- a/docs/build_docs.py
+++ b/docs/build_docs.py
@@ -745,7 +745,7 @@ def print_build_errors_and_exit(message) -> None:
 CHANNEL_INVITATION = """\
 If you need help, write to #documentation channel on Airflow's Slack.
 Channel link: https://apache-airflow.slack.com/archives/CJ1LVREHX
-Invitation link: https://s.apache.org/airflow-slack/\
+Invitation link: https://s.apache.org/airflow-slack\
 """
 
 print_build_errors_and_exit("The documentation has errors. Fix them to build documentation.")


Test patch: