Repository: apache/airflow

Issue:
Scheduler Pickling - conflict with 'session' parameter defined in both args and kwargs
Dear Airflow Maintainers,

Before I tell you about my issue, let me describe my environment:
# Environment
- **Version of Airflow:** master (1db892b)
- **Example code to reproduce the bug:** Using airflow cli: `airflow scheduler -p`
- **Stack trace:**

```
[2016-03-28 19:11:19,952] {jobs.py:656} ERROR - pickle() got multiple values for keyword argument 'session'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow-1.6.2-py2.7.egg/airflow/jobs.py", line 653, in _execute
    self.process_dag(dag, executor)
  File "/usr/local/lib/python2.7/dist-packages/airflow-1.6.2-py2.7.egg/airflow/jobs.py", line 459, in process_dag
    pickle_id = dag.pickle(session).id
  File "/usr/local/lib/python2.7/dist-packages/airflow-1.6.2-py2.7.egg/airflow/utils.py", line 143, in wrapper
    result = func(*args, **kwargs)
TypeError: pickle() got multiple values for keyword argument 'session'
```
- **Operating System:** Ubuntu 14.04
- **Python Version:** 2.7.6

Now that you know a little about me, let me tell you about the issue I am having:

The Airflow Scheduler is unable to pickle DAGs using `airflow scheduler -p`. The reason is as follows:

For each DAG, the pickle-enabled scheduler attempts to pickle the DAG:

```
def process_dag(self, dag, executor):  # shortened for brevity 
        session = settings.Session()

        # picklin'
        pickle_id = None
        if self.do_pickle and self.executor.__class__ not in (
                executors.LocalExecutor, executors.SequentialExecutor):
            pickle_id = dag.pickle(session).id
```

Here is the header for the `pickle` function of the DAG class:

```
@provide_session
def pickle(self, session=None):
```

Note that `pickle`
1. has an optional argument named 'session' which is used as a positional argument in `process_dag`
2. uses a decorator function `provide_session`, which is seen below

```
def provide_session(func):  # shortened for brevity
    @wraps(func)
    def wrapper(*args, **kwargs):
        needs_session = False
        if 'session' not in kwargs:  # args is not checked!
            needs_session = True
            session = settings.Session()
            kwargs['session'] = session
        result = func(*args, **kwargs)
        if needs_session:
            session.expunge_all()
            session.commit()
            session.close()
        return result
    return wrapper
```

Because `session` is used as a positional argument in `process_dag`, the `provide_session` wrapper does not find a 'session' argument among the keyword arguments and therefore provides one. When the interior function is called, the positional and keyword arguments are reconciled and the two respective `session`s collide to produce the error shown above.

For reference, see [this stack overflow post](http://stackoverflow.com/questions/21764770/typeerror-got-multiple-values-for-argument)
- **What did you expect to happen?** No errors, DAGs should be scheduled normally.
- **What happened instead?** An error with aforementioned stack trace.
## Reproduction Steps
1. Modify airflow.cfg to use `MesosExecutor`
2. from the CLI, input `airflow scheduler -p`



Discussion:

Resolving patch:
diff --git a/airflow/utils/db.py b/airflow/utils/db.py
--- a/airflow/utils/db.py
+++ b/airflow/utils/db.py
@@ -42,10 +42,14 @@ def provide_session(func):
     @wraps(func)
     def wrapper(*args, **kwargs):
         needs_session = False
-        if 'session' not in kwargs:
+        arg_session = 'session'
+        func_params = func.__code__.co_varnames
+        session_in_args = arg_session in func_params and \
+            func_params.index(arg_session) < len(args)
+        if not (arg_session in kwargs or session_in_args):
             needs_session = True
             session = settings.Session()
-            kwargs['session'] = session
+            kwargs[arg_session] = session
         result = func(*args, **kwargs)
         if needs_session:
             session.expunge_all()


Test patch: