Repository: apache/airflow

Issue:
depend_on_past strange behavior
Dear Airflow Maintainers,

my environment is 
- Airflow version: 1.7.0
- Airflow components:  webserver and scheduler with a postgres database and LocalExecutor
- Airflow config

> arallelism = 32
> dag_concurrency = 16
> max_active_runs_per_dag = 16
- Python Version: 2.7.6
- Operating System: Linux ubuntu 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
- Python packages: 
  iabel==1.3, Flask==0.10.1, Flask-Admin==1.4.0, Flask-Cache==0.13.1, Flask-Login==0.2.11, Flask-WTF==0.12, Jinja2==2.8, Landscape-Client==14.12, Mako==1.0.3, Markdown==2.6.5, MarkupSafe==0.23, PAM==0.4.2, Pygments==2.0.2, SQLAlchemy==1.0.9, Twisted-Core==13.2.0, WTForms==2.0.2, Werkzeug==0.11.2, airflow==1.7.0, alembic==0.8.3, apt-xapian-index==0.45, argparse==1.2.1, cffi==1.3.1, chardet==2.0.1, chartkick==0.4.2, colorama==0.2.5, configobj==4.7.2, croniter==0.3.10, cryptography==1.1.2, dill==0.2.4, enum34==1.1.1, future==0.15.2, gunicorn==19.3.0, html5lib==0.999, idna==2.0, ipaddress==1.0.15, itsdangerous==0.24, numpy==1.10.1, pandas==0.17.1, psycopg2==2.6.1, pyOpenSSL==0.13, pyasn1==0.1.9, pycparser==2.14, pyserial==2.6, python-apt==0.9.3.5ubuntu1, python-dateutil==2.4.2, python-debian==0.1.21-nmu2ubuntu2, python-editor==0.5, python-telegram-bot==3.4, pytz==2015.7, requests==2.2.1, setproctitle==1.1.9, six==1.5.2, ssh-import-id==3.21, thrift==0.9.3, urllib3==1.7.1, vertica-python==0.5.5, wheel==0.24.0, wsgiref==0.1.2, zope.interface==4.0.5

I have the following DAG:

``` python
from airflow import DAG
from airflow.operators import PythonOperator
from datetime import datetime

import logging
import time

default_args = {
    'owner': 'airflow',
    'depends_on_past': True,
    'start_date': datetime(2016, 4, 24),
}

dag_name = 'dp_test'

dag = DAG(
        dag_name,
        default_args=default_args,
        schedule_interval='10 1 * * *')

def cb(**kw):
        time.sleep(10)
        logging.info('Done %s' % kw['ds'])

d = PythonOperator(task_id="delay", provide_context=True, python_callable=cb, dag=dag)
```

It is run by the scheduer the following way:

the first run scheduled__2016-04-24T00:00:00 completes successfully
the second run scheduled__2016-04-24T01:10:00  is marked as running but the task is not run and it keeps hanging in this state.
I tried the same dag with the SequentialExecuter and also from the latest Airflow version from git. The behavior doesn't change.

Another strange thing that bothers me is the run scheduled at 2016-04-24T00:00:00. The dag schedule interval doesn't suggest such run.



Discussion:
Hi @aadim , thanks for being the thorough with your description. It allowed my to dive in properly.

On why the scheduler starts the job 2016-04-24T00:00:00 (I am _not_ saying it is correct, I dont think it is). This is because the tasks in the DAG also get as start date "2016-04-24" and this is used to set the next_run_date in models.py:

```
                    # set start_date based on tasks
                    task_start_dates = [t.start_date for t in dag.tasks]
                    if task_start_dates:
                        next_run_date = min(task_start_dates)
                    else:
                        next_run_date = None
                    logging.info("Set next run date to {} based on task_start_date".format(next_run_date))
```

At the moment I, personally, think that next_run_date should be set to min(task_start_dates) + schedule_interval. What do you think @mistercrunch @jlowin ?

On the first issue it gets interesting here:

```
[2016-04-25 12:32:58,252] {models.py:2661} INFO - Marking run <DagRun dp_test @ 2016-04-24 00:00:00: scheduled__2016-04-24T00:00:00, externally triggered: False> successful
[2016-04-25 12:32:58,254] {jobs.py:520} INFO - Getting list of tasks to skip for active runs.
[2016-04-25 12:32:58,254] {jobs.py:536} INFO - Checking dependencies on 0 tasks instances, minus 0 skippable ones
[2016-04-25 12:32:58,266] {jobs.py:785} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-04-25 12:32:58,266] {jobs.py:788} INFO - Loop took: 0.261704 seconds
[2016-04-25 12:32:58,269] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-04-25 12:32:58,269] {models.py:311} INFO - Failing jobs without heartbeat after 2016-04-25 12:30:43.269621
/Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages/airflow-1.7.0-py2.7.egg/airflow/bin/cli.py:287: DeprecationWarning: The S3_LOG_FOLDER conf key has been replaced by REMOTE_BASE_LOG_FOLDER. Your conf still works but please update airflow.cfg to ensure future compatibility.
  DeprecationWarning)
[2016-04-25 12:33:03,010] {jobs.py:618} INFO - Prioritizing 0 queued jobs
[2016-04-25 12:33:03,019] {jobs.py:772} INFO - Starting 1 scheduler jobs
[2016-04-25 12:33:03,031] {jobs.py:387} INFO - Scheduling dag: dp_test Schedule interval:10 1 * * *
[2016-04-25 12:33:03,036] {jobs.py:420} INFO - last scheduled run: 2016-04-24 00:00:00
[2016-04-25 12:33:03,036] {models.py:2479} INFO - Shedule interval:10 1 * * * ddtm:2016-04-24 00:00:00
[2016-04-25 12:33:03,037] {models.py:2482} INFO - Cron Next: 2016-04-24 01:10:00
[2016-04-25 12:33:03,038] {jobs.py:455} INFO - next_run_date: 2016-04-25 01:10:00
[2016-04-25 12:33:03,038] {jobs.py:458} INFO - (after dag.check_start) next_run_date: 2016-04-25 01:10:00
[2016-04-25 12:33:03,038] {models.py:2479} INFO - Shedule interval:10 1 * * * ddtm:2016-04-25 01:10:00
[2016-04-25 12:33:03,038] {models.py:2482} INFO - Cron Next: 2016-04-26 01:10:00
[2016-04-25 12:33:03,038] {jobs.py:470} INFO - final next_run_date: 2016-04-25 01:10:00
```

So, on my machine, the next run date gets set to 2016-04-25 01:10:00 which wont happen as it is in the past. It is also not marked running. I will dive in a little further now with dates in the future.

Closing in favor of Jira issue. Please keep an eye on that one and/or add a vote.


Resolving patch:
diff --git a/airflow/models.py b/airflow/models.py
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -868,6 +868,7 @@ def set_state(self, state, session):
         self.start_date = datetime.now()
         self.end_date = datetime.now()
         session.merge(self)
+        session.commit()
 
     def is_queueable(
             self,
@@ -1106,7 +1107,6 @@ def are_dependencies_met(
             session=session, successes=successes, skipped=skipped,
             failed=failed, upstream_failed=upstream_failed, done=done,
             flag_upstream_failed=flag_upstream_failed)
-        session.commit()
         if verbose and not satisfied:
             logging.warning("Trigger rule `{}` not satisfied".format(task.trigger_rule))
         return satisfied


Test patch: