Issue 1:
I want to know how to handle cases where everything goes wrong, I am not too much worry for the workers, queue or db. I already know how their up time or how they handle crashes.

But, I am still not sure to understand how works airflow. I assume that you a server or a process that know when to add stuff in the queue to respect the dag. Am I right ? if yes, how can I handle a crashes ? (can I duplicate this server/process on a other container ?).

Issue 2:
-->

Apache Airflow version: 1.10.10

Kubernetes version (if you are using kubernetes) (use kubectl version): NA

Environment:

Cloud provider or hardware configuration: AWS

OS (e.g. from /etc/os-release): CentOS Linux release 7.7.1908 (Core)

Kernel (e.g. uname -a): 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools:

Others:

What happened:

After upgrading 1.10.2 to 1.10.10 cant get into WebUI

no RBAC

no DAG Serialization

postgres source\dest 11.5

prepare new db from 1.10.2 snapshot

connect it to Airflow 1.10.10 python env

double check no live connections to db

run airflow upgradedb

Alembic log

(airflow)$ airflow upgradedb
/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/configuration.py:631: DeprecationWarning: Specifying both AIRFLOW_HOME environment variable and airflow_home in the config file is deprecated. Please use only the AIRFLOW_HOME environment variable and remove the config file entry.
warnings.warn(msg, category=DeprecationWarning)
[2020-06-14 08:30:14,925] {rest_api_plugin.py:48} WARNING - [rest_api_plugin/REST_API_PLUGIN_EXPECTED_HTTP_TOKEN] value is empty
DB: postgresql://airflow:***@airflow-aws-test-airflowdb.*.us-east-1.rds.amazonaws.com:5432/airflow
[2020-06-14 08:30:15,214] {db.py:378} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> c8ffec048a3b, add fields to dag
INFO [alembic.runtime.migration] Running upgrade c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag
INFO [alembic.runtime.migration] Running upgrade dd4ecb8fbee3 -> 939bb1e647c8, task reschedule fk on cascade delete
INFO [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 6e96a59344a4, Make TaskInstance.pool not nullable
INFO [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> d38e04c12aa2, add serialized_dag table
Revision ID: d38e04c12aa2
Revises: 6e96a59344a4
Create Date: 2019-08-01 14:39:35.616417
INFO [alembic.runtime.migration] Running upgrade d38e04c12aa2 -> b3b105409875, add root_dag_id to DAG
Revision ID: b3b105409875
Revises: d38e04c12aa2
Create Date: 2019-09-28 23:20:01.744775
INFO [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 74effc47d867, change datetime to datetime2(6) on MSSQL tables
INFO [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
INFO [alembic.runtime.migration] Running upgrade c8ffec048a3b -> a56c9515abdc, Remove dag_stat table
INFO [alembic.runtime.migration] Running upgrade a56c9515abdc, 004c1210f153, 74effc47d867, b3b105409875 -> 08364691d074, Merge the four heads back together
INFO [alembic.runtime.migration] Running upgrade 08364691d074 -> fe461863935f, increase_length_for_connection_password
INFO [alembic.runtime.migration] Running upgrade fe461863935f -> 7939bcff74ba, Add DagTags table
INFO [alembic.runtime.migration] Running upgrade 7939bcff74ba -> a4c2fd67d16b, add pool_slots field to task_instance
INFO [alembic.runtime.migration] Running upgrade a4c2fd67d16b -> 852ae6c715af, Add RenderedTaskInstanceFields table
INFO [alembic.runtime.migration] Running upgrade 852ae6c715af -> 952da73b5eff, add dag_code table

After auth getting next screen constantly

Crush log from web-ui

-------------------------------------------------------------------------------
Node: airflow-master-test-1-10-10.dev.somehost.net
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/www/views.py", line 2330, in index
auto_complete_data=auto_complete_data)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/www/views.py", line 389, in render
return super(AirflowViewMixin, self).render(template, **kwargs)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask_admin/base.py", line 308, in render
return render_template(template, **kwargs)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/templating.py", line 140, in render_template
ctx.app,
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/www/templates/airflow/dags.html", line 20, in top-level template code
{% extends "airflow/master.html" %}
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/www/templates/airflow/master.html", line 20, in top-level template code
{% extends "admin/master.html" %}
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/www/templates/admin/master.html", line 20, in top-level template code
{% extends 'admin/base.html' %}
File "/opt/python-envs/airflow/lib/python3.6/site-packages/flask_admin/templates/bootstrap3/admin/base.html", line 38, in top-level template code
{% block page_body %}
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/www/templates/admin/master.html", line 191, in block "page_body"
{% block body %}
File "/opt/python-envs/airflow/lib/python3.6/site-packages/airflow/www/templates/airflow/dags.html", line 84, in block "body"
<a href="{{ url_for('airflow.'+ dag.get_default_view(), dag_id=dag.dag_id) }}" title="{{ dag.description[0:80] + '...' if dag.description|length > 80 else dag.description }}">
TypeError: object of type 'NoneType' has no len()

What you expected to happen: WebUI works

How to reproduce it:

Prepare some DAGs without description executed under Airflow 1.10.2

Upgrade to Airflow 1.10.10

Anything else we need to know:

DB data which wont work (sensitive data removed)

airflow=> select dag_id,default_view,description from dag where (description = '') IS NOT FALSE limit 10;
dag_id | default_view | description
---------------------------------------------------------------------+--------------+-------------
hello_world | |

(10 rows)

Quick fix which helped (sensitive data removed)

airflow=> update dag SET description = 'Desc' where (description = '') IS NOT FALSE;
UPDATE 14

airflow=> select dag_id,default_view,description from dag where (description = '') IS NOT FALSE limit 10;
dag_id | default_view | description
--------+--------------+-------------
(0 rows)

airflow=> select dag_id,default_view,description from dag limit 10;
dag_id | default_view | description
---------------------------------------------------------------------+--------------+-------------
hello_world | | Desc

(10 rows)