Repository: apache/airflow

Issue:
Dag Dependency page not showing anything
**Apache Airflow version**: 2.1.

**Environment**: Ubuntu 20.04

- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): UBUNTU 20.04 LTS
- **Kernel** (e.g. `uname -a`): Linux 20.04.1-Ubuntu SMP Tue Jun 1 09:54:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**: python and pip


**What happened**: After performing the upgrade from 2.0.2 to 2.10 using the guide available in the documentation, Airflow upgraded successfully, Dag dependency page isn't  working as expected.
The DAG dependency page doesn't show the dependency graph.

<!-- (please include exact error messages if you can) -->

**What you expected to happen**: I expected the dag dependency page to show the dags and their dependency in a Graph view

<!-- What do you think went wrong? -->

**How to reproduce it**: Its reproduced by opening these pages every time. 

![Dag Dependency Page](https://user-images.githubusercontent.com/43160555/123028222-6eee0000-d422-11eb-9664-a8c0ee2a6723.png)


How often does this problem occur? Once? Every time etc?
Every time

Any relevant logs to include? Put them here in side a detail tag:
<details><summary>Upgrade Check  Log</summary> 

/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/configuration.py:34                                            
6 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/configuration.py:34                                            
6 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/plugins_manager.py:                                            
239 DeprecationWarning: This decorator is deprecated.In previous versions, all subclasses of BaseOperator must use apply_default decorator for the`default_args` feature to work properly.
In current version, it is optional. The decorator is applied automatically using the metaclass.
/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/configuration.py:34                                           
6 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.

 </details>




Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

@SamiRehmanNostra Could you open the browser console and screenshot any error messages in the logs?

Edit:
I was able to reproduce on a fresh build. Please check that your scheduler is running. It seems that the scheduler is required to load the dag dependency data. We should see if we can get the data with just the webserver.

Hi @bbovenzi ,

Please see the screenshot attached below:
![Browser Screenshot](https://user-images.githubusercontent.com/43160555/123727628-46a84a80-d8d5-11eb-9b0e-cc5c17c51dec.png)


`d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,20)scale(0)".
i @ d3.min.js:1
Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
i @ d3.min.js:1
d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
i @ d3.min.js:1
d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
i @ d3.min.js:1
d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
i @ d3.min.js:1
d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
i @ d3.min.js:1
d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)".
i @ d3.min.js:1
`
Let me know if you need any more information.

Thank you,
Sami
@SamiRehmanNostra Do you by chance have no dags at all?
@ashb I have about 13 active dags, all running on different schedules.
Weirdly enough I was installing a bunch of ODBC drivers today and restarted the web-server about 3 times. The issue seems to have resolved itself.

![image](https://user-images.githubusercontent.com/43160555/123765008-ac61fa00-d908-11eb-80af-297a9f8269e9.png)

 Not sure what has changed. Let me know if I need to share any logs.
Hmmm tricky. Without having a reliable way of reproducing this it's going to be tricky to fix.

Did you clear the DB when restarting the webserver? (Not likely, just checking)
@ashb I have been able to reliably replicate it. If I only run the webserver the returned [nodes and edges](https://github.com/apache/airflow/blob/main/airflow/www/views.py#L4130-L4131`) will be empty arrays. As soon as I run the scheduler both will populate and the graph will render.
@bbovenzi Cool -- lets protect against that case as it is _a_ cause of this problem.
Yeah, I'll add some sort of "Dependency data could not be loaded right now" error message.
@ashb I didn't reset the DB. Seems like @bbovenzi has narrowed it down. Thank you
@bbovenzi Its back people!

![image](https://user-images.githubusercontent.com/43160555/124052826-82244f80-da62-11eb-9a87-92a9749b9337.png)

@bbovenzi @ashb - It appears like for a fresh build this piece of code is setting value as None.
https://github.com/apache/airflow/blob/9a1fc19a627662122dd8dacee2ef61c3edf97b85/airflow/serialization/serialized_objects.py#L682

We just upgraded to 2.1.1 and see the dag dependencies empty. Tried restarting the scheduler multiple times but doesn't help.

Could it be possible that task_type in the below piece of code is BaseOperator instead of the actual operator? IDK, haven't been able to reproduce this issue in my local.
https://github.com/apache/airflow/blob/9a1fc19a627662122dd8dacee2ef61c3edf97b85/airflow/serialization/serialized_objects.py#L341

I see this error message on the UI. Not sure if it is an underlying UI code issue or issue with the way things are being set.
<img> 
![image](https://user-images.githubusercontent.com/9946408/125763690-c101ebd6-147f-405e-8735-e1c4051537b9.png)



Resolving patch:
diff --git a/airflow/www/views.py b/airflow/www/views.py
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -4294,7 +4294,10 @@ def list(self):
         """Display DAG dependencies"""
         title = "DAG Dependencies"
 
-        if timezone.utcnow() > self.last_refresh + self.refresh_interval:
+        if not self.nodes or not self.edges:
+            self._calculate_graph()
+            self.last_refresh = timezone.utcnow()
+        elif timezone.utcnow() > self.last_refresh + self.refresh_interval:
             max_last_updated = SerializedDagModel.get_max_last_updated_datetime()
             if max_last_updated is None or max_last_updated > self.last_refresh:
                 self._calculate_graph()
@@ -4305,7 +4308,7 @@ def list(self):
             title=title,
             nodes=self.nodes,
             edges=self.edges,
-            last_refresh=self.last_refresh.strftime("%Y-%m-%d %H:%M:%S"),
+            last_refresh=self.last_refresh,
             arrange=conf.get("webserver", "dag_orientation"),
             width=request.args.get("width", "100%"),
             height=request.args.get("height", "800"),


Test patch: