Repository: apache/airflow

Issue:
UndefinedJinjaVariablesRule fails on non-standard python objects
**Apache Airflow version**: 1.10.14

**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):

**Environment**: 

- **Cloud provider or hardware configuration**: X
- **OS** (e.g. from /etc/os-release): X
- **Kernel** (e.g. `uname -a`): X
- **Install tools**: X
- **Others**: X

**What happened**:

The `UndefinedJinjaVariablesRule` appears to expect non-standard python objects to have a `template_fields` attribute, when many (`pathlib.Path`, `None`) do not.

<details><summary>Path example</summary>

```python
========================================================================================================================================================== STATUS ==========================================================================================================================================================

Check for latest versions of apache-airflow and checker...........................................................................................................................................................................................................................................................SUCCESS
Remove airflow.AirflowMacroPlugin class...........................................................................................................................................................................................................................................................................SUCCESS
Chain between DAG and operator not allowed........................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_id is not unique..................................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_type is not nullable..............................................................................................................................................................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom operators.................................................................................................................................................................................................................................................SUCCESS
Fernet is enabled by default......................................................................................................................................................................................................................................................................................FAIL
GCP service account key deprecation...............................................................................................................................................................................................................................................................................SUCCESS
Unify hostname_callable option in core section....................................................................................................................................................................................................................................................................FAIL
Changes in import paths of hooks, operators, sensors and others...................................................................................................................................................................................................................................................SUCCESS
Users must delete deprecated configs for KubernetesExecutor.......................................................................................................................................................................................................................................................FAIL
Legacy UI is deprecated by default................................................................................................................................................................................................................................................................................SUCCESS
Logging configuration has been moved to new section...............................................................................................................................................................................................................................................................FAIL
Removal of Mesos Executor.........................................................................................................................................................................................................................................................................................SUCCESS
No additional argument allowed in BaseOperator....................................................................................................................................................................................................................................................................SUCCESS
Users must set a kubernetes.pod_template_file value...............................................................................................................................................................................................................................................................SKIPPED
SendGrid email uses old airflow.contrib module....................................................................................................................................................................................................................................................................SUCCESS
Changes in import path of remote task handlers....................................................................................................................................................................................................................................................................SUCCESS
Traceback (most recent call last):
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow", line 37, in <module>
    args.func(args)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 88, in run
    all_problems = check_upgrade(formatter, rules)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 37, in check_upgrade
    rule_status = RuleStatus.from_rule(rule)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py", line 44, in from_rule
    result = rule.check()
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 141, in check
    dag_messages = self.iterate_over_dag_tasks(dag)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 128, in iterate_over_dag_tasks
    error_messages = self.iterate_over_template_fields(task)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 117, in iterate_over_template_fields
    self._check_rendered_content(rendered_content, set())
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 63, in _check_rendered_content
    debug_error_messages.update(self._check_rendered_content(value))
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 69, in _check_rendered_content
    return self._nested_check_rendered(rendered_content, seen_oids)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 75, in _nested_check_rendered
    nested_template_fields = rendered_content.template_fields
AttributeError: 'PosixPath' object has no attribute 'template_fields'
```

</details>

<details><summary>None example</summary>

```python
========================================================================================================================================================== STATUS ==========================================================================================================================================================

Check for latest versions of apache-airflow and checker...........................................................................................................................................................................................................................................................SUCCESS
Remove airflow.AirflowMacroPlugin class...........................................................................................................................................................................................................................................................................SUCCESS
Chain between DAG and operator not allowed........................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_id is not unique..................................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_type is not nullable..............................................................................................................................................................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom operators.................................................................................................................................................................................................................................................SUCCESS
Fernet is enabled by default......................................................................................................................................................................................................................................................................................FAIL
GCP service account key deprecation...............................................................................................................................................................................................................................................................................SUCCESS
Unify hostname_callable option in core section....................................................................................................................................................................................................................................................................FAIL
Changes in import paths of hooks, operators, sensors and others...................................................................................................................................................................................................................................................SUCCESS
Users must delete deprecated configs for KubernetesExecutor.......................................................................................................................................................................................................................................................FAIL
Legacy UI is deprecated by default................................................................................................................................................................................................................................................................................SUCCESS
Logging configuration has been moved to new section...............................................................................................................................................................................................................................................................FAIL
Removal of Mesos Executor.........................................................................................................................................................................................................................................................................................SUCCESS
No additional argument allowed in BaseOperator....................................................................................................................................................................................................................................................................SUCCESS
Users must set a kubernetes.pod_template_file value...............................................................................................................................................................................................................................................................SKIPPED
SendGrid email uses old airflow.contrib module....................................................................................................................................................................................................................................................................SUCCESS
Changes in import path of remote task handlers....................................................................................................................................................................................................................................................................SUCCESS
Traceback (most recent call last):
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow", line 37, in <module>
    args.func(args)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 88, in run
    all_problems = check_upgrade(formatter, rules)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 37, in check_upgrade
    rule_status = RuleStatus.from_rule(rule)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py", line 44, in from_rule
    result = rule.check()
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 146, in check
    dag_messages = self.iterate_over_dag_tasks(dag)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 130, in iterate_over_dag_tasks
    error_messages = self.iterate_over_template_fields(task)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 118, in iterate_over_template_fields
    self._check_rendered_content(rendered_content, set())
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 64, in _check_rendered_content
    debug_error_messages.update(self._check_rendered_content(value))
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 70, in _check_rendered_content
    return self._nested_check_rendered(rendered_content, seen_oids)
  File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 76, in _nested_check_rendered
    nested_template_fields = rendered_content.template_fields
AttributeError: 'NoneType' object has no attribute 'template_fields'
```

</details>

**What you expected to happen**:

I expect these types to be handled appropriately by the upgrade rule and not assume that a `template_fields` attribute exists on them.

**How to reproduce it**:

Create a simple DAG with a `PythonOperator` task that has `op_kwargs: {"foo": None}` and run the upgrade check.


Discussion:

Resolving patch:
diff --git a/airflow/upgrade/rules/undefined_jinja_varaibles.py b/airflow/upgrade/rules/undefined_jinja_varaibles.py
--- a/airflow/upgrade/rules/undefined_jinja_varaibles.py
+++ b/airflow/upgrade/rules/undefined_jinja_varaibles.py
@@ -48,9 +48,6 @@ def _check_rendered_content(self, rendered_content, seen_oids=None):
         if isinstance(rendered_content, six.string_types):
             return set(re.findall(r"{{(.*?)}}", rendered_content))
 
-        elif isinstance(rendered_content, (int, float, bool)):
-            return set()
-
         elif isinstance(rendered_content, (tuple, list, set)):
             debug_error_messages = set()
             for element in rendered_content:
@@ -63,11 +60,15 @@ def _check_rendered_content(self, rendered_content, seen_oids=None):
                 debug_error_messages.update(self._check_rendered_content(value))
             return debug_error_messages
 
-        else:
+        elif hasattr(rendered_content, "template_fields"):
             if seen_oids is None:
                 seen_oids = set()
             return self._nested_check_rendered(rendered_content, seen_oids)
 
+        else:
+            # Rendered content is not actually rendered
+            return set()
+
     def _nested_check_rendered(self, rendered_content, seen_oids):
         debug_error_messages = set()
         if id(rendered_content) not in seen_oids:


Test patch:
diff --git a/tests/upgrade/rules/test_undefined_jinja_varaibles.py b/tests/upgrade/rules/test_undefined_jinja_varaibles.py
--- a/tests/upgrade/rules/test_undefined_jinja_varaibles.py
+++ b/tests/upgrade/rules/test_undefined_jinja_varaibles.py
@@ -19,7 +19,6 @@
 from unittest import TestCase
 
 import jinja2
-import pytest
 
 from airflow import DAG
 from airflow.models import DagBag
@@ -67,6 +66,12 @@ def setUpValidDag(self):
                 "float": "{{ params.float }}",
                 "string": "{{ params.string }}",
                 "boolean": "{{ params.boolean }}",
+                "integer_direct": 1,
+                "float_direct": 1.0,
+                "string_direct": "test_string",
+                "boolean_direct": True,
+                "none_direct": None,
+                "object_direct": object(),
             },
             params={
                 "integer": 1,
@@ -156,7 +161,6 @@ def test_skipping_dag_check(self):
 
         assert len(messages) == 0
 
-    @pytest.mark.quarantined
     def test_invalid_check(self):
         dagbag = DagBag(dag_folder=self.empty_dir, include_examples=False)
         dagbag.dags[self.invalid_dag.dag_id] = self.invalid_dag