Repository: reata/sqllineage

Issue:
Column lineage does not traverse through CTE containing uppercase letters
For SQL code with a CTE, column lineage is ending at the CTE and not traversing through to the source tables or expressions referenced in the CTE.

For example, I have the following code in test.sql:

```sql
WITH EmployeeDepartmentCTE AS (
    SELECT
        E.EmployeeName,
        D.DepartmentName
    FROM Employees AS E
    INNER JOIN Departments AS D
        ON E.DepartmentID = D.DepartmentID
)

INSERT INTO EmployeeDepartment

SELECT
    EmployeeName,
    DepartmentName
FROM EmployeeDepartmentCTE;
```

When I run the following command I get the subsequent output:
```shell
sqllineage -f test.sql -l column
```

output:
```
<default>.employeedepartment.departmentname <- <default>.employeedepartmentcte.departmentname
<default>.employeedepartment.employeename <- <default>.employeedepartmentcte.employeename
```

expected output:
```
<default>.employeedepartment.departmentname <- <default>.department.departmentname
<default>.employeedepartment.employeename <- <default>.employee.employeename
```

**Python version:**
 - 3.12.1
 
**SQLLineage version:**
 - 1.4.9

**Additional context**
If there are multiple CTE's with one referencing another, the lineage stops at the first CTE.


Discussion:
Caused by capitalization, if the SQL is modified to all lowercase, the result will be accurate.
Thanks @maoxingda for your helpful response!

Would you consider it a bug that it needs to be lowercase, or is transforming to lowercase a best-practice that is (or should be) documented?
It's because most databases are case-insensitive, so the underlying layer of sqllineage uniformly converts all identifiers to lowercase. Therefore, currently, lineage analysis is not sensitive to uppercase SQL.
Thanks. I'll close this issue since it sounds like it's a known bug that has already been considered.
Thanks @maoxingda for the background. Yes we want to enforce case-insensitivity in lineage analysis in short term.

However, I'm afraid this is indeed a bug introduced via fbad73abf65be8ef0f6e60f34b52d0675bf8abbc and released in v1.4.9.

Converting all identifiers to lower case is just implementation details, the real intention is that for all the following SQL, they should generate the same lineage result:
```sql
WITH CTE AS (SELECT name FROM person) SELECT name FROM cte;
WITH cte AS (SELECT name FROM person) SELECT name FROM CTE;
WITH cte AS (SELECT name FROM person) SELECT name FROM cTe;
WITH ctE AS (SELECT name FROM person) SELECT name FROM cte;
WITH CtE AS (SELECT name FROM person) SELECT name FROM Cte;
```
because in a case-insensitive SQL database, they're indeed the same.



Patch:
diff --git a/sqllineage/core/models.py b/sqllineage/core/models.py
index fd96c609..8eb012fb 100644
--- a/sqllineage/core/models.py
+++ b/sqllineage/core/models.py
@@ -58,7 +58,7 @@ def __init__(self, name: str, schema: Schema = Schema(), **kwargs):
             self.raw_name = escape_identifier_name(table_name)
             if schema:
                 warnings.warn("Name is in schema.table format, schema param is ignored")
-        self.alias = kwargs.pop("alias", self.raw_name)
+        self.alias = escape_identifier_name(kwargs.pop("alias", self.raw_name))
 
     def __str__(self):
         return f"{self.schema}.{self.raw_name}"
diff --git a/sqllineage/core/parser/sqlfluff/extractors/base.py b/sqllineage/core/parser/sqlfluff/extractors/base.py
index 1f1ca364..275e3f16 100644
--- a/sqllineage/core/parser/sqlfluff/extractors/base.py
+++ b/sqllineage/core/parser/sqlfluff/extractors/base.py
@@ -156,7 +156,7 @@ def _add_dataset_from_expression_element(
                     )
                 if "." not in table_identifier.raw:
                     cte_dict = {s.alias: s for s in holder.cte}
-                    cte = cte_dict.get(table_identifier.raw)
+                    cte = cte_dict.get(escape_identifier_name(table_identifier.raw))
                     if cte is not None:
                         # could reference CTE with or without alias
                         tables.append(
diff --git a/sqllineage/core/parser/sqlparse/handlers/source.py b/sqllineage/core/parser/sqlparse/handlers/source.py
index 8f5b77cd..f6c62b48 100644
--- a/sqllineage/core/parser/sqlparse/handlers/source.py
+++ b/sqllineage/core/parser/sqlparse/handlers/source.py
@@ -27,6 +27,7 @@
     is_values_clause,
 )
 from sqllineage.exceptions import SQLLineageException
+from sqllineage.utils.helpers import escape_identifier_name
 
 
 class SourceHandler(SourceHandlerMixin, NextTokenBaseHandler):
@@ -151,7 +152,9 @@ def _add_dataset_from_identifier(
             else:
                 cte_dict = {s.alias: s for s in holder.cte}
                 if "." not in identifier.value:
-                    cte = cte_dict.get(identifier.get_real_name())
+                    cte = cte_dict.get(
+                        escape_identifier_name(identifier.get_real_name())
+                    )
                     if cte is not None:
                         # could reference CTE with or without alias
                         read = SqlParseSubQuery.of(


Test patch:
diff --git a/tests/sql/column/test_column_select_column.py b/tests/sql/column/test_column_select_column.py
index f3aa3ea2..c946aa4b 100644
--- a/tests/sql/column/test_column_select_column.py
+++ b/tests/sql/column/test_column_select_column.py
@@ -96,6 +96,30 @@ def test_select_column_with_table_qualifier():
     )
 
 
+def test_select_column_with_table_qualifier_case_insensitive():
+    sql = """INSERT INTO tab1
+SELECT TA.col1
+FROM tab2 AS ta"""
+    assert_column_lineage_equal(
+        sql,
+        [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
+    )
+    sql = """INSERT INTO tab1
+SELECT ta.col1
+FROM tab2 AS TA"""
+    assert_column_lineage_equal(
+        sql,
+        [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
+    )
+    sql = """INSERT INTO tab1
+SELECT Ta.col1
+FROM tab2 AS tA"""
+    assert_column_lineage_equal(
+        sql,
+        [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
+    )
+
+
 def test_select_columns():
     sql = """INSERT INTO tab1
 SELECT col1,
diff --git a/tests/sql/column/test_column_select_from_cte.py b/tests/sql/column/test_column_select_from_cte.py
index d892e448..17fa5812 100644
--- a/tests/sql/column/test_column_select_from_cte.py
+++ b/tests/sql/column/test_column_select_from_cte.py
@@ -101,3 +101,27 @@ def test_cte_inside_insert_in_parenthesis():
         ],
         test_sqlparse=False,
     )
+
+
+def test_select_column_from_cte_case_insensitive():
+    sql = """WITH CTE1 AS (SELECT col1 FROM tab2)
+INSERT INTO tab1
+SELECT col1 FROM cte1"""
+    assert_column_lineage_equal(
+        sql,
+        [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
+    )
+    sql = """WITH cte1 AS (SELECT col1 FROM tab2)
+INSERT INTO tab1
+SELECT col1 FROM CTE1"""
+    assert_column_lineage_equal(
+        sql,
+        [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
+    )
+    sql = """WITH CTe1 AS (SELECT col1 FROM tab2)
+INSERT INTO tab1
+SELECT col1 FROM cTE1"""
+    assert_column_lineage_equal(
+        sql,
+        [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
+    )
diff --git a/tests/sql/column/test_column_select_from_subquery.py b/tests/sql/column/test_column_select_from_subquery.py
index 7264e3e6..389b0a04 100644
--- a/tests/sql/column/test_column_select_from_subquery.py
+++ b/tests/sql/column/test_column_select_from_subquery.py
@@ -126,3 +126,10 @@ def test_select_column_in_subquery_alias_and_qualifier_case_insensitive():
         sql,
         [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
     )
+    sql = """INSERT INTO tab1
+SELECT Dt.col1
+FROM (SELECT col1 FROM tab2) dT"""
+    assert_column_lineage_equal(
+        sql,
+        [(ColumnQualifierTuple("col1", "tab2"), ColumnQualifierTuple("col1", "tab1"))],
+    )