Repository: reata/sqllineage

Issue:
Silent Mode Option to Suppress UnsupportedStatementException
Currently when using any dialect other than non-validating, namely sqlfluff parse implementation, SQLLineage needs to know the statement type before analyzing it. 

For example, SelectExtractor supports statement type of `["select_statement", "set_expression", "bracketed"]`. And we have a special-purpose NoopExtractor to whitelist a bunch of statement types where no lineage can be extracted.

This whitelist approach solves a problem from non-validating dialect, where we don't know what statement we support. And when user reports a exception, it's hard to tell whether it's a bug or feature.

However, this comes at cost that for statement type we haven't included in the whitelist an UnsupportedStatementException will be thrown. Catch the exception is one way. 

In this proposal we'd also like to introduce a new silent_mode option to throw warnings instead of exception.

Discussion:
maoxingda: Is the silent_mode option also required for the command line?
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/reata/sqllineage/pull/509?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
Attention: `2 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`c0cf6aa`)](https://app.codecov.io/gh/reata/sqllineage/commit/c0cf6aa6fc5976a61ed534040b44b21cafb41094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) 99.41% compared to head [(`12ee3f4`)](https://app.codecov.io/gh/reata/sqllineage/pull/509?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) 99.31%.

| [Files](https://app.codecov.io/gh/reata/sqllineage/pull/509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Patch % | Lines |
|---|---|---|
| [sqllineage/core/parser/sqlfluff/analyzer.py](https://app.codecov.io/gh/reata/sqllineage/pull/509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-c3FsbGluZWFnZS9jb3JlL3BhcnNlci9zcWxmbHVmZi9hbmFseXplci5weQ==) | 60.00% | [2 Missing :warning: ](https://app.codecov.io/gh/reata/sqllineage/pull/509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master     #509      +/-   ##
==========================================
- Coverage   99.41%   99.31%   -0.10%     
==========================================
  Files          40       40              
  Lines        2052     2056       +4     
==========================================
+ Hits         2040     2042       +2     
- Misses         12       14       +2     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/reata/sqllineage/pull/509?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).

kkozhakin: > Is the silent_mode option also required for the command line?

Optional, default is False

Patch:
diff --git a/.readthedocs.yml b/.readthedocs.yml
index 5fa7401..6178ddd 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -5,6 +5,12 @@
 # Required
 version: 2
 
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
   configuration: docs/conf.py
@@ -13,9 +19,9 @@ sphinx:
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: all
 
-# Optionally set the version of Python and requirements required to build your docs
+# We recommend specifying your dependencies to enable reproducible builds:
+# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
 python:
-  version: 3.8
   install:
     - method: pip
       path: .
diff --git a/sqllineage/cli.py b/sqllineage/cli.py
index e5ff05c..15b27f2 100644
--- a/sqllineage/cli.py
+++ b/sqllineage/cli.py
@@ -86,6 +86,11 @@ def main(args=None) -> None:
         help="list all the available dialects",
         action="store_true",
     )
+    parser.add_argument(
+        "--silent_mode",
+        help="skip unsupported statements",
+        action="store_true",
+    )
     args = parser.parse_args(args)
     if args.e and args.f:
         warnings.warn("Both -e and -f options are specified. -e option will be ignored")
@@ -100,6 +105,7 @@ def main(args=None) -> None:
                 "port": args.port,
                 "f": args.f if args.f else None,
             },
+            silent_mode=args.silent_mode,
         )
         if args.graph_visualization:
             runner.draw(args.dialect)
diff --git a/sqllineage/core/analyzer.py b/sqllineage/core/analyzer.py
index a30ea65..8f41ffb 100644
--- a/sqllineage/core/analyzer.py
+++ b/sqllineage/core/analyzer.py
@@ -13,8 +13,10 @@ class LineageAnalyzer:
     SUPPORTED_DIALECTS: List[str] = []
 
     @abstractmethod
-    def analyze(self, sql: str) -> StatementLineageHolder:
+    def analyze(self, sql: str, silent_mode: bool) -> StatementLineageHolder:
         """
         to analyze single statement sql and store the result into
-        :class:`sqllineage.core.holders.StatementLineageHolder`.
+        :class:`sqllineage.core.holders.StatementLineageHolder`
+
+        :param silent_mode: skip unsupported statements.
         """
diff --git a/sqllineage/core/parser/sqlfluff/analyzer.py b/sqllineage/core/parser/sqlfluff/analyzer.py
index 15af2fc..e716f3d 100644
--- a/sqllineage/core/parser/sqlfluff/analyzer.py
+++ b/sqllineage/core/parser/sqlfluff/analyzer.py
@@ -35,7 +35,7 @@ class SqlFluffLineageAnalyzer(LineageAnalyzer):
             sqls.append(segment.raw)
         return sqls
 
-    def analyze(self, sql: str) -> StatementLineageHolder:
+    def analyze(self, sql: str, silent_mode: bool = False) -> StatementLineageHolder:
         if sql in self.tsql_split_cache:
             statement_segments = [self.tsql_split_cache[sql]]
         else:
@@ -56,10 +56,17 @@ class SqlFluffLineageAnalyzer(LineageAnalyzer):
                     )
                     return StatementLineageHolder.of(lineage_holder)
             else:
-                raise UnsupportedStatementException(
-                    f"SQLLineage doesn't support analyzing statement type [{statement_segment.type}] for SQL:"
-                    f"{sql}"
-                )
+                if silent_mode:
+                    warnings.warn(
+                        f"SQLLineage doesn't support analyzing statement type [{statement_segment.type}] for SQL:"
+                        f"{sql}"
+                    )
+                    return StatementLineageHolder()
+                else:
+                    raise UnsupportedStatementException(
+                        f"SQLLineage doesn't support analyzing statement type [{statement_segment.type}] for SQL:"
+                        f"{sql}"
+                    )
 
     def _list_specific_statement_segment(self, sql: str):
         parsed = Linter(dialect=self._dialect).parse_string(sql)
diff --git a/sqllineage/core/parser/sqlfluff/models.py b/sqllineage/core/parser/sqlfluff/models.py
index 6c60309..921600e 100644
--- a/sqllineage/core/parser/sqlfluff/models.py
+++ b/sqllineage/core/parser/sqlfluff/models.py
@@ -209,13 +209,9 @@ class SqlFluffColumn(Column):
     def _get_column_from_parenthesis(
         sub_segment: BaseSegment,
     ) -> List[ColumnQualifierTuple]:
-        """
-        :param sub_segment: segment to be processed
-        :return: list of columns and alias from the segment
-        """
-        col, _ = SqlFluffColumn._get_column_and_alias(sub_segment)
-        if col:
-            return col
+        # windows function has an extra layer, get rid of it so that it can be handled as regular functions
+        if window_specification := sub_segment.get_child("window_specification"):
+            sub_segment = window_specification
         col, _ = SqlFluffColumn._get_column_and_alias(sub_segment, False)
         return col if col else []
 
@@ -223,6 +219,10 @@ class SqlFluffColumn(Column):
     def _get_column_and_alias(
         segment: BaseSegment, check_bracketed: bool = True
     ) -> Tuple[List[ColumnQualifierTuple], Optional[str]]:
+        """
+        check_bracketed is True for top-level column definition, like (col1 + col2) as col3
+        set to False for bracket in function call, like coalesce(col1, col2) as col3
+        """
         alias = None
         columns = []
         sub_segments = list_child_segments(segment, check_bracketed)
diff --git a/sqllineage/core/parser/sqlfluff/utils.py b/sqllineage/core/parser/sqlfluff/utils.py
index 406b17d..c9b732c 100644
--- a/sqllineage/core/parser/sqlfluff/utils.py
+++ b/sqllineage/core/parser/sqlfluff/utils.py
@@ -48,7 +48,7 @@ def is_subquery(segment: BaseSegment) -> bool:
 
 def is_wildcard(segment: BaseSegment) -> bool:
     return segment.type == "wildcard_expression" or (
-        segment.type == "symbol" and segment.raw == "*"
+        segment.type == "symbol" and segment.raw == "*" and segment.get_type() == "star"
     )
 
 
diff --git a/sqllineage/core/parser/sqlparse/analyzer.py b/sqllineage/core/parser/sqlparse/analyzer.py
index 13095f1..ede14f6 100644
--- a/sqllineage/core/parser/sqlparse/analyzer.py
+++ b/sqllineage/core/parser/sqlparse/analyzer.py
@@ -38,7 +38,7 @@ class SqlParseLineageAnalyzer(LineageAnalyzer):
     PARSER_NAME = "sqlparse"
     SUPPORTED_DIALECTS = ["non-validating"]
 
-    def analyze(self, sql: str) -> StatementLineageHolder:
+    def analyze(self, sql: str, silent_mode: bool = False) -> StatementLineageHolder:
         # get rid of comments, which cause inconsistencies in sqlparse output
         stmt = sqlparse.parse(trim_comment(sql))[0]
         if (
diff --git a/sqllineage/runner.py b/sqllineage/runner.py
index e65bce6..e90af88 100644
--- a/sqllineage/runner.py
+++ b/sqllineage/runner.py
@@ -41,6 +41,7 @@ class LineageRunner(object):
         metadata_provider: MetaDataProvider = DummyMetaDataProvider(),
         encoding: Optional[str] = None,
         verbose: bool = False,
+        silent_mode: bool = False,
         draw_options: Optional[Dict[str, str]] = None,
     ):
         """
@@ -68,6 +69,7 @@ class LineageRunner(object):
         self._stmt: List[str] = []
         self._dialect = dialect
         self._metadata_provider = metadata_provider
+        self._silent_mode = silent_mode
 
     @lazy_method
     def __str__(self):
@@ -190,7 +192,9 @@ Target Tables:
                 )
             self._stmt = split(self._sql.strip())
 
-        self._stmt_holders = [analyzer.analyze(stmt) for stmt in self._stmt]
+        self._stmt_holders = [
+            analyzer.analyze(stmt, self._silent_mode) for stmt in self._stmt
+        ]
         self._sql_holder = SQLLineageHolder.of(
             self._metadata_provider, *self._stmt_holders
         )


Test patch:
diff --git a/tests/core/test_runner.py b/tests/core/test_runner.py
index 01a41b1..ead0ef6 100644
--- a/tests/core/test_runner.py
+++ b/tests/core/test_runner.py
@@ -17,3 +17,8 @@ def test_statements_trim_comment():
     comment = "------------------\n"
     sql = "select * from dual;"
     assert LineageRunner(comment + sql).statements()[0] == sql
+
+
+def test_silent_mode():
+    sql = "begin; select * from dual;"
+    LineageRunner(sql, dialect="greenplum", silent_mode=True)._eval()
diff --git a/tests/sql/column/test_column_select_expression.py b/tests/sql/column/test_column_select_expression.py
index f54c998..020e445 100644
--- a/tests/sql/column/test_column_select_expression.py
+++ b/tests/sql/column/test_column_select_expression.py
@@ -37,6 +37,21 @@ FROM tab2"""
     )
 
 
+def test_select_column_multiply_expression():
+    sql = """INSERT INTO tab1
+SELECT col1 * 1 AS col1
+FROM tab2"""
+    assert_column_lineage_equal(
+        sql,
+        [
+            (
+                ColumnQualifierTuple("col1", "tab2"),
+                ColumnQualifierTuple("col1", "tab1"),
+            ),
+        ],
+    )
+
+
 def test_select_column_using_expression_in_parenthesis():
     sql = """INSERT INTO tab1
 SELECT (col1 + col2)