Package: collate-sqllineage
SQLLineageSQL Lineage Analysis Tool powered by Python.This is a fork authored by the OpenMetadata community, where we are addingsqlfluffas a parsing backend instead ofsqlparse.Never get the hang of a SQL parser? SQLLineage comes to the rescue. Given a SQL command, SQLLineage will tell you its
source and target tables, without worrying about Tokens, Keyword, Identifier and all the jagons used by SQL parsers.Behind the scene, SQLLineage pluggable leverages parser library (sqlfluffandsqlparse) to parse the SQL command, analyze the AST, stores the lineage
information in a graph (using graph librarynetworkx), and brings you all the
human-readable result with ease.Demo & DocumentationTalk is cheap, show me ademo.Documentationis online hosted by readthedocs, and you can check therelease notethere.Quick StartInstall sqllineage via PyPI:$pipinstallsqllineageUsing sqllineage command to parse a quoted-query-string:$ sqllineage -e "insert into db1.table1 select * from db2.table2"
Statements(#): 1
Source Tables:
    db2.table2
Target Tables:
    db1.table1Or you can parse a SQL file with -f option:$ sqllineage -f foo.sql
Statements(#): 1
Source Tables:
    db1.table_foo
    db1.table_bar
Target Tables:
    db2.table_bazAdvanced UsageMultiple SQL StatementsLineage result combined for multiple SQL statements, with intermediate tables identified:$ sqllineage -e "insert into db1.table1 select * from db2.table2; insert into db3.table3 select * from db1.table1;"
Statements(#): 2
Source Tables:
    db2.table2
Target Tables:
    db3.table3
Intermediate Tables:
    db1.table1Verbose Lineage ResultAnd if you want to see lineage result for every SQL statement, just toggle verbose option$ sqllineage -v -e "insert into db1.table1 select * from db2.table2; insert into db3.table3 select * from db1.table1;"
Statement #1: insert into db1.table1 select * from db2.table2;
    table read: [Table: db2.table2]
    table write: [Table: db1.table1]
    table cte: []
    table rename: []
    table drop: []
Statement #2: insert into db3.table3 select * from db1.table1;
    table read: [Table: db1.table1]
    table write: [Table: db3.table3]
    table cte: []
    table rename: []
    table drop: []
==========
Summary:
Statements(#): 2
Source Tables:
    db2.table2
Target Tables:
    db3.table3
Intermediate Tables:
    db1.table1Dialect-Awareness LineageBy default, sqllineage doesn't validate your SQL and could give confusing result in case of invalid SQL syntax.
In addition, different SQL dialect has different set of keywords, further weakening sqllineage's capabilities when
keyword used as table name or column name. To reduce the impact, user are strongly encouraged to pass the dialect to
assist the lineage analyzing.Take below example,analyzeis a reserved keyword in PostgreSQL. Default non-validating dialect gives incomplete result,
while ansi dialect gives the correct one and postgres dialect tells you this causes syntax error:$ sqllineage -e "insert into analyze select * from foo;"
Statements(#): 1
Source Tables:
    <default>.foo
Target Tables:
    
$ sqllineage -e "insert into analyze select * from foo;" --dialect=ansi
Statements(#): 1
Source Tables:
    <default>.foo
Target Tables:
    <default>.analyze

$ sqllineage -e "insert into analyze select * from foo;" --dialect=postgres
...
sqllineage.exceptions.InvalidSyntaxException: This SQL statement is unparsable, please check potential syntax error for SQLUsesqllineage --dialectsto see all available dialects.Column-Level LineageWe also support column level lineage in command line interface, set level option to column, all column lineage path will
be printed.INSERTOVERWRITETABLEfooSELECTa.col1,b.col1AScol2,c.col3_sumAScol3,col4,d.*FROMbaraJOINbazbONa.id=b.bar_idLEFTJOIN(SELECTbar_id,sum(col3)AScol3_sumFROMquxGROUPBYbar_id)cONa.id=sq.bar_idCROSSJOINquuxd;INSERTOVERWRITETABLEcorgeSELECTa.col1,a.col2+b.col2AScol2FROMfooaLEFTJOINgraultbONa.col1=b.col1;Suppose this sql is stored in a file called foo.sql$ sqllineage -f foo.sql -l column
<default>.corge.col1 <- <default>.foo.col1 <- <default>.bar.col1
<default>.corge.col2 <- <default>.foo.col2 <- <default>.baz.col1
<default>.corge.col2 <- <default>.grault.col2
<default>.foo.* <- <default>.quux.*
<default>.foo.col3 <- c.col3_sum <- <default>.qux.col3
<default>.foo.col4 <- col4Lineage VisualizationOne more cool feature, if you want a graph visualization for the lineage result, toggle graph-visualization optionStill using the above SQL filesqllineage -g -f foo.sqlA webserver will be started, showing DAG representation of the lineage result in browser:Table-Level LineageColumn-Level Lineage