Package: acryl-sqlglot
SQLGlot is a no-dependency SQL parser, transpiler, optimizer, and engine. It can be used to format SQL or translate between20 different dialectslikeDuckDB,Presto/Trino,Spark/Databricks,Snowflake, andBigQuery. It aims to read a wide variety of SQL inputs and output syntactically and semantically correct SQL in the targeted dialects.It is a very comprehensive generic SQL parser with a robusttest suite. It is also quiteperformant, while being written purely in Python.You can easilycustomizethe parser,analyzequeries, traverse expression trees, and programmaticallybuildSQL.Syntaxerrorsare highlighted and dialect incompatibilities can warn or raise depending on configurations. However, it should be noted that SQL validation is not SQLGlot’s goal, so some syntax errors may go unnoticed.Learn more about SQLGlot in the APIdocumentationand the expression treeprimer.Contributions are very welcome in SQLGlot; read thecontribution guideto get started!Table of ContentsInstallVersioningGet in TouchFAQExamplesFormatting and TranspilingMetadataParser ErrorsUnsupported ErrorsBuild and Modify SQLSQL OptimizerAST IntrospectionAST DiffCustom DialectsSQL ExecutionUsed ByDocumentationRun Tests and LintBenchmarksOptional DependenciesInstallFrom PyPI:pip3install"sqlglot[rs]"# Without Rust tokenizer (slower):# pip3 install sqlglotOr with a local checkout:make installRequirements for development (optional):make install-devVersioningGiven a version numberMAJOR.MINOR.PATCH, SQLGlot uses the following versioning strategy:ThePATCHversion is incremented when there are backwards-compatible fixes or feature additions.TheMINORversion is incremented when there are backwards-incompatible fixes or feature additions.TheMAJORversion is incremented when there are significant backwards-incompatible fixes or feature additions.Get in TouchWe'd love to hear from you. Join our communitySlack channel!FAQI tried to parse SQL that should be valid but it failed, why did that happen?You need to specify the dialect to read the SQL properly, by default it is SQLGlot's dialect which is designed to be a superset of all dialectsparse_one(sql, dialect="spark"). If you tried specifying the dialect and it still doesn't work, please file an issue.I tried to output SQL but it's not in the correct dialect!You need to specify the dialect to write the sql properly, by default it is in SQLGlot's dialectparse_one(sql, dialect="spark").sql(dialect="spark").I tried to parse invalid SQL and it should raise an error but it worked! Why didn't it validate my SQL.SQLGlot is not a validator and designed to be very forgiving, handling things like trailing commas.ExamplesFormatting and TranspilingEasily translate from one dialect to another. For example, date/time functions vary between dialects and can be hard to deal with:importsqlglotsqlglot.transpile("SELECT EPOCH_MS(1618088028295)",read="duckdb",write="hive")[0]'SELECT FROM_UNIXTIME(1618088028295 / 1000)'SQLGlot can even translate custom time formats:importsqlglotsqlglot.transpile("SELECT STRFTIME(x, '%y-%-m-%S')",read="duckdb",write="hive")[0]"SELECT DATE_FORMAT(x, 'yy-M-ss')"As another example, let's suppose that we want to read in a SQL query that contains a CTE and a cast toREAL, and then transpile it to Spark, which uses backticks for identifiers andFLOATinstead ofREAL:importsqlglotsql="""WITH baz AS (SELECT a, c FROM foo WHERE a = 1) SELECT f.a, b.b, baz.c, CAST("b"."a" AS REAL) d FROM foo f JOIN bar b ON f.a = b.a LEFT JOIN baz ON f.a = baz.a"""print(sqlglot.transpile(sql,write="spark",identify=True,pretty=True)[0])WITH`baz`AS(SELECT`a`,`c`FROM`foo`WHERE`a`=1)SELECT`f`.`a`,`b`.`b`,`baz`.`c`,CAST(`b`.`a`ASFLOAT)AS`d`FROM`foo`AS`f`JOIN`bar`AS`b`ON`f`.`a`=`b`.`a`LEFTJOIN`baz`ON`f`.`a`=`baz`.`a`Comments are also preserved on a best-effort basis when transpiling SQL code:sql="""/* multilinecomment*/SELECTtbl.cola /* comment 1 */ + tbl.colb /* comment 2 */,CAST(x AS INT), # comment 3y               -- comment 4FROMbar /* comment 5 */,tbl #          comment 6"""print(sqlglot.transpile(sql,read='mysql',pretty=True)[0])/* multilinecomment*/SELECTtbl.cola/* comment 1 */+tbl.colb/* comment 2 */,CAST(xASINT),/* comment 3 */y/* comment 4 */FROMbar/* comment 5 */,tbl/*          comment 6 */MetadataYou can explore SQL with expression helpers to do things like find columns and tables:fromsqlglotimportparse_one,exp# print all column references (a and b)forcolumninparse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Column):print(column.alias_or_name)# find all projections in select statements (a and c)forselectinparse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Select):forprojectioninselect.expressions:print(projection.alias_or_name)# find all tables (x, y, z)fortableinparse_one("SELECT * FROM x JOIN y JOIN z").find_all(exp.Table):print(table.name)Read theast primerto learn more about SQLGlot's internals.Parser ErrorsWhen the parser detects an error in the syntax, it raises a ParseError:importsqlglotsqlglot.transpile("SELECT foo( FROM bar")sqlglot.errors.ParseError: Expecting ). Line 1, Col: 13.
  select foo( FROM bar
              ~~~~Structured syntax errors are accessible for programmatic use:importsqlglottry:sqlglot.transpile("SELECT foo( FROM bar")exceptsqlglot.errors.ParseErrorase:print(e.errors)[{'description':'Expecting )','line':1,'col':16,'start_context':'SELECT foo( ','highlight':'FROM','end_context':' bar','into_expression':None,}]Unsupported ErrorsPrestoAPPROX_DISTINCTsupports the accuracy argument which is not supported in Hive:importsqlglotsqlglot.transpile("SELECT APPROX_DISTINCT(a, 0.1) FROM foo",read="presto",write="hive")APPROX_COUNT_DISTINCTdoesnotsupportaccuracy'SELECT APPROX_COUNT_DISTINCT(a) FROM foo'Build and Modify SQLSQLGlot supports incrementally building sql expressions:fromsqlglotimportselect,conditionwhere=condition("x=1").and_("y=1")select("*").from_("y").where(where).sql()'SELECT * FROM y WHERE x = 1 AND y = 1'You can also modify a parsed tree:fromsqlglotimportparse_oneparse_one("SELECT x FROM y").from_("z").sql()'SELECT x FROM z'There is also a way to recursively transform the parsed tree by applying a mapping function to each tree node:fromsqlglotimportexp,parse_oneexpression_tree=parse_one("SELECT a FROM x")deftransformer(node):ifisinstance(node,exp.Column)andnode.name=="a":returnparse_one("FUN(a)")returnnodetransformed_tree=expression_tree.transform(transformer)transformed_tree.sql()'SELECT FUN(a) FROM x'SQL OptimizerSQLGlot can rewrite queries into an "optimized" form. It performs a variety oftechniquesto create a new canonical AST. This AST can be used to standardize queries or provide the foundations for implementing an actual engine. For example:importsqlglotfromsqlglot.optimizerimportoptimizeprint(optimize(sqlglot.parse_one("""SELECT A OR (B OR (C AND D))FROM xWHERE Z = date '2021-01-01' + INTERVAL '1' month OR 1 = 0"""),schema={"x":{"A":"INT","B":"INT","C":"INT","D":"INT","Z":"STRING"}}).sql(pretty=True))SELECT("x"."a"<>0OR"x"."b"<>0OR"x"."c"<>0)AND("x"."a"<>0OR"x"."b"<>0OR"x"."d"<>0)AS"_col_0"FROM"x"AS"x"WHERECAST("x"."z"ASDATE)=CAST('2021-02-01'ASDATE)AST IntrospectionYou can see the AST version of the sql by callingrepr:fromsqlglotimportparse_oneprint(repr(parse_one("SELECT a + 1 AS z")))Select(expressions=[Alias(this=Add(this=Column(this=Identifier(this=a,quoted=False)),expression=Literal(this=1,is_string=False)),alias=Identifier(this=z,quoted=False))])AST DiffSQLGlot can calculate the difference between two expressions and output changes in a form of a sequence of actions needed to transform a source expression into a target one:fromsqlglotimportdiff,parse_onediff(parse_one("SELECT a + b, c, d"),parse_one("SELECT c, a - b, d"))[Remove(expression=Add(this=Column(this=Identifier(this=a,quoted=False)),expression=Column(this=Identifier(this=b,quoted=False)))),Insert(expression=Sub(this=Column(this=Identifier(this=a,quoted=False)),expression=Column(this=Identifier(this=b,quoted=False)))),Keep(source=Identifier(this=d,quoted=False),target=Identifier(this=d,quoted=False)),...]See also:Semantic Diff for SQL.Custom DialectsDialectscan be added by subclassingDialect:fromsqlglotimportexpfromsqlglot.dialects.dialectimportDialectfromsqlglot.generatorimportGeneratorfromsqlglot.tokensimportTokenizer,TokenTypeclassCustom(Dialect):classTokenizer(Tokenizer):QUOTES=["'",'"']IDENTIFIERS=["`"]KEYWORDS={**Tokenizer.KEYWORDS,"INT64":TokenType.BIGINT,"FLOAT64":TokenType.DOUBLE,}classGenerator(Generator):TRANSFORMS={exp.Array:lambdaself,e:f"[{self.expressions(e)}]"}TYPE_MAPPING={exp.DataType.Type.TINYINT:"INT64",exp.DataType.Type.SMALLINT:"INT64",exp.DataType.Type.INT:"INT64",exp.DataType.Type.BIGINT:"INT64",exp.DataType.Type.DECIMAL:"NUMERIC",exp.DataType.Type.FLOAT:"FLOAT64",exp.DataType.Type.DOUBLE:"FLOAT64",exp.DataType.Type.BOOLEAN:"BOOL",exp.DataType.Type.TEXT:"STRING",}print(Dialect["custom"])<class '__main__.Custom'>SQL ExecutionOne can even interpret SQL queries using SQLGlot, where the tables are represented as Python dictionaries. Although the engine is not very fast (it's not supposed to be) and is in a relatively early stage of development, it can be useful for unit testing and running SQL natively across Python objects. Additionally, the foundation can be easily integrated with fast compute kernels (arrow, pandas). Below is an example showcasing the execution of a SELECT expression that involves aggregations and JOINs:fromsqlglot.executorimportexecutetables={"sushi":[{"id":1,"price":1.0},{"id":2,"price":2.0},{"id":3,"price":3.0},],"order_items":[{"sushi_id":1,"order_id":1},{"sushi_id":1,"order_id":1},{"sushi_id":2,"order_id":1},{"sushi_id":3,"order_id":2},],"orders":[{"id":1,"user_id":1},{"id":2,"user_id":2},],}execute("""SELECTo.user_id,SUM(s.price) AS priceFROM orders oJOIN order_items iON o.id = i.order_idJOIN sushi sON i.sushi_id = s.idGROUP BY o.user_id""",tables=tables)user_idprice14.023.0See also:Writing a Python SQL engine from scratch.Used BySQLMeshFugueibismysql-mimicQuerybookQuokkaSplinkDocumentationSQLGlot usespdocto serve its API documentation.A hosted version is on theSQLGlot website, or you can build locally with:make docs-serveRun Tests and Lintmake style  # Only linter checks
make unit   # Only unit tests
make check  # Full test suite & linter checksBenchmarksBenchmarksrun on Python 3.10.12 in seconds.Querysqlglotsqlglotrssqlfluffsqltreesqlparsemoz_sql_parsersqloxidetpch0.00944 (1.0)0.00590 (0.625)0.32116 (33.98)0.00693 (0.734)0.02858 (3.025)0.03337 (3.532)0.00073 (0.077)short0.00065 (1.0)0.00044 (0.687)0.03511 (53.82)0.00049 (0.759)0.00163 (2.506)0.00234 (3.601)0.00005 (0.073)long0.00889 (1.0)0.00572 (0.643)0.36982 (41.56)0.00614 (0.690)0.02530 (2.844)0.02931 (3.294)0.00059 (0.066)crazy0.02918 (1.0)0.01991 (0.682)1.88695 (64.66)0.02003 (0.686)7.46894 (255.9)0.64994 (22.27)0.00327 (0.112)Optional DependenciesSQLGlot usesdateutilto simplify literal timedelta expressions. The optimizer will not simplify expressions like the following if the module cannot be found:x+interval'1'month