Package: pylinsql
Language-Integrated SQL Queries in Pythonpylinsqlhelps you write SQL queries in Python that integrate with the type checker and produce standard SQL query strings as an end result. The main idea is to take a Python generator expression such asselect(asc(p.given_name)forp,ainentity(Person,Address)ifinner_join(p.address_id,a.id)and((p.given_name=="John"andp.family_name!="Doe")or(a.city!="London")))and transform it into a SQL query such asSELECTp.given_nameFROM"Person"ASpINNERJOIN"Address"ASaONp.address_id=a.idWHEREp.given_name='John'ANDp.family_name<>'Doe'ORa.city<>'London'ORDERBYp.given_nameASCUsing a language-integrated query formalism (analogous to LINQ in C#), users can write queries in a format that is transparent to lint tools, and identify errors early. The query expressions map to SQL statement strings, which allows for constant-time look-up, makingpylinsqlincur almost zero additional run-time cost over writing raw SQL statements while providing type safety.ObjectivesThe inspiration forpylinsqlhas been to employ efficient asynchronous communication with the database engine (such as inasyncpg) while providing a type-safe means to formulate SELECT and INSERT queries (as inPonyORM).This work is no substitute for an all-in-one boxed solution that handles database connections, performs pooling, caching, manages entity relationships, etc. (such asSQLAlchemy). Its purpose is to help write a SQL query in the style of C# language-integrated queries that you can then execute with a(n asynchronous) SQL engine client (e.g.asyncpgin Python).UsageExpressions precedingforin a Python generator expression go intoSELECTin SQL:select((p.family_name,p.given_name)forpinentity(Person))SELECTp.family_name,p.given_nameFROM"Person"ASpIf you have an entity variable precedingfor, it will expand into all properties of that entity:select(pforpinentity(Person))SELECT*FROM"Person"ASpBoolean expressions in the condition part of a Python generator expression (i.e. followingif) normally go into theWHEREclause:select(pforpinentity(Person)ifp.given_name=="John"andp.family_name!="Doe"oryear(p.birth_date)>=1982)SELECT*FROM"Person"ASpWHEREp.given_name='John'ANDp.family_name<>'Doe'OREXTRACT(YEARFROMp.birth_date)>=1982The conditional part also accepts special functionsinner_join,left_join,right_join, etc. to create join expressions in SQL. These special functions are only allowed in the condition part of the generator expression but not elsewhere. You can combine several join conditions with Python'sand.select(pforp,a1,a2inentity(Person,Address,Address)ifinner_join(p.perm_address_id,a1.id)andleft_join(p.temp_address_id,a2.id))SELECT*FROM"Person"ASpINNERJOIN"Address"ASa1ONp.perm_address_id=a1.idLEFTJOIN"Address"ASa2ONp.temp_address_id=a2.idYou can also use aggregation functions. Expressions that are not aggregated automatically go into theGROUP BYclause. If you have a condition that involves an aggregated expression, it becomes part of theHAVINGclause.select((a.city,min(p.birth_date))forp,ainentity(Person,Address)ifinner_join(p.perm_address_id,a.id)andmin(p.birth_date)>=date(1989,10,23))SELECTa.city,MIN(p.birth_date)FROM"Person"ASpINNERJOIN"Address"ASaONp.perm_address_id=a.idGROUPBYa.cityHAVINGMIN(p.birth_date)>=MAKE_DATE(1989,10,23)Join expressionspylinsqlsupports (inner) join, left join, right join and full (outer) join via the Python functionsinner_join,left_join,right_joinandfull_join. These go into condition part of the generator expression. They take two parameters, both of which must be table attribute references, e.g.p.perm_address_idora1.id:select(pforp,a1,a2inentity(Person,Address,Address)ifinner_join(p.perm_address_id,a1.id)andleft_join(p.temp_address_id,a2.id)and((a1.city!="London")or(a2.city!="Zürich")))If two tables are listed as entities but not referenced by a join in the condition part, they are assumed to expand to a cross product, as in SQL.Select expressionsIn addition to a scalar expression (single column per row) and a tuple expression (for multiple columns per row),pylinsqloffers a convenience syntax with a@dataclassannotated type acting as the output.Assume that you have a custom@dataclasstype calledPersonCity:@dataclassclassPersonCity:family_name:strgiven_name:strcity:strWhen executed against a database engine, the following query will produce a list ofPersonCityinstances:select(PersonCity(p.family_name,p.given_name,a.city)forp,ainentity(Person,Address)ifinner_join(p.perm_address_id,a.id))Positional and keyword arguments in the@dataclassinitializer are both supported.Sort orderIf you specify the sort order of a column with special Python functionsasc(column)anddesc(column),pylinsqlwill append the appropriateORDER BYclause at the end of the SQL query:select((asc(p.family_name),desc(p.given_name),p.birth_date)forpinentity(Person))SELECTp.family_name,p.given_name,p.birth_dateFROM"Person"ASpORDERBYp.family_nameASC,p.given_nameDESCAggregation functionsSeveral aggregation functions are available, includingavg,count,max,min,sum,avg_if,count_if,max_if,min_ifandsum_if.The following example illustrates how to use simple aggregation functions:select((count(p.birth_date),min(p.birth_date),max(p.birth_date))forpinentity(Person))Conditional aggregation functions take a Boolean filter expression as a second parameter:select((count_if(p.birth_date,p.given_name!="John"))forpinentity(Person))SELECTCOUNT(p.birth_date)FILTER(WHEREp.given_name<>'John')FROM"Person"ASpDate and timeA date constructed withdatetime.date(y, m, d)in Python is translated toMAKE_DATE(y, m, d)in PostgreSQL. Likewise,datetime.time(h, m, s)is translated toMAKE_TIME(h, m, s). Parts of a date or time can be extracted with functions likeyear(dt)orhour(dt), which map to the appropriateEXTRACTclause in SQL. Date and time differences are also supported.select(pforpinentity(Person)ifyear(now()-p.birth_date)>=18)SELECT*FROM"Person"ASpWHEREEXTRACT(YEARFROM(CURRENT_TIMESTAMP-p.birth_date))>=18String matchingString matching with the SQL-standard LIKE operator and PostgreSQL's regular expression match operators~(case sensitive) and~*(case insensitive match) are both supported, use Python functionslike,ilike,matchandimatch. The following example matches all people records whose family name ends incan(with a case sensitive match):select(pforpinentity(Person)ifmatches(p.family_name,r"can$"))SELECT*FROM"Person"ASpWHEREp.family_name~'can$'Executing a queryThe packageasync_databasecontains functions to create a database connection, acquire a connection from a connection pool, and run queries in a transaction. Member functions of the classDatabaseConnectionaccept Python generator expressions the same way thatselectdoes in the examples above.asyncwithasync_database.connection()asconn:results=awaitconn.select(pforpinentity(Person))result=awaitconn.select_first(pforpinentity(Person))Code generator for data classespylinsqldepends on Python data classes for its language-integrated query mechanism. For example, in order to executeselect(asc(p.given_name)forp,ainentity(Person,Address)ifinner_join(p.address_id,a.id)and((p.given_name=="John"andp.family_name!="Doe")or(a.city!="London")))one has to define data classes corresponding to entitiesAddressandPerson:@dataclassclassAddress:id:intcity:str@dataclassclassPerson:id:intfamily_name:strgiven_name:strbirth_date:datetimeperm_address_id:int=field(default=...)temp_address_id:Optional[int]=field(default=...)Defining these classes manually would be tedious work. Fortunately,pylinsqlcomes with a code generator utility that scans table schema definitions in a database, and writes corresponding Python code:$python3-mpylinsql.generator.code_generatorexample.py--schemapublicThe generated code takes into account type mappings, nullable types, table references and even table and column comments.Use the switch--helpto learn more:$python3-mpylinsql.generator.code_generator--helpBackground and related workpsycopg2provides a way to piece together SQL queries using composable primitive objects likeIdentifier(e.g. a table name),Literal(e.g. an integer or string value),Placeholder(in a prepared statement) andSQL(represents a SQL statement segment). It also provides a mechanism to establish a synchronous connection to a PostgreSQL server.asyncpgis a library that exposes an asynchronous connection to a PostgreSQL server utilizing Python'sasyncioservices. If queries or parameterized queries are available as a string,asyncpgcan execute them efficiently.PonyORMis an object-relational mapping (ORM) library that uses a similar syntax based on Python generator expressions. It is a full-fledged ORM solution that uses a synchronous connection to a SQL server.SQLAlchemyis the most widely-used object-relational mapping with a rich set of features (organized in a hierarchy), and an ability to use asynchronous database connections. Unfortunately, the query syntax is rather verbose and does not look like a neat Python expression.The disassembling approach to reverse-engineer the abstract syntax tree (AST) from the control flow graph (CFG) is similar to that used inPonyORM.The consistent coloring of incoming green/red edges of nodes in the abstract node graph is discussed in detail inDecompiling Boolean Expressions from Java Bytecode, specificallyAlgorithm 2.For further reading, check outNo More Gotos: Decompilation Using Pattern-Independent Control-Flow Structuring
and Semantics-Preserving Transformations. Also,Solving the structured control flow problem once and for allmight be of interest.Implementation detailspylinsqlutilizes some more advanced features and programming language concepts such as Python intermediate language, low-level code analysis, graph theory and parsers/generators.As an example, let's consider the following Python generator expression:((p.family_name,p.given_name)forpinentity(Person)ifp.given_name=="John"andp.family_name!="Doe")This has aconditional part:p.given_name=="John"andp.family_name!="Doe"and ayield part:(p.family_name,p.given_name)pylinsqlperforms several steps to construct a SQL query string from a Python generator expression:Disassembly.pylinsqluses the Python moduledisto retrieve a Python generator expression as a series of instructions, which are low-level intermediate language statements such as BINARY_ADD (to add two numbers on the top of the stack), CALL_FUNCTION (to call a function with arguments on the stack), LOAD_GLOBAL (push a global variable to the top of the stack), or POP_JUMP_IF_TRUE (jump to a label if the value on the top of the stack is true).Extract basic blocks.A basic block is a series of instructions that starts with a label (that jump instructions point to) and/or ends with a (conditional or unconditional) jump statement (e.g. POP_JUMP_IF_TRUE). For example, the following snippet shows the disassembly of our sample Python generator expression (including both the conditional and the yield part), with horizontal bars separating basic blocks. Target labels for jump instructions are shown with>>. The number in the first column is the instruction address.0 LOAD_FAST                0 (.0)
------------------------------------------------------------
>>       2 FOR_ITER                38 (to 42)
         4 STORE_FAST               1 (p)
         6 LOAD_FAST                1 (p)
         8 LOAD_ATTR                0 (given_name)
        10 LOAD_CONST               0 ('John')
        12 COMPARE_OP               2 (==)
        14 POP_JUMP_IF_FALSE        2
------------------------------------------------------------
        16 LOAD_FAST                1 (p)
        18 LOAD_ATTR                1 (family_name)
        20 LOAD_CONST               1 ('Doe')
        22 COMPARE_OP               3 (!=)
        24 POP_JUMP_IF_FALSE        2
------------------------------------------------------------
        26 LOAD_FAST                1 (p)
        28 LOAD_ATTR                1 (family_name)
        30 LOAD_FAST                1 (p)
        32 LOAD_ATTR                0 (given_name)
        34 BUILD_TUPLE              2
        36 YIELD_VALUE
        38 POP_TOP
        40 JUMP_ABSOLUTE            2
------------------------------------------------------------
>>      42 LOAD_CONST               2 (None)
        44 RETURN_VALUECreate control flow graph (CFG).The control flow graph has basic blocks as nodes, and jump instruction targets as edges. For example, a basic block that ends with POP_JUMP_IF_TRUE has two outgoing edges: one points to the basic block targeted when the condition is true, and the other points to the next basic block (i.e. the next statement in the program).pylinsqluses a jump resolver to translate numeric instruction addresses into control flow graph edges.Merge nodes that correspond to conditional expressions and loop conditions.When you have a conditional expression such asp.given_name=="John"andp.family_name!="Doe"then the expression is represented in low-level instructions as a series of basic blocks, interconnected with jump instructions.pylinsqlmerges nodes corresponding to these basic blocks into a single compound node. For example, the above expression would becomeNodeConjunction(a, b)whereastands for the node representing the equality test andbstands for that of the inequality test, andNodeConjunctioncaptures the intent of the Python keywordand.pylinsqlmerges nodes in a well-defined order. First, it merges all conditions that act like expressions (e.g. they are part of a function call). Second, it merges the condition that constitutes the loop condition of the Python generator expression. The end result is a chain of nodes, where each node is a simple node (a single basic block), or a composite node: asequence, aconjunctionor adisjunction. All composite nodes are also chains inside, with no branches.Create an abstract syntax tree (AST).pylinsqlconverts the node chains into an abstract representation with the help of an evaluator. The evaluator builds a symbolic expression (e.g.0 * 1 + 2ora and b and c) from a chain of nodes and the low-level instructions stored in them.The evaluator maintains a stack, mimicking how the Python interpreter works. It goes through the instructions of a basic block, and manipulates the stack following the instructions. Whenever global or local symbols are referenced (e.g. constants or variable names), the evaluator pushes their symbolic representation. Any further operations are performed with this symbolic representation. For example, when encountering the instruction BINARY_ADD, which adds two numbers popping off items from the top of the stack and pushing the result, the evaluator will pop off two symbolic expressions (e.g.Variable(a)andConstant(2)), and push a new symbolic expression (e.g.Addition(Variable(a), Constant(2))).Conjunctions and disjunctions are handled in a special way. These are represented by multiple basic blocks, interconnected by conditional jump instructions.Whenever the AST builder encounters a conjunction, it tells the evaluator to process jump instructions as if the condition evaluated to true. In a structured expression such asa and b and c, this would force evaluating the rest of the expression, and not short-circuit ataora and b. Likewise, jump instructions in blocks of a disjunction are processed as if conditions evaluated to false. In either case, the top of the stack is going to contain a symbolic expression for all the sub-expressions combined into a conjunction or disjunction, respectively, as the evaluator jumps through all basic blocks that comprise them.Analyze the abstract syntax tree.pylinsqlchecks if the expression is well-formed, e.g. whether you join objects along existing properties (e.g.Personhasgiven_name).Emit an SQL statement.pylinsqlmaps Python function calls into SQL statement equivalents, e.g.asc()becomesORDER BY,inner_join()maps to anINNER JOINin aFROMclause, a condition on amin()becomes part ofHAVING,GROUP BYis generated based on the result expressions in the original Python generator expression, etc.