Package: querky
querkyTurn your SQL queries into type annotated Python functions and autogenerated types with a single decorator.ShowcaseThis example shows whatquerkySQL functions look like.Consider this PostgreSQL database schema:CREATETABLEaccount(idBIGSERIALPRIMARYKEY,usernameTEXTUNIQUENOTNULL,first_nameTEXTNOTNULL,last_nameTEXT,phone_numberTEXT,balanceBIGINTNOTNULLDEFAULT0,join_tsTIMESTAMPTZNOTNULLDEFAULTNOW(),referred_by_account_idBIGINTREFERENCESaccount(id));CREATETABLEpost(idBIGSERIALPRIMARYKEY,poster_idBIGINTNOTNULLREFERENCESaccount(id),messageTEXTNOTNULL,tsTIMESTAMPTZNOTNULLDEFAULTNOW());CREATETABLEpost_comment(idBIGSERIALPRIMARYKEY,post_idBIGINTNOTNULLREFERENCESpost(id),commenter_idBIGINTNOTNULLREFERENCESaccount(id),messageTEXTNOTNULL,tsTIMESTAMPTZNOTNULLDEFAULTNOW());And these are the queries defined on it:fromquerky_defimportqrk# an UPDATE query: no value returned@qrk.query# or @qrk.query(shape='status')defupdate_account_phone_number(account_id,new_phone_number):returnf'''UPDATEaccountSETphone_number ={+new_phone_number}WHEREid ={+account_id}'''# an INSERT query to always return a single value@qrk.query(shape='value',optional=False)definsert_account(username,first_name,last_name,phone_number,balance,referred_by_account_id):returnf'''INSERT INTOaccount(username,first_name,last_name,phone_number,balance,referred_by_account_id)VALUES({+username},{+first_name},{+last_name},{+phone_number},{+balance},{+referred_by_account_id})RETURNINGid'''# a SELECT query to return an array of single values@qrk.query(shape='column')defselect_top_largest_balances(limit):returnf'''SELECTbalanceFROMaccountORDER BYbalance DESCLIMIT{+limit}'''# now for the most interesting part: fetching rows# a SELECT query to return a single (one) AccountReferrer row or None (optional)@qrk.query('AccountReferrer',shape='one',optional=True)defget_account_referrer(account_id):returnf'''SELECTreferrer.id,referrer.username,referrer.first_name,referrer.last_name,referrer.join_tsFROMaccountINNER JOINaccount AS referrerONaccount.referred_by_account_id = referrer.idWHEREaccount.id ={+account_id}'''# a SELECT query to return many (an array of) AccountPostComment rows@qrk.query('AccountPostComment',shape='many')defselect_last_post_comments(post_id,limit):returnf'''SELECTaccount.first_name,account.last_name,post_comment.id,post_comment.messageFROMpost_commentINNER JOINaccountONpost_comment.commenter_id = account.idWHEREpost_comment.post_id ={+post_id}ORDER BYpost_comment.ts DESCLIMIT{+limit}'''So, as you can see, all you need is3 simple steps:Write a Python functionreturning the desired SQL query.Insert the argumentsexactly where you want them to be.Don't forget to prepend your arguments with a plus sign(+). Even though it is a regular Python format string,the resulting query is not SQL-injectable, as you'll later see.Add the@qrk.querydecoratorusing arguments to describe the expected shape and type of result set.Before you can use this code,you'll need theqrkobject.Bear with me, I'll show the full configuration in the next section, but, firstly, I would like to showthe results of runningquerky's code generator. Here it is:# ~ AUTOGENERATED BY QUERKY ~ #importdatetimefromasyncpgimportConnectionfromdataclassesimportdataclassfromsql.exampleimportupdate_account_phone_numberas_q0fromsql.exampleimportinsert_accountas_q1fromsql.exampleimportselect_top_largest_balancesas_q2fromsql.exampleimportget_account_referreras_q3fromsql.exampleimportselect_last_post_commentsas_q4asyncdefupdate_account_phone_number(__conn:Connection,/,account_id:int,new_phone_number:str)->str:returnawait_q0.execute(__conn,account_id,new_phone_number)asyncdefinsert_account(__conn:Connection,/,username:str,first_name:str,last_name:str,phone_number:str,balance:int,referred_by_account_id:int)->int:returnawait_q1.execute(__conn,username,first_name,last_name,phone_number,balance,referred_by_account_id)asyncdefselect_top_largest_balances(__conn:Connection,/,limit:int)->list[int]:returnawait_q2.execute(__conn,limit)@dataclass(slots=True)classAccountReferrer:id:intusername:strfirst_name:strlast_name:strjoin_ts:datetime.datetimeasyncdefget_account_referrer(__conn:Connection,/,account_id:int)->AccountReferrer|None:returnawait_q3.execute(__conn,account_id)_q3.bind_type(AccountReferrer)@dataclass(slots=True)classAccountPostComment:first_name:strlast_name:strid:intmessage:strasyncdefselect_last_post_comments(__conn:Connection,/,post_id:int,limit:int)->list[AccountPostComment]:returnawait_q4.execute(__conn,post_id,limit)_q4.bind_type(AccountPostComment)__all__=["select_last_post_comments","AccountPostComment","AccountReferrer","insert_account","update_account_phone_number","get_account_referrer","select_top_largest_balances",]So, let's analyze what we got:We have all of our input and output types defined. The linter can now help us whenever we use any of these functions and types in our code.Whenever the database schema changes, the types and function arguments will accommodate automatically: just run the generation script again - and you're set.All the types were inferred from a live database connection, because your database is the single source of truth for your data, not the application.Our "models" are database rows. At last.Do not be discouraged, if you don't like using dataclasses in your projects, as this is just an example!So, if you like what you're seeing, let's configure your project!Basic ConfigurationasyncpgTo install, runpip install querky[asyncpg]Consider this project structure:src
|__ querky_def.py
|__ querky_gen.py
|__ sql
    |__ example.pysqlfolder contains.pyfiles with the query functions. Generated code will be placed in thesql/queriesfolder under the same name as the inital script (example.pyin this case).querky_gen.pyfile is the code generation script. You run it when you want to regenerate the query functions:importasynciofromquerky.presets.asyncpgimportgeneratefromquerky_defimportqrkimportsqlfromenvimportCONNECTION_STRINGif__name__=="__main__":asyncio.run(generate(qrk,CONNECTION_STRING,base_modules=(sql,)))querky_def.pyis code generator configuration. We'll use a preset for the sake of simplicity.importosfromquerky.presets.asyncpgimportuse_presetqrk=use_preset(os.path.dirname(__file__),type_factory='dataclass+slots')The first argument should be the path to the root directory of your project.If you'd like more fine-grained control over theQuerkyobject, there will be an explaination in the later sections.After the configuration of theqrkobject it's time to run thequerky_gen.pyscript.Each of your queries will become type hinted, each of them will return a real Python object,
and you can call these queries as regular Python functions.Every time you change your database schema or queries, you can now expect the changes to propagate throughout your code.
Because of that,refactoring SQL-dependent code has never been easier.This time the linter is on your side.Do not change the generated files, as they are transient and will be overwritten.
If you need to modify the generated code, consider usingon_before_func_code_emitandon_before_type_code_emithooks passed in to theQuerkyobject constructor.Type Hinting ExtensionsArgumentsOptionalsA careful reader might have noticed, that the generated argument types are never optional.
If we go back to the database schema, we will notice, that some of the columns areNULLABLE.
And yet, ininsert_accountquery the respective arguments are notOptional.Why is that?Unfortunately, there is nostraightforwardway for the library to automate this process,
because SQL-wise these are constraints, and not data types.So, it's our job to hint the library to do the right thing.Let's look at the signature of this function again:@qrk.query(shape='value',optional=False)definsert_account(username,first_name,last_name,phone_number,balance,referred_by_account_id):...We know thatphone_number,last_nameandreferred_by_account_idare optional. So we just hint them like this:importtyping@qrk.query(shape='value',optional=False)definsert_account(username,first_name,last_name:typing.Optional,phone_number:typing.Optional,balance,referred_by_account_id:typing.Optional):...Then, the generated function's signature will look like this:asyncdefinsert_account(__conn:Connection,/,username:str,first_name:str,last_name:str|None,phone_number:str|None,balance:int,referred_by_account_id:int|None)->int:...Default valuesLet's consider the sameinsert_accountquery.Let's makereferred_by_account_idNoneby default:@qrk.query(shape='value',optional=False)definsert_account(username,first_name,last_name:typing.Optional,phone_number:typing.Optional,balance,referred_by_account_id=None):...We'll get this signature:asyncdefinsert_account(__conn:Connection,/,username:str,first_name:str,last_name:str|None,phone_number:str|None,balance:int,referred_by_account_id:int|None=_q1.default.referred_by_account_id)->int:..._q1is a reference to the originalinsert_accountQueryobject, which contains the default value for this argument.This way any kind of default argument can be used,
since the generated function always references the original value.Notice that we got rid oftyping.Optionalfromreferred_by_account_id's annotation,
yet the generated signature is stillint | None.
This "type-inference" behavior holds only for theNonedefault value.Enforce typeSometimes even type inference from the database itself does not help.
A prime example would be the infamouspostgres'ARRAY[]type.
Values of this type can have an arbitrary number of dimensions.Postgres docs:The current implementation does not enforce the declared number of dimensions either. Arrays of a particular element type are all considered to be of the same type, regardless of size or number of dimensions. So, declaring the array size or number of dimensions in CREATE TABLE is simply documentation; it does not affect run-time behavior.But oftentimes we find ourselves with a well-known structure,
even though the type itself is permissive (hello, Python!).Consider this query:@qrk.query(shape='value',optional=False)defare_2d_matrices_equal(m):returnf"SELECT{+m}= ARRAY[[1,2,3], [1,2,3], [1,2,3]]::INTEGER[]"It yields this signature:asyncdefare_2d_matrices_equal(__conn:Connection,/,m:list[int])->bool:...Now, let's enforce our knowledge, that this array is two-dimensional.The way we do it is by regular PEP484 annotations:@qrk.query(shape='value',optional=False)defare_2d_matrices_equal2(m:'list[list[int]]'):returnf"SELECT{+m}= ARRAY[[1,2,3], [1,2,3], [1,2,3]]::INTEGER[]"And get this in return:asyncdefare_2d_matrices_equal2(__conn:Connection,/,m:list[list[int]])->bool:...Overriding annotations must always be strings.This is because they are literally copied from the source file into the generated one by using function annotation introspection.
If they were objects, this wouldn't be reliable.Return typesThe same problems apply to fields of a row.They, same as arguments, can be optional, can have a different type from the inferred one.For example, let's pimp theget_account_referrerquery, sincelast_nameis nullable.To do that, we need to import theattrobject:fromquerkyimportattr@qrk.query('AccountReferrer',shape='one',optional=True)defget_account_referrer(account_id):returnf'''SELECTreferrer.id,referrer.username,referrer.first_name,referrer.last_name AS{attr.last_name(optional=True)},referrer.join_tsFROMaccountINNER JOINaccount AS referrerONaccount.referred_by_account_id = referrer.idWHEREaccount.id ={+account_id}'''The generated type will now look like this:@dataclass(slots=True)classAccountReferrer:id:intusername:strfirst_name:strlast_name:str|Nonejoin_ts:datetime.datetimeNotice, howlast_nameis now optional.You can also use-attr.last_namesyntax for optional fields.To override the generated annotation use this syntax:@qrk.query(shape='one')defget_row():returnf'''SELECTARRAY[[1,2,3], [1,2,3], [1,2,3]]::INTEGER[] AS{attr.matrix2d('list[list[int]]')},ARRAY[[[1,2,3],[1,2,3]], [[1,2,3],[1,2,3]], [[1,2,3],[1,2,3]]]::INTEGER[] AS{attr.matrix3d('list[list[list[int]]]')},'Looks like text' AS{attr.definitely_not_text('float')}'''Generates this:@dataclass(slots=True)classSimpleRow:matrix2d:list[list[int]]matrix3d:list[list[list[int]]]definitely_not_text:floatThis looks a bit like magic, but here is how it works:attrobject is a singleton, which records each__getattr__invocation, returning anAttrobject in its stead.
When theAttrobject is called, it records the arguments, and returns a string with which the__getattr__was invoked.
Once the code inside@qrk.querydecorated function is run,
it flushes all the recordedAttrobjects into the query. Then, when the code is being generated,
those objects serve to complete the type information about their respective fields.Different queries - same return typeSometimes there is a couple of different ways to get to the same data.Consider these queries:@qrk.query('AccountInfo',shape='one')defget_account(account_id):returnf'''SELECTfirst_name,last_name,username,phone_numberFROMaccountWHEREid ={+account_id}'''@qrk.query('???',shape='one')defget_last_joined_account():returnf'''SELECTfirst_name,last_name,username,phone_numberFROMaccountORDER BYjoin_ts DESCLIMIT1'''What do we call the second type to not clash with the first one?AccountInfo2?It can be done this way, however this would potentially
lead to many weirdly named duplicate types across the project.Instead, just pass in theget_accountquery in place of the first parameter:@qrk.query(get_account,shape='one')defget_last_joined_account():...This way the type won't be generated the second time,
and the code will use the "parent type" in runtime.You can use queries from other modules, too. Just import them and use as shown,
andquerkywill infer the required imports and place them in the generated file.Query reuseSincequerkyqueries are simple f-strings, there is no limit to combining them together via CTEs or
simply replacing a part of your query with another one.You can use subqueries as arguments to the main query with this technique:fromquerkyimportsubquery@qrk.query(shape='value',optional=False)defadd(a,b):returnf"SELECT{+a}::INTEGER +{+b}"@qrk.query(shape='value',optional=False)defmultiply(a,b):returnf"SELECT{+a}::INTEGER *{+b}"@qrk.query(shape='value',optional=False)defdivide(a,b):returnf"SELECT{+a}::INTEGER /{+b}"@qrk.query(shape='value',optional=False)defoperation(a,b):@subquerydefadd_ab():returnadd.query(a,b)@subquerydefmult_ab():returnmultiply.query(a,b)returndivide.query(add_ab,mult_ab)The resulting SQL will be:SELECT(SELECT$1::INTEGER+$2)::INTEGER/(SELECT$1::INTEGER*$2)We use::INTEGERcast to explicitly tell the database that in this query we work with integers.
Otherwise, query "compilation" will fail, because+,-and/operators exist for many types,
so there is no definitive way for the DBMS to infer what we meant.If it's not enough for you, you can generate SQL by joining strings,
the universal oldest method of how the ancients did it.
Or use another library for this particular task.Just remember thatthe decorated function actually runs only onceto generate the SQL query.
At runtimequerkyalways uses that SQL query, never rerunning the function again.
Except, of course, for explicitQuery#query(...)calls, but these don't changeQuerystate in any way.If you need query folding capabilities, e.g.INSERTa variable number of rows with a single query,
be sure to look into thequerky.tools.query_foldermodule.How it WorksTheQuerkyclassIt is the class to configure how the library will talk to the database and provide code generator configurations.Let's go over constructor's arguments one by one:basedirThe absolute path to the base directory of your project.This path serves as an anchor, so that the generated files are placed properly.annotation_generatorIt's an instance ofAnnotationGeneratorsubclass, which, well, generates annotations both for arguments and for return types,
based onTypeKnowledgecollected.It's safe to use theClassicAnnotationGeneratorfor every use-case here.contractThis is an instance ofContractsubclass. ThisContractis betweenquerkyand your database driver of choice.TheContractdescribes a common interface forquerkyto talk to the database, execute your queries and infer the types.conn_param_configThis directive tellsquerkywhere you want to place your database connection argument for every generated function.Available subclasses:FirstandLast.type_factoryThis function createsoneandmanyqueries' return types. The currently implemented types are underquerky/type_constructors.TypedDictConstructor- generates a subclass oftyping.TypedDict. It's a regular dictionary with linter support.DataclassConstructor- generates a class decorated with@dataclasses.dataclass(...). Any additional**kwargspassed toDataclassConstructor's constructor will be reflected in the decorator. E.g.kw_only=True,slots=True.EveryTypeConstructorhas arow_factoryargument, which should be provided in case your database driver does not return the expected type.Therow_factoryis simply a converter from whatever the database driver returns to the type you need.In this example, we convert nativeasyncpg'sRecordobjects to Pythondataclasses.It is up to the user to implementrow_factory.subdirName of the subdirectory, where all the generated files will go.E.g., consider this project layout:payment_service
|__ payments.py
|__ withdrawals.py
|__ __init__.py
delivery_service
|__ deliveries.py
|__ products.py
|__ __init__.pyConsidering every.pyfile has queries andsubdir="queries",
we're going to end up with the following structure:payment_service
|__ queries
    |__ payment.py
    |__ withdrawal.py
|__ payment.py
|__ withdrawal.py
|__ __init__.py
delivery_service
|__ queries
    |__ delivery.py
    |__ product.py
|__ delivery.py
|__ product.py
|__ __init__.pyIf not specified, files will be placed under the same directory as the source file, but with_queriespostfix appended.The previously mentioned structure would become:payment_service
|__ payment.py
|__ payment_queries.py
|__ withdrawal.py
|__ withdrawal_queries.py
|__ __init__.py
delivery_service
|__ delivery.py
|__ deliviry_queries.py
|__ product.py
|__ product_queries.py
|__ __init__.pyYou can change the naming behavior by overridingQuerky#generate_filename.Custom Database Typesasyncpgtype_mapperIf you define custom types in yourpostgresdatabase,
you should also put conversions into theAsyncpgNameTypeMapperobject using theset_mappingmethod,
otherwisequerkywouldn't know about them.asyncpg's basic conversions are definedhere.These are all availableasyncpgdefault types:@qrk.query(shape='one',optional=False)defall_default_types():returnf'''SELECTNULL::INTEGER[] AS _anyarray,NULL::TSRANGE AS _anyrange,NULL::NUMMULTIRANGE AS _anymultirange,NULL::RECORD AS _record,NULL::VARBIT AS _bitstring,NULL::BOOL AS _bool,NULL::BOX AS _box,NULL::BYTEA AS _bytes,NULL::TEXT AS _text,NULL::CIDR AS _cidr,NULL::INET AS _inet,NULL::MACADDR AS _macaddr,NULL::CIRCLE AS _circle,NULL::DATE AS _date,NULL::TIME AS _time,NULL::TIME WITH TIME ZONE AS _timetz,NULL::INTERVAL AS _interval,NULL::FLOAT AS _float,NULL::DOUBLE PRECISION AS _double_precision,NULL::SMALLINT AS _smallint,NULL::INTEGER AS _integer,NULL::BIGINT AS _bigint,NULL::NUMERIC AS _numeric,NULL::JSON AS _json,NULL::JSONB AS _jsonb,NULL::LINE AS _line,NULL::LSEG AS _lseg,NULL::MONEY AS _money,NULL::PATH AS _path,NULL::POINT AS _point,NULL::POLYGON AS _polygon,NULL::UUID AS _uuid,NULL::TID AS _tid'''@dataclass(slots=True)classAllDefaultTypes:_anyarray:list[int]_anyrange:_Range_anymultirange:list[_Range]_record:_Record_bitstring:_BitString_bool:bool_box:_Box_bytes:bytes_text:str_cidr:Union[IPv4Network,IPv6Network]_inet:Union[IPv4Interface,IPv6Interface,IPv4Address,IPv6Address]_macaddr:str_circle:_Circle_date:datetime.date_time:datetime.time_timetz:datetime.time_interval:datetime.timedelta_float:float_double_precision:float_smallint:int_integer:int_bigint:int_numeric:Decimal_json:str_jsonb:str_line:_Line_lseg:_LineSegment_money:str_path:_Path_point:_Point_polygon:_Polygon_uuid:UUID_tid:tupleParameter Substitutionquerkynever uses string concatenation to put arguments into a query. SQL injection is impossible.What it does instead is create a parametrized query based on yourContract(database driver)
and your decorated function's signature.You can make sure by checking out any of your queries'sqlfield -
it contains the actual SQL query that will always be used for this particularQueryobject.The core of parameter substitution are theParamMapperandMappedParamclasses.When a function is decorated with@qrk.query, it gets wrapped with aQueryobject.TheQueryobject in turn calls the function inside its__init__method with all arguments
substituted withMappedParamobjects generated by yourContractbackend'sParamMapperobject. EachParamMapperobject is unique per query.The f-string gets run and theMappedParamobjects "remember" the positions of the arguments inside the query.After that theParamMapperobject is always ready to remap any incoming arguments into your database driver's format, since it knows the signature of the function, the order of arguments and which arguments they were.Query "compilation"Once you run thequerky_gen.pyscript, all the queries arePREPAREd against the DBMS to
let it infer the types and check the correctness of the queries.
You can call it a kind of "compilation step".
Which, by the way,conveniently checks query correctness before the program is run, meaning, that if you have
a syntax error, the DBMSitselfwill tell you that, before you ship your code.Tips and TricksUsing ORM models for schema hintingI personally found it very convenient to use ORM models to have
a copy of the expected database schema state.This way you can use the linter to your advantage when refactoring code: e.g. if a column gets dropped,
the linter will be screaming "this field does not exist" for every query you used it in.I suggest exploring thesqlacodegenpackage to tackle the issue.Code generation is not requiredQueryobjects are actually callable, if you look at the sample generated code. So, you don't need the code generation
procedure to use the argument mapping capabilities of this package.Issues, Help and DiscussionsIf you encountered an issue, you can leave it here, on GitHub.If you want to join the discussion, there are Telegram chats:English Language Chat 🇺🇸🇬🇧Russian Language Chat 🇷🇺If you need to contact me personally:Email:verolomnyy@gmail.comTelegram:@datscrazyLicenceMIT License

Copyright (c) 2023 Andrei Karavatski