Package: yandex-query-magic
yandex_query_magicYandex Query cell (%%yq) and line (%yq) magics for Jupyter and VSCodeMotivationyandex_query_magic was created to:Yandex Query magics for Jupyter Lab/Notebook simplify the process of writing and executing queries directly within notebooks, enhancing productivity and interactive data exploration.They enable seamless integration with Yandex Query services, allowing users to effortlessly access and analyze large datasets hosted on Yandex’s cloud platform without leaving the Jupyter environment.By providing shortcuts for common operations and queries, these magics reduce the necessity for boilerplate code, making data analysis workflows more efficient and user-friendly.InstallationInstall pip package:%pip install yandex_query_magic --upgrade --quietJupyter LabNothing more needs to be done, Jupyter Lab contains everything.Jupyter NotebookEnable jupyter extension for UI controls in Jupyter Notebooks:%jupyter contrib nbextension install --userIf you encounter "No module named 'notebook.base'" error, consider switching to Jupyter notebook version 6.4.12.
Discussion can be foundat Stackoverflow.Check the extensionRun extension in Jupyter:%load_ext yandex_query_magic
%yq_settings --folder-id <yandex_cloud_folder_id>
%yq select 1i.e.%yq_settings --folder-id b1gjt6gku4aa28p61fstUsageGlobal settingsExample:%yq_settings --folder-id <yandex_cloud_folder_id>Parameters:--folder-id <folder_id>:Required. Default folder to execute Yandex Query queries.--vm-auth:Default mode. If set sets authentication mode to VM account key. Seemore.--sa-file-auth <sa_key.json>: If set sets authentication mode to authorized keys. Seemore.Basic usage%yqselect1Here%yqis the magic's name andselect 1is the query text.Advanced usage%%yq--folder-id b1ggt6geu4aa38p61kst --name "My query" --description "Test query" --raw-resultsselectcol1,count(*)fromtablegroupbycol1Parameters:--folder-id <folder_id>: folder to execute Yandex Query the query.--name "<name>": query name.--description "<description>": query description.--raw-results: returns raw unconverted results from Yandex Query. Specification can be foundhere.Variables expansionyandex_query_magics support several ways of variables expansion.Jinja2 templates%%yq<otherparameters>--jinja2select*fromtablewherecol1like"{{val}}"Parameters:--jinja2: turns on rendering query text withJinjatemplates. RequiresJinja2package to be installed.Capture DataFrame variable from Jupyteryandex_query_magic can capture variables specified in query in mustache syntax.Suppose, you have a DataFrame:df=pd.DataFrame({'_float':[1.0],'_int':[1],'_datetime':[pd.Timestamp('20180310')],'_string':['foo']})Thendfcan be used as variable in YQ queries. While running DataFrame is created like table with DataFrame structure and named like a variable,dfin current example.%%yqselect*frommytableinnerjoin{{df}}onmytable.id=df._intCurrently supported Pandas types:int64float64datetime64[ns]stringIf you do not want to use variable expansion functionality, it can be disable using--no-var-expansionparameter, like this:%%yq--no-var-expansionselect*frommytableinnerjoin{{df}}onmytable.id=df._intCapture Dict variable from Jupyteryandex_query_magic can capture variables specified in query in mustache syntax.Suppose, you have a Dict:dct={"a":1,"b":2.0,"c":"test","d":datetime(2022,2,2,21,12,12)}Thendctcan be used as variable in YQ queries.
It will be transformed to single row table of structure:abcd12.0"test"DateTime("2022-02-02 21:12:12")Dictdctcan be used in queries named as variable,dct.%%yqselect*frommytableinnerjoin{{dct}}onmytable.id=dct.aCurrently supported types:intfloatdatetimestringCapture List variable from Jupyteryandex_query_magic can capture variables specified in query in mustache syntax.Suppose, you have a List:lst = [1,2,3]Thenlstcan be used as variable in YQ queries. While running List is created and named like a variable,lstin current example.%%yqselect*frommytablewheremytable.idin{{lst}}Currently supported types:intfloatdatetimestringCapturing outputLine Magics can be captured with assignment:varname = %yq <query>Cell magics can be captured specifying variable name at query text start with<<operator%%yqvarname<<<query>To silence a cell, you can stack %%capture:%%capture
%%yq
<query>