Package: jupyter-vvp
jupyter-vvpThis extension to Jupyter lets you write and execute Flink SQL statements right from your Jupyter notebooks.
It is backed by Ververica Platform's SQL REST API.PrerequisitesIn order to use the Jupyter magics, you will require access to an installation of the Ververica Platform.To set up Ververica Platform follow the instructions athttps://docs.ververica.com/getting_started/index.htmlCompatibilityVerverica Platform VersionJupyter-VVP Version2.30.1.0InstallationThe package can be installed from PyPi withpip install jupyter-vvpJupyter VVP comes with a custom kernel that extendsipykernelwith code-completion for SQL Code through Ververica Platform.
In order to use the kernel you need to install it:First locate the installation directory of the package:pip show jupyter-vvpGo to the installation directory and install the kernel:jupyter-kernelspec install jupytervvp/flinksqlkernelUsageLoading the extensionFrom within IPython (ipython3) or an IPython3 kernel in any local Jupyter instance,
run%load_ext jupytervppto load the extension and register the magics.Connecting to Ververica PlatformIn order to use the Ververica Platform Jupyter magics, you will first need to connect to a Ververica Platform instance.
The%connect_vvpmagic can be used for that:%connect_vvp localhost --port 8080 --namespace default -s <your-session-name>SQL StatementsThis will set up your notebook to communicate with the Ververica Platform.
We can test this by trying a DDL statement, e.g. to display all existing tables:%%flink_sql
SHOW TABLESTheflink_sqlmagic can, of course, also be used to execute DML statements via the Ververica Platform:%%flink_sql
CREATE TEMPORARY TABLE my_source_table (
id INT,
data STRING
) WITH (
'connector' = 'datagen'
);
CREATE TEMPORARY TABLE my_sink_table (
data STRING
) WITH (
'connector' = 'blackhole'
);
INSERT INTO my_sink_table SELECT data FROM my_source_tableExamplesA few example notebook can be found in theexample_notebooks folderAdvanced UsageSessionsA connection to a Ververica Platform instance is represented by asession,
specifying its hostname and port,
and an API token if required.
The session has a name for convenient reference.From within IPython (ipython3) or an IPython3 kernel in a Jupyter instance,
run%connect_vvp localhost -p 8080 -n default -s mysessionThis will connect and create a session with the namemysession.
The Ververica Platform host in this example islocalhostand the port is8080.
The hostname should be the name under which Ververica Platform is accessible from the Jupyter server.
To connect to Ververica Platform through HTTPS, use the--secure(or-S) parameter.
In case Ververica Platform uses a self-signed certificate, it is possible to deactivate certificate checking using--secure_self_signedinstead of--secure.
If no session exists then this session will be set as the default.Session names are treated by the magics as strings.
The corresponding session object, with that name,
is found and taken from the local user context.Session names can be listed by executingfromvvpmagicsimportvvpsessionvvpsession.VvpSession.get_sessions()When executing statements a session can be specified via:%%flinksql mySession
...Using API TokensThe argument-k <API-Key>(or--key <API-Key>)
will use the given value in<API-Key>as the API Key.To avoid having keys in notebooks,
the argument-Kor--prompt_keycan be specified,
which asks the user to enter the key,
and overrides any value specified by-k.If no keys are specified, no API keys are used.Examples:%connect_vvp HOSTNAME -n default -s mySession -k 10504c2d-55f0-4969-ba83-26fad5f1640f
%connect_vvp HOSTNAME -n default -s mySession -KSubstituting user variablesUser variables defined in the notebook can be referenced inflink_sqlcells.
To reference the variable, surround it with braces.
For example, settingtopic_name="myTopic"allows the user (as in the example above) to do%%flink_sql 

    .....

   ...: WITH ( 
   ...:     'connector.topic' = '{topic_name}', 

    .....

   ...: )The cell is treated as a string,
and variables are replaced using Python'sstring.format()method,
so in principle all variables
that have a reasonable representation as a string can be used.
The scope is theuser_nsdictionary,
accessed by Python via the IPython shell object.Take care to avoid nesting braced expressions,
but note that double-brace placeholders may also be used (double-braced placeholders are used in the Flink configuration,
for more information see the Flink configuration section below).Setting deployment parametersDeployments of SQL INSERT jobs can be customised by setting parameters.
The possible settings keys are listed in a parameters dictionary in the example notebook,
and its use is shown there.
To use these parameters, the switch-p [parameters-variable-name]is used in theflink_sqlMagic.
If no switch is specified, the default variablevvp_default_parametersis used.You may find the following documentation generally useful:Deployment Template settingsLifecycle Management settingsImportant parametersPlease find some frequently used parameters below:SettingPossible valuesCommentDocumentationmetadata.nameArbitrary stringIf not specified, then this will be the cell contents.metadata.annotations.license/testingBoolean:TrueorFalseTheflink_sqlmagic will set this toFalseif not specified.spec.template.spec.parallelismIntegerLinkspec.restoreStrategyString:"LATEST_STATE","LATEST_SAVEPOINT", or"NONE".Linkspec.upgradeStrategyString:"STATELESS","STATEFUL", or"NONE".LinkFlink ConfigurationIn the deployment settings,
keys of the formspec.template.spec.flinkConfiguration.<FlinkConfigurationKey>can be used.
The user can specify Flink configuration parameters in place of<FlinkConfigurationKey>.
For example,"spec.template.spec.flinkConfiguration.state.savepoints.dir": "s3://flink/savepoints"Seeherefor deployment configuration documentation.Note that the placeholders (e.g.,{{Namespace}}) appearing inflinkConfigurationsettings
are left untouched by%%flink_sql, so can be used as normal;
e.g.:"spec.template.spec.flinkConfiguration.state.savepoints.dir": "s3://flink/savepoints/{{ namespace }}"Seeherefor further details on placeholders.Error messagesBoth the%connect_vvpand the%%flink_sqlmagics support the-d/--debugflag
to show full error messages.
In case of error results from the Ververica Platform it will display the full JSON response.Code completionWhen using the FlinkSql kernel, SQL queries in a%%flink_sqlcell will be completed with suggestions from Ververica Platform.
In order for code completion to work, a connect_vvp session needs to exist.
The default session will be used to communicate with Ververica Platform unless the another session is set for the%%flink_sqlcell.
If any problems occur in the communication with Ververica Platform, code completion will attempt to use the standard Jupyter completion.Help%connect_vvp?and%%flink_sql?will show help texts for the magics.BugsReport an issue athttps://github.com/ververica/jupyter-vvp/issues