Package: vdk-ipython
vdk-ipythonIpython extension for VDKThis extension introduces a magic command for Jupyter.
The command enables the user to load job_input for his current data job and use it freely while working with Jupyter.See more about magic commands:https://ipython.readthedocs.io/en/stable/interactive/magics.htmlInstallationTo use the extension it must be firstly installed with pip as a python package.pip install vdk-ipythonUsageThen to load the extension in Jupyter the user should use:%reload_ext vdk.plugin.ipythonData Job Python coding cellsAnd to load the VDK (Job Control object):%reload_VDKThe %reload_VDK magic can be used with arguments:ArgumentDescription--paththe path of the data job. Usually you want to leave the default (the directory of Notebook file)--namethe name of the data job. Usually you want to leave the default (the directory name of the Notebook file)--argumentsArguments (in json format) to be passed to the job--log-level-vdkThe log level of the VDK logsData Job SQL CellsYou can also specify%%vdksqlcell magic to convert cell into SQL cell
which will using Job Input Managed Connection%vdksql
select * from my_tableThe output of cell will be a table.
Ifipyaggridis installed then the table would ipyaggrid type of table
which allows filtering, search and other cool thingsExampleThe output of this example is "myjob"%reload_ext vdk.plugin.ipython

%reload_VDK --name=myjobresponse = requests.get("https://jsonplaceholder.typicode.com/todos/1")

job_input.send_object_for_ingestion(
    payload=response.json(), destination_table="placeholder_todo"
)%%vdksql
select * from placeholder_todo
where completed = TrueIngesting data with %%vdkingest%%vdkingest# Data Source Configuration[sources.yourSourceId]## Data Source Name. Installed dta sources can be seen using vdk data-sources --listname="<data-source-name>"## The singer tap we will useconfig={## Set the configuration for the data source.## You can see what config options are supported with vdk data-sources --config <data-source-name>}[sources.yourSourceId_2]# repeat this for as many sources you want# ...# Data Destination Configuration.## Ingestion methods and targets are the same one as those accepted by send_object_for_ingestion## See https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/api/job_input.py#L183[destinations.yourDestinationId]## the only required parameter is methodmethod="<method-name>"## Optionally specify target## target =[destinations.yourDestinationId_2]# repeat this for as many destinations you want# ...# Data Flows from Source to Destination[[flows]]from="yourSourceId"to="yourDestinationId"[[flows]]from="yourSourceId_2"to="yourDestinationId_2"Complete the full self-paced tutorial athttps://bit.ly/vdk-ingestBuild and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9, 3.10 and 3.11).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.