Package: vdk-heartbeat
Versatile Data Kit Heartbeat toolHeartbeat tool for verifying deployed SDK and Control Service are functional and working correctly.It  checks that a job can be created, deployed, run and deleted.Health monitoring - one can schedule this run every few minutes and get heartbeat of their installationRegression testing after change in Versatile Data Kit, as clients would be able to customize vdk, they should have a way to check for regressions.Installation/Upgrade of VDK Control Service acceptance test.What it does ?It simulates Data Engineer workflow:Creates a data job, downloads keytabDeploys the data job with pre-defined scripts to run on a scheduled basis (every minute)Different data jobs and run tests can be run depending on the configuration.This way it can be run in different modes. Seeconfig.pyand the DATAJOB_DIRECTORY_* and JOB_RUN_TEST_* configuration options.Undeploys and deletes the data job.PrerequisitesSeeheartbeat_config_example.iniand complete the TODOs insideInstallation# TODO: Change to public PYPI indexpipinstall-ihttps://test.pypi.org/simple/vdk-heartbeatConfigurationSeeconfig.pyfor details on what can be configured.RunningYou can run the test locally, part of your CICD or schedule it to run periodically.The test is passed or fail test.If it fails it returns non-zero error code and prints the error.It also produces a tests.xml file in junit xml format.Specify configuration in environment variables or in a file (use the file for things that can be in source control)Example:exportDATABASE_PASS=xxx
vdk-heartbeat-fheartbeat_config.iniExtensibilityUsers can replace the data job being deployed and executed and the run test which is used to verify the job run/execution.Seeconfig.pyDATAJOB_DIRECTORY_* and JOB_RUN_TEST_* configuration options.Build, test, and releaseSee or runcicd/build.shto build and test the project locally.ReleaseReleases are made to PyPI.Versioning followshttps://semver.org.A release step in Gitlab CI is automatically triggered after merging changes if build/tests are successful.To trigger a control-service integration tests image rebuild, commit anycontrol_service_change_locations-defined change inCONTRIBUTING.mdexceptversion.txt- until automatedUpdate major or minor version when necessary only.TestsDatabase ingestionThe testing job ingests data into a database and reads it from that database to verify the results.ConfigurationTargetTarget identifies where the data should be ingested into.The value for this parameter depends on the ingest method chosen.For "http" method, it would require an HTTP URL.
Example:http://example.com///For "file" method, it would require a file name or path.export VDK_HEARTBEAT_INGEST_TARGET="datasource"MethodIndicates the ingestion method to be used. Example:method="file" -> ingest to filemethod="http" -> ingest using HTTP POST requestsmethod="kafka" -> ingest to kafka endpointexport VDK_HEARTBEAT_INGEST_METHOD="http"Destination tableThe name of the table, where the data should be ingested into.
This parameter does not need to be passed, in case the table is
included in the payload itself.export VDK_HEARTBEAT_INGEST_DESTINATION_TABLE="destination_table"Database typeexport DB_DEFAULT_TYPE="trino"Database nameexport DATABASE_TEST_DB="memory.default"ScenariosTrino ingestionVDK_HEARTBEAT_INGEST_METHOD is set to "TRINO" and DB_DEFAULT_TYPE is set to "TRINO"
and the connection settings for both is the same (same Trino database instance).export VDK_HEARTBEAT_INGEST_TARGET="trino-http-datasource"
export VDK_HEARTBEAT_INGEST_METHOD="TRINO"
export VDK_HEARTBEAT_INGEST_DESTINATION_TABLE="sample_destination_table"
export DB_DEFAULT_TYPE="trino"
export DATABASE_TEST_DB="memory.default"Trino HTTP ingestionVDK_HEARTBEAT_INGEST_METHOD is set to "HTTP" and DB_DEFAULT_TYPE is set to "TRINO"
and connection settings are set to same Trino instance.export VDK_HEARTBEAT_INGEST_TARGET="trino-http-datasource"
export VDK_HEARTBEAT_INGEST_METHOD="http"
export VDK_HEARTBEAT_INGEST_DESTINATION_TABLE="sample_destination_table"
export DB_DEFAULT_TYPE="trino"
export DATABASE_TEST_DB="memory.default"Ping the frontendNo additional configuration is needed for this test.Point heartbeat to the correct module and classJOB_RUN_TEST_MODULE_NAME=vdk.internal.heartbeat.ping_frontend_test
JOB_RUN_TEST_CLASS_NAME=PingFrontendTestThe CONTROL_API_URL environment variable doubles as the frontend url, since both
th control service and the frontend are deployed to the same host by the helm
chart.CONTROL_API_URL=http://cicd-control-service-svc.cicd.svc.cluster.local:8092The test sends aGETrequest to the URL and expects a success response