Package: ds-planner
glancefeed-reco-spaces-zappcontent-personalizationOverviewThis is a repository with the code for building pipelines for Zapp Content Personalization.
Currently, the pipeline for BreakingNews Zapp is built using Spark Structured Streaming.
In general, any Zapp pipeline should consist the following components:Consuming data from KafkaA High Pass Filter to filter out the data that is not relevant to the Zapp (Static filters)A Low Pass Filter to filter out the data that is not relevant to the Zapp (Dynamic filters)A ranking algorithm to rank the data based on the Zapp's requirementsWriting the output to KafkaPOCsEmbedded Model in the PipelineRPC Call from the PipelineMessage Queue based ML Inference ServerPreparing the pipeline for a new ZappUse this repo as the templateIn the config.ini fileupdate the following fields:[kafka-config]group.id=<consumer-group-id># Consumer group ID should in the format of consumer-glance-ds*zapp.widget.id=<zapp-widget-id># Zapp Widget ID to be deployed into[gcp-config]checkpoint.path=<gcs-checkpoint-path># for checkpointing the streaming datacheckpoint.analytics.path=<gcs-analytics-checkpoint-path># for checkpointing the streaming data - streaming[env-config]env=<DEV/STAGING/PROD>Running the pipeline1. Environment setupDev environmentIn this environment, output will be displayed on the console and not written to the output topic.Set the env config in config.ini to DEV and project number to the project number of the non-prod projectCreate a GCP DataProc cluster on your non-prod project with the spark properties as follows:--properties  spark:spark.jars.packages=org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2Staging environmentIn this environment, output will be written to the staging output topic.Set the env config in config.ini to STAGING and project number to the project number of the non-prod projectCreate a GCP DataProc cluster on your non-prod project with the spark properties as follows:--properties  spark:spark.jars.packages=org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2Production environmentTBD2. Run the pipelineThe pipeline can be run in two ways:Running the pipeline as a spark jobRunning the pipeline as a serverless streaming pipeline1. Running Zapp Pipeline as a spark jobFirst make necessary config changes in config.ini and prepare your script item_fan_out.pygsutilcp-rsrcgs://pipelines-staging/spark_job/item_fan_outNext, run the pipeline as a spark jobgclouddataprocjobssubmitpyspark--cluster<cluster-name>--region<region>--propertiesspark:spark.jars.packages=org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2gs://pipelines-staging/spark_job/item_fan_out/item_fan_out.py--filesgs://pipelines-staging/spark_job/item_fan_out/config.ini,gs://pipelines-staging/spark_job/item_fan_out/utils.py2. Running Zapp Pipeline as a serverless steaming pipelineFirst prepare your scriptstreaming_zap_pipeline.pygsutilcpsrc/streaming_zap_pipeline.pygs://pipelines-staging/serverless/streaming_zap_pipeline.pyNext trigger a serverless jobOpen notebooks/launch_pipeline_poc.ipynbReferencesCMS <> DS ContractDS <> Planner-Ingestion ContractCreating a secret in Google Secret Manager -https://github.tools.inmobi.com/shivjeet-bhosale/secret_manager/blob/main/generate_secrets.ipynbKafka topic creation stepsGo tohttps://confluent.cloud/homeFor Dev/staging, create a topic in CONFLUENT-KAFKA-NON-PROD-SG-001 cluster under Glance-Staging environmentTopic name should be in the format of glance-ds-non-prodtopic-name