Package: iuindycar
Setting up IndyCar on KubernetesSTEP 1 : Setting up the Kubernetes clusterA minikube setup can be used to test the distribution locally while it’s recommended to allocate at least  6-8 CPUs and 12-16GB RAM for the minikube VM.A comprehensive guide on setting up minkube can be found from the blow URL.https://kubernetes.io/docs/setup/learning-environment/minikube/STEP 2 : Setting up the Kubernetes DashboardClone the IndyCar repository to your machine.gitclonehttps://github.com/DSC-SPIDAL/IndyCar.gitInside the “containerize” directory, you would find a bash script called “setup_k8.sh”. Navigate to the "containerize" directory and execute setup_k8.sh script to set up the kubernetes dashboard../setup_k8.shNow, from a web browser of your choice, you should be able to access the Kubernetes dashboard through below URL.http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/loginUse the token printed in the previous step to Sign In.STEP 3 : Deploying ZookeeperWe perform IndyCar anomaly detection using an opensource HTM implementation called HTM Java. We have been running HTM Java networks on Apache Storm for convenience and to handle future data stream preprocessing requirements. Apache Storm relies on Zookeeper for coordination between Nimbus and the Supervisors.Zookeeper can be deployed using kubectl as follows.kubectlcreate-fzookeeper.json
kubectlcreate-fzookeeper-service.jsonAllow few minutes to initialize Zookeeper.STEP 3 : Deploying Storm NimbusStorm nimbus can be deployed as follows.kubectlcreate-fstorm-nimbus.json
kubectlcreate-fstorm-nimbus-service.jsonBy default, this definition maps the following directories to the nimbus. If you want to override the default storm configs,
copy a modified storm.yaml configuration file to /nfs/indycar/config directory. Note that this directory will be created inside the minikube VM.{"volumes":[{"name":"topologies","hostPath":{"path":"/nfs/indycar/data"}},{"name":"config","hostPath":{"path":"/nfs/indycar/config"}}]}STEP 4 : Deploying Storm Dashboard(Optional)Storm dashboard can be deployed as follows.kubectlcreate-fstorm-ui.json
kubectlcreate-fstorm-ui-service.jsonNow execute following command to get the IP of the minikube VM.minikubeipExecute following command to determine the port that has been assigned for strom ui.kubectlgetservicesThis should give an output similar to below.storm-uiNodePort10.98.10.86<none>8080:30336/TCP5dNow you should be able to access the storm ui via a web browser.http://<minikubeip>:30366STEP 5 : Deploy Strom SupervisorsStrom supervisors should be dynamically scalable. So we deploy them as a Kubernetes replication controller.In storm-worker-controller.json adjust the resource requirements as per your requirement.{"resources":{"limits":{"cpu":"11","memory":"100G"}}}kubectlcreate-fstorm-worker-controller.jsonSTEP 6 : Deploy Apollo BrokerWe use Apache Apollo broker to enable communication between components(Processes, Pods, Nodes).Apollo broker can be deployed as follows.kubectlcreate-factivemq-apollo.json
kubectlcreate-factivemq-apollo-service.jsonSTEP 7 : Deploy IndyCar Socket ServerIndyCar socket server will be used as both the log file streamer, and the backend for the IndyCar web application.kubectlcreate-fsocket-server.yamlChange the below parameters appropriately.args:["java","-jar","server.jar","/data/logs/eRPGenerator_TGMLP_20170528_Indianapolis500_Race.log","tcp://activemq-apollo:61613","33","2017","compact_topology_out_2017"]/data/logs/eRPGenerator_TGMLP_20170528_Indianapolis500_Race.logis the absolute path to the log file. Copy the log files to the/nfs/indycar/datalogsof the  minkube VM.tcp://activemq-apollo:61613is the URL of the apollo broker. This can be kept with this default setting.33is the number of cars to stream. Reduce this number, if you don't want to stream all 33 cars.2017is the prefix that will be used when creating broker topics. This can be used to support multiple streams of multiple years within the same cluster.compact_topology_out_2017is the output topic name. Storm workers should publish the processed data back to this topic.Similar to the Storm-UI you can determine the port assigned for socket server by executingkubectl get servicescommand.STEP 8 : Building Strom TopologyPre requisite: Apache MavenNavigate to thestreamingdirectory at the root of IndyCar repository and execute below command to build the storm topology.mvncleaninstallNow copy thetarget/Indycar500-33-HTMBaseline-1.0-SNAPSHOT.jarto the/nfs/indycar/datadirectory of minikube VM.STEP 9 : Starting dashboard client application.Pre requisite: Node Package Manager(NPM)Navigate to thedashboarddirectory at the root of IndyCar repository.Change the following line ofsrc/index.jsfile to point to the IP and the port of the Socket Sever.letsocketService=newSocketService("149.165.150.51",31623,store);//let socketService = new SocketService("<minikube ip>", <socket server port>, store);Now execute below command within thedashboarddirectory to download the required dependencies.npminstallNow execute below command to start the dashboard locally.npmstartNow you will be able to access the dashboard from a web browser.http://localhost:3000STEP 8 : Deploy Jupiter NotebookJupyter notebook can be used to easily deploy the IndyCar anomaly detection cells and test new components.kubectlcreate-fjupyter.yamlSimilar to Storm-UI usekubectl get servicescommand to determine the port assigned for the notebook.Now upload IndyCar-API.ipynb to your notebook server and use it appropriately.STEP 9 : Using python packageAfter completing upto Step 8, you have all the tools deployed to test/run your streaming processing application.Optionally, you can use iuindycar package to make the deployment easier.pipinstalliuindycarNow import the iuindycar package to your notebook as followsimportiuindycar.OrchestratorasiuiYou can create an instance of Orchestrator as follows.oc=iui.Orchestrator(iui.Config(<passyourk8tokenhere>))The only mandatory parameter to the Config object is kubernetes token. But you have the flexibility to change
the default parameters of the config object. Please refer below URL for more information.https://github.com/DSC-SPIDAL/IndyCar/blob/070bd2ac546a39756f4a5c839dbf1f3a1d700d20/containerize/python/iuindycar/Orchestrator.py#L225Deploying a new streamcell=iui.DetectionCell(storm_jar="Indycar500-33-HTMBaseline-1.0-SNAPSHOT.jar",class_name="com.dsc.iu.streaming.LSTMAnomalyDetectionTask")oc.deploy_stream(<uniquenameforthestream>,<inputdatatopic>,<outputdatatopic>,detection_cell=cell)oc.deploy_stream("indycar-22","22","compact_topology_out")Killing an existing streamoc.kill_stream(<uniquenameforthestream>)oc.kill_stream("indycar-22")Inspecting a broker topicTo make the debugging and testing easier, you can listen to the broker topics using the Orchestrator object as follows.stream=oc.probe_topic(<topicname>)# stream = oc.probe_topic("22")stream.loop_forever()In addition to the loop_forver() option, you can use any of the functions provided by paho-mqtt package.https://pypi.org/project/paho-mqtt/