Package: ml-pipe-core
ML PipelineMachine learning pipeline.How to create and run an Agent?To create an Agent you need to clone the ml-pipe-service repository and install ml-pipe-core.git clonemkdir .venvpipenv install .Afterwards you create a class which inherits form SimpleService.classSillyAgent(SimpleService):def__init__(self,name,adapter):super().__init__(name,adapter)self.factor=1.0defreconfig_event(self,msg):factor=msg.get('factor')iffactorisNone:_logger.error("reconfig message doesn't have 'factor' as a key.")else:_logger.debug(f"set new factor to:{factor}")self.factor=factordefproposal(self,params):_logger.debug(f"use factor:{self.factor}")hcors=self.machine_adapter.get_hcors(names=self.machine_adapter.get_hcor_device_names(),is_group_call=True)self.set_machine(SetMachineMessage({'hcor':{"names":self.machine_adapter.get_hcor_device_names(),"values":[val*self.factorforvalinhcors]}}))if__name__=='__main__':agent=SillyAgent('silly_agent',PetraSimulationAdapter())agent.init_local()Inside this class you need to overloaddefproposal(self,params)which will be called when the Agent is triggered. The Machine/Simulation can be set withdefset_machine(self,data:SetMachineMessage)This function is taking an SetMachineMessage as input parameter. SetMachineMessage holds a python dictionary with which you can set the Machine/Simulation. Currently only setting horizontal/vertical correctors, machine parameters and twiss parameters are supported. The structure of the dictionary have to look like this:{'hcor':{'name':str,'values':List[float]},'vcor':{'name':str,'values':List[float]},'machineparams':{'name':str,'values':List[float]},'twiss':{'name':str,'mat':List[List[float]]}# only for simulation}Optionally you can overloaddefreconfig_event(self,msg)which give you the possibility to reset internal parameters without restarting your Agent.After creating an Agent you need configure the agent in a yaml file.Here is an example of config.yaml file.Notes: You have to replace localhost with the real server name.observer:
  # used to check if jbb is done
  url: localhost:5003
  event_db_pw: xxxxx
  # event_db_url:
  event_db_usr: xxxxx
register:
  # registers all services
  url: localhost:5004
simulation:
  # sql database which maps the machine parameter
  sim_db_pw: xxxxx
  sim_db_usr: xxxxx
  sim_db_url: localhost:3306
msg_bus:
  # message bus
  broker_urls: localhost:9094,localhost:9096Afterwards, set the path to the config yaml file in the environment value PIPE_CONFIG_FILEPATH.PIPE_CONFIG_FILEPATH=/path/to/ml-pipe-config.yamlTo run the Agent you you can activate the python env.pipenv shellAfterwards you can run the Agentpython agents/testing/silly_agent.pyTo interact with the pipeline (e.g trigger, start, list, stop Agents) you can used KafkaPipeClient.fromml_pipe_core.client.kafka_clientimportKafkaPipeClientpipe=KafkaPipeClient()print(pipe.get_running_service_info())p_id=pipe.run_service('SillyAgent')pipe.get_observation(p_id)# wait for responseAn GUI will be come soon.Developer: Core Services and ml_pipe_core PackageThe ML-Pipeline is using docker to run the infrastructure and test it locally.
For this you need docker and docker-compose >= 1.28.0.Build Docker imagesTo build all docker images run:makebuild-allThis can take a while...Notes: After changing code you just need to rebuild the service images, which is much faster.make build-service-imagesStart Core ServicesTo Start the core services of the ml-pipeline you need to set the following environment valuesPIPE_EXTERNAL_HOST_ADDR=localhost
PIPE_DB_PW=xxxx
PIPE_DB_USER=xxxx
PIPE_CONFIG_FILEPATH = /path/to/ml-pipe-config.yaml
PATH_TO_ML_PIPE_SERVICES_ROOT=/path/to/ml-pipe-servicesPIPE_CONFIG_FILEPATH is the path to the yaml config file, which look like this:observer:
  # used to check if jbb is done
  url: observer:5003
  event_db_pw: xxxx
  # event_db_url:
  event_db_usr: xxxx
register:
  # registers all services
  url: register:5004
simulation:
  # sql database which maps the machine parameter
  sim_db_pw: xxxx
  sim_db_usr: xxxx
  sim_db_url: simulation_database:3306
msg_bus:
  # message bus
  # external_host_addr: localhost
  broker_urls: kafka_1:9092,kafka_2:9096In production replace PIPE_EXTERNAL_HOST_ADDR=localhost with the server url and set PATH_TO_ML_PIPE_SERVICES_ROOT to the location of the cloned ml-pipe-services repository.
The environment values PIPE_DB_PW and PIPE_DB_USER define the credentials for the databases used by the pipeline.Stop Core ServicesYou have to stop the docker container at the end.makedownTests locallyNote: Docker and docker-compose => 2.80 is required to run tests locallyRun all tests:make tests ENV_FILE=.envRun end to end tests:make e2e-tests ENV_FILE=.envRun integration tests:make integration-tests ENV_FILE=.envRun unit testsmake unit-tests ENV_FILE=.envAdditional StuffMaxwellLog in via ssh to max-wgs.desy.deInstall python 3.8.8wget https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh
sh Anaconda3-2021.05-Linux-x86_64.sh
export PATH=$HOME/anaconda3/bin:$PATHPyTine and K2I2K_os on Machine PETRA IIIThe machine observer and controller for PETRA III are using the PetraAdapter which is using the libs PyTine and K2I2K_os.
The Path to this libs have to be added to the PYTHONPATH.For PyTine have a look athttps://confluence.desy.de/display/HLC/Developing+with+Python.K2I2K_os can be cloned via git from:gitclonehttps://username@stash.desy.de/scm/pihp/petra3.optics.tools.gitJupyter notebookTo use KafkaPipeClient in a Jupyter notebook you need to add the virtual environment to Jupyter.First activate the python virtual environment.For PipenvpipenvshellStart the jupyter notebook:pipinstall--useripykernelNote: You have to add the virtual enviroment to jupyter. First, activate the virutal environment. Then run:python-mipykernelinstall--user--name=<myenv>