Package: mc2-galileo
Galileo: MC2 Experimentation EnvironmentThis project allows users to define operational experiments for the MC2 (Mini Compute Cluster),
and interact with the experimentation environment during runtime.
Galileo consists of two major components a user can interact with:
the experiment controller shell and the galileo dashboard.
The experiment controller can spawn emulated clients on workers, and control the amount of load they generate.
Furthermore, a user can interact with Symmetry's routing table shell to control to which cluster node a service request
is sent to.BuildCreate and activate a new virtual environmentvirtualenv .venv
source .venv/bin/activateInstall requirementspip install -r requirements.txtIf you have symmetry locally in the parent folder, link it viapip install -e ../symmetryDockerTo create a Docker image that can run galileo components, runmake dockerTo create a arm32v7 Docker image that can run galileo components, runmake docker-armStart a worker withcd docker/galileo-worker
docker-compose upCompose files for arm32v7 are located indocker/armStart a local dev environment, including: API, ExperimentDaemon, 1 worker, redis and database:cd docker/dev
docker-compose upPreparing the Example ApplicationWe prepare the cluster with an example application. Specifically a image classification service.Run the mxnet-model-server as a Docker container named 'mms', exposed on port 8080.
For example, to start mxnet-model-server with models squeezenet and alexnet, run the following command on a cluster node:docker run -itd --name mms -p 8080:8080 -p 8081:8081 awsdeeplearningteam/mxnet-model-server:1.0.0-mxnet-cpu mxnet-model-server --start \
--models squeezenet=https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model,alexnet=https://s3.amazonaws.com/model-server/model_archive_1.0/alexnet.marPrepare the Experiment Worker HostsThe devices hosting the workers that generate load need to run the experiment controller host application.bin/run worker --logging=INFOAll runtime parameters are controlled viagalileo_*environment variables. Checkdocker/galileo-worker/worker.envfor some examples.Run the Experiment Controller Shell(.venv) pi@graviton:~/mc2/galileo $ bin/run shell
                                   __  __
 .-.,="``"=.          ____ _____ _/ (_) /__  ____
 '=/_       \        / __ `/ __ `/ / / / _ \/ __ \
  |  '=._    |      / /_/ / /_/ / / / /  __/ /_/ /
   \     `=./`.     \__, /\__,_/_/_/_/\___/\____/
    '=.__.=' `='   /____/


Welcome to the galileo shell!

Type `usage` to list available functions

galileo> usage
the galileo shell is an interactive python shell that provides the following commands

Commands:
  usage         show this message
  env           show environment variables
  pwd           show the current working directory

Functions:
  sleep         time.sleep wrapper

Objects:
  g             Galileo object that allows you to interact with the system
  show          Prints runtime information about the system to system out
  exp           Galileo experiment
  rtbl          Symmetry routing table

Type help(<function>) or help(<object>) to learn how to use the functions.Configure the routing tableThertblobject in the shell allows you to set load balancing policies. Runhelp(rtbl)in the galileo shell.
Here is an example of how to set a record for the servicemyservice.galileo> rtbl.set('myservice', ['host1:8080', 'host2:8080'], [1,2])
RoutingRecord(service='myservice', hosts=['host1:8080', 'host2:8080'], weights=[1, 2])
galileo> rtbl
+---------------------------+----------------------+-----------+
| Service                   |                Hosts |   Weights |
+---------------------------+----------------------+-----------+
| myservice                 |           host1:8080 |       1.0 |
|                           |           host2:8080 |       2.0 |
+---------------------------+----------------------+-----------+Run the Experiment DaemonFIXME: THIS IS OUTDATEDThe experiment daemon continuously reads from the blocking redis queuegalileo:experiments:instructions.
After receiving instructions, the controller will execute the commands and record the telemetry data
published via Redis. At the end the status of the experiment will be set to 'FINISHED' and the traces,
that were saved in the db by the clients, will be updated to reference the experiment.Example Redis command to inject a new experiment (whereexphostis the hostname of the experiment host):LPUSH galileo:experiments:instructions '{"instructions": "spawn exphost squeezenet 1\nsleep 2\nrps exphost squeezenet 1\nsleep 5\nrps exphost squeezenet 0\nsleep 2\nclose exphost squeezenet"}'you can also specifyexp_id,creator, andname, otherwise some generated/standard values will be used.You can change the database used to store the experiment data via the envgalileo_expdb_driver(sqliteormysql).
To write the changes into MySQL (or MariaDB), set the following environment variables:galileo_expdb_mysql_host,galileo_expdb_mysql_port,galileo_expdb_mysql_db,galileo_expdb_mysql_user,galileo_expdb_mysql_passwordYou can create a mariadb docker instance with:./bin/run-db.shThen execute the daemon with:python -m galileo.cli.experimentdOr run the script, which exports the mariadb setup from the docker container (add--logging DEBUGfor output)./bin/experimentd-mysql.shRun the Galileo REST APIServe the app with gunicorngunicorn -w 4 --preload -b 0.0.0.0:5001 \
    -c galileo/webapp/gunicorn.conf.py \
    galileo.webapp.wsgi:api