Package: deepcoord
DeepCoord: Self-Learning Multi-Objective Service Coordination Using Deep Reinforcement LearningUsing deep reinforcement learning (with DDPG), for online service coordinating, including scaling and placement of services and scheduling of rapidly incoming flows.
Services consist of chained components that need to be instantiated at nodes in the substrate network and that incoming flows need to traverse in a predefined order.
Our approach learns how to do this by itself just from experience, optimizing individual objectives (e.g., flow success rate) or multiple,
even competing objectives (e.g., throughput, QoS, energy, costs).
It works with realistically available monitoring information, containing partial and delayed observations of the full network state.CitationIf you use this code, please cite ourconference paper(Best Student Paper at IEEE/IFIP CNSM 2020) andjournal paper:@inproceedings{schneider2020selfdriving,
	title={Self-Driving Network and Service Coordination Using Deep Reinforcement Learning},
	author={Schneider, Stefan and Manzoor, Adnan and Qarawlus, Haydar and Schellenberg, Rafael and Karl, Holger and Khalili, Ramin and Hecker, Artur},
	booktitle={International Conference on Network and Service Management (CNSM)},
	year={2020},
	publisher={IFIP/IEEE}
}

@article{schneider2021selflearning,
	title={Self-Learning Multi-Objective Service Coordination Using Deep Reinforcement Learning},
	author={Schneider, Stefan and Khalili, Ramin and Manzoor, Adnan and Qarawlus, Haydar and Schellenberg, Rafael and Karl, Holger and Hecker, Artur},
	journal={Transactions on Network and Service Management (TNSM)},
	volume={18},
	number={3},
	year={2021},
	publisher={IEEE}
}SetupRecommended for development: Clone and installcoord-simandcommon-utilslocally first in the same venv before running the installation of the RL agent.You need to havePython 3.6 or 3.7andvenvmodule installed.
The installation is tested and works on Ubuntu 16.04 and 20.04 withPython 3.6andcoord-sim v2.1.0.
It does not with Python 3.8 becausetensorboard 1.14.0, which is a required dependency, is not available for Python 3.8.Create a venvOn your local machine:# check versionpython3--version# if not 3.6 or 3.7, install python 3.6sudoaptupdate
sudoadd-apt-repositoryppa:deadsnakes/ppa
sudoaptinstallpython3.6python3.6-devpython3.6-venv# create venv oncepython3.6-mvenv./venv# activate the venv (always)sourcevenv/bin/activate# update setuptoolspipinstall-UsetuptoolsInstall dependencies# from within the repo directorypipinstall-rrequirements.txtThis also installs the requiredcoord-simandcommon-utilspackage
if they were not installed manually before.Use the RL agentAll options:$deepcoord-h
Usage:deepcoord[OPTIONS]AGENT_CONFIGNETWORKSERVICESIM_CONFIGSTEPSdeepcoordcliforlearningandtesting

Options:--seedINTEGERSpecifytherandomseedfortheenvironmentandthelearningagent.-t,--testTEXTNameofthetrainingrunwhoseweightsshouldbeusedfortesting.-w,--weightsTEXTContinuetrainingwiththespecifiedweights(similartotesting)-a,--append-testAppendatestrunofthepreviouslytrainedagent.-v,--verboseSetconsoleloggerleveltodebug.(DefaultisINFO)-b,--bestTestthebestofthetrainedagentssofar.-e,--test-episodesINTEGERSetthenumberoftestingepisodes-ss,--sim-seedINTEGERSetthesimulatorseed-gs,--gen-scenarioPATHDiff.simconfigfileforadditionalscenariotest-h,--helpShowthismessageandexit.Ignore potentialtensorflowwarnings.Training and testingExample for short training then testing:deepcoordres/config/agent/sample_agent.yamlres/networks/sample_network.graphmlres/service_functions/abc.yamlres/config/simulator/sample_config.yaml10--append-testResults are stored underresults/according to the input arguments and the current time stamp.
There, you'll find copies of the used inputs, the trained weights, logs, and all result files of any test runs that you performed with these weights.TestingTo run another test run with the trained weights, specify the<timestamp_seed>of the training run. For testing, it is recommended to use 200 steps as it is the duration of one episode and use-eto specify the number of testing episodes.
For example:deepcoordres/config/agent/sample_agent.yamlres/networks/sample_network.graphmlres/service_functions/abc.yamlres/config/simulator/sample_config.yaml200-t<timestamp_seed>-e1Testing with a different simulator configuration (Generalization)To train an agent and test it on multiple scenarios (simulator configurations), use the-gsto specify a different simulator config file to test in combination with--append-test.Example for testing with generalization:deepcoordres/config/agent/sample_agent.yamlres/networks/sample_network.graphmlres/service_functions/abc.yamlres/config/simulator/sample_config.yaml1000--append-test-gsres/config/simulator/sample_config.yamlLearning Curves using TensorboardTo view the learning curve of all agents, i.e., the episode reward over time, usetensorboard:tensorboard--logdir==./graphYou can also filter to only show curves of a specific agent config, network (and service and config) by setting the--logdircorrespondingly:tensorboard--logdir==./graph/<agent_config>/<network>/<service>/<simulator_config>Visualizing/Analyzing ResultsTo get a better understanding of what the agent is doing, there is an Juypter notebookeval_example.ipynb.
It's just an example; you won't be able to run it without all the results (which are too large for the repo).To create a similar notebook for evaluation:# first time installationpipinstall-reval_requirements.txt# run jupyter serverjupyterlabNote:If you're running on the server, you should start the Jupyter server in a screen with the following command:jupyternotebook--ip0.0.0.0--no-browserYou can then access it over the server's URL at port 8888 at the/labendpoint. For authentication, copy and paste the token that is displayed whed starting the Jupyter server.Additionally, thecoord-simsimulator provides the option to generate animations of the learned policy:
Seecoord-sim Readme.Training and testing on multiple scenariosThere is script provided in thescriptsfolder that utilizes theGNU Parallelutility to run multiple agents at the same time to speed up the training process../scripts/run_parallel.shTo run long training sessions in remote environments without risking to stop the sessions due to possible connectivity issues, it is recommended to run the experiments with thescreenlinux tool.For that, start a newscreenwithscreen -S rl-parallel.Configure the agent and the network, service, config files in thescriptsdirectory to match the scenarios that you want to run. Here, the lines of network, service, config are matched by lines (not all permutations), eg, 1. network is matched with 1. service and 1. config. Then all seeds are used for all scenarios.Inside the screen, with the venv activated, run./scripts/run_parallel.shfrom the project root.AcknowledgementThis project has received funding from German Federal Ministry of Education and Research (BMBF) through Software Campus grant 01IS17046 (RealVNF).