Package: pyloa
PyLoa - Learning on-line Algorithms with Pythonpyloais a research repository for analyzing the performance of classic on-line algorithms vs. modern Machine
Learning, specifically Reinforcement Learning, approaches. PyLoa ships with an implementation of two commonly known
on-line problems asenvironments:(k,n)-paging-problemwith acache_size kandn pagesfor a sequence of page-requests(k,n)-coloring-problemwithk colorsfor a graph withn verticesPyLoa allows foragentsto betrained on suchenviroments(problem definitions) that require on-line solutions,evaluated against commonly used heuristics or any state-of-the-art algorithm,exploited (extrapolation of apotentiallyworst case problem instances) to determine a solution'scompetitve ratio.Dependenciespyloais developed for Python 3.5+ and has the following package dependencies:matplotlib==3.0.3scipy==1.2.1tensorflow==1.13.1tqdm==4.31.1numpy==1.16.2InstallationWe recommend usingpyloawithin avirtual environment:mkdir myproject
cd myproject
python3 -m venv virtualenv/
source virtualenv/bin/activateUpdatepipandsetuptoolsbefore continuing:pip install --upgrade pip setuptoolsAfterwards you can installpyloaeither from its latestPyPI stablereleasepip install pyloaorfrom its latestdevelopment releaseon GitHubpip install git+https://github.com/pyloa/PyLoa.gitGeneral Usagepyloacan be used in three different ways to analyze an on-line problem; each depicted via a so called runmode
(train,eval,gen). Any runemode can be invoked via its positional argument and requires a python-configuration-file.pyloa {train,gen,eval} --config path/to/hyperparams.pyhyperparams depicts the setting of the experiment at hand; it must hold a dictionary namedparams, which moreovermust
containdictionaries for the keysinstance,environmentandagent.params["ìnstance"]: Must define a configuration of a subclass implementation ofpyloa.instance.InstanceGenerator,
which generates problem instances for the domain. As an example, for the(k,n)-paging-problema simple generator could
randomly generate a sequence of requests of lengthsequence_size, whereas each request is within [1, n].params["agent"]: Must define a configuration of a subclass implementation ofpyloa.agent.Agent,
which observes a statesof its environment, acts with actionaaccordingly, receives rewardrand observes
transitioned states'. For toy problem instances a simple Q-learning table implementation would suffice.params["environment"]: Must define a configuration of a subclass implementation ofpyloa.environment.Environment,
which consumes a problem instance and let's the agentplayuntil it terminates. Anenvironmentconstitutes as a problem
definition.A minimal example for learning the(5,6)-paging-problemwith aQTableAgenton aPagingEnvironmentcan be invoked withpyloa train --config hyperparams.pyand the hyperparams.py as following:frompyloa.instanceimportRandomSequenceGeneratorfrompyloa.environmentimportDefaultPagingEnvironmentfrompyloa.agentimportQTableAgent# varssequence_size=1000max_page=6min_page=1episodes=250# hyperparamsparams={'checkpoint_step':episodes//10,'instance':{'type':RandomSequenceGenerator,'sequence_size':sequence_size,'sequence_number':episodes,'min_page':min_page,'max_page':max_page,},'environment':{'type':DefaultPagingEnvironment,'sequence_size':sequence_size,'cache_size':5,'num_pages':max_page-min_page+1,},'agent':{'type':QTableAgent,'discount_factor':0.55,'learning_rate':0.001,'epsilon':0.0,'epsilon_delta':13/(episodes*10),'epsilon_max':0.99,'save_file':"/home/me/models/",},}This example is defined inexamples/0_train_qtable_paging/hyperparams.pyand can be run withpyloa train --config examples/0_train_qtable_paging/hyperparams.pyThe resulting run can be seenhere. In total there are five toy examples, which can be run on any system,
defined in the examples directory.RunmodesPyLoa has three different runmodes:train,evalandgen. There are slight adaptions to be made for the configuration file
depending on the selected runmode; we encourage checking the examples for reference (on a site note: hyperparams are loaded and validated
inpyloa.utils.load). Semantically the three different runmodes stand for:train: AnRLAgentwill be trained forepisode-many instances, generated by anInstanceGenerator, on hisenvironment.
Everycheckpoint_step-many instances a checkpoint ofRLAgentwill be saved.eval: All trainedRLagentsnested withinroot_dirwill be evaluated onepisode-many instances, generated by anInstanceGenerator.
Additionally non-trainable agents may be defined and evaluated alongside.gen: Currentlyonly applicablefor the(k,n)-paging-problem. A genetic algorithm empirically determines aPagingAgent's
(approximate) competitive ratio.Each runmode will create TFEvent-files for TensorBoard in its experiment's output directory.