Package: alpyperl
ALPypeRLALPypeRLorAnyLogic Python Pipe for Reinforcement Learningis an open source library for connectingAnyLogicsimulation models withreinforcement learningframeworks that are compatible withOpenAI Gymnasiuminterface (single agent).WithALPypeRLyou will be able to:Connect your AnyLogic model to a reinforcement learning framework of your choice (e.g. rayrllib).Scale your training by launching many AnyLogic models simultaneously (requires an exported model).Deploy and evaluate your trained policy from AnyLogic.Debug your AnyLogic models during training (this is a special feature unique to ALPypeRL that improves the user experience during model debugging remarkably).Identify and replicatefailed runsby having control on theseedused for each run.Leverage on the AnyLogic rich visualization while training or evaluating.There is a more comprehensivedocumentationavailable that includes numerous examples to help you understand the basic functionalities in greater detail.No licenceis required for single instance experiments.AnyLogic PLEis free!.NOTE: ALPypeRL has been developed usingray rllibas the base RL framework. Ray rllib is an industry leading open source package for Reinforcement Learning. Because of that, ALPypeRL has certain dependencies to it (e.g. trained policy deployment and evaluation).EnvironmentsALPypeRL includes 2 environments that make the connection betweenAnyLogicand yourpython scriptpossible:ALPypeRLConnector- The AnyLogic connector ('agent') library to be dropped into your simulation model.alpyperl- The library that you will use after configuring yourpolicyin yourpython scriptto connect to theAnyLogicmodel (includes functionalities totrainandevaluate).InstallationTo install the baseALPypeRLlibrary in python, usepip install alpyperl.To useALPypeRLConnectorinAnyLogic, you can add thelibraryto yourPalette. That will allow you to drag and drop the connector into your model.Note that furtherinstructionsare required to be followed in order for the connector to work.RequirementsTheALPypeRLrequires you to have theAnyLogic software(or a valid exported model). AnyLogic is a licensed software for building simulations that includes an ample variety of libraries for modelling many industry challenges. At the moment, AnyLogic provides afreelicense under the name PLE (Personal Learning Edition). There are other options available. For more information, you can visit theAnyLogic website.Note: This is not a package that is currently backed by the AnyLogic support team.The python packagealpyperlrequires (among others) 4 packages that are relatively heavy (and might take longer times to install depending on the host machine specs):rayray[rllib]tensorflowtorchAPI basicsTrainingTo be able to train your policy, you must have the following:AnAnyLogic modelthat requires decisions to be taken as the simulation runs. Using theCartPole-v0example, a decision must be taken on the direction of the force to be applied so the pole can be kept straight for as long as possible. For that, the AnyLogic model will be making requests to theALPypeRLConnectorand consuming the returned/suggested action.Apython scriptthat contains the RL framework. Here is where the policy is going to be trained. For that, you will need to create yourcustom environmenttaking into consideration what your AnyLogic model expects to return and receive. By default, you must define theactionandobservationspaces. Please visit theCartPole-v0example for a more detailed explanation.fromalpyperlimportAnyLogicEnvfromray.rllib.algorithms.ppoimportPPOConfig# Set checkpoint directory.checkpoint_dir="./resources/trained_policies/cartpole_v0"# Initialize and configure policy using `rllib`.policy=(PPOConfig().rollouts(num_rollout_workers=2,num_envs_per_worker=2).fault_tolerance(recreate_failed_workers=True,num_consecutive_worker_failures_tolerance=3).environment(AnyLogicEnv,env_config={'run_exported_model':True,'exported_model_loc':'./resources/exported_models/cartpole_v0','show_terminals':False,'verbose':False,'checkpoint_dir':checkpoint_dir,'env_params':{'cartMass':1.0,'poleMass':0.1,'poleLength':0.5,}}).build())# Perform training.for_inrange(10):result=policy.train()# Save policy checkpoint.policy.save(checkpoint_dir)print(f"Checkpoint saved in directory '{checkpoint_dir}'")# Close all enviornments.# NOTE: This is required to be called for correct checkpoint saving by ALPypeRL.policy.stop()EvaluationThe evaluation of yourtrained policyis made simple inalpyperl. See the example:fromalpyperl.serve.rllibimportlaunch_policy_serverfromalpyperlimportAnyLogicEnvfromray.rllib.algorithms.ppoimportPPOConfig# Load policy and launch server.launch_policy_server(policy_config=PPOConfig(),env=AnyLogicEnv,trained_policy_loc='./resources/trained_policies/cartpole_v0',port=3000)Once the server is on, you can run your AnyLogic model and test your trained policy. You are expected to select modeEVALUATEand specify the serverurl.Bugs and/or development roadmapAt the moment, ALPypeRL is at its earliest stage. You can join thealpyperl projectand raise bugs, feature requests or submit code enhancements via pull request.Support ALPypeRL's developmentIf you are financially able to do so and would like to support the development ofALPypeRL, please reach out tomarcescandellmari@gmail.com.LicenseThe ALPypeRL software suite is licensed under the terms of the Apache License 2.0. SeeLICENSEfor more information.