Package: crete
Crete_       
  ___ _ __ ___| |_ ___ 
 / __| '__/ _ \ __/ _ \
| (__| | |  __/ ||  __/
 \___|_|  \___|\__\___|
Reinforcement Learning CLI assistant.InstallationCrete is now officially on PyPI!pip install creteIntroductionCrete is a CLI for RL training.
Named after the island that Talos, the first AI in human mythology, walked upon.It can be used as easily as:python -m crete --helpRegistering Agents, Environments and WrappersTo start using Crete, you must register (at least) one agent.
Crete already comes preloaded with all the OpenAI Gym(nasium) environments.
You can register your own environments too, as long as they use the OpenAI Gym(nasium) API.
More information on making enivironment can be found on
theGym(nasium) website.Agents must inherit thecrete.Agentclass. For example;cool_rl_project
 +- crete_profile.yaml  <- config file, more on this later.
 +- module_a 
 |  +- agents
 |  |  +- __init__.py
 |  |  +- agent_1.py
 |  +- etc...# module_a/agent_1.pyfromcreteimportAgent,ExtraState,ProfileConfigclassDQNAgent(Agent):def__init__(self,obs,n_actions,device='cpu')->None:super().__init__("DQN")...defget_action(self,obs,extra_state:ExtraState=None)->Tuple[int,ExtraState]:"""Request an action."""...defsave(self)->bytes:"""Extract all policy data."""...defload(self,agent_data:bytes):"""Load policy data."""...defdqn_training_wrapper(env_factory:Callable[[int],gym.Env],# Function that creates the environment.agent:DQNAgent,# The agent itself.config:ProfileConfig,# Object with config values.artifacts:Dict,# Empty map for storing training information.save_callback# Function for saving agents, for instance at peak rewards.):"""The DQN training procedure."""# Note that this function is outside the agent definition....And then register that agent in the module's initialisation file.
There is alsoregister_envandregister_wrapperwhich work the same way.
Environments and wrappers should use the OpenAI Gym(nasium) API.# module_a/__init__.pyfromcreteimportregister_agentfrom.dqn_agentimportDQNAgent,dqn_training_wrapperregister_agent(agent_id="DQN",agent_factory=lambdaobs,n_actions,device:DQNAgent(obs,n_actions,device=device),training_wrapper=dqn_training_wrapper)Then, register that module with crete (this will create a.crete.yamlfile);python -m crete module add module_a.agentsVerify the module has been loaded correctly by running the following, which should list DQN.python -m crete list agentsTraining AgentsPreviously we saw the training wrapper for the example DQN agent. It contained 5 parameters;env_factory: Callable[[int], gym.Env]. This is the function that creates the environment.A factory is given instead of the environment itself to allow fresh environments to be created for evaluation.agent: Agent. The agent itself.config: ProfileConfig. The configuration profile containing agent and training parameters.artifacts: Dict. An empty dictionary for storing training artifacts.Training artifacts are things like loss over time. These artifacts are saved in the concrete file.save_callback: SaveCallback. Utility function for saving snapshots of the agent.Used likesave_callback(agent, artifacts, step_iteration, "autosave-name")Agents can be trained using thetrainorbatchcommand, which trains one or several 'profiles' respectively.
A 'profile' is contained within a.yamlfile, and may look like the following;---# crete_profile.yamldefaults:# Contains default `config` values that can be overridden.batch_size:32init_epsilon:1final_epsilon:0.1eval_freq:1000gather_freq:50final_dqn_map_0:# Arbitrary profile ID, can be anything.agent_id:DQN# The ID of the registered agent to use.env_id:CartPole-v1# The ID of the environment to use.env_args:# Arbitrary environment arguments that will be passed to the constructorrender_mode:humanconfig:# Config values that can be accessed through the ProfileConfig object via the training wrapper.total_steps:40000decay_steps:38000replay_buffer_size:10000batch_size:128refresh_target_network_freq:1000hidden_layers:-64-64Training is engaged by;python -m crete train crete_profile.yaml final_dqn_map_0 ; to train a specific profile or,
python -m crete batch crete_profile.yaml                 ; to train all profiles within a file.Saving Agents and their PerformancesCrete allows the entire training process and outcomes to be saved, for replaying, evaluating and comparing.
All that is required is for the agent to implement thesaveandloadmethods, which serialise the agents weight data
to bytes.
Everything else is handled by Crete.
One would usually usepickleto achieve this.For example, with a PyTorch-based agent:defsave(self)->bytes:data={"data":self.net.state_dict(),"layers":self.net.hidden_layers}returnpickle.dumps(data)defload(self,agent_data:bytes):agent_dict=pickle.loads(agent_data)data,layers=itemgetter("data","layers")(agent_dict)self.net.set_hidden_layers(layers)self.net.load_state_dict(data)Crete automatically saves the agent on completion of the training wrapper, or on keyboard interrupt.