Package: hnp
Data center temperature control is a critical process for maintaining high quality of service. While maintaining temperatures in appropriate temperature bands is paramount, it is also important to consider that every heating or cooling strategy has associated environmental and economic costs. For example, a cooling strategy that makes excessive or unnecessary use of air conditioning will consume more energy than a strategy that better leverages free cooling. As long as operational constraints are satisfied, opportunities to discover temperature control solutions that minimize energy utilization are highly valuable. Co-developed by theVector InstituteandTELUS, this repository provides a solution for training a reinforcement learning (RL) agent to perform temperature control in a room. Clickherefor detailed documentation.FeaturesInnovative discretization method: The provided solution leverages an innovation in state space design proposed by TELUS calledHyperspace Neighbour Penetration(HNP), which allows slowly-changing continuous variables to be approximated as discrete variables.Support for building control environments: The HNP package has built-in support for Sinergym - a building control simulation environment, as well as Beobench - a toolkit providing unified access to building control environments.Ready-to-use RL agents: A HNP-enabled Q-Learning agent and two baseline agents are provided in the package.HNPA key assumption in HNP is that continuous state spaces with very incremental, locally linear transitions can be effectively discretized into relatively coarse ranges or bands of values astiles. When this type of coarse discretization is applied to a slowly-changing variable (e.g. temperature), it can lead to situations where an action (e.g. changing the setpoints) results in such a small change to the subsequent state observation that no discrete state transition has actually occurred. A naïve solution could be to increase the granularity of the state space, i.e. to consider much smaller changes in temperature as distinct elements of the state space, but it is computationally expensive or impossible to establish an extremely granular grid system. Alternatively, HNP computes multiple values from tileboundariesand then aggregates them using a weighted norm. This enables state-action pairs to result insteps towardsother states and their corresponding values.HNP is fully described in itsfoundational paper.Supported EnvironmentsSinergymSinergymis a building control environment that follows OpenAI Gym interface and uses EnergyPlus simulator. To use Sinergym, see detailed instruction on how to installhere.BeobenchBeobenchis a toolkit providing unified access to building control environments for RL (Sinergym also supported). It uses docker to manage all environment dependencies in the background. See detailed instruction on how to use Beobenchhere.Supported AgentsThe HNP package provides the following agents:Random Action Agent: An agent that takes a random actionFixed Action Agent: An agent that always take a pre-defined actionHNP-enabled Q-Learning Agent: A Q-learning agent with built-in HNP that allows different types of observation variablesQuickstartRequirementsPython ≥ 3.9Sinergym == 2.2.0InstallationTo installhnpfromPyPI:pip install hnpExample UsageThis is a minimalist example of using the HNP Q-Learning agent in Sinergymimportnumpyasnpfromhnp.agentsimportQLearningAgentfromhnp.environmentimportObservationWrapper,create_envconfig={"agent":{"num_episodes":100,"horizon":24,"gamma":0.99,"num_tiles":20,"initial_epsilon":1,"epsilon_annealing":0.999,"learning_rate":0.1,"learning_rate_annealing":0.999},"env":{"name":"Eplus-5Zone-hot-discrete-v1","normalize":True,"obs_to_keep":[0,1,8,10],"mask":[0,0,0,0]}}obs_to_keep=np.array(config["env"]["obs_to_keep"])mask=np.array(config["env"]["mask"])env=create_env(config["env"])env=ObservationWrapper(env,obs_to_keep)agent=QLearningAgent(env,config["agent"]["params"],mask,)agent.train()agent.save_results()env.close()DocumentationDetailed package documentation:HNP DocsLicenseThe code in this repository is published under 3-Clause BSD license (seeLICENSEfile).