Question: What is the definition of macro-scale?
Evidence: There have been many recent successes in the field of Reinforcement Learning. In the online RL setting, an agent takes actions, observes the outcome from the environment, and updates its policy based on the outcome. This repeated access to the environment is not feasible in practical applications; it may be unsafe to interact with the actual environment, and a high-fidelity simulator may be costly to build. Instead, offline RL, consumes fixed training data which consist of recorded interactions between one (or more) agent(s) and the environment to train a policy). An agent with the trained policy is then deployed in the environment without further evaluation or modification. Notice that in offline RL, the deployed agent must consume data in the same format (for example,having the same features) as in the training data. This is a crippling restriction in many large-scale applications, where, due to some combination of resource/system constraints, all of the features used for training cannot be observed (or misspecified) by the agent during online operation. In this work, we lay the foundations for studying this Resource-Constrained setting for offline RL. We then provide an algorithm that improves performance by transferring information from the full-featured offline training set to the deployed agent's policy acting on limited features. We first illustrate a few practical cases where resource-constrained settings emerge.
System Latency A deployed agent is often constrained by how much time it has to process the state of the environment and make a decision. For example, in a customer-facing web application, the customer will start to lose interest within a fraction of a second. Given this constraint, the agent may not be able to fully process more than a few measurements from the customer before making a decision. This is in contrast to the process of recording the training data for offline RL, where one Power Constraints Consider a situation where an RL agent is being used in deep space probes or nano-satellites). In this case an RL agent is trained on Earth with rich features and a large amount of sensory information. But when the agent is deployed and being used on these probes, the number of sensors is limited by power and space constraints. Similarly, consider a robot deployed in a real world environment. The limited compute power of the robot prevents it from using powerful feature extractors while making a decision. However, such powerful feature extractors can be used during the offline training of the robot (Fig).
In the resource-constrained setting, one can simply ignore the offline features and only train the offline agent with the online features that are available during deployment. This strategy has the drawback of not utilizing all of the information available during training and can lead to a sub-optimal policy. To confirm this, we performed the following simple experiment. We consider an offline RL dataset for the OpenAI gym MuJoCo HalfCheetah-v2 environment and simulate the resourceconstrained setting by removing a fixed set of randomly selected features during deployment (see Sections 5.1.1, C.1 for more details). We train an offline RL algorithm, TD3+BC using only the online features and collect online data in the environment using the trained policy. We repeat this assuming all features available during deployment, train a TD3+BC agent using the same offline dataset with all features, and collect online data in the environment. We plot the histogram of rewards in the two datasets in Fig 1b We observe that the agent trained only with online features obtains much smaller reward than the agent trained with offline features.
Traditionally, scenarios where the observability of the state of the system is limited are studied under the Partially Observable Markov Decision Process (POMDP) setting by assuming a belief over the observations. In contrast, we have an offline dataset (which records rich but not necessarily full state transitions) along with partially obscured (with respect to the offline dataset) observations online. Our goal is to leverage the offline dataset to reduce the performance gap caused by the introduction of resource constraints. Towards this, we advocate using a teacherstudent transfer algorithm. Our main contributions are summarized below:
• We identify a key challenge in offline RL: in the resource-constrained setting, datasets with rich features cannot be effectively utilized when only a limited number of features are observable during online operation.
• We propose the transfer approach that trains an agent to efficiently leverage the offline dataset while only observing the limited features during deployment.
• We evaluate our approach on a diverse set of tasks showing the applicability of the transfer algorithm. We also highlight that when the behavior policy used by the data-collecting agent is trained using a limited number of features, the quality of the dataset suffers. We propose a data collection procedure (RC-D4RL) to simulate this effect.
There have been many recent successes in the field of Reinforcement Learning. In the online RL setting, an agent takes actions, observes the outcome from the environment, and updates its policy based on the outcome. This repeated access to the environment is not feasible in practical applications; it may be unsafe to interact with the actual environment, and a high-fidelity simulator may be costly to build. Instead, offline RL, consumes fixed training data which consist of recorded interactions between one (or more) agent(s) and the environment to train a policy). An agent with the trained policy is then deployed in the environment without further evaluation or modification. Notice that in offline RL, the deployed agent must consume data in the same format (for example,having the same features) as in the training data. This is a crippling restriction in many large-scale applications, where, due to some combination of resource/system constraints, all of the features used for training cannot be observed (or misspecified) by the agent during online operation. In this work, we lay the foundations for studying this Resource-Constrained setting for offline RL. We then provide an algorithm that improves performance by transferring information from the full-featured offline training set to the deployed agent's policy acting on limited features. We first illustrate a few practical cases where resource-constrained settings emerge.
System Latency A deployed agent is often constrained by how much time it has to process the state of the environment and make a decision. For example, in a customer-facing web application, the customer will start to lose interest within a fraction of a second. Given this constraint, the agent may not be able to fully process more than a few measurements from the customer before making a decision. This is in contrast to the process of recording the training data for offline RL, where one Power Constraints Consider a situation where an RL agent is being used in deep space probes or nano-satellites). In this case an RL agent is trained on Earth with rich features and a large amount of sensory information. But when the agent is deployed and being used on these probes, the number of sensors is limited by power and space constraints. Similarly, consider a robot deployed in a real world environment. The limited compute power of the robot prevents it from using powerful feature extractors while making a decision. However, such powerful feature extractors can be used during the offline training of the robot (Fig).
In the resource-constrained setting, one can simply ignore the offline features and only train the offline agent with the online features that are available during deployment. This strategy has the drawback of not utilizing all of the information available during training and can lead to a sub-optimal policy. To confirm this, we performed the following simple experiment. We consider an offline RL dataset for the OpenAI gym MuJoCo HalfCheetah-v2 environment and simulate the resourceconstrained setting by removing a fixed set of randomly selected features during deployment (see Sections 5.1.1, C.1 for more details). We train an offline RL algorithm, TD3+BC using only the online features and collect online data in the environment using the trained policy. We repeat this assuming all features available during deployment, train a TD3+BC agent using the same offline dataset with all features, and collect online data in the environment. We plot the histogram of rewards in the two datasets in Fig 1b We observe that the agent trained only with online features obtains much smaller reward than the agent trained with offline features.
Traditionally, scenarios where the observability of the state of the system is limited are studied under the Partially Observable Markov Decision Process (POMDP) setting by assuming a belief over the observations. In contrast, we have an offline dataset (which records rich but not necessarily full state transitions) along with partially obscured (with respect to the offline dataset) observations online. Our goal is to leverage the offline dataset to reduce the performance gap caused by the introduction of resource constraints. Towards this, we advocate using a teacherstudent transfer algorithm. Our main contributions are summarized below:
• We identify a key challenge in offline RL: in the resource-constrained setting, datasets with rich features cannot be effectively utilized when only a limited number of features are observable during online operation.
• We propose the transfer approach that trains an agent to efficiently leverage the offline dataset while only observing the limited features during deployment.
• We evaluate our approach on a diverse set of tasks showing the applicability of the transfer algorithm. We also highlight that when the behavior policy used by the data-collecting agent is trained using a limited number of features, the quality of the dataset suffers. We propose a data collection procedure (RC-D4RL) to simulate this effect.
In this work, we aim at exploring this accuracy versus time trade-off of anytime learning, not at the level of a single batch of data, but at the macroscale of the entire sequence of batches.****In this work, we aim at exploring this accuracy versus time trade-off of anytime learning, not at the level of a single batch of data, but at the macroscale of the entire sequence of batches. This is a setting which more closely mimics practical applications, that we call anytime learning at mascroscale (ALMA). In this learning setting, we assume that the time to train a model is negligible compared to the interval of time between two consecutive batches of data (and therefore we do not care about how quickly a learner adapts to a new batch), yet efficiency matters in the sense that for the same performance a predictor that uses less compute and memory is preferable. In summary, we are interested in a learner that i) yields high accuracy, ii) can make non-trivial predictions at any point in time while iii) limiting its computational and memory resources.
Answer: We used the term macro-scale to indicate the level of granularity of our analysis, which is dictated by the arrival time of mega-batches as opposed to individual data points (or mini-batches). In classical anytime learning, we would look at the anytime performance while learning from each mega-batch (if I stop training after N mini-batches, what error rate do I get?). Instead, we focus on the anytime performance at the stream level where ticks are given by mega-batches. By doing so, we assume negligible the time to learn from each mega-batch and only report the learning curve across mega-batches (If I stop training after M mega-batches, what is the error rate?), folding the cost of learning on each mega-batch in the computational cost.