# World-Models for Bitrate Streaming
## 2 Background
### Reinforcement Learning  
Reinforcement learning is a decision making paradigm. In reinforcement learning, an agent takes actions in an environment to optimize a cumulative reward. Reinforcement learning problems can be stated as a Markov Decision Process, being formally defined with a four-tuple:  
* \(\mathcal{S}\) is the set of possible states;* \(\mathcal{A}\) is the set of possible actions available to an agent;
* \(\mathcal{P}_{s_{t},a_{t},s_{t+1}}\) is the transition kernel for states and actions, the probability that an action \(a\) at state \(s\) leads to state \(s^{\prime}\); and,
* \(\mathcal{R}_{s_{t},a_{t},s_{t+1}}\) is the reward function, the immediate reward from transitioning to state \(s^{\prime}\) from \(s\) due to action \(a\).  
A policy \(\pi\), produces an action \(a\) given current state \(s\): \(\pi(s)\to a\). The objective of a reinforcement learning algorithm is to learn a policy that maximizes the expected return in an environment.  
\[\pi\quad\mathbf{E}[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t+1}))]\]  
Here, \(R(s_{t},a_{t},s_{t+1})\) is the reward given by the policy \(\pi\) selecting action \(a\) at time \(t\) and \(\gamma\in[0,1]\) is the discount factor that determines the importance of future rewards. For \(\gamma=1\), if the environment does not contain a terminal state, or the agent never reaches one, rewards can become infinite. Typically, reinforcement learning algorithms are evaluated in their performance across episodes, sequences of actions that end with a terminal state. Initially, agents do not have any knowledge of environments; agents must balance between the exploration of unseen states and exploitation of current knowledge to find an optimal policy.  
Partially Observable Markov Decision Processes (POMDPs) are MDPs where the state space may be noisy or not fully observable. Reinforcement learning is concerned with learning a policy when the environment model is not known in advance. There are two broad classes of reinforcement learning algorithms: model-free and model-based [3]. Generally, model-free methods learn a policy without learning a model of environment dynamics, and model-based methods learn a model of environment dynamics and use the learned model for deriving a controller. In practice, there are no distinct boundaries between these two classes, and some algorithms blur the lines between the categories [13]. Classical reinforcement learning methods can be untenable on large state and action spaces; the state space may be too large for the agent to visit exhaustively, and there are large memory overheads for dynamic programming methods. Deep reinforcement learning has been used in order to achieve state-of-the-art results on tasks that were previously computationally prohibitive, such as Atari games [14] and Go, a complex board game [15]. Agents that are trained with deep reinforcement learning can learn complex functions, handle unstructured environment data, and predict quality actions in states that have never been visited. There are a wide variety of reinforcement learning algorithms with different use cases and advantages, such as parallelization or properly handling discrete action spaces. Actor-critic methods [16] have achieved recent success on Atari games, StarCraft [17], and visual navigation [18], among others. In actor-critic algorithms, a critic network estimates a value-function, and an actor network updates a policy-network that is based on the estimations from the critic [19]. This work uses the A2C algorithm, a synchronous version of the A3C algorithm. For brevity, this work omits the mathematical details of the A2C algorithm, and it refers the reader to the original A3C paper [16] for more information.