# World-Models for Bitrate Streaming
## 3 World-Models
### MDN-RNN World-Model  
[PERSON] & [PERSON] [5] used the MDN-RNN to create a hallucinogenic world-model in reinforcement learning environments, achieving state-of-the-art results on the Car Racing and VizDoom tasks. Their work is the primary inspiration for the work in this paper. The Car Racing and VizDoom tasks are stochastic, and a stochastic input process, the network bandwidth, partially determine the bitrate streaming environment. The MDN-RNN can handle random discrete events, such as a monster shooting a fireball in VizDoom. While sampling in MDN-RNN is probabilistic, it can still form approximations for deterministic properties. The MDN-RNN can fit naturally to bitrate streaming, as bitrate streaming contains discrete events, such as bandwidth dropping while passing through a tunnel. [PERSON] & [PERSON] [5] first used a variational autoencoder to compress the state space \(s\) into a latent vector \(z\), and learned a world model with this compressed version of the state space. The authors train an MDN-RNN to model \(P(z_{t+1}\mid z_{t},a_{t},h_{t})\), where \(a_{t}\) is the action take at time \(t\), \(z_{t}\) is the state of the environment that is encoded as a latent vector at time \(t\), and \(h_{t}\) is the hidden state of the MDN-RNN  
Figure 2: Recurrent Neural Networks allow for the network to retain information over time. Figure adapted from [26]. at time \(t\). The world model is trained using recorded episodes in the teacher forcing method, where the inputs to the MDN-RNN model in training only consist of the ground truth at each step. Sampling only occurs during the training of the controller. For inference, the MDN-RNN uses a softmax layer on the output \(\pi\)'s to form a categorical distribution. The softmax function, \(\text{Softmax}(x_{i})=\frac{\exp(x_{i})}{\sum_{j}\exp(x_{i})}\), converts a vector of numbers into a probability distribution, where each component is in the interval (0,1) and the entire vector sums to 1. A sample drawn from this distribution determines which Gaussian distribution pair, \((\mu_{i},\sigma_{i})\), is used to sample the actual output. [PERSON] & [PERSON] [5] additionally incorporate a temperature hyperparameter \(\tau\), which is used to adjust the randomness during sampling. The logits, components of the \(\pi\) vector before the application of the softmax function, are divided by the temperature parameter before being scaled into probabilities. \(\tau\) is typically set between 0 and 1; however, larger values are used in practice. When \(\tau\) trends to zero, the output of the MDN-RNN becomes deterministic, and samples only consist of the most likely point in the probability density function. Lower temperature values make sampling more conservative, and the unlikely candidates become even more unlikely to be sampled. Conversely, higher temperature values make unlikely candidates more likely to be sampled, which results in more stochastic predictions. During sampling, increasing the temperature parameter beyond one makes it possible to add more uncertainty and stochasticity into the dream environment. A controller trained in a hallucinogenic world-model with more stochasticity may be trained in a more difficult version of the real environment. [PERSON] & [PERSON] [5] found that increasing \(\tau\) helps to prevent a controller from taking advantage of the imperfections in the world-model, and it can guide the controller to more conservative policies when deployed to the real environment. While the VizDoom and Car Racing environments have complex visual inputs, [PERSON] & [PERSON] did not model reward functions in their experiments. For the hallucinogenic VizDoom world-model, the reward is simply incremented by a constant, 1, at each time step. Most systems problems do not contain reward functions that are this trivial. [PERSON] & [PERSON] [5] combined an incremental reward function and MDN-RNN state predictor to form a fully hallucinogenic world model. A controller receives current state \(s\) and takes an action \(a\).