# World-Models for Bitrate Streaming
## 5 Results and Discussion
### Training in a Dream  
A script collects traces from 10,000 episodes (five million total actions) episodes using a random agent to construct the world-model; all of the observations are normalized using min-max scaling from the bounds of the Park environment. This project uses a version of the MDN-RNN from [PERSON] [5], which was modified to operate in the native bitrate streaming state space, without an autoencoder. The MDN-RNN is wrapped by a class to advance the state of the world-model, and compute the reward function, as described above. The wrapped world-model implements the OpenAI Gym interface; any controller compatible with this interface can train inside the hallucinogenic world-model. The following section contains the results from training both the MDN-RNN and training a controller inside the hallucinogenic world-model. Appendix A lists the parameters used in these experiments, and they were the same parameters used by [PERSON] [5], unless otherwise noted. This project trained the MDN-RNNs using a smaller hidden state size to reduce possibilities of overfitting because the state space for bitrate streaming contains 12 values. Figure 5 shows the log-likelihood loss during training for an MDN-RNN with 64 hidden units and five Gaussians. We trained the established baseline agent inside of the fully hallucinogenic world-model. We performed a sweep on the temperature values in order to determine how increased difficulty in the world-model provides benefits for translation to the real environment. Table 1 contains the results of the parameter sweep. For each temperature, an A2C controller was entirely trained in the world-model with that temperature. Next, the controller was evaluated across 500 episodes in the dream environment, the actual environment, and a test environment. This procedure was repeated three times for each temperature value. The first three columns in Table 1 show the mean of the average score from each controller instance. At low temperatures, the agent can reach extremely high scores in the dream environment, but these scores do not translate to reality; an agent trained in a world-model with a temperature of 0.1 performs nearly four times worse than the performance of a random agent. As temperature increases, it becomes progressively more difficult for the agent to earn high scores in the dream environment. Noticeably, the agents that translate the best to the real environment (see temperature of 2.0) receive poor overall scores in the dream environment. The temperature hyperparameter can be tuned in order to find the optimal translation from the dream environment to the real environment. The average across runs in Table 1 shows that increasing the temperature increases scores in the real environment, and then decreases scores as the environment becomes too difficult to find a translatable policy. However, this behavior is not entirely consistent if one were to run a single sweep on temperatures. The rightmost column presented in Table 1 shows the average  
Figure 4: Training curve of an Actor-Critic (A2C) agent in the bitrate streaming Park environment. scores and standard deviations a single temperature sweep, tested in the actual environment. While a single sweep follows the general trend of increasing to a peak and then decreasing, the increase and subsequent decrease in scores are not always monotonic; the rightmost column presented in Table 1 highlights these discrepancies. In training, it is possible that an agent learns to exploit the world-model and fails to generalize to the real environment. The best A2C agent trained in the world-model received an average score of \(408.71\pm 280.6\) in the real environment, which is close, but does not beat the average score of \(500.5\pm 358.8\) for the same model-free agent. The world-model was trained with a total of five-million actions in the real environment, and the controller trained for ten-million actions inside of the world-model. The strictly model-free controller used ten-million actions in the real environment.