# World-Models for Bitrate Streaming
## 6 Conclusions  
This work does show that it can be straightforward to apply world-models to networking problems. The MDN-RNN world-model was able to capture the dynamics of the environment; however, these experiments required proper tuning of the temperature parameter in order to learn a policy that can translate into the real environment. Finding the optimal temperature hyperparameter for other environments may require sweeping several values, and these runs must be repeated. The optimal temperature was 70% higher than the optimal temperature that was used by [PERSON] (2018). Furthermore, these experiments show a rapid decrease in performance after the sweep passes the  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Num. Actions** & **Agent** & **Mean Episode Score** & **STDEV** \\ \hline - & Random Agent & \(-\)5729.3 & \(-\)4250.1 \\ - & Buffer-based Agent & 237.7 & 435.4 \\ - & RobustMPC Agent & 286.2 & 502.5 \\ \hline
5 M & A2C Model-free Agent & 500.5 & 358.8 \\
5 M & Best World-model Agent & 408.71 & 280.5 \\ \hline
1 M & Best World-model Agent & 441.0 & 316.8 \\
1 M & Best Rainbow Agent & 306.3 & 316.3 \\ \hline
500 k & Best World-model Agent & 399.7 & 245.1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison between heuristics, model-free, and world-model agents for bitrate streaming.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Temperature** & **Dream Score** & **Real Score** & **Test Score** & **Max Average Real Score** \\ \hline
0.75 & 550.04 \(\pm\) 22.86 & \(-\)813.62 \(\pm\) 764.66 & \(-\)403.87 \(\pm\) 659.83 & 68.85 \(\pm\) 826.89 \\ \hline
1 & 453.31 \(\pm\) 12.76 & 96.09 \(\pm\) 50.03 & 382.93 \(\pm\) 30.09 & 151.88 \(\pm\) 826.91 \\ \hline
1.25 & 291.33 \(\pm\) 61.87 & \(-\)37.10 \(\pm\) 132.88 & 247.92 \(\pm\) 87.29 & 82.31 \(\pm\) 648.22 \\ \hline
1.5 & 158.27 \(\pm\) 126.67 & 280.53 \(\pm\) 182.38 & 428.88 \(\pm\) 83.72 & 399.79 \(\pm\) 245.12 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Temperature parameter sweep for a world-model trained from 500,000 recorded actions. The data follow the same format as Table 2.  
optimal temperature. It may take careful tuning to find the optimal temperature in other environments. The experiments in this work demonstrate that world-models can successfully learn simulators from the recorded data. Furthermore, world-models can outperform model-free approaches in low sample regimes. World-models are a promising area of research that may provide benefits for many other problems in the networking and systems domains.  
This paper was produced from [PERSON]'s MPhil dissertation that was supervised by [PERSON] and [PERSON]; [PERSON] prepared the draft; [PERSON] and [PERSON] worked with [PERSON] to edit and review the paper. All authors have read and agreed to the published version of the manuscript.  
This research was partially funded by the Alan Turing Institute.  
This research was supported by the Alan Turing Institute.  
The authors declare no conflict of interest.  
The following abbreviations are used in this manuscript:  
\begin{tabular}{l l} ALE & Arcade Learning Environment \\ A2C & Advantage Actor-Critic \\ A3C & Asynchronous Advantage Actor-Critic \\ GMM & Gaussian Mixture Model \\ LSTM & Long Short-Term Memory \\ MDN-RNN & Mixture Density Network - Recurrent Neural Network \\ MDP & Markov Decision Process \\ MPC & Model Predictive Control \\ MSE & Mean Squared Error \\ Mbps & Megabits per second \\ POMDP & Partially Observable Markov Decision Process \\ RL & Reinforcement Learning \\ RNN & Recurrent Neural Network \\ TPE & Tree-structured Parzen Estimator \\ QoE & Quality-of-Experience \\ \end{tabular}