SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We launched a measurement campaign using H3B over 5 different geographical locations. The locations are: Delhi, Bangalore, New York, Germany, and Singapore. Such a measurement was conducted using two different web browsers Chrome/Chromium and Firefox. We stream once over HTTP/3 and once over legacy HTTP/2. One of the testbeds is set up inside our campus premises, and the rest are set up using Digital Ocean machines. We emulate 5 different bandwidth patterns namely Dynamic High (DH): a good bandwidth, Dynamic Low (DL): a poor bandwidth and Dynamic Very Low (DVL): a very poor bandwidth, Real: real packet captures under mobility over WiFi and over cellular network.
Bandwidth patterns: To emulate DH, DL, and DVL bandwidth patterns, we created user-specified packet delivery trace files. Table 4 shows the bandwidth patterns where each pattern has the starting bandwidth, last bandwidth, and the Jump required to move from the starting bandwidth to the last bandwidth. After each jump, that bandwidth stays at that bandwidth for the Jump duration. This pattern from start to last bandwidth and then back from last to start bandwidth repeats in a cyclic fashion. Note that to emulate these bandwidth patterns, we use the Mahimahi network emulation tool. We also replay real packet captures. The real packet captures are of two categories, over WiFi and cellular network. Since we focus on a poor network, we utilize 105 mobility traces collected while watching YouTube videos over WiFi [16]. For cellular networks, we collected "in-the-wild" traces from 10 Android smartphone users watching YouTube videos of their choice using the cellular network. The volunteers are from 4 developing countries namely Kenya, Saudi Arabia, Pakistan, and India. The data was collected from 14 cities. Note that we focus on low and middle-income developing countries for collecting the data. The volunteers were instructed to collect network traces (pcap) using a pcapDroid application on their smartphones while watching the YouTube videos. The data was collected while the volunteers were traveling to/from their workspace to home. The phones used by volunteers had android versions 9-12
Table 4: Bandwidth Patterns
Bandwidth Pattern	Starting Bandwidth	Last Bandwidth	Jump (Kbps)	Jump Duration
Dynamic High (DH)	1152Kbps	896Kbps	-256	240 sec
Dynamic Low (DL)	640Kbps	128Kbps	-256	240 sec
Dynamic Very Low (DVL)	64Kbps	256Kbps	+64	60 sec
Real	Mobility Traces from [16] and "in-the-wild" volunteer traces
Dataset structure: As part of the campaign, we obtain application and network level dataset of 5936 sessions (Firefox: 100 over HTTP/3 + 100 over HTTP/2, Chromium: 1864 over HTTP/3 + 1864 over HTTP/2, Original browser: 1004 over HTTP/3 + modified browser: 1004 over HTTP/3) for a total duration of 5464 hours. Fig. 2 (a) shows the hierarchy of the dataset collected.  $FS1$ ,  $FS2 \dots$ , and  $FS100$  are the streaming session pairs over the Firefox browser.  $S1$ ,  $S2$ ,  $S3 \dots$ , and  $S1864$  are the streaming session pairs over Chromium browser. Each streaming session pair consists of streaming once over HTTP/3 and once over HTTP/2. Fig. 2(b) shows the hierarchy of the dataset collected for original and modified chromium browser over HTTP/3.  $S1865$ ,  $S1866 \dots$ , and  $S2868$  are the streaming session pairs over the original & modified chromium browser. Table 5 shows the duration of the collected dataset over different browsers, locations, and network conditions.
(b)
Figure 2: Dataset Hierarchy of (a) 1864 video session pairs over chromium and 100 video session pairs over Firefox browser on DH, DL, DVL and Real bandwidth patterns across Delhi, Bangalore, Singapore, Germany, and New York. (b) 1004 video session pairs over modified and original HTTP/3 enabled chromium-browser in Delhi on DVL bandwidth pattern.
Table 5: Dataset Details
Configuration	Duration	Configuration	Duration
Chromium Delhi duration	3941 hours	Firefox DVL duration	142 hours
Chromium Bangalore duration	490 hours	Chromium Total DH duration	278 hours
Chromium Singapore duration	253 hours	Chromium Total DL duration	1712 hours
Chromium Germany duration	459 hours	Chromium Total DVL duration	157 hours
Chromium New York duration	218 hours	Chromium Total Real duration	894 hours
Chromium Total OB duration	738 hours	Chromium Total MB duration	715 hours
(c)
Figure 3: Hypothesis testing results over 1964 streaming session pairs on QoE over (a) Chromium and Firefox Browser, (b) different bandwidth patterns, and (c) TCP Fallback over Chrome/Chromium: Percentage of bytes transferred over TCP across all HTTP/3 streaming sessions for various bandwidth patterns. DH: Dynamic High, DL: Dynamic Low, DVL: Dynamic Very Low, and Real.
5 Dataset Analysis
We now use our dataset for analyzing the performance of HTTP/3. For the same, we compare the performance of HTTP/3 with legacy HTTP/2.
QoE of HTTP/3 vs legacy HTTP/2: To compare the QoE obtained over HTTP/3 and HTTP/2 statistically, we perform hypothesis testing over all 1964 streaming session pairs. We find out in what percentage of sessions HTTP/3 provides (a) better, (b) statistically similar, and (c) worse QoE compared to HTTP/2. Case (b) is considered the null hypothesis and vice versa as an alternative hypothesis. If the null hypothesis gets rejected, then we perform the two-tail test [30] to check whether the HTTP/3-enabled browser performs better. Fig. 3(a) shows the hypothesis-testing results on computed QoE values over Chromium and Firefox browsers. In the case of Chromium, we observe that for  $81.5\%$  of the cases, the two browsers behave differently. For  $41\%$  of the cases, we found the legacy HTTP/2 browser outperforms HTTP/3 one. In the case of Firefox, we have a similar observation that HTTP/3 is not always a winner. When we look into the network logs for these  $41\%$
cases where HTTP/3 underperforms legacy HTTP/2: there were several instances of QUIC to TCP and TCP to QUIC protocol switching. From Fig. 3(b), we observe that for the scenarios under DL and DVL, HTTP/2 performs better than HTTP/3 for more than  $40\%$  of the video session pairs. Fig. 3(c) shows the CDF graph of the percentage of TCP bytes experienced at various bandwidth patterns. We observe that for  $DH$ , there is the almost negligible presence of TCP traffic for  $80\%$  times. Indeed, it is expected that QUIC does not face many connection failures for high bandwidth. For  $DL$ ,  $DVL$  and Real, the presence of TCP traffic is  $32\%$ ,  $52\%$ , and  $98\%$  respectively for  $70\%$  times. Fig. 4(a) indicates that HTTP/3 outperformed legacy for more than  $40\%$  cases in Bangalore and Germany. However, across all five geographical regions, there are more than  $30\%$  cases when legacy yielded better application QoE compared to HTTP/3. Again, correlating this TCP fallback in Fig. 4(b), we observe that Delhi has more instances of TCP traffic within HTTP/3 enabled streaming, compared to other locations; Fig 4(a) shows that HTTP/2 provides better QoE for these two cities (Delhi and Singapore) compared to HTTP/3. Thus, we observe that protocol switching impacts video streaming performance.
(b)
Figure 4: (a) Hypothesis testing results over different geographical locations. (b) TCP Fallback over Chrome/Chromium across locations. D: Delhi, B: Bangalore, N: New York, G: Germany and S: Singapore and (c) Hypothesis testing results over 1004 streaming session pairs on QoE of Original Chromium Browser (OB) and Modified Chromium Browser (MB)
QoE of HTTP/3 supported original vs modified browser: We observe that such a protocol switching occurs due to faulty implementation of fallback at the browser. Hence, an HTTP/3 browser tends to fall back to TCP at a low-bandwidth network. We, therefore, modify the Chromium source code by disabling the fallback completely. We then launch another campaign using H3B. We obtain 1004 YouTube session pairs over original and modified browser. We then perform hypothesis testing and observe that HTTP/3 modified browser outperforms original browser for  $60\%$  of cases. Thus, we conclude that fallback to TCP hinders achieving the benefits of HTTP/3.
We now discuss the limitations of our dataset, ethical considerations, and how the dataset can be utilized for future research.
6.1 Data Limitations
(1) More diversity in networking conditions: Though our dataset comprises various bandwidth patterns with a specific focus on poor bandwidth, there can be more diversity by including high bandwidth network types and other poor network bandwidth types. We allow replaying real packet traces; collected under mobility for both WiFi and cellular. This can be extended further with diverse traces collected under various mobility/poor bandwidth scenarios. (2) More diversity in locations: We have collected data for 5 locations. There can be much more diversity in locations focusing on locations from developing countries. (3) Diversity in platform: Our data collection was performed from a desktop platform; it can also be extended to include mobile platforms.
6.2 Ethical Considerations
This paper does not directly interact with human subjects or use any network services beyond their usage restrictions. All the network services used in this work (Digital Ocean machines) have been paid as per the usage.
6.3 Research Topics
Dynamic tuning of protocol configurations based on learning the network environment: Given the diversity of our dataset in terms of different network conditions, this can help better configure the protocol hyper-parameters used for video streaming. For example, to provide better application QoE, one can analyze the network conditions to tune hyperparameters like deciding the optimal transport protocol i.e., QUIC or TCP, tuning the transport protocol's congestion control or tuning the ABR (Adaptive Bitrate Streaming) parameters, and so on.
Intelligent network-application integration: We believe our dataset can allow intelligent network and application integration. To provide better application QoE, the applications should adapt themselves to network conditions. Since our dataset has both types of logs, this can enable better design of applications that will possibly look for signatures in the network behavior and adapt accordingly.
Predicting and optimizing YouTube's QoE: Our dataset provides the QoE of YouTube applications under different network bandwidth patterns. Utilizing such data, one can develop a learning model to predict YouTube's QoE. Further, our dataset for poor or variable networks provides an opportunity to use intelligent learning techniques for optimizing YouTube QoE.
Predicting QoE of HTTP/3 based video streaming application: Our dataset contains the network packet exchanges and the corresponding QoE. We believe our dataset will be valuable in training a learning model by observing the network packet exchanges and QoE. Such a model can predict the QoE of other video streaming applications that use similar packet exchanges like YouTube.
In this paper, we presented H3B, a toolbox to collect application and network layer logs for YouTube video streaming. Such a tool can emulate any given network pattern, which is one of its major features. We utilized this tool to launch a measurement campaign for 5 different geographical locations and 5 different bandwidth patterns. We obtained a dataset of 5464 streaming hours over 5936 sessions of YouTube video streaming. We further analyzed the dataset and observed that HTTP/3 is not always a winner compared to legacy HTTP/2. We believe our dataset will be valuable to the community for developing various solutions to provide better application QoE on top of the HTTP/3 browsers.
[1] S. Ahmad, A. L. Haamid, Z. A. Qazi, Z. Zhou, T. Benson, and I. A. Qazi. A view from the other side: Understanding mobile phone characteristics in the developing world. In Proceedings of the 2016 Internet Measurement Conference.
[2] Z. Akhtar, Y. S. Nam, R. Govindan, S. Rao, J. Chen, E. Katz-Bassett, B. Ribeiro, J. Zhan, and H. Zhang. Oboe: Auto-tuning video ABR algorithms to network conditions. In Proceedings of SIGCOMM'18. ACM.
[3] S. Arisu and A. C. Begen. Quickly starting media streams using quic. In Proceedings of the 23rd Packet Video Workshop, 2018.
[4] V. Arun, M. T. Arashloo, A. Saeed, M. Alizadeh, and H. Balakrishnan. Toward formally verifying congestion control behavior. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference.
[5] C. G. Bampis, Z. Li, I. Katsavounidis, T.-Y. Huang, C. Ekanadham, and A. C. Bovik. Towards perceptually optimized adaptive video streaming-a realistic quality of experience database. IEEE Transactions on Image Processing, 30:5182-5197, 2021.
[6] C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron, and A. C. Bovik. Study of temporal effects on subjective video quality of experience. IEEE Transactions on Image Processing, 26(11):5217-5231, 2017.
[7] N. Barman and M. G. Martini. Qoe modeling for http adaptive video streaming-a survey and open challenges. IEEE Access, 7:30831-30859, 2019.
[43] YouTube. YouTube Embedded Players and Player Parameters. https://developers.google.com/youtube/player_parameters, 2021. [Online; accessed December 19, 2023].
[44] A. Yu. Benchmarking QUIC. https://medium.com/@the.real.yushuf/benchmarking-quic-1fd043e944c7, 2020. [Online; accessed 19-October-2020].
[45] Z. Zheng, Y. Ma, Y. Liu, F. Yang, Z. Li, Y. Zhang, J. Zhang, W. Shi, W. Chen, D. Li, et al. Xlink: Qoe-driven multi-path QUIC transport in large-scale video services. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference.
[46] X. Zuo, J. Yang, M. Wang, and Y. Cui. Adaptive bitrate with user-level QoE preference for video streaming. In IEEE INFOCOM, pages 1279-1288. IEEE, 2022.