# A Novel Proprietary Internet Video Traffic Dataset Generation Algorithm
## 3 Methodology
### Proprietary Internet Video Traffic Dataset Generation  
To generate proprietary video quality category datasets for YouTube, Facebook, and Twitch, we incorporated prior knowledge from PCAP files, including attributes such as video streaming platforms, resolution, and video playback modes. This knowledge was used to develop the YouTube, Twitch, and Facebook video quality category datasets. Each dataset was further divided into four subsets based on four different nDPI accounting modes. Each dataset contains eight categories: 720P_30 FPS_VoD, 720P_30 FPS_Live, 720P_60 FPS_VoD, 720P_60 FPS_Live, 1080P_30 FPS_Live, 1080P_30 FPS_VoD, 1080P_60 FPS_VoD, and 1080P_60 FPS_Live. Due to limitations in capturing 60 frames per second on the Facebook platform, no packets for relevant 60 FPS videos were collected, resulting in only four 30 FPS classes being included for this platform. The generation of nDPI-detected YouTube, Facebook, and Twitch video quality category datasets (nYFTQC) is outlined in Algorithm 2. The following are detailed steps. Initial Filtering: The first step involves retaining all flow samples associated with the transport layer QUIC and TLS protocols, which primarily handle video traffic packets. For instance, the Google application YouTube interacts with Google's Chrome browser and servers using the QUIC protocol. Additionally, network video traffic is typically encrypted, with the TLS protocol widely used by users and video providers to ensure secure and reliable web communication. Handling Unknown Categories: In the second step, flow samples from unknown application categories that the nDPI library cannot parse should be retained if they display certain characteristics, such as a high volume of bidirectional packets, large data sizes, and long durations of bidirectional flows. This retention specifically includes flow samples related to Google and Facebook. Removing Irrelevant Flow Samples: Flow samples irrelevant to video content are deleted based on the requested server name. These include samples related to automatic updates, background processes, advertising statistics, account interfaces, and other non-video-related activities. Filtering is performed using the TLS application name and Web application category name. Filtering Specific Protocols: The fourth step involves filtering out minor protocols associated with non-video traffic, such as those related to Microsoft, Skype, and similar applications, as indicated in the application name column. Using the elimination method, we delete flow records if the application category name is not from the following list: AmazonAWS, Amazon-Video, Azure, Twitch, Facebook, and YouTube. Additionally, flow samples are removed if the application category does not belong to one of these categories: Cloud, Media, Social Network, Video, Unspecified, and Web. Excluding Inconsistent Flows: The fifth step removes flow samples with bidirectional flow durations of zero milliseconds or zero-count accumulators of bidirectional packets, as these are inconsistent with video traffic characteristics. Final Compilation: The final step involves saving all remaining flow samples under a new column labeled Video Quality Category. These steps are repeated for all classes, and the remaining flow samples are merged to form a new dataset. Overall, the process of selecting video-related flow samples using the nDPI tool is complex and meticulous. However, noise flow records that lack sufficient metadata for nDPI inspection or exhibit flow-level time-temporal features similar to normal samples are difficult to clean. Additional potential methods, such as flow aggregation or increasing the flow active timeout threshold, may help address this issue.