Toward Timeliness-Enhanced Loss Recovery for Large-Scale Live Streaming

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:28:02

Abstract: Due to the limited permissions for upgrading dual-side (i.e., server-side and client-side) loss tolerance schemes from the perspective of CDN vendors in a multi-supplier market, modern large-scale live streaming services are still using the automatic-repeat-request (ARQ) based paradigm for loss recovery, which only requires server-side modifications. In this paper, we first conduct a large-scale measurement study with a collection of up to 50 million live streams. We find that loss shows dynamics and live streaming contains frequent on-off mode switching in the wild. We further find that the recovery latency, enlarged by the ubiquitous retransmission loss, is a critical factor affecting client-side QoE (e.g., video freezing) of live streaming. 
We then propose an enhanced recovery mechanism called AutoRec, which can transform the disadvantages of on-off mode switching into an advantage for reducing loss recovery latency without any modifications on the client side. 
AutoRec also adopts an online learning-based scheduler to fit the dynamics of loss, balancing the tradeoff between the recovery latency and the incurred overhead. We implement AutoRec upon QUIC and evaluate it via both testbed and real-world deployments of commercial services. The experimental results demonstrate the practicability and profitability of AutoRec, in which the 95th-percentile times and duration of client-side video freezing can be lowered by 34.1\% and 16.0\%, respectively.

Main paper: 
 introduction: 
 Internet live services such as Youtube Live, TikTok Live, and Twitch have gradually become a fundamental element for enriching daily life and work [1, 2] . This also introduces an urgent requirement to promote the transmission performance of live streaming. The ubiquitous packet loss is an essential factor affecting client-side quality-of-experience (QoE) [3, 4] , which will introduce head-ofline (HOL) blocking and even incur long-time video freezing if the available frames in the player buffer are all consumed. Therefore, loss tolerance control matters in live-streaming services.The existing loss tolerance schemes mainly focus on designing dual-side (i.e., service-side CDN and client-side application) control policies, including Forward Error Correction (FEC) [5] [6] [7] , multipath retransmissions [8, 9] , semi-reliable transmissions [10, 11] , and application-level controls [12] [13] [14] . However, as shown in Figure 1 , live-streaming application operators (e.g., TikTok Live) usually apply the Multi-Supplier Strategy [15] in the CDN market. As a result, Figure 1 : An example of a multi-supplier market for CDN vendors in large-scale live streaming services.it is the CDN vendor's duty that optimize the transmission performance (e.g., loss tolerance), according to which the application operators will choose the better-performed CDN vendors to carry more traffic (i.e., larger market share). In this case, only serverside sending policies can be adjusted by the selected CDN vendors, which lack the proper authority to synchronize client-side control rules. Thus, the above-mentioned arts for loss tolerance control suffer from deployment issues in the multi-supplier CDN market.In this case, most modern CDN vendors only apply the automaticrepeat-request (ARQ) paradigm [16] to control loss tolerance as the commercial solution, which retransmits only one replica of the lost packet when a loss is detected. However, we find that the legacy ARQ-based loss recovery is far from satisfactory according to our performed large-scale measurements. For example, the proportion of connections with maximum retransmission times of two or more exceeds 43%. Among them, a considerable portion of the connections has certain packets that are retransmitted even more than 10 times ( §2.1). The retransmission loss enlarges the loss recovery latency by 123.2ms on average and 279.3ms in the worst case. This enlarged recovery latency further increases the probability of empty buffer space on the client side, thereby increasing the risk of video freezing ( §2.3). Thus from a philosophical standpoint, it is worth asking: Can we accelerate loss recovery solely through CDN servers without modifying clients? How can it be addressed without adding significant additional overhead?Our key insight is that the on-off mode switching ubiquitously occurs in current live streams ( §2.2), where the bandwidth during the "off" periods is not fully utilized. Based on our measurements, it can be observed that each live stream spends 464 ms in off-mode per second and enters off-mode 20 times per second on average. It is well-studied that the on-off traffic pattern is not conducive to transmission control [17] [18] [19] [20] [21] [22] . However, we argue that it can act as an advantage for the loss tolerance control of live streaming. In this paper, we present AutoRec, an enhanced loss recovery mechanism that can transform the disadvantages of on-off mode into an advantage of loss tolerance controls under large-scale live streaming.A straightforward way of AutoRec is to directly reinject a fixed number of replicas of loss packets once stepping into off-streaming mode. However, this approach faces two challenges. First, the fixed settings of redundant replicas cannot well adapt to the dynamics of packet loss, while excessive reinjection of packets results in non-trivial recovery overhead. Second, according to our largescale measurement study, the off-mode is unevenly distributed throughout an entire live session, the sending of replicas might be delayed when the stream doesn't enter off-mode in time. To tackle these issues, we propose a two-step solution: Redundancy Adaption and Reinjection Control.Redundancy Adaption smartly determines how many replicas of the lost packets should be reinjected. Specifically, it adopts an online learning-based policy to dynamically set the number of replicas. The goal is to send the least number of replicas that adapt to the dynamics of packet loss. This assures a minimized redundancy overhead while accelerating loss recovery.Reinjection Control determines when to retransmit the replicas. Generally, it enables the replica reinjection during the off-modes. AutoRec further adopts the opportunistic reinjection to trigger loss reinjection even though lacking the desired opportunity of off-mode. This assures that each replica can be reinjected in time even under uneven distribution of off-modes.We implement the AutoRec prototype in the user-space QUIC protocol and deploy it on both testbed and real-network CDN proxy for 6 months. The experimental results demonstrate the practicability and profitability of AutoRec, in which the average (95thpercentile) times and duration of client-side video freezing can be lowered by 11.4% and 5.2% (34.1% and 16.0%), respectively.The rest of the paper is organized as follows: §2 motivates our work with a large-scale measurement study. Then, the high-level architecture and design details of AutoRec are depicted in §3 and §4, respectively. §5 gives the experimental evaluations of AutoRec. §6 overviews the related work and §7 concludes the paper. 
measurement study: 
 In this section, we conduct a large-scale measurement study to motivate our work. We first explore the characteristics of loss ( §2.1) and the characteristics of live streaming ( §2.2) in the wild. We then analyze the performance of live streaming under the context of packet loss, especially under retransmission loss ( §2.3). We dissected the characteristics of loss using real-world logs spanning up to two weeks. These logs documented specific details related to packet loss, including the frequency of loss events and the magnitude of each loss. Our measurements encompass over 200,000 connections, spanning a variety of application scenarios and regions worldwide. The findings are as follows.Observation #1: Retransmission loss is ubiquitous. A packet may undergo multiple retransmissions before the receiver correctly receives it. We can delve deeper into the number of times each packet was retransmitted before successful reception. For any given connection, the maximum retransmission times can be calculated as the highest count of retransmissions across all packets within that connection. Figure 2 illustrates the distribution of the maximum retransmission times in the production network. We discover that the proportion of connections with maximum retransmission times of two or more exceeds 43%. Among them, a considerable portion of the connections have certain packets that are retransmitted even more than 10 times. Traditional ARQ mechanism focuses solely on promptly retransmitting lost packets after each loss event, but it overlooks the total number of retransmissions and the total time required for the retransmitted data to be successfully received by the receiver. Therefore, the traditional ARQ mechanism hardly meets the timeliness requirements for data in certain scenarios (e.g., live streaming).Observation #2: Loss shows dynamics. We then investigate the distribution of the loss rate (every 5 minutes) for each connection across various global regions. The results are depicted in Figure 3 (a). While some regions exhibit similar average packet loss rates, their packet loss rate deviations differ (i.e., the violin shapes are different).For instance, both Brazil and Japan have an average packet loss rate of 3.78%, but Japan has the highest packet loss rate of 7.1% while Argentina has only 5.7%. Figure 3 (b) further illustrates how the packet loss rate evolves. Specifically, the loss rate is never static, varying between 2% and 5% over 24 hours. This confirms that packet loss exhibits dynamic behavior in real-world scenarios. This also reveals that an adaptive loss tolerance scheme should adapt to differentiated conditions and the always-changed status of networks.Unlike video-on-demand or file traffic, the current live streams frequently and extensively reveal off-mode, in which a sender temporarily has no data (i.e., becomes application-limited [19][23] ) for continuous transmissions after sending one or more frames. On the one hand, the new generation rate of live data might be slower (e.g., than the sending rate) so senders have to wait for a while until As Figure 4 shows, the data of live streaming is encoded (by video/audio encoder) into frame i which is delivered to the sending queue (from t 0 to t 1 ) and is sent out before t 2 . In this case, the sender has to keep waiting (from t 2 to t 3 ) for frame i+1 which will be encoded based on the follow-up live data. In this paper, the on-off mode that appears on the sender side can be recognized with the following conditions.• On-mode: A mode that occurs when enough data exists in the sending queue (i.e., t 0 ∼ t 2 and t 3 ∼ t 5 in Figure 4 ), which can be sent by a sender at the subsequent time. • Off-mode: A mode that occurs when no data can be obtained for traffic transmissions (i.e., t 2 ∼t 3 in Figure 4 ), making senders have to wait for the follow-up frame.We make large-scale measurements and gain the following observations to explore the characteristics of on-off modes in live streaming.Observation #1: The on-off mode switching commonly exists in live streaming. Figure 5 (a) presents our measurements of the duration each stream spends in off-mode every second, as well as the frequency of each stream entering off-mode on a per-second basis, where we can learn more than 95% of the live streams can be in off-mode for 394ms per second and enter off-mode 17 times per second. On average, each live stream spends 464ms in off-mode per second and enters off-mode 20 times per second.Observation #2: The off-modes are unevenly distributed throughout the live-streaming lifetime. Figure 5 (b) depicts the cumulative values of both off-mode duration and detected loss amount in our testbed experiments. We can find (i) off-mode mainly occurs after 2s while only 2 out of 7 packets are detected lost during this period; (ii) more (i.e., 5 out of 7) packet losses "meet" less off-mode within the first second of this measured stream, in which, especially, this streaming is always keeping on-mode traffic transmission when the 4-th to 6-th losses are detected.Live streaming has a high requirement for data timeliness that can affect and reflect client-side QoE, in which the traffic data that fails to reach receivers in time will cause longer intra-stream HOL blocking time, especially in the common designs of TCP and QUIC protocol [24] , introducing video freezes of players, e.g., Tiktok Live and Youtube Live. Due to the characteristics of packet loss and the limitations of traditional ARQ ( §2.1), the timeliness issues caused by packet loss in live streaming deserve attention. We introduce recovery latency to measure the timeliness of packet loss recovery and conduct large-scale measurements on live streams in real networks.Recovery latency is defined as the duration from when any data is detected lost to when resending a recovery packet that will be successfully received. T unit is defined as the sum of the delayed time of ACK packets, RTT, and loss detection time, representing the time elapsed for a data packet from being sent to being retransmitted. recovery latency consists of zero or more T unit , reflecting the additional time for loss recovery besides first T unit . In Figure 6(a) , recovery latency is close to 0, in which both the sender-side loss identification and retransmission all occur at t 3 . Figure 6(b) shows the resent data is detected lost again and another recovery packet (that will be successfully received at t 7 ) is sent at t 6 , where recovery latency = t 6 -t 3 . Besides, the maximum recovery latency is employed to evaluate the largest recovery latency for loss recoveries.We make large-scale measurements and collect the transmission logs of 50 million live streams. We then classify the average values of measured recovery latency and maximum recovery latency based on the ranges of loss rate and smooth RTT (SRTT), as Table 1 shows.Observation #1: The current recovery latency of live streaming is far from satisfactory. For the lost data that requires more than 1-time retransmissions, traffic senders waste 123.2 ms (i.e., recovery latency=123.2 ms), on average, before sending out the recovery packets that their receivers will indeed acknowledge. Worse still, the average maximum recovery latency is up to 279.3 ms, which can easily cause video freezing.Observation #2: Higher loss rate and larger SRTT can easily introduce more deteriorated recovery latency. As Table 1 shows, with the increase in loss rate, the recovery latency is also gradually increasing. This is because the recovery data can easily suffer from another packet loss in the networks with higher loss rate. By contrast, SRTT has serious impacts on recovery latency. For example, maximum recovery latency will be improved to ∼1 s, on average, under the SRTT of 500 ∼ 2000 ms, which is unbearable for the timeliness of loss recovery. Therefore, an enhanced recovery scheme is highly required to further optimize the current unsatisfied recovery latency and promote the recovery timeliness of live streams. 
the autorec overview: 
 In this section, we first discuss the design principles of AutoRec. Then we give an overview of the architecture of AutoRec.AutoRec regards the control-unfriendly on-off mode as an essential opportunity for enhancing the recovery latency of live streaming and mitigating the negative effects on player freezing caused by potential HOL blocking. AutoRec should follow two design principles for optimizing the timeliness of loss recovery in live streams.Principle #1: Maximize the utilization of specific off-mode to enhance recovery latency. Concretely, more off-mode that is unfriendly to transmission controls can be leveraged to reinject loss duplicates and further lower recovery latency of live streaming.Principle #2: Minimize the non-trivial effects on the followup traffic transmissions. Concretely, the off-mode reinjection cannot consume much more sender-side or network resources so that the transmission efficiency will not seriously deteriorate.AutoRec is a sender-side modification to the protocol stack whose key modules are illustrated in Figure 7 . Particularly, once a loss is detected, AutoRec adopts redundancy adaption to compute the number of replicas of the lost packet that should be retransmitted next. Given the number of replicas, AutoRec then adopts reinjection control to determine the specific order and time for sending out each replica from the sender.Redundancy adaptation. This module answers the question of how many replicas should be sent to accelerate loss recovery. The redundancy level is defined as the number of replicas that should be resent for a specific lost packet. We define the reinjection overhead as the total number of replicas that are sent during transmission. To adapt to the dynamics of loss, we incorporate redundancy adaptation to allow the redundancy level to vary dynamically. To achieve this, we introduce the Redundancy Adapter, which gradually learns the feature of loss dynamics and carefully selects the most appropriate redundancy level for each retransmission round of lost packets. This ensures that AutoRec can adapt to the dynamics of packet loss while minimizing the redundancy cost.Reinjection control. This module answers the question of how to send the given number of replicas determined by the Redundancy Adapter. For each lost packet, more than one replica might be injected into the network. To avoid bandwidth contention for unlost data transmission, we introduce the Reinjection Controller to enable sending replicas of the lost packet if the stream is in off-mode. To further reduce the loss recovery latency, the Reinjection Controller also opportunistically captures the ideal chance to reinject replicas even when the stream is not in the off-mode. The Reinjection Controller can also keep the bandwidth contention for unlost data transmission within a safe limit. This ensures that AutoRec can adapt to the uneven distribution of off-modes. 
detailed design: 
 In this section, we give the detailed design of AutoRec for its practical deployment.To better balance the tradeoff between the targeted recovery latency and the potential reinjection overhead, we introduce the Redundancy Adapter to applies an online learning-based scheme to adjust the redundancy level (denoted by A thres ) for each loss recovery dynamically. The goal is to reinject "few but enough" replicas for optimizing data timeliness affected by packet losses. Specifically, as illustrated in Figure 8 , for each decision interval (denoted by DI), AutoRec performs the following three steps.Step #1: Evaluate the utility and its change ratio.Step #2: Estimate the expected waiting time at the receiver. Step #3: Determine the redundancy level.Calculation of the utility. The utility function U(A thres ) can be evaluated using the incurred recovery latency and the obtained network status, e.g., minimum RTT (minRTT) and goodput, as Eq. 1 shows. The decreased u shows receivers can take shorter time for loss recoveries without significantly lowering goodput or incurring cost-sensitive A thres . Then the utility can be evaluated as follows: REQUATIONEQUATIONCalculation of the expected waiting time. We use E waiting to denote the expected time at the receiver to wait for the loss recovery. Since AutoRec is a sender-side modification, we estimate E waiting at the sender instead as follows:EQUATIONwhere r is the actual loss rate when performing loss recoveries, and n is the maximum retransmission times. The smaller the SRTT, r, and n values are, the better the current network condition is, indicating that the receiver's expected waiting time for a single packet loss is relatively small.Determination of the redundancy level. We use DI i to denote the 𝑖 𝑡ℎ decision interval. The Redundancy Adapter determines the redundancy level of DI i+1 according to the redundancy level of DI i . Specifically, if E waiting less than its threshold Θ thres , especially for the live streams with lower loss rate or shorter SRTT, we will set A 𝑖+1 thres equal to 0 (i.e., do not perform reinjection). Note that Θ thres tries to reflect the time length of player cached available data. By setting a larger Θ thres , AutoRec can achieve targeted optimizations for those live streams that take longer to complete loss recovery. Else, we will calculate A 𝑖+1 thres as follows:EQUATIONwhere 𝜆 records the adjustment direction from D i-1 's decision (A 𝑖 -1 thres ) to D i 's decision A 𝑖 thres , where 𝜆 = 1 if A 𝑖 thres > A 𝑖 -1 thres , and 𝜆 = -1, otherwise. R nor is the normal disturbance ratio that tries to filter the non-AutoRec impacts on optimization effects. For example, if A 𝑖 thres keeps unchangeable (compared to its previous value), the utility u can also become better or worse, which is caused by the dynamic network status or congestion control.To avoid the local optima issue incurred by this utility-powered optimization, the Redundancy Adapter enables the decision module to randomly choose a different redundancy level (A rand thres ) from [0, 4] if the redundancy levels (A old thres ) selected in the past several DIs (e.g., 10) keep unchangeable. In this case, if the newly introduced R Δ ≤ R nor , the configured T thres will fall back from T rand thres to T old thres ; otherwise (i.e., R Δ > R nor ), the decision module will continue to determine the next A thres based on Eq. 4.We use D ul to denote the lost data that has been resent but unacknowledged by its receiver. The data D ul is stored and managed in a reinjection queue added at the sender. To decide when to retransmit the replicas, we introduce the Reinjection Controller which enables sending replicas of the lost packet if the stream is in off-mode. The Reinjection Controller also adopts opportunistic reinjection to further reduce the loss recovery latency while keeping the bandwidth contention for unlost data transmission within a safe limit.Queue management. In AutoRec, a reinjection queue will be created by a traffic sender when a new connection of live streaming is established, which will be released once this connection is closed. AutoRec enables senders to update the reinjection queue (using the following operations) for each live stream if any lost data has been resent or acknowledged, as Figure 9 shows. The detected lost data will be inserted into the end of the reinjection queue after it has been retransmitted. The D ul will be deleted from reinjection queue when (i) the resent D ul is acknowledged by its receiver, or (ii) reinjection times A i exceeds its threshold A thres . The D ul will be moved from the head to the end of the reinjection queue if it has been resent again, which is sorted by its last retransmission time. Besides, the AutoRec sender enables a status table for each reinjection queue, which records the reinjection amount (A i ) that has been performed, timestamp (T stmp ) of D ul 's last (reinjected) retransmission and D ul identification (Data_ID) 1 , as Figure 9 shows. Opportunistic reinjection.To address the issue of unevenly distributed on-off mode and conduct replica reinjection more timely, AutoRec enables traffic senders to inject packets even during onmode. As illustrated in Figure 9 , in the status table of the reinjection queue, T stmp can be leveraged to compute D ul 's "silence" duration since its last retransmission (or reinjection). Once it exceeds the threshold T thres , D ul will be fetched from the head of the reinjection queue and then resent out regardless of whether the off-mode is entered or not. Note that for achieving well-distributed loss reinjections (the challenge in §2.2), the threshold T thres is updated based on the latest A thres as follows:T thres = T unit A thres + 1 (5)It is also worth noting that the smaller the T thres , the larger the reinjection overhead. The goodput might decrease due to the bandwidth contention from the reinjected packets. However, the Redundancy Adapter in AutoRec has already considered the impact of reinjection on goodput (see Eq. 1), this helps AutoRec maintain the incurred overhead within an acceptable range. 
experimental evaluation: 
 We perform the experimental evaluations on our established testbed and the real networks, respectively, whose traffic senders (deploying AutoRec) can pull and transmit the requested live streaming from our live CDN to its client or real-network users, as Figure 10 shows. The AutoRec prototype is implemented based on QUIC protocol [23] (with LSQUIC Q043) [25] and NGINX architecture (with nginx 1.17.3) [26] , which consists of 900+ lines of code without any client-side modification. The testbed server and client are running on CentOS Linux release 7.9 with Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz (E5-2620 v3 @ 2.40GHz), 48 (24) processors, 62GB memory and 1000Mbps NIC. The employed network emulator is HoloWAN ultimate 2600u which supports 0∼1000Mbps bandwidth, 0∼10s delay, 0∼100% loss rate and 0∼1000GB buffer length. In this section, BBR (with version 1) [27] scheme is leveraged for congestion controls. The baseline scheme is the typical ARQ paradigm that will retransmit one recovery packet once a packet is detected lost. Each-stream recovery latency in this section is displayed only for the lost data, whose recovery requires at least two retransmissions.To better evaluate AutoRec, we define a new metric called recovery deterioration rate, which refers to the ratio between the amount of lost data (D k ) that takes two or more T unit (i.e., recovery latency ≥ T unit ) to be recovered, to the amount of all lost data. For example, there are 2 packets (one is in Figure 6 (a) and the other is in Figure 6(b) ) detected lost, which require 1 and 2 T unit for their successful recoveries (at t 4 and t 7 ), respectively. Then, we can learn D k = 1 and recovery deterioration rate = 50%.The testbed experiments will be performed to demonstrate the practicability of AutoRec. Unless otherwise declared, we use R nor = 3%, A thres ∈ [0, 4] and Θ thres = 50ms. The basic environment is configured as follows: ∼8Mbps bitrate of live streams, 5% loss rate, 50ms RTT, 8Mbps bandwidth (BW) and 500KB network buffer. Each obtained metric is the average value of over 100 sets of experiments, in which each live stream will last for 60 seconds. Benefits under different network environments. AutoRec can introduce consistent recovery latency enhancements under variable network conditions as Figure 11 shows. We can learn the following results. (i) The changes of either video bitrate or bottleneck link buffer have fewer effects on our cared recovery latency of interest, in which recovery deterioration rate can be optimized by 69.0% from 4.94% (of baseline) to 1.53% (of AutoRec) and recovery latency will be lowered by 43.0% from 76.1ms to 43.4ms, on average. (ii) Higher loss rate can easily cause worse recovery deterioration rate that is approximately equal to the actual loss rate, which can be optimized to 0.8%, 1.6%, 3.5% and 5.7% under the loss rate values of 3%, 5%, 10% and 15%, respectively. By contrast, recovery latency reductions keep a stable range of 27ms∼32ms when loss rate≤10%. (iii) Larger-RTT live streaming actually experiences more deteriorated recovery latency, which is lowered by 34.7% from 134.9ms to 88.0ms when RTT=100ms. (iv) The bandwidth has significant effects on recovery latency, where 4Mps configuration brings recovery latency=167.1ms compared to 64.3ms of 16Mbps bandwidth.Performance of specially-designed functions. To evaluate the specially-designed functions in §4, we develop three AutoRec variants, i.e., AutoRec (1), ( 2) and (3), to achieve (i) off-mode reinjection, (ii) off-mode and opportunistic reinjection with A thres = 2 and (iii) AutoRec that reinjection will perform even E waiting ( §4.1) being too small, respectively. Figure 12 shows the benefits of recovery latency and utility as well as the recovery cost compared to the baseline. We can learn AutoRec (2) can obtain higher recovery latency benefits while incurring unacceptable overhead (e.g., 21.8% goodput deterioration). Besides, off-mode reinjection can introduce smaller reinjection costs but fail to achieve more significant recovery latency enhancements. By contrast, AutoRec with the functions of §4.1 can well balance the tradeoff between recovery latency and reinjection overhead.The sensitivity to pre-configured parameters. In AutoRec, some essential parameters (i.e., 𝜃 thres and 𝛼 in §4.1) should be firstly configured to decide loss reinjection activation and evaluate last-DI's performance of AutoRec. To further explore AutoRec's sensitivity to these specific parameters, we perform sensitivity experiments on our testbed, as Figure 13 shows. The smaller value of either 𝜃 thres or 𝛼 can result in a more satisfied recovery latency while introducing much more reinjection cost (e.g., intolerable extra retran_ratio and deteriorated goodput). For example, 𝜃 thres = 50ms can achieve significant optimization for recovery latency (= 43.4ms) and recovery deterioration rate (=1.6%), but also incurs an extra 3.9% retran_ratio and 8.6% goodput decrease, compared to 𝜃 thres = 100ms. Meanwhile, recovery deterioration rate = 1.51% and recovery latency = 44.3ms are actually incurred with 𝛼 = 0.01, compared to the corresponding 2.32% and 53.8ms of 𝛼 = 1, respectively. However, the configuration of 𝛼 = 0.01 also introduces an extra ∼1.0% retran_ratio and the decreased goodput by 2.5%, compared to the values of 𝛼 = 1, respectively. For 𝛼 ∈ {0.01, 0.1, 0.5}, AutoRec keeps insensitive to these parameters, in which the obtained recovery latency benefits and the caused recovery cost are close to each other.To further explore AutoRec's performance, we deploy AutoRec prototype in our CDN proxy and evaluate receiver-side video freezes, recovery latency benefits and reinjection cost.Deployment experiences. For better deploying AutoRec in the real networks, the following rules are obeyed, which are based on our experiences for optimizing player video freezing. The specific parameter 𝛼 of the AutoRec utility function is configured as 0.1. Meanwhile, 𝜃 thres should be set as the value between the 50th-and 80th-percentile SRTT. To avoid the issues caused by higher loss rate, the deployed AutoRec is recommended to be turned off or kept inactivated if the newly monitored loss rate > 10%.Recovery latency benefits. AutoRec keeps continuous optimizations for recovery latency and recovery deterioration rate in the real network, as Figure 14 (a) shows. We can learn the average recovery latency and recovery deterioration rate can be lowered by the ratio of 24.6% and 37.1%, whose values are reduced from 123.2ms and 6.8% to 92.9ms and 4.3%, respectively. In particular, high-percentile (i.e., 95th-) recovery latency and maximum recovery latency can be decreased by 145ms and 384ms, which means 5% receivers take 145ms (up to 384ms) less to wait for recovering some loss if the first-time resent data is detected lost again. As E waiting represents the expected time for loss recovery from the client side ( §4.1), we evaluate the recovery latency benefits under the different ranges of E waiting . Figure 14(b) shows the optimization ratios of recovery deterioration rate (30% ∼ 50%) and recovery latency (10%∼35%) become obvious, especially for those live streams with larger E waiting values. In particular, 174.0ms (314.2ms) maximum recovery latency is lowered at the above ranges of E waiting . Therefore, AutoRec can achieve targeted optimization for recovery latency of large-E waiting live streams.Utility and reinjection overhead. In the real network, the deployed AutoRec enables utility optimizations without incurring non-trivial reinjection overhead. From Figure 15 (a), we can learn the average utility can be optimized by 6.3%, in which the optimization ratios of 13.4% are achieved for 80th-percentile utility value, respectively. As for the incurred cost, Figure 15(b) shows the utility optimization will first become more obvious (up to ∼17%) with the increased (top 80%) E waiting , which also introduces the controllable goodput deterioration of 2.5%∼5.1% and retran_ratio of up to 3.6%. Besides, the tradeoff between recovery latency and recovery overhead will become difficult to balance in those streams with the last 10% E waiting , whose goodput is seriously affected by reinjection.Video freezing. The AutoRec performance can be further evaluated by the observed client-side player video freezing, including its frequency and duration. As Figure 16 (a) shows, the average freezing times (per 100s) can be optimized by 11.4% from 0.57 to 0.51, whose 90th-and 95th-percentile can be lowered by 24.4% and 34.1%, respectively. Besides, AutoRec also optimizes freezing duration (per 100s) by 5.2%, in which the 95th-percentile value is reduced by ∼80ms (with the ratio of 16.0%), as Figure 16 (b) shows. These results demonstrate AutoRec is worthwhile for optimizing the timeliness of loss recovery and client-side video freezing. 
related work: 
 Adaption to on-off traffic pattern. To the best of our knowledge, almost all prior works [28] [29] [30] [31] [32] [33] [34] [35] [36] fall into the category of adapting congestion control to on-off traffic patterns. For example, Zhang et al. [33] proposed a TCP variant to overcome the challenges, in which the on-off traffic pattern disturbs the increase of the TCP congestion window and triggers packet loss at the beginning of the ON period. This paper does not focus on the congestion control issues of live streaming. Instead, we take a first step toward taming the loss tolerance control under the on-off pattern for live streaming.Loss tolerance control for live streaming. To achieve efficient loss recovery and optimize client-side video freezes, many studies have been proposed to enhance the data timeliness of live streaming. The key ideas of these works include injecting supplement data (e.g., FEC [5] [6] [7] and multi-path retransmissions [8, 9] ), ignoring some non-critical losses (e.g., application-level controls [12] [13] [14] and semi-reliable transmissions [10, 11] ). However, in commercial large-scale live-streaming product networks, the CDN vendor is mainly responsible for optimizing the end-to-end transmission performance and has control rights only on the sender side rather than the client side. Therefore, the above arts that apply client-side modification cannot meet the requirements of real-world deployment. In this paper, we propose a sender-side approach from the perspective of CDN vendors.Advancements upon ARQ. Modern CDN vendors simply apply the ARQ for loss tolerance control in commercial live-streaming services. However, the legacy ARQ-based loss recovery is far from satisfactory. There also exist many studies [37] [38] [39] [40] [41] [42] [43] that improve the performance of ARQ by introducing redundancy to loss recovery. These works, however, suffer from obviously-deteriorated goodput since the inserted extra packets occupy the sender-side and innetwork resources. This paper overcomes the above challenges by taking full advantage of the off-mode of live streaming. 
conclusion: 
 Retransmission in itself doesn't impede slow loss recovery; rather, it's the loss of retransmission that presents the true challenge. This paper proposes AutoRec to accelerate loss recovery by allowing senders to reinject loss duplicates smartly. AutoRe's ingenuity is demonstrated by its transformation of weaknesses into strengths. It employs on-off mode switching -a feature typically challenging for transmission control -to its advantage. This approach not only facilitates quicker recovery from packet loss but also ensures that the existing data transmission on the connection remains unaffected. Both testbed evaluations and real-world deployments demonstrate the practicability and profitability of AutoRec. Currently, AutoRec has been deployed on one of the world's largest CDN vendors [reference hidden anonymously], serving thousands of millions of live-streaming users worldwide. 
 
 Appendix: * The displayed recovery latency (maximum recovery latency) only records the each-stream's average recovery latency (maximum recovery latency) of the lost data, whose recovery requires two or more retransmissions.In this paper, Data_ID can recognized as the packet number (pkt_num) in TCP or the stream offset (stream_offset) in QUIC. 


Decision: This paper has received extremely consistent ratings: all four reviewers recommend acceptance. Its overall average rating places it in the top 22% of the papers, and therefore it should be considered for acceptance.

Reviews: review: paper_summary: paper_summary: This paper investigates loss recovery mechanisms for large-scale live streaming systems. It leverages the insight that the typically disadvantageous on-off mode switching can be transformed into a beneficial feature to reduce loss recovery latency. Building on this observation, the paper introduces an enhanced recovery mechanism called AutoRec. AutoRec employs an online learning-based policy to adapt to the dynamic nature of packet loss, effectively balancing the trade-off between recovery latency and overhead. Experimental results from both testbed and real-world deployments in commercial services demonstrate that AutoRec significantly reduces client-side video freezing.
strength_weakness: strength_weakness: 1. The introduction presents a new approach to addressing packet loss in live streaming services. The insight of utilizing on-off mode switching for loss tolerance control transforms a typically disadvantageous traffic pattern into a strategic advantage for enhancing loss recovery. This perspective offers a fresh solution to the problem.

2. Experiments conducted on both testbed and real-world deployments validate the practicability and effectiveness of AutoRec. These evaluations demonstrate its potential to  improve performance in commercial live streaming services.
limitations: limitations: 1. A lack of comparison with existing solutions: the paper  mentions the limitations of current solutions but does not provide a detailed comparison with AutoRec on the experiment section . While the novelty of AutoRec is emphasized, a more detailed discussion of its advantages over existing ARQ-based solutions and other loss tolerance mechanisms would clarify its value proposition.

2. Weak theoretical analysis: loss recovery in live streaming has been extensively studied and can be modeled as a constrained optimization problem. However, this paper falls short in providing a precise and formal problem formulation. A well-defined problem statement is essential for setting clear objectives and constraints for the proposed solution, including specifying the parameters, metrics, and conditions under which the solution is expected to operate.
paper_summary: paper_summary: Proposes a way of taking advantage of off-mode live streaming to retransmit lost packets more efficiently than standard ARQ schemes: the receiver should be able to recover from the lost packet more quickly than using ARQ. Retransmissions can also be done during on-mode as long as they don't significantly impact new packet transmissions. Experimental results from a testbed and an actual network show that the proposed approach can achieve improved performance relative to a standard ARQ approach.
strength_weakness: strength_weakness: The idea of sending multiple retransmissions of lost packets while not disturbing new packet transmissions is innovative, and the proposed approach (AutoRec) is well explained.

The experimental results show that, at least in the conditions studied, AutoRec can improve the performance seen by live stream users.
limitations: limitations: Equation (3) computes the expected waiting time but this expression may be inaccurate e.g. requires knowing/estimating r and SRTT
 - could an observation-based approach be used instead to estimate E_waiting ?

Lots of parameters to be chosen e.g. 𝛼 , Θ_thres , R_nor
- some sensitivity results are shown for some of these parameters, but there is still a question over how they should be chosen in practice.

"To better evaluate AutoRec, we define a new metric called recovery deterioration rate"
- why not some measure or estimate of user's QoE?

How would AutoRec work for packets of different priority levels? e.g. lowest-priority packet should not be retransmitted, highest-priority packet should be retransmitted at max level, etc. This does not appear to have been considered in the proposed approach.

Instead of sending replicas at different times, what about sending them simultaneously on different routes? (spatial diversity)
paper_summary: paper_summary: This paper introduces AutoRec, a mechanism to improve loss recovery in large-scale live streaming services without client-side modifications. AutoRec leverages the on-off mode switching in live streams to reduce recovery latency, transforming a disadvantage into an advantage for loss tolerance control. It employs an online learning-based policy to dynamically adjust the number of packet replicas, balancing recovery latency and overhead. The implementation of AutoRec shows improvements in reducing the times and duration of video freezing during live streaming.
strength_weakness: strength_weakness: 1.	The topic is interesting. The paper includes a measurement study with live streams to understand loss dynamics and live streaming characteristics.

2.	The background is clearly presented. The authors provide adequate information about the topic.

3.	The authors demonstrate practicability and profitability through testbed and real-world deployments, reducing video freezing times and duration.
limitations: limitations: 1.	The major flaw of this paper is that the motivation is not clear to me. It is tough to understand the limitations of existing schemes and the challenges you want to overcome from the Abstract. 

2.	What are the disadvantages of on-off mode switching?

3.	In the introduction, the authors claim that the legacy ARQ-based loss recovery is far from satisfactory according to the performed large-scale measurements. But under what scenario? You have spent too much effort on introducing these quantitative results without briefly introducing some crucial settings, making it hard for readers to appreciate the significance of these quantitative results.

4.	Many important terminologies need further introduction. For example, what is on-off mode switching?

5.	In Section 2, more details about the large-scale measurements should be given. More information about the real-world logs should also be provided.

6.	Further, I needed clarification about the observation of the dynamics of loss. What is the purpose of you showing the loss rate across different regions?

7.	Many important parameters in your experiments need further discussion/justification.
paper_summary: paper_summary: The authors investigate how to deal with the interruptions of media being played out due to retransmissions that typically use ARQ-style error recovery.  The authors have conducted a measurement study, proposed a solution, and then evaluated the solution.

The basics of the idea are to recognize that for live streaming there are periods of time for which there is no encoded video available for transmission (called an off period).  This time can be used for preempting losses in the authors' proposed solution.
strength_weakness: strength_weakness: There are some interesting ideas here.  And if the results are accurate, there seems to be some benefit to the solution proposed by the authors.
limitations: limitations: The challenge with this paper is that there are some descriptions that are missing that would both improve readability, but more importantly, make the results easier to understand and digest.

For example, not a lot is said about the measurement study.  There are some fairly high-level graphs from the study, but the section leaves almost as many questions as it tries to answer.  The graphs show results for a number of countries, but they are very high level, so high level that it is hard to understand the point of the results.  One question in particular is how much of a problem is video freezing for live streaming from CDNs.  In most developed countries it seems like a pretty rare event.  The authors would do well to better motivate just how significant of a problem video freezing is.

The intuition behind the authors' solution makes sense, but there seems very little (if any) discussion of tradeoffs.  It seems the authors are essentially proposing guessing at what data might be lost or at least which lost data is likely to cause interruptions and then re-re-transmit that data.  Sending more data into the network generally doesn't help when there is congestion, but obviously not all loss is due to congestion (the assumptions TCP make, for example, are quite out-dated).  But it would be useful to better understand the tradeoffs and overhead of the solution.

Finally, the authors work is largely predicated on an assumption that no changes can be made to the client.  It isn't clear that this is an accurate assumption.  For example, decoders and the surrounding execution environment are quite flexible, to the point where forward error correct is a fairly widely deployed.  How does FEC compare to the authors' solution?

Overall, while I think there are some open issues for the paper, it might generate some discussion if I correctly understand what the authors are proposing.