# Keyframe Insertion: Enabling Low-Latency Random Access and Packet Loss Repair
## 3 Materials and Methods  
The main idea is to generate a normal video stream for which random-access and packet-loss considerations are minimized (see Figure 1). This normal stream is accompanied by a companion stream designed to solve random access and packet loss situations. The benefit of this strategy is that such a separation in responsibility makes the design decisions about compression settings more straightforward. Within the normal stream, encoders will still introduce keyframes when content specific changes occur, such as high-motion events or scene changes. Bounding the interval between keyframes is necessary here to completely recover the normal stream from artifacts. On the other hand, limiting the number of keyframes is beneficial for compression efficiency resulting in an efficient normal video stream. Additionally, the companion stream is generated consisting solely of keyframes such as Instantaneous Decoder Refresh (IDR) or Continuous Random Access (CRA) frames. Both of these framertypes are keyframes as defined in H.265/HEVC. The frequency of these keyframes must be chosen corresponding to the required random-access latency, the packet loss recovery latency, and the bitrate overhead. The data to make such a trade-off are provided in Section 4. Whenever a random-access event or packet-loss event occurs, the following procedure needs to be followed (see Figure 2). After a switch or loss event, let us denote the Picture Order Count (POC) of the first occurrence of a keyframe in the companion stream as \(f\). At that moment, the keyframe at position \(f\) will be used for starting the decoding process instead of the predicted frame at \(POC=f\). After that, the regular decoding process can continue with decoding \(f+1\). With packet loss, this procedure assumes that the regular decoding process continues from the moment packets are lost (implying packet-loss drift errors) until the first keyframe in the normal stream or the companion stream, whichever comes first. Only then, at \(POC=f\), the keyframe is decoded and regular decoding of the normal stream continues from \(POC=f+1\). The normal stream will be encoded from the original source content (NSsrc). For the encoding of companion streams, two possibilities for input selection will be considered and measured. First, the companion streams can be encoded from the source content similar to what happened with the normal stream (CSsrc). Second, a closer match with the normal stream can be found by decoding the normal stream and encoding these reconstructed as a companion stream (CSNS). This process is illustrated in Figure 3.  
Figure 1: A source video is encoded as a low-latency normal video stream accompanied by a companion stream providing keyframes at specific intervals. Figure 2: After a random access or packet loss event, a companion stream keyframe replaces the collocated predicted frame. Although the basic idea of inserting keyframes has been mentioned in previous work with respect to MPEG-2 and H.264/AVC, it is important to consider the restrictions associated with the recent H.265/HEVC standard when performing the proposed operations. * POC: When starting the decoding process at a companion stream keyframe, the POC of this keyframe should be identical to the POC of the replaced frame in the normal video stream. The synchronization of keeping these POCs identical is preferably performed during the encoding process, but can alternatively be performed right after the encoding process through an entropy rewrite step. * Resolution: In contrast to H.266/VVC, H.265/HEVC cannot accommodate resolution changes within a Group of Pictures (GOP), such that the resolution of the normal stream and the companion stream should be identical. * Sequence Parameter Set (SPS) and Picture Parameter Set (PPS): The parameter sets of both the normal and the companion stream should be identical or a retransmission of parameter sets should occur after the introduced keyframe from the companion stream.