Patent Publication Number: US-2023153952-A1

Title: Deep learning framework for video remastering

Description:
CROSS REFERENCE TO RELATED APPLICATION 
     This application claims the benefit of U.S. Provisional Pat. App. No. 63/279,386, filed Nov. 15, 2021, the disclosure of which is hereby incorporated by reference herein by its entirety. 
    
    
     BACKGROUND 
     Streaming services require expansive catalogs to be competitive. Old legacy films can enrich and supplement the content of such catalogs. However, the video content of legacy films is typically degraded – that is, video content, captured by low-resolution cameras, based on old sensor technologies, may be blurry, noisy, and scratched. To meet current expectations of quality and current streaming and display technologies, remastering (restoration) of these legacy films is required. 
     Current restoration techniques, based on deep learning technologies, provide tools that separately tackle video denoising or video upscaling. Such specialized tools can be applied sequentially to denoise and, then, to upscale a video into higher resolution, for example. However, applying independently optimized restoration tools, in a cascading manner, may lead to sub-optimal performance in terms of restoration quality and computational complexity. Thus, techniques that restore video content by jointly addressing different types of degradations are needed. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings wherein: 
         FIG.  1    is a block diagram of an example device, based on which one or more features of the disclosure can be implemented; 
         FIG.  2    is a functional block diagram of an example system for video remastering, based on which one or more features of the disclosure can be implemented; 
         FIG.  3    is a functional block diagram of an example system for tuning video restoration, based on which one or more features of the disclosure can be implemented; 
         FIG.  4    is a functional block diagram of an example system for training a degradation encoder, based on which one or more features of the disclosure can be implemented; 
         FIG.  5    is a functional block diagram of an example system for training a blur kernel encoder and a noise encoder, based on which one or more features of the disclosure can be implemented; 
         FIG.  6    is a functional block diagram of an example system for training a mutator, based on which one or more features of the disclosure can be implemented; and 
         FIG.  7    is a flowchart of an example method for video remastering, based on which one or more features of the disclosure can be implemented. 
     
    
    
     DETAILED DESCRIPTION 
     Systems and methods disclosed herein provide pipelined video processing that can enhance the quality of corrupted and low-resolution legacy films. Techniques disclosed herein restore an input video of a legacy film, including the removal of scratches, denoising, and upscaling the input video into higher resolution. Furthermore, techniques for manual refinement of the video restoration, for artistic tuning, are provided. To remove content degradation consisting of various types of degradations that may be present in a legacy film, aspects of the present disclosure include extracting a representation of the content degradation. Further aspects include manipulating the extracted degradation representation to artistically adjust the restored (output) video. The degradation representation may then be used for conditioning a backbone network that feeds restoration-specific networks, such as a denoising network and a super-resolution network. 
     Disclosed in the present application are video restoration models that jointly target common degradations that are typically present in legacy films. These video restoration models utilize a new contrastive training strategy to learn interpretable and controllable representations of different types of content degradation. Techniques disclosed herein employ contrastive learning to learn degradation representations (namely, latent vectors) in a discriminative representation space. Training of networks described herein is based on pairs of degraded video samples, forming positive, negative, and hard negative examples. Given a low-resolution corrupted input video, the remastering systems described herein produce a denoised low-resolution output video as well as a denoised high-resolution output video. The denoised high-resolution output video can be produced at any scale—a feature that is useful when the input video is to be restored to various video standards (e.g., NTSC). 
     Aspects disclosed herein describe methods for video remastering by a restoration system. The methods comprise receiving, by the system, a video sequence. For each frame of the video sequence, the methods further comprise encoding, by a degradation encoder, a video content associated with the frame into a latent vector. The latent vector is a representation of the degradation present in the video content; the degradation present in the video content includes one or more degradation types. Then, generating, by a backbone network, based on the latent vector and the video content, one or more feature maps, and, restoring, based on the one or more feature maps, the frame. 
     Aspects disclosed herein also describe restoration systems for video remastering. The systems comprise at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the processor to receive, by the system, a video sequence. For each frame of the video sequence, the instructions further cause the processor to encode, by a degradation encoder, a video content associated with the frame into a latent vector. The latent vector is a representation of the degradation present in the video content; the degradation present in the video content includes one or more degradation types. Then, to generate, by a backbone network, based on the latent vector and the video content, one or more feature maps, and to restore, based on the one or more feature maps, the frame. 
     Further, aspects disclosed herein describe a non-transitory computer-readable medium comprising instructions executable by at least one processor to perform methods for video remastering by a restoration system. The methods comprise receiving, by the system, a video sequence. For each frame of the video sequence, the methods further comprise encoding, by a degradation encoder, a video content associated with the frame into a latent vector. The latent vector is a representation of the degradation present in the video content; the degradation present in the video content includes one or more degradation types. Then, generating, by a backbone network, based on the latent vector and the video content, one or more feature maps, and, restoring, based on the one or more feature maps, the frame. 
       FIG.  1    is a block diagram of an example device  100 , based on which one or more features of the disclosure can be implemented. The device  100  can be, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. The device  100  includes a processor  102 , an accelerated processing unit (APU)  104 , memory  106 , storage  116 , an input device  108 , and an output device  110 . The device  100  can also include an input driver  112  and an output driver  114 . In an aspect, the device  100  can include additional components not shown in  FIG.  1   . 
     The processor  102  can include a central processing unit (CPU) or one or more cores of CPUs. The APU  104  can represent a highly parallel processing unit, a graphics processing unit (GPU), or a combination thereof. The processor  102  and the APU  104  may be located on the same die or on separate dies. The memory  106  can be located on the same die as the processor  102 , or can be located separately from the processor  102 . The memory  106  can include volatile or non-volatile memory, for example, random access memory (RAM), dynamic RAM (DRAM), a cache, or a combination thereof. 
     The storage  116  can include fixed or removable storage, for example, a hard disk drive, a solid-state drive, an optical disk, or a flash drive. The input device  108  can represent one or more input devices, such as a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for receipt of wireless IEEE 802 signals). The output device  110  can represent one or more output devices, such as a display, a speaker, a printer, a haptic feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless local area network card for transmission of wireless IEEE 802 signals). 
     The input driver  112  communicates with the processor  102  and the input device  108 , and facilitates the receiving of input from the input device  108  to the processor  102 . The output driver  114  communicates with the processor  102  and the output device  110 , and facilitates the sending of output from the processor  102  to the output device  110 . In an aspect, the input driver  112  and the output driver  114  are optional components, and the device  100  can operate in the same manner when the input driver  112  and the output driver  114  are not present. 
     The APU  104  can be configured to accept compute (dispatch) commands and graphics (draw) commands from processor  102 , to process those compute and graphics rendering commands, and/or to provide output to a display (output device  110 ). As described in further detail below, the APU  104  can include one or more parallel processing units configured to perform computations, for example, in accordance with a single instruction multiple data (SIMD) paradigm. A SIMD paradigm is one in which the same one or more instructions (associated with a computational task) are applied in parallel to different data elements. 
       FIG.  2    is a functional block diagram of an example system  200  for video remastering, based on which one or more features of the disclosure can be implemented. The system  200  includes processing components, such as a degradation encoder  210 , a degradation tuner  220 , a backbone network  230 , a denoising network  240 , and a super-resolution network  250 . The device  100  of  FIG.  1    may be employed to implement the functions described herein with respect to the system’s components  210 ,  220 ,  230 ,  240 ,  250 . Modules associated with these components may be stored in memory  106 , retrieved from storage  116 , executed by one or more processors  102  or APU  104 , and may be implemented by hardware, firmware, or software. Input (legacy) video  205  may be fed into the degradation encoder  210 , from which a degradation representation (a latent vector)  215  may be encoded. Optionally, the encoded degradation representation  215  may be adjusted by the degradation tuner  220 . Then, the (optionally adjusted) degradation representation  225  may be used by the backbone network  230  to generate one or more feature maps  235 . To restore the input video  205 , based on the feature maps  235 , the denoising network  240  may generate a denoised version  245  of the input video  205 , and the super-resolution network  250  may generate a denoised and upscaled version  255  of the input video  205 . The backbone network  230  and the denoising and super-resolution networks  240 ,  250  may be part of a network architecture, starting with several shared layers -namely, a backbone network – that feed multiple heads of specialized networks. Hence, the backbone network  230  may learn features that are beneficial for different restoration tasks (top-level features) and the heads may learn features that are beneficial for specific restoration tasks (low-level features), such as the denoising  240  and the super-resolution  250  restoration tasks. 
     To carry out the video restoration, the remastering system  200  may utilize a degradation model that is trained according to principles of contrastive learning. The degradation model may be used to remove (and, optionally, adjust) content degradation that is typically present in the video content of a legacy film, including, for example, scratches, noise, and implicit blur that may exist in low-resolution films. A degradation model may be formulated as follows: 
     
       
         
           
             y 
             = 
             S 
             ∘ 
             
               
                 
                   
                     x 
                     ∗ 
                     k 
                   
                 
                 
                   ↓ 
                   s 
                 
                 + 
                 n 
               
             
             , 
           
         
       
     
      where, y is a low-resolution degraded input video to be restored into a high-resolution output video x. As modeled, x (a ground-truth of the output video) is first degraded by a blurring operation, employed by a convolution (denoted by ∗) with a blur kernel k, and, then, by a down-sampling operation (denoted by ↓)by a factor s. The output video x is further degraded by adding noise, n, followed by a scratching operation (denoted by °). The scratching may be employed by a mask S that sets the values of randomly selected pixels in the degraded video ((x ∗ k) ↓ s  + n) to 0. Based on such a model and based on contrastive learning principles, the degradation encoder  210  may be trained to produce a latent vector  215  that discriminatively characterizes the degradation present in an input video y  205 . Such a latent vector may then be used by the backbone network  230  to extract features (or one or more feature maps)  235 . Feature maps  235  generated by the backbone network  230  may then be used to generate both a low-resolution denoised video  245  by the denoising network  240  and a denoised and up-scaled video  255  by the super-resolution network  250 ,as illustrated by the system  200  of  FIG.  2   . Additionally, degradation parameters – such as the modeled blur kernel k and the additive noise n – can be decoded from the latent vector and can be used, by the degradation tuner  220 , to adjust the resulting restored video  245 ,  255 , as further discussed below. 
       FIG.  3    is a functional block diagram of an example system  300  for tuning video restoration, based on which one or more features of the disclosure can be implemented. The system  300  may be employed by the degradation tuner  220  of  FIG.  2    to artistically adjust the blurring and the level of noise in the restored video  245 ,  255 . The system  300  may include a kernel encoder  310 , a noise encoder  320 , respective adjusters  330 ,  340 , and a mutator  350 . The kernel encoder  310  and the noise encoder  320  may be trained to generate, based on a latent vector  305 , estimates for the degradation parameters – that is, a blur kernel estimate k̂  315  and a noise estimate n̂  325 , respectively. The training of the kernel encoder  310  and the noise encoder  320  is further described in reference to  FIG.  5   . The adjusters  330 ,  340  may represent any means by which the generated blur kernel estimate  315  or noise estimate  325  may be tuned, resulting in an adjusted blur kernel  335  or an adjusted noise level  345 . For example, the adjusters  330 ,  340  may represent a graphical user interface with which a user may tune the blur kernel estimate  315  or the noise estimate  325 . The mutator  350  is trained to generate, out of the adjusted degradation parameters  335 ,  345 , a corresponding altered latent vector  355 . The training of the mutator  350  is further described in reference to  FIG.  6   . The altered latent vector  355  (that is, the output  225  of the degradation tuner  220 ) may then be used in lieu of the latent vector  215  by the backbone network  230  of the remastering system  200  of  FIG.  2   . 
     As disclosed herein, the degradation encoder  210  is configured to learn to extract, from the content of the input video  205 , degradation representations (latent vectors) that are discriminative. That is, two latent vectors of similarly degraded respective video contents will be located in close proximity in the representation space, while two latent vectors of differently degraded respective video content will be located in remote proximity in the representation space. Furthermore, the discriminative nature of the representation space should not be content dependent – the training of the degradation encoder  210  should provide a clear decoupling between the video content and the degradation that may exist in the video. Thus, in training the degradation encoder  210 , an important objective is to disentangle the degradation present in the training video samples from the content of those training video samples. The training of the degradation encoder  210  of  FIG.  2    is further described in reference to  FIG.  4   . 
       FIG.  4    is a functional block diagram of an example system  400  for training the degradation encoder  210 , based on which one or more features of the disclosure can be implemented. The training of the encoder  210  may be carried out using a set V of training videos from which training samples may be extracted. For example, a training sample extracted from a video x p  may be a video content (also referred to herein as an image content) that is associated with a frame of the video x p  (for simplicity, such a training sample is referred to herein as x p ). The extracted training samples may be degraded by a wide range of degradations, defined as follows. A set D of degradations: d i  ∈ D may be formed, parametrized by blur kernels k i  and noise levels n i . Accordingly, degrading a training sample x p  by degradation d i  results in a respective degraded sample  
     
       
         
           
             
               y 
               p 
               i 
             
           
         
       
     
     , in accordance with 
     
       
         
           
             
               y 
               p 
               i 
             
             = 
             
               d 
               i 
             
             
               
                 
                   x 
                   p 
                 
               
             
             = 
             
               x 
               p 
             
             ∗ 
             
               k 
               i 
             
             + 
             
               n 
               i 
             
           
         
       
     
      . Encoding the degraded sample  
     
       
         
           
             
               y 
               p 
               i 
             
           
         
       
     
     by the encoder  210 , denoted E d , results in a latent vector  
     
       
         
           
             
               z 
               p 
               i 
             
             = 
             
               E 
               d 
             
             
               
                 
                   y 
                   p 
                   i 
                 
               
             
           
         
       
     
     . In training the network parameters of E d , learning is focused on generating latent vectors that discriminately represent degradations that are applied to respective training samples, as further disclosed below. 
     Training video samples are typically captured with different camera exposure levels, by sensors of various resolutions, that output images with various additive noise levels. Therefore, these samples already contain inherent degradation before the application of the additional degradation (e.g., 
     
       
         
           
             
               y 
               p 
               i 
             
             = 
             
               d 
               i 
             
             ( 
             
               x 
               p 
             
             ) 
           
         
       
     
     ). Separating between these two sources of degradation (the inherent and the applied ones) is an ill-posed problem. Therefore, as disclosed herein, the degradation encoder  210  is trained by pairs of degraded video samples, where each pair is produced by a video sample that is degraded differently. By doing so, the learning is focused on differences between degradations introduced to video samples during the training (the applied degradations), rather than focusing on differences between degradations already present in the video samples (the inherent degradations). 
     Accordingly, to train the encoder  210 , two pairs of degradations are sampled from D, for example, (d i , d j ) and (d k , d l ), and a pair of videos x p  and x q  are sampled from a training video set. Then, the two pairs of degradations are applied to the pair of videos, and, then, the degraded videos are encoded, as follows: 
     
       
         
           
             
               x 
               p 
             
             → 
             
               
                 
                   d 
                   i 
                 
                 
                   
                     
                       x 
                       p 
                     
                   
                 
                 , 
                 
                   d 
                   j 
                 
                 
                   
                     
                       x 
                       p 
                     
                   
                 
               
             
             → 
             
               
                 
                   y 
                   p 
                   i 
                 
                 , 
                 
                   y 
                   p 
                   j 
                 
               
             
             → 
             
               
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       p 
                       i 
                     
                   
                 
                 , 
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       p 
                       j 
                     
                   
                 
               
             
             → 
             
               
                 
                   z 
                   p 
                   i 
                 
                 , 
                 
                   z 
                   i 
                   j 
                 
               
             
             , 
           
         
       
     
     
       
         
           
             
               x 
               q 
             
             → 
             
               
                 
                   d 
                   i 
                 
                 
                   
                     
                       x 
                       q 
                     
                   
                 
                 , 
                 
                   d 
                   j 
                 
                 
                   
                     
                       x 
                       q 
                     
                   
                 
               
             
             → 
             
               
                 
                   y 
                   q 
                   i 
                 
                 , 
                 
                   y 
                   q 
                   j 
                 
               
             
             → 
             
               
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       q 
                       i 
                     
                   
                 
                 , 
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       q 
                       i 
                     
                   
                 
               
             
             → 
             
               
                 
                   z 
                   q 
                   i 
                 
                 , 
                 
                   z 
                   q 
                   j 
                 
               
             
             , 
           
         
       
     
     
       
         
           
             
               x 
               p 
             
             → 
             
               
                 
                   d 
                   k 
                 
                 
                   
                     
                       x 
                       p 
                     
                   
                 
                 , 
                 
                   d 
                   l 
                 
                 
                   
                     
                       x 
                       p 
                     
                   
                 
               
             
             → 
             
               
                 
                   y 
                   p 
                   k 
                 
                 , 
                 
                   y 
                   p 
                   l 
                 
               
             
             → 
             
               
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       p 
                       k 
                     
                   
                 
                 , 
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       p 
                       l 
                     
                   
                 
               
             
             → 
             
               
                 
                   z 
                   p 
                   k 
                 
                 , 
                 
                   z 
                   p 
                   l 
                 
               
             
             . 
           
         
       
     
      Note that the pairs  
     
       
         
           
             
               
                 
                   z 
                   p 
                   i 
                 
                 , 
                 
                   z 
                   p 
                   j 
                 
               
             
               
             and 
               
             
               
                 
                   z 
                   q 
                   i 
                 
                 , 
                 
                   z 
                   q 
                   j 
                 
               
             
           
         
       
     
     are obtained by degrading two different videos x p  and x q , respectively, with the same pair of degradations (d i , d j ). Therefore, they form a positive example. The pairs  
     
       
         
           
             
               
                 
                   z 
                   p 
                   k 
                 
                 , 
                 
                   z 
                   p 
                   l 
                 
               
             
               
             and 
               
             
               
                 
                   z 
                   q 
                   i 
                 
                 , 
                 
                   z 
                   q 
                   j 
                 
               
             
           
         
       
     
     are obtained by degrading two different videos x p  and x q , with different pairs of degradations (d k , d l ) and (d i , d j ). Therefore, they form a negative example. And, the pairs  
     
       
         
           
             
               
                 
                   z 
                   p 
                   i 
                 
                 , 
                 
                   z 
                   p 
                   j 
                 
               
             
               
             and 
               
             
               
                 
                   z 
                   p 
                   k 
                 
                 , 
                 
                   z 
                   p 
                   l 
                 
               
             
           
         
       
     
     are obtained by degrading the same video x p  with different pairs of degradations (d i , d j ) and (d k , d l ). Therefore, they form a hard-negative example. Positive, negative, and hard-negative examples are utilized herein to force a contrastive learning of latent vectors that focuses on differences in degradations, rather than differences in content. 
       FIG.  4    illustrates this process of generating encoded pairs of degraded video samples, in accordance with equations (2)-(4). To train the degradation encoder  210 , samples of high-resolution video content  405 , sampled from the set V of training videos, may be used. As illustrated, image content  405 . 1  of video x p  is degraded by degradation operators  410 . 1  and  410 . 2 , resulting in a pair of degraded content d 1 (x p ) and d 2 (x p ) – degraded by a pair of degradations d 1  and d 2  (sampled from the degradation set D), respectively. The pair of degraded content, d 1 (x p ) and d 2 (x p ), may then be subsampled and encoded  420 . 1 ,  420 . 2 , resulting in a pair of latent vectors z 1 (x p ) and z 2 (x p ), in accordance with equation (2). Image content  405 . 2  of video xq is degraded by degradation operators  410 . 3  and  410 . 4 , resulting in a pair of degraded content d 1 (x q ) and d 2 (x q ) – degraded by the pair of degradations d 1  and d 2 , respectively. The pair of degraded content, d 1 (x q ) and d 2 (x q ), may then be subsampled and encoded  420 . 3 ,  420 . 4 , resulting in a pair of latent vectors z 1 (x q ) and z 2 (x q ), in accordance with equation (3). Image content  405 . 3  of video x p  (same content as  405 . 1 ) is degraded by degradation operators  410 . 5  and  410 . 6 , resulting in a pair of degraded content d 3 (x p ) and d 4 (x p ) – degraded by a sampled pair of degradations d 3  and d 4 , respectively. The pair of degraded content, d 3 (x p ) and d 4 (x p ), may then be subsampled and encoded  420 . 5 ,  420 . 6 , resulting in a pair of latent vectors z 3 (x p ) and z 4 (x p ), in accordance with equation (4). In the same manner, pairs of latent vectors (in accordance with equations (2), (3), and (4)) may be generated from image content  405  sampled from the training video set V to be used in the training of the degradation encoder  210 , as further disclosed below. 
     The degradation encoder  210  may be trained based on contrastive learning. To that end, a MoCo framework may be used, where encoded pairs of degraded samples,  
     
       
         
           
             
               
                 
                   z 
                   p 
                   i 
                 
                 , 
                 
                   z 
                   p 
                   j 
                 
               
             
             , 
               
             
               
                 
                   z 
                   q 
                   i 
                 
                 , 
                 
                   z 
                   q 
                   j 
                 
               
             
             , 
               
             and 
               
             
               
                 
                   z 
                   p 
                   k 
                 
                 , 
                 
                   z 
                   p 
                   l 
                 
               
             
             , 
           
         
       
     
     are concatenated and fed into multilayer perceptron (MLP) projection heads  430 , denoted F, as follows: 
     
       
         
           
             
               ψ 
               p 
               
                 i 
                 j 
               
             
             = 
             F 
             
               
                 
                   z 
                   p 
                   i 
                 
                 , 
                 
                   z 
                   p 
                   j 
                 
               
             
             , 
             
               ψ 
               q 
               
                 i 
                 j 
               
             
             = 
             F 
             
               
                 
                   z 
                   q 
                   i 
                 
                 , 
                 
                   z 
                   q 
                   j 
                 
               
             
             , 
               
             and 
               
             
               ψ 
               p 
               
                 k 
                 l 
               
             
             = 
             F 
             
               
                 
                   z 
                   p 
                   k 
                 
                 , 
                 
                   z 
                   p 
                   l 
                 
               
             
             . 
           
         
       
     
      In a contrastive learning, the objective is to optimize for  
     
       
         
           
             
               ψ 
               p 
               
                 i 
                 j 
               
             
               
             and 
               
             
               ψ 
               q 
               
                 i 
                 j 
               
             
           
         
       
     
     that are similar, since they share the same degradation (in spite of the different video contents) and to optimize for  
     
       
         
           
             
               ψ 
               p 
               
                 i 
                 j 
               
             
               
             and 
               
             
               ψ 
               p 
               
                 k 
                 l 
               
             
           
         
       
     
     that are dissimilar, since they do not share the same degradations. To achieve that objective, a cost metric L c  is minimized  440 , such as the InfoNCE loss function that is defined as follows: 
     
       
         
           
             
               L 
               c 
             
             = 
             
               ∑ 
               
                 
                     
                   
                     p 
                     , 
                     q 
                   
                   V 
                 
                 
                   ∑ 
                   
                     
                         
                       
                         i 
                         , 
                         j 
                       
                       D 
                     
                   
                 
               
             
             − 
             log 
             
               
                 
                   e 
                   
                     
                       
                         
                           ψ 
                           p 
                           
                             i 
                             j 
                           
                         
                         ⋅ 
                         
                           ψ 
                           q 
                           
                             i 
                             j 
                           
                         
                         / 
                         τ 
                       
                     
                   
                 
               
               
                 
                   e 
                   
                     
                       
                         
                           ψ 
                           p 
                           
                             i 
                             j 
                           
                         
                         ⋅ 
                         
                           ψ 
                           p 
                           
                             k 
                             l 
                           
                         
                         / 
                         τ 
                       
                     
                   
                 
                 + 
                 
                   ∑ 
                   
                     
                         
                       
                         t 
                         = 
                         1 
                       
                       N 
                     
                   
                 
                 
                   e 
                   
                     
                       
                         
                           ψ 
                           p 
                           
                             i 
                             j 
                           
                         
                         ⋅ 
                         
                           ψ 
                           τ 
                         
                         / 
                         τ 
                       
                     
                   
                 
               
             
           
         
       
     
      where N is the number of samples in the MoCo queue, V is the set of training videos from which image contents  405  are sampled, D is the set of degradations, τ is a temperature parameter, and the operator “ · ” denotes the dot product between two vectors. The metric L c  may be minimized  440 , for example, by applying gradient descent optimization techniques. 
     As mentioned above, positive, negative, and hard-negative examples may be used for contrastive learning. As illustrated in  FIG.  4   , latent vectors  
     
       
         
           
             
               ψ 
               p 
               
                 12 
               
             
           
         
       
     
     and  
     
       
         
           
             
               ψ 
               q 
               
                 12 
               
             
           
         
       
     
     (outputs of MLP  430 . 1  and  430 . 2 , respectively) form a positive example  450 , latent vectors  
     
       
         
           
             
               ψ 
               q 
               
                 12 
               
             
               
             and 
               
             
               ψ 
               p 
               
                 34 
               
             
           
         
       
     
     (outputs of MLP  430 . 2  and  430 . 3 , respectively) form a negative example  470 , and latent vectors  
     
       
         
           
             
               ψ 
               p 
               
                 12 
               
             
               
             and 
               
             
               ψ 
               p 
               
                 34 
               
             
           
         
       
     
     (outputs of MLP  430 . 1  and  430 . 3 , respectively) form a hard-negative example  460 . These examples are fed into the optimizer  440 . The network parameters of the degradation encoder E d   210  may be learned by an optimization process, employed by the optimizer  440 . That is, by minimizing the cost metric  
     
       
         
           
             
               L 
               c 
             
           
         
       
     
      (e.g., the InfoNCE loss function of equation (6)). 
       FIG.  5    is a functional block diagram of an example system  500  for training a blur kernel encoder and a noise encoder, based on which one or more features of the disclosure can be implemented.  FIG.  5    illustrates the training of the kernel encoder  310  and the noise encoder  320  that may be used for refining the restored video output  245 ,  255 , as described in reference to  FIG.  2    and  FIG.  3   . The system  500  includes a degradation operator  510  that degrades samples of image content  505 , x p , sampled from the set V of training videos, according to respective degradations, d i , sampled from the set D of degradations. Degraded samples  515  may then be fed into a degradation encoder  520  (trained as described in reference to  FIG.  4   ). Then, the encoded samples  525 , E d  (d i (x p )), may be used for the training of a kernel encoder  530 , denoted E k , and a noise encoder  540 , denoted E n . To that end, the encoders  530 ,  540  (e.g., implemented by MLPs) are trained by optimizing  550  respective cost functions, such as cost functions  
     
       
         
           
             
               L 
               k 
             
           
         
       
     
     and  
     
       
         
           
             
               L 
               n 
             
           
         
       
     
     : 
     
       
         
           
             
               L 
               
                   
                 k 
               
             
             = 
             
               
                 ∑ 
                 p 
                 V 
               
               
                 
                   
                     ∑ 
                     i 
                     D 
                   
                   
                     
                       
                         
                           E 
                           k 
                         
                         
                           
                             
                               E 
                               d 
                             
                             
                               
                                 
                                   d 
                                   i 
                                 
                                 
                                   
                                     
                                       x 
                                       p 
                                     
                                   
                                 
                               
                             
                           
                         
                           
                         − 
                           
                         
                           k 
                           i 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     
       
         
           
             
               L 
               
                   
                 n 
               
             
             = 
             
               
                 ∑ 
                 p 
                 V 
               
               
                 
                   
                     ∑ 
                     i 
                     D 
                   
                   
                     
                       
                         
                           E 
                           n 
                         
                         
                           
                             
                               E 
                               d 
                             
                             
                               
                                 
                                   d 
                                   i 
                                 
                                 
                                   
                                     
                                       x 
                                       p 
                                     
                                   
                                 
                               
                             
                           
                         
                           
                         − 
                           
                         
                           n 
                           i 
                         
                       
                     
                   
                 
               
             
           
         
       
     
      Thus, the encoded samples E d  (d i (x p )) 525  are supplied to the encoders E k  and E n  and respective outputs E k  (E d  (d i (x p ))) and E n  (E d  (d i (x p ))) are trained to match the respectively applied distortion parameters  512 , that is, the blur kernel k i  and the noise n i . 
     In an aspect, the training of the degradation encoder  210 ,  420 , E d , the kernel encoder  310 ,  530 , E k , and the noise encoder  320 ,  540 , E n , may be carried out concurrently by optimizing the cost functions in equations (6)-(8) jointly: 
     
       
         
           
             L 
             = 
             
               λ 
               c 
             
             
               L 
               
                   
                 c 
               
             
             + 
             
               λ 
               k 
             
             
               L 
               
                   
                 k 
               
             
             + 
             
               λ 
               n 
             
             
               L 
               
                   
                 n 
               
             
           
         
       
     
      where λ c , λ k , λ n  weigh the respective contributions of  
     
       
         
           
             
               L 
               c 
             
           
         
       
     
     ,  
     
       
         
           
             
               L 
               k 
             
           
         
       
     
     , and  
     
       
         
           
             
               L 
               n 
             
           
         
       
     
      to the overall cost function  
     
       
         
           L 
         
       
     
      (e.g., λ c  = 1, λ k  = 400, λ n  = 1). 
       FIG.  6    is a functional block diagram of an example system  600  for training a mutator, based on which one or more features of the disclosure can be implemented. The mutator  630  is trained to provide fine-grained control over the process of restoring a degraded video  205 . Once trained, the mutator  330  can be used by the degradation tuner  220  to artistically adjust the blurring and the noise level in the restored video  245 ,  255  (as discussed in reference to  FIG.  3   ). As illustrated by  FIG.  6   , high-resolution image content  605 , x p , sampled from the set V of training videos, is degraded  610 . 1 , according to degradation d i , resulting in degraded content  615 . 1  d i (x p ). Similarly, the high-resolution image content  605 , x p , is degraded  610 . 2 , according to degradation d j , resulting in degraded content  615 . 2  d j (x p ). Then, both degraded contents are subsampled and encoded by encoders  620 . 1  and  620 . 2 , into latent vector zp = E d  (d i (x p ))  625 . 1  and latent vector  
     
       
         
           
             
               z 
               p 
               j 
             
             = 
             
               E 
               d 
             
             
               
                 
                   d 
                   j 
                 
                 
                   
                     
                       x 
                       p 
                     
                   
                 
               
             
           
         
       
     
       625 . 2 , respectively. The mutator  630  is trained to provide a latent vector  630  that corresponds to new parameters k i  and n i  (of d i   612 ) that deviate from parameters k j  and n j , to which the latent vector  625 . 2  corresponds. Accordingly, the training of the mutator  630  is performed by optimizing  640  the cost function  
     
       
         
           
             
               L 
               m 
             
           
         
       
     
     , as follows: 
     
       
         
           
             
               L 
               m 
             
             = 
             
               
                 ∑ 
                 p 
                 V 
               
               
                 
                   
                     ∑ 
                     
                       i 
                       , 
                       j 
                     
                     D 
                   
                   
                     
                       
                         M 
                         
                           
                             
                               z 
                               p 
                               j 
                             
                             , 
                             
                               k 
                               i 
                             
                             , 
                             
                               n 
                               i 
                             
                           
                         
                         − 
                         
                           z 
                           p 
                           i 
                         
                       
                     
                   
                 
               
             
           
         
       
     
      Hence, the mutator  630  (or  350 ), when presented with an adjusted degradation parameter  612  (or  335 ,  345 ) and a current latent vector  625 . 2  (or  305 ), is trained to produce an altered latent vector  630  (or  355 ) that matches a latent vector  625 . 1  that represents a degradation  615 . 1  that is present in a video content when the video content is degraded by the adjusted degradation parameter  612 . 
     As illustrated in  FIG.  2   , the remastering system  200  receives as an input a corrupted (degraded) video y p  to be restored. Image contents representing consecutive frames of y p  are encoded  210 , resulting in corresponding representations of the degradation present in the video. For example, for each frame of y p , the degradation encoder  210  may generate a respective latent vector that discriminatively represents the degradation that may exist in image content corresponding to that frame. As described above, if desired, that latent vector may be adjusted by the degradation tuner  220 . The (optionally adjusted) latent vector may then be used by the backbone network  230 , denoted R B , for conditioning the restoration performed by the denoising network  240 , denoted R DN , and the super-resolution network  250 , denoted R SR . In an aspect, the R DN  and the R SR  networks are task-specific networks that branch from a shared R B  network. In this way, feature maps, generated by the R B  network, can be simultaneously learned for different restoration tasks, while features specific for denoising and super-resolution can be learned by the R DN  and the R SR  networks, respectively. 
     Hence, a restoration process begins by encoding a corrupted input y p  into a latent vector  
     
       
         
           
             
               E 
               d 
             
             
               
                 
                   y 
                   p 
                   i 
                 
               
             
           
         
       
     
     . Then, both the corrupted video y p  and the latent vector  
     
       
         
           
             
               E 
               d 
             
             
               
                 
                   y 
                   p 
                   i 
                 
               
             
           
         
       
     
     are provided to the restoration backbone R B   230 , based on which the restoration backbone may generate feature maps. These feature maps are fed into the denoising network R DN  and the super-resolution network R SR . Thus, two outputs can be produced by the system  200 . A first output is the denoised low-resolution video  245  (that is, an estimate of the original low-resolution video), generated by the denoising network R DN   240 . A second output is the denoised high-resolution video  255 , generated by the super-resolution network R SR   250 . To generate the two outputs, the system  200  may first remove scratches that may be present in the video y p . Thus, during training, the networks R SR  and R DN  may be trained to minimize the cost functions  
     
       
         
           
             
               L 
               
                 S 
                 R 
               
             
           
         
       
     
      and  
     
       
         
           
             
               L 
               
                 D 
                 N 
               
             
           
         
       
     
     , as follows: 
     
       
         
           
             
               L 
               
                 S 
                 R 
               
             
             = 
             
               
                 ∑ 
                 p 
                 V 
               
               
                 
                   
                     ∑ 
                     i 
                     D 
                   
                   
                     
                       
                         
                           R 
                           
                             S 
                             R 
                           
                         
                         
                           
                             
                               E 
                               d 
                             
                             
                               
                                 
                                   y 
                                   p 
                                   i 
                                 
                               
                             
                             , 
                             
                               y 
                               p 
                               i 
                             
                           
                         
                         − 
                         
                           
                             x 
                             ^ 
                           
                           p 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     
       
         
           
             
               L 
               
                   
                 D 
                 N 
               
             
             = 
             
               
                 ∑ 
                 p 
                 V 
               
               
                 
                   
                     ∑ 
                     i 
                     D 
                   
                   
                     
                       
                         
                           R 
                           
                             D 
                             N 
                           
                         
                         
                           
                             
                               E 
                               d 
                             
                             
                               
                                 
                                   y 
                                   p 
                                   i 
                                 
                               
                             
                             , 
                             
                               y 
                               p 
                               i 
                             
                           
                         
                         − 
                         
                           
                             
                               
                                 x 
                                 ^ 
                               
                               p 
                             
                             ∗ 
                             
                               k 
                               i 
                             
                           
                         
                           
                         
                           ↓ 
                           s 
                         
                       
                     
                     . 
                   
                 
               
             
           
         
       
     
      Where,  
     
       
         
           
             
               R 
               
                 S 
                 R 
               
             
             
               
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       p 
                       i 
                     
                   
                 
                 , 
                 
                   y 
                   p 
                   i 
                 
               
             
           
         
       
     
     is the output of super-resolution network  250  that is optimized to match x̂ p , an enhanced version of x p  (the high-resolution ground-truth video). The enhancement may be implemented by a filter, for example, to sharpen the content of x p .  
     
       
         
           
             
               R 
               
                 D 
                 N 
               
             
             
               
                 
                   E 
                   d 
                 
                 
                   
                     
                       y 
                       p 
                       i 
                     
                   
                 
                 , 
                 
                   y 
                   p 
                   i 
                 
               
             
           
         
       
     
     is the output of the denoising network  240  that is optimized to match (x̂ p  ∗ k i ) ↓ s , a down-sampled version of x̂ p  (generated by first blurring x̂ p  by a blur kernel k i  and, then, down-sampling by a scale s). 
     In an aspect, the models disclosed herein may be fine-tuned jointly by optimizing the parameters of the respective networks. Thus, the cost function  
     
       
         
           L 
         
       
     
      of equation (9) can be extended as follows: 
     
       
         
           
             L 
             = 
             
               λ 
               
                 S 
                 R 
               
             
             
               L 
               
                   
                 S 
                 R 
               
             
             + 
             
               λ 
               
                 D 
                 N 
               
             
             
               L 
               
                   
                 D 
                 N 
               
             
             + 
             
               λ 
               c 
             
             
               L 
               
                   
                 c 
               
             
             + 
             
               λ 
               k 
             
             
               L 
               
                   
                 k 
               
             
             + 
             
               λ 
               n 
             
             
               L 
               
                   
                 n 
               
             
           
         
       
     
      Where λ SR , λ DN , λ c , λ k , and λ n  weigh the respective contributions of  
     
       
         
           
             
               L 
               
                 S 
                 R 
               
             
           
         
       
     
     ,  
     
       
         
           
             
               L 
               
                 D 
                 N 
               
             
           
         
       
     
     ,  
     
       
         
           
             
               L 
               c 
             
           
         
       
     
     ,  
     
       
         
           
             
               L 
               k 
             
           
         
       
     
     , and  
     
       
         
           
             
               L 
               n 
             
           
         
       
     
      to the overall cost function  
     
       
         
           L 
         
       
     
      (e.g., λ SR  = 1, λ DN  = 1, λ c  = 1, λ k  = 400, λ n  = 1). 
       FIG.  7    is a flowchart of an example method  700  for video remastering, based on which one or more features of the disclosure can be implemented. The method  700  may begin with receiving, by the system  200 , a video sequence, in step  710 . The video sequence may be degraded by one or more degradation types (e.g., scratches, noise, or blur). Then, for each frame of the video sequence, the method  700  may employ a process for restoring the frame, as follows. In step  720 , a video content associated with a frame of the video sequence  205  may be encoded, by a degradation encoder  210 , into a latent vector. A video content associated with a frame may be an image content from the frame or an image content from frames within a temporal neighborhood centered at that frame, where the image content may be derived from one or more channels of a frame or a frame region, for example. The latent vector is a representation of the degradation present in the video content - a degradation which may include one or more degradation types. In step  730 , one or more feature maps may be generated by a backbone network  230  based on the latent vector and the video content associated with the frame. And, in step  740 , the frame is restored based on the one or more feature maps. In an aspect, the frame may be restored by a denoising network  240 , resulting in a denoised frame. In another aspect, the frame may be restored by a super-resolution network  250 , resulting in a denoised frame at a higher resolution frame. 
     Further, according to the method  700 , for each frame of the video sequence, the latent vector may be tuned  220  and the generation of the one or more feature maps may be based on the tuned latent vector. In an aspect, the tuning may be performed by estimating, based on the latent vector, a degradation parameter (such as the blur kernel and noise level); adjusting the estimate of the degradation parameter; and tuning, based on the adjusted estimates of the degradation parameter, the latent vector (e.g., as described in reference to  FIG.  3   ). The degradation parameter may be estimated by training a degradation parameter encoder (e.g., kernel encoder  530  and noise encoder  540 , as described in reference to  FIG.  5   ). The tuning of the latent vector may be done by altering the latent vector by a mutator  350 , where the mutator may be trained to produce an altered latent vector that matches a latent vector that represents a degradation that is present in a video content when the video content is degraded by the adjusted estimate of the degradation parameter (e.g., as described in reference to  FIG.  6   ). 
     It should be understood that many variations are possible based on the disclosure herein. The techniques disclosed herein for restoring degraded input video are not limited to removal of scratches, denoising, and upscaling the input video. Rather, the disclosed techniques can be similarly applied to other types of video degradations, such as those caused by video compression and interlacing operations. Although features and elements are described above in particular combinations, each feature or element can be used alone without the other features and elements or in various combinations with or without other features and elements. 
     The methods provided can be implemented in a general-purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general-purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediary data including netlists (such as instructions capable of being stored on a computer readable media). The results of such processing can be maskworks that are then used in a semiconductor manufacturing process to manufacture a processor which implements aspects of the embodiments. 
     The methods or flow charts provided herein can be implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general-purpose computer or a processor. Examples of a non-transitory computer-readable medium include read only memory (ROM), random-access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs).