Implicit Transformer Network for Screen Content Image Continuous Super-Resolution

1 Introduction . Nowadays , screen content images are becoming ubiquitous due to the wide application of screen sharing and wireless display . Meanwhile , due to the limited bandwidth , screen content images received by users may be in low-resolution ( LR ) and users may need to zoom in the content for detail inspection . Therefore , screen content image super-resolution ( SCI SR ) is to improve the quality of LR SCIs . However , different from natural scene images , SCIs are dominated by the contents generated or rendered by computers , such as texts and graphics . Such contents highly demanded are characterized by thin and sharp edges , little color variance , and high contrast . In contrast , the natural scene images are relatively smooth , and contain rich colors and textures . Conventional image SR methods designed for nature images are good at modeling the local smoothness of natural images other than the thin and sharp edges in SCIs . Very recently , Wang et al . [ 1 ] proposed a SR method for compressed screen content videos , which addressed the compression artifacts of screen content videos by introducing a distortion differential guided reconstruction module . However , their network is still composed of fully convolution layers without designing specific structures for the thin and sharp edges in screen contents . In addition , they utilize previous frames to help reconstruct the current frame , which makes it unsuitable for frame-wise SCI SR. ∗Corresponding author : Huanjing Yue 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . On the other hand , conventional SR methods are designed for discrete ( i.e. , several fixed ) magnification ratios [ 2 , 3 , 4 ] , making them hard to fit screens with various sizes . Recently , a few SR methods for continuous magnification have been proposed [ 5 , 6 ] . Hu et . al . [ 5 ] proposed to perform arbitrary-scale SR with a learnable up-sampling weight matrix based on meta-learning . The work LIIF [ 6 ] introduced the concept of implicit function [ 7 , 8 , 9 ] to image SR . The implicit function , which attempts to represent images with continuous coordinates and directly maps the coordinates to values , enables continuous magnification . In this work , we observe that convolution filters could be harmful to sharp and thin edges in SCIs since the weight sharing strategy makes them tend to produce a smooth reconstruction result . Therefore , we propose to render the pixel values by a point-to-point implicit function , which adapts to image content according to image coordinates and pixel features . Fortunately , this also enables us to perform continuous magnification for SCIs . We would like to point out that even with the point-to-point implicit function , reconstructing dense edges are still quite challenging . As shown in Fig . 1 , LIIF [ 7 ] can not reconstruct the dense edges well since it directly concatenates the pixel coordinates and features together to predict the pixel values , which is not optimal since the two variables have different physical meanings . As a departure , we reformulate the interpolation process as a transformer and introduce implicit mapping to model the relationship between pixel coordinates , which are used to aggregate the pixel feature . Our main contributions are summarized as follows . • First , we propose a novel Implicit Transformer Super-Resolution Network ( ITSRN ) for SCI SR . The LR and HR image coordinates are termed as the “ key ” and “ query ” , respectively . Correspondingly , the LR image pixel features are termed as “ value ” . In this way , we can infer pixel values by an implicit transformer , where implicit means we model the relationship between LR and HR images in terms of coordinates instead of pixel values . • Second , instead of directly concatenating the coordinates and pixel features to predict the pixel value , we propose to predict the transform weights with query and key coordinates via nonlinear mapping , which are then used to transform the pixel features to pixel values . In addition , we propose an implicit position encoding to aggregate similar neighboring pixel values to the central pixel . • Third , we construct a benchmark dataset with various screen contents for SCI SR. Extensive experiments demonstrate that the proposed method outperforms the competitive continuous and discrete SR methods for both compressed and uncompressed screen content images . Fig . 1 presents an example of our SR results , which demonstrates that the proposed method is good at reconstructing thin and sharp edges for various magnification ratios . 2 Related Work . 2.1 Screen Content Processing . The screen content is generally dominated by texts and graphics rendered by computers , making the pixel distribution of screen contents totally different from that of natural scenes . Therefore , many works specifically designed for screen contents are proposed , such as screen content image quality assessment [ 10 , 11 , 12 , 13 ] , screen content video ( image ) compression [ 14 , 15 ] . However , there is still no work exploring screen content image SR . Very recently , Wang et al . [ 1 ] proposed screen content video SR , which reconstructed the current frame by taking advantage of the correlations between neighboring frames , making it can not deal with image SR . In addition , its main motivation is solving the SR problem when the videos are degraded by compression other than designing specific structures for continuously recovering thin and sharp edges in screen contents . In this work , we address this issue by introducing point-to-point implicit transformation . 2.2 Continuous Image Super-Resolution . Image SR refers to the task of recovering HR images from LR observations . Many deep learning based methods have been proposed for super-resolving the LR image with a fixed scale [ 16 , 3 , 2 , 17 , 18 , 4 , 19 , 20 ] . Since the screen contents are usually required to be displayed on screens with various sizes . Therefore , continuous SR is essential for screen contents . In recent years , several continuous image SR methods [ 5 , 6 ] are proposed in order to achieve arbitrary resolution SR. MetaSR [ 5 ] introduces a meta-upscale module to generate continuous magnification but it has limited performance in dealing with out-of-training-scale upsampling factors . LIIF [ 6 ] reformulates the SR process as an implicit neural representation ( INR ) problem , which achieves promising results for both in-distribution and out-of-distribution upsampling ratios . Inspired by LIIF , we utilize the point-to-point implicit function for SCI SR . 2.3 Implicit Neural Representation . Implicit Neural Representation ( INR ) usually refers to continuous and differentiable function ( e.g. , MLP ) , which can map coordinates to a certain signal . INR was widely used in 3D shape modeling [ 21 , 22 , 23 , 24 ] , volume rendering ( i.e. , neural radiance fields ( Nerf ) ) [ 9 , 25 ] , and 3D reconstruction [ 8 , 26 ] . Very Recently , LIIF [ 6 ] was proposed for continuous image representation , in which networks took image coordinates and the deep features around the coordinate as inputs , and then map them to the RGB value of the corresponding position . Inspired by LIIF , we propose an implicit transformer network to achieve continuous magnification while retaining the sharp edges of SCIs well . 2.4 Positional Encoding ( Mapping ) . Positional encoding is critical to exploit the position order of the sequence in transformer networks [ 27 , 28 , 29 ] . We have to utilize positional encoding to indicate the order of the sequence since there are no other modules to model the relative or absolute position information in the sequenceto-sequence model . In the literature , sine and cosine functions are used for positional encoding in transformer [ 27 ] . Coincidentally , we find that the Fourier feature ( combined with a set of sinusoids ) based positional mapping is used in the implicit function to improve the convergence speed and generalization ability [ 30 , 9 , 31 ] . Specifically , a Fourier feature mapping is applied on the input coordinates to map them to a higher dimensional hypersphere before going through the coordinatebased MLP . Although the two schemes are proposed based on different motivations , they show significant effects in representing position information , which further boost the final results . Inspired by them , we propose implicit positional encoding to model the relationship between neighboring pixel values . Here `` implicit '' means that we do not explicitly encode ( map ) the coordinates but encode the pixel values of neighboring coordinates . 3 Approach . Figure 2 illustrates the framework of the proposed ITSRN . In the following , we give details for the proposed implicit transformer network and implicit position encoding . 3.1 Implicit Transformer Network . Let ’ s first review the process of image interpolation . Suppose we have an LR image IL that needs to be upsampled . The pixel value of query point q ( i , j ) in the HR image IH is obtained by fusing pixel values of its closest key points k ( i′ , j′ ) in IL with a weighting matrix . Denoted by Q the query points in upsampled image , K the key points in the input LR image , and V the feature values on the corresponding key points . Then , the image interpolator can be reformulated as a transformer [ 27 ] . Different from the explicit transformer which takes pixel values as Q and K , the interpolation transformer deals with pixels ’ coordinates instead of their values . Inspired by the implicit function in NeRF [ 9 ] , which uses the pixel coordinates to generate RGB values , we reformulate the interpolation process as Implicit Transformer , and propose a novel Implicit Transformer Network for SCI SR . The super-resolution process as well as image interpolation is reformulated in terms of implicit function as Iq = Φ ( q , k , v ) , ( 1 ) where Iq is the target RGB value that need to be predicted , q is the query coordinate ( s ) in IH , k is the key coordinate ( s ) in IL , and v is the pixel value ( s ) or feature ( s ) corresponding to the key coordinate ( s ) . Note that both q and k are coordinates in the continuous image domain . Φ is a mapping function which maps coordinates and features to RGB values . In image interpolation , k is the neighboring key coordinate of q , v are the pixel value of k , Φ is the weighting matrix . In implicit function based SR method LIIF [ 6 ] , k is the nearest neighbor coordinate in IL for q in IH , and v is the corresponding pixel feature of k. LIIF directly concatenates v and the relative coordinate of q from k , and then utilize the nonlinear mapping Φ realized by a multi-layer perceptron ( MLP ) to render the pixel value Iq . It achieves promising results due to the strong fitting ability of the MLP . However , we observe that directly concatenating the pixel feature and the relative coordinates is not optimal since they have different physical meanings . To solve this problem , following the idea of transformer [ 27 ] , we reorganize Formula ( 1 ) as follows . Iq = Φ ( q , k , v ) = φ ( q , k ) v. ( 2 ) Different from the explicit transformer , here q and k are pixel coordinates other than pixel values . In other words , we are not learning the the RGB → RGB mapping , but the coordinate→ RGB mapping . Due to the continuity of coordinates , φ ( · ) is a continuous projection function , which computes the transform weights φ ( q , k ) to aggregate the feature v , i.e. , Iq is computed with the multiplication of φ and v . To further illustrate it , we decompose φ as : φ ( q , k ) = f ( δ ( q , k ) ) , ( 3 ) where the function δ produces a vector that represents the relationship between q and k. Then the mapping function f projects this vector into another vector , which is then multiplied with feature v as indicated in Eq . 2 . There are many ways to model the relation function δ , such as dot product and hadamard product commonly used in transformers [ 27 ] . Different from their approaches , in this paper , we utilize subtraction operation , i.e. , δ ( q , k ) = γ ( q ) − τ ( k ) , ( 4 ) where γ and τ can use trainable functions or identity mapping . In this work , we utilize identity mapping since it generates similar results as that of trainable functions . Inspired by ViT ’ s [ 32 ] [ class ] token , which is extra global information for classification , we augment the input coordinates with scale information , and name it as [ scale ] token . It represents the global magnification factor . With the [ scale ] token , the implicit transformer can take the shape of the query pixel as additional information for reconstructing the target RGB value . Although it is feasible to predict RGB values without [ scale ] token , it is not optimal since the predicted RGB value should not be completely independent of its shape 2 . The output of δ ( q , k ) is concatenated with the [ scale ] token , and is then fed to the mapping function f . In this way , φ in Formula ( 3 ) is reformulated as φ ( q , k ) = f ( [ δ ( q , k ) , s ] ) , ( 5 ) where s = ( sh , sw ) is a two-dimensional scale token , representing the upsampling scale along the row and column for the query pixel . [ δ ( q , k ) , s ] means the concatenation of δ ( q , k ) and s along the channel dimension . Note that , to avoid large coordinates in HR space , we normalize the coordinates as p ( i , j ) = [ −1 + 2i+ 1 H , −1 + 2i+ 1 W ] , i ∈ [ 0 , H − 1 ] , j ∈ [ 0 , W − 1 ] , ( 6 ) where ( i , j ) represents the spatial location within the RH×W space .

Summary: This paper addresses the problem of arbitrary screen content image super-resolution, which is quite different from natural image super-resolution.  The author proposed two datasets specifically designed for this scenario, which can greatly motivate research  effort into this field. Besides, the authors proposed Implicit Transformer Super-Resolution Network (ITSRN) to achieve arbitrary scaling for screen content images. The ITSRN is largely inspired by LIIF, the main novelty of ITSRN includes the proposed  implicit transformer and the implicit position encoding, these schemes are empirically evaluated to show its effectiveness.