Implicit Transformer Network for Screen Content Image Continuous Super-Resolution
Authors: Jingyu Yang
Sheng Shen
Huanjing Yue
Kun Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Implicit fuction
Transformer
Continuous super-resolution
Screen content images

Abstract: Nowadays, there is an explosive growth of screen contents due to the wide application of screen sharing, remote cooperation, and online education. To match the limited terminal bandwidth,  high-resolution (HR) screen contents may be downsampled and compressed.  At the receiver side, the super-resolution (SR)of low-resolution (LR) screen content images (SCIs) is highly demanded by the HR display or by the users to zoom in for detail observation.  However, image SR methods mostly designed for natural images do not generalize well for SCIs due to the very different image characteristics as well as the requirement of SCI browsing at arbitrary scales. To this end, we propose a novel Implicit Transformer Super-Resolution Network (ITSRN) for SCISR. For high-quality continuous SR at arbitrary ratios, pixel values at query coordinates are inferred from image features at key coordinates by the proposed implicit transformer and an implicit position encoding scheme is proposed to aggregate similar neighboring pixel values to the query one. We construct benchmark SCI1K and SCI1K-compression datasets withLR and HR SCI pairs.  Extensive experiments show that the proposed ITSRN significantly outperforms several competitive continuous and discrete SR methods for both compressed and uncompressed SCIs.

Implicit Transformer Network for Screen Content Image Continuous Super-Resolution
Jingyu Yang'
Sheng Shen'
Huanjing Yue!" Kun Li
School of Electrical and Information Engineering, Tianjin University
College of Intelligence and Computing, Tianjin University {yjy, codyshen, , muanjingg yue, yue, 1ik]@tju. edu. https://github com/ codsshee00000///SRRR
Abstract
Nowadays, there is an explosive growth of screen contents due to the wide ap- plication of screen sharing, remote cooperation, and online education. To match the limited terminal bandwidth, high-resolution (HR) screen contents may be downsampled and compressed. At the receiver side, the super-resolution (SR) low-resolution (LR) screen content images (SCIs) is highly demanded by the HR display by the users to zoom in for detail observation. However, image SR methods mostly designed for natural images do not generalize well for SCIs due to the very different image characteristics browsing at arbitrary scales. To this end, we propose a novel Implicit Transformer Super-Resolution Network (ITSRN) for SCISR. For high-quality continuous SR arbitrary ratios, pixel values at query coordinates are inferred from image features key coordinates by the proposed implicit transformer and implicit position encoding scheme is proposed to aggregate similar neighboring pixel values to the query one. We construct benchmark SCIIK and SCII compressso datasets with LR and HR SCI pairs. Extensive experiments show that the proposed ITSRN significantly outperforms several competitive continuous and discrete SR methods for both compressed and uncompressed SCIs.
well the requirement of SCI
Introduction
Nowadays, screen content images are becoming ubiquitous due to the wide application of screen sharing and wireless display. Meanwhile, due to the limited bandwidth, screen content images received by users may be in low-resolution (LR) and users may need to zoom in the content for detail inspection. Therefore, screen content image super-resolution (SCI SR) is improve the quality of LR SCIs.
However, different from natural scene images, SCIs are dominated by the contents generated or rendered by computers, such as texts and graphics. Such contents highly demanded are characterized by thin and sharp edges, little color variance, and high contrast. In contrast, the natural scene images are relatively smooth, and contain rich colors and textures. Conventional image SR methods designed for nature images are good modeling the local smoothness of natural images other than the thin and sharp edges in SCIs. Very recently, Wang et al. [1] proposed a SR method for compressed screen content videos, which addressed the compression artifacts of screen content videos by introducing distortion differential guided reconstruction module. However, their network is still composed of fully convolution layers without designing specific structures for the thin and sharp edges in screen contents. In addition, they utilize previous frames to help reconstruct the current frame, which makes unsuitable for frame-wise SCI SR.
"Corresponding author: Huanjing Yue
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

GT
ghotes
grates an
Blabic
grates and
MetaSR
imine grates and drain
ggates and
undine grotes and dnain
grates and
ITSNN
grates and
XL0
XB6
X305
Comparison of the proposed Implicit Transformer SR Network (ITSRN) with state-of- the-art image continuous magnification methods. The ground truth (GT), which has the same resolution with that of the x4 upsampling, is visualized at the top row, and its magnification results xx2.3, 10) are obtained by cruubccinterpolaatinn
Figure 1:
On the other hand, conventional SR methods are designed for discrete (ī.e. several fixed) magni- fication ratios 3 making them hard to fit screens with various sizes. Recently, few SR methods for continuous magnification have been proposed []] Hu et. al. [5] proposed to perform arbitrary-scale SR with learnable up-sampling weight matrix based on meta-learning. The work LIIF [6] introduced the concept of implicit function 7
g 9 to image SR. The implicit function, which attempts to represent images with continuous coordinates and directly maps the coordinates to values, enables continuous magnification.
In this work, we observe that convolution filters could be harmful to sharp and thin edges in SCIs since the weight sharing strategy makes them tend to produce smooth reconstruction result. Therefore, we propose render the pixel values by point-to-point implicit function, which adapts to image content according to image coordinates and pixel features. Fortunately, this also enables us to perform continuous magnification for SCIs. We would like to point out that even with the point-to-poing implicit function, reconstructing dense edges still quite challenging. As shown in Fig, [ LIIF 7 cannot reconstruct the dense edges well since it directly concatenates the pixel coordinates and features together to predict the pixel values, which is not optimal since the two variables have different physical meanings. As a departure, we reformulate the interpolation process as a transformer and introduce implicit mapping to model the relationship between pixel coordinates, which used to aggregate the pixel feature. Our main contributions are summarized as follows.
First, we propose novel Implicir Transformer Ressoutinn Network (ITSRN) for SCI SR. The LR and HR image coordinates termed as the "key" and "query", respectively. Correspondingly. the LR image pixel features are termed as "value". infer pixel values by implicit transformer, where implicit means model the relationship between LR and HR images terms of coordinates instead of pixel values.
this way, we can
Second, instead of directly concatenating the coordinates and pixel features to predict the pixel value, we propose to predict the transform weights with query and key coordinates nonlinear mapping, which are then used to transform the pixel features to pixel values. In

---

addition, we propose an implicit position encoding to aggregate similar neighboring pixel values to the central pixel.
Third, we construct a benchmark dataset with various screen contents for SCI SR. Extensive experiments demonstrate that the proposed method outperforms the competitive continuous and discrete SR methods for both compressed and uncompressed screen content images. Fig. presents an example of our SR results, which demonstrates that the proposed method is good reconstructing thin and sharp edges for various magnification ratios.
2 Related Work
2.1 Screen Content Processing
The screen content is generally dominated by texts and graphics rendered by computers, making the pixel distribution of screen contents totally different from that of natural scenes. Therefore, many works specifically designed for screen contents are proposed, such as screen content image quality assessment [1O 11 [3], screen content video (image) compression [14] [15.. However, there still no work exploring screen content image SR. Very recently, Wang el al. I proposed screen content video SR, which reconstructed the current frame by taking advantage of the correlations between neighboring frames, making it cannot deal with image SR. In addition, its main motivation is solving the SR problem when the videos are degraded by compression other than designing specific structures for continuously recovering thin and sharp edges in screen contents. In this work, address this issue by introducing point-to- point implicit transformation.
2.2
Continuous Image
Image SR refers the task of recovering HR images from LR observations. Many deep learning based methods have been proposed for super-resolving the L.R image with a fixed scale [16] 3 18 4
19 20
Since the screen contents are usually required to be displayed on screens with various sizes. Therefore, continuous SR is essential for screen contents. In recent years, several continuous image
SR methods [ 6 are proposed in order to achieve arbitrary resolution SR. MetaSR (5) introduces ueta-upscale module to generate continuous magnification but it has limited performance in dealing with titlo-of-taiiin-sscee -saale upsampling factors. LIIF [6) reformulates the SR process as an implicit neural representation (INRR problem, which achieves promising results for both in-distribution and out-of-distribution upsampling ratios. Inspired by LII,, we utilize the point-to-point implicit function for SCI SR.
2.3 Implicit Neural Representation
Implicit Neural Representation (INR) usually refers to continuous and differentiable function (e.g..
MLP), which can map coordinates certain signal. INR was widely used in 3D shape modeling [2]
23 24), volume rendering (i.e., neural radiance fields(Nerf)) (9 [25), and 3D reconstruction [8] 26). Very Recently, LIIF (6 was proposed for continuous image representation, in which networks took image coordinates and the deep features around the coordinate as inputs, and then map them to the RGB value of the corresponding position. Inspired by LIIF, propose implicit transformer network to achieve continuous magnification while retaining the sharp edges of SCIs well.
2.4 Positional Encoding (Mapping)
Positional encoding is critical to exploit the position order of the sequence transformer networks (27] (28 29). We have to utilize positional encoding to indicate the order of the sequence since there are no other modules to model the relative or absolute position information in the sequence- to-sequence model. In the literature, sine and cosine functions are used for positional encoding in transformer (27]. Coincidentally, we find that the Fourier feature (combined with a of sinusoids) based positional mapping is used in the implicit function to improve the convergence speed and generalization ability 30
Specifically, Fourier feature mapping applied on the input coordinates to map them to higher dimensional hypersphere before going through the coordinate- based MLP. Although the two schemes are proposed based on different motivations, they show significant effects in representing position information, which further boost the final results. Inspired
3

---

by them, we propose implicit positional encoding to model the relationship between neighboring pixel values. Here "implicit" means that we do not explicitly encode (map) the coordinates but encode the pixel values of neighboring coordinates.
Implicit Transformer
CNN Backbone
Flatten
Values
Implicit Position Facoding
Key Coodinates
R Reshape
Flatten
MEP
MEAT- FFREE
Flatten
EX"
Query
"[Sealee
Figure 2: The framework of our proposed ITSRN. The input LR image first goes through a CNN backbone generate pixel features. Then the features and key coordinates are upsampled by nearest neighbor interpolation based on query coordinates. Hereafter, we utilize the proposed implicit transformer learn the transform weights with query. key coordinates and scale token, and the pixel value is obtained via transforming the pixel features with transform weights. Finally, the pixel value further refined by the proposed implicit position encoding. The symbols (.. O.C, and & refer to upsampling, subtraction, concatenation, and matrix product operations respectively.
Approach
Figure 2 iillustrates the framework of the proposed ITSRN. In the following, give details for the proposed implicit transformer network and implicit position encoding.
3.1 Implicit Transformer Network
Let's first review the process of image interpolation. Suppose have LR image _ that needs to be upsampled. The pixel value query point g(i.j) the HR image 1 is obtained by fusing pixel values of its closest key points k(d', j') in I' with weighting matrix. Denoted by the query points in upsampled image, K the key points in the input LR image, and V the feature values on the corresponding key points. Then, the image interpolator can be reformulated as a transformer (27). Different from the explicit transformer which takes pixel values as Q and K, the interpolation transformer deals with pixels' coordinates instead of their values. Inspired by the implicit function in NeRF [ which uses the pixel coordinates to generate RGB values, we reformulate the interpolation process Implicit Transformer, and propose novel Implicit Transformer Network for SCI SR.
The super-resolution process as well as image interpolation is reformulated terms of implicit function as
(1) where Ig is the target RGB value that need to be predicted, is the query coordinate(s) in !" k is the key coordinate(s) in 4 and is the pixel value(s) or feature(s) corresponding to the key coordinate(s). Note that both g and & are coordinates in the continuous image domain. is mapping function which maps coordinates and features to RGB values. In image interpolation, l is the neighboring key coordinate g. are the pixel value of i. the weighting matrix. implicit function based SR method LIIF [6), & is the nearest neighbor coordinate in I for in I" and is the corresponding pixel feature A: LIIF directly concatenates v and the relative coordinate
Ig = (bq k)
from k, and then utilize the nonlinear mapping & realized by multi-layer perceptron (MLP) to render the pixel value Tq- It achieves promising results due to the strong fitting ability of the MLP. However, we observe that directly concatenating the pixel feature and the relative coordinates is not optimal since they

---

have different physical meanings. To solve this problem, following the idea of transformer 277, we reorganize Formula as follows.
- k. v) Φ(q, k)v.
(2)
Different from the explicit transformer, here q and are pixel coordinates other than pixel values. In other words, we are not learning the the RGB + RGB mapping, but the coordinate - RGB mapping. Due the continuity of coordinates, o(.) is a continuous projection function, which computes the transform weights ((g. k) to aggregate the feature v, iee, Ig is computed with the multiplication of & and v. To further illustrate it, we decompose & as:
(gg, k) = f(ō(g,))
where the function produces a vector that represents the relationship between y and k. Then the mapping function f projects this vector into another vector, which is then multiplied with feature as indicated in Eq.
There are many ways to model the relation function 6, such as dot product and hadamard product commonly used in transformers [27]. Different from their approaches, in this paper, we utilize subtraction operation, i.e.
8(g. k)) =
where 7 and r can use trainable functions or identity mapping. In this work, we utilize identity mapping since it generates similar results as that of trainable functions.
Inspired by ViT's 32 [class] token, which is extra global information for classification, we augment the input coordinates with scale information, and name it [scale] token. It represents the global magnification factor. With the [scale] token, the implicit transformer can take the shape of the query pixel as additional information for reconstructing the target RGB value. Although it is feasible to predict RGB values without [scale] token, it is not optimal since the predicted RGB value should The output of 8(q, k) is concatenated with the [scale] token, and is then fed to the mapping function f. In this way, ø in Formula (3) is reformulated as
not be completely independent of its shape [
(5) where: s = (sp, sw) is a wwo-dimensional scale token, representing the upsampling scale along the row and column for the query pixel. [p(y, k), s) means the concatenation of 8(q, k) and along the channel dimension. Note that, to avoid large coordinates in HR space, we normalize the coordinates
6(g. k) = f(6((q ()) s]),
2i + 11
2i +
p(i.j) = [-1+
1. i 10, H - 11.j E [0, W - 1]. -1].
(6)
H
W
where (i,j) represents the spatial location within the R"* space.
3.2 Implicit Position Encoding
Although we have considered the positional relationship between q and k. we still ignore the relative position of different query points. Therefore, inspired by the position encoding in explicit transformer, propose introduce implicit position encoding (IPE) avoid the discontinuity of neighboring predictions. Here, "Implicit Position Encoding" means that we do not explicitly encode the positional relationship among the pixel coordinates in the Q sequence since they are already absolute position coordinates. On the contrary, we encode the pixel value relationships within a local neighborhood. Chu al. claimed that convolution, which models the relationship between neighboring pixels, could be considered as an implicit position encoding scheme [33]. Therefore, we termed Eq [ as implicit position encoding. Specifically, we unfold the query within local window. The final predict value of the query coordinate is conditioned on its neighbor values. The IPE process is denoted as
f, = Σ w(p, I
(7)
pEN())
where i, is the refined pixel value, 2(g) is local window centered at q, w represents the neighbors' (denoted by p) contribution to the target pixel. In traditional image filtering, w(p, q) is generally realized by a Gaussian filter or bilateral filter. In this work, we utilize an MLP to learn adaptive weighting parameters for each q.
'It is surprised to find that our formulation in terms of transformer for the |scale] token is similar to the cell decoding strategy in LIIF.
5

---

the stage in w the stage in w whe stage in w the stage in w the stage in w the stage in w the stage in w
scientific data scientific data scientiffi data scientific data scientific data scientific data scientific data
to transform to transóorm to transdorm to transform to transóorm to transform to transform
Bicubic
RDN
RCAN
MetaSR (26.66 (.9468)
LUF
FTSRN(OurS) (27.23 (0.954))
GT
(23.41 (0.8239
(26.51 (0.9455)
(26.78 (.9497)
(26.71 (.95519)
(SNN SSIM
HE SQUADS HE SQUADS HE SQUADS HE SQUADS HE SQUADS HE SQUADS HE SQUADS
16 squads.
16 squads.
16 squads.
16 squads.
16 squads.
16 squads.
16 squads.
layers per squad layers per squad layers per squad layers per squad layers per squad squadlayers per squad layers per squad total players.
total players.
total players.
total players.
total players.
total players.
total players.
Bicabic (24.50/0.827)) /0.8274)
RDN
RCAN (09974))
MetaSR (3..0/00.8888) (33,05/ 0.99829
LIF
ITSRN(Ours) (33.87 (0.9657)
GT (PSNR/ SANR/SSIM SSIM)
(31.04/09925)
Figure 3: Visual comparison with state-of-the-arts at x4 SR.
4 Architecture Details of ITSRN
As shown in Figure 2 the proposed ITSRN has three parts, i.e., a CNN backbone to extract compact feature representations for each pixel, an implicit transformer for mapping coordinates target values, and an implicit position encoding that further enhances the target values. In the following, we give details for the three modules.
Backbone. Given an LR image 1 E R3xkxw we utilize CNN to extract its feature map
V E Rexhxw our experiments, c = 64. Normally, any CNN without downsampling upsampling can be adopted as the feature extraction backbone. To be consistent with LIIF [6]. we utilize RDN [18] (excluding its up-sampling layers) as the backbone. The extracted features V will be used in the following implicit transformer.
Implicit Transformer. First, inspired by LIIF 6 and MetaSR [5] to enlarge each feature's local receptive field, we apply feature unfolding, namely concatenating the features for the pixels in local region (3 x 3 in this work) to get o' E Rex9x1x1, After that, v' replaces v for the following processes. All the key coordinates corresponding to the feature vectors construct coordinate matrix K E R22xxx with the same spatial shape as V. The channel dimension is 2, which includes the row and column coordinates. Similarly, the query coordinates build matrix Q E R²* HWW where H x W is larger than & x w. Therefore, we first utilize the nearest neighbor interpolation to upsample the coordinates K to the size of Q. Hereafter, we utilize an MLP as the nonlinear mapping function f()) (mentioned in Eq. 3 to learn the relationship between the coordinate pairs in upsampled K and Q. As demonstrated in Eq. 5 f-)) maps the 4-dimensional (coordinates and scale token) input to a 9c-dimensional out x 3 for RGB channels). Finally, the 9c- De-dimensional output is multiplied by the corresponding pixel feature v, generating the coarse result for Ig.
Implicit Position Encoding. After obtaining the coarse value Lę. we further utilize IPE to refine it. this way, the central pixel can be more continuous with its neighbors. As mentioned in Eq. 7 we set Ω to 3 x 3 and for each point is learned via an MLP. The MLP includes two layers, i.e., [Linear > Gelu 27 + Limear), and the numbers of neurons for the two layers are 256 and 1, respectively.
5 Experiments
5.1 Datasets
To the best of our knowledge, there still no SCI SR dataset for public usage. Therefore, we first build dataset named SCIIK. It contains 1000 screenshots with various screen contents, including but not limited to web pages, game scenes, cartoons, slides, documents, etc. Among them, 800 images with x280 720 are used for training and validation. The other 200 images with resolution ranging

---

mnjoy hyaeeles ta
ogjoy / izz, dregons, Fpaceutips.
enjoy typewriless, ez, dregons, spaceships.
oajoy typewnlor,, ez, dragons, Ap spaceships
Lhk
shark is the Dw sharkr is the Dv shark Is the Dwshark Is the Dv
At just 6 inc At just 6 inc At just 6 inc At just 6 inc
fit in your hane
fit in your han fit in your han fit in your han
Input (48px)
MctaSR
L
(TSPN)480px)
Figure 4: Visual comparison with state-of-the-arts for arbitrary SR results. The input is a 48 x 48 patch from image SCID [12 test set. All the three models are trained with continuous scales in the range x1 ~ x4 and are tested for x 10 magnification. Note that the LR input is generated with x4 downsampling. and there no groud truth for its x10 magnification.
from 1280 720 to 2560 1440 are used for testing. To be consistent with previous works 4 18
utilize bicubic downsampling to synthesize the LR images. In addition, to simulate the degradation introduced in transmission and storage, we build another dataset, named SCII K-compreessoon by utilizing JPEG compression further degrade the LR images. The quality factor of JPEG is randomly selected from 75, 85, and 95.
To evaluate the generalization of the trained model, besides test set, also test two other screen content datasets constructed for image quality assessment, i.e., SCID (including 40 images with resolution x 720) [122 and SIQAD (including 20 images with resolution around 600 800)
5.2
Training Details
In the training phase, to simulate continuous magnification, the downsampling scale is sampled in uniform distribution (1,,, 4). We then randomly crop 48 48 patches from the LR images and augment them via flipping and rotation.
Following [8], we utilize the fl distance between the reconstructed image and the ground truth the loss function. The Adam 34 optimizer is used with betal=0.9 and beta2=0.999, All the parameters initialized with He initialization and the whole network is trained end-to-end. Following (6), the learning rate starts with 1e 4 for all modules and decays in half every 200 epochs. We parallelly I'TSRN-RDN two GeForce GTX 1080Ti GPU with mini-batch size 16 and it cost 2 days to reach convergence (about 500 epochs). During the test, we directly feed the whole LR image into network (as long the graphic memory is sufficient) to generate the SR result.
5.3 Comparison with State-of-the-arts
To demonstrate the effectiveness of the proposed SR strategy, we compare our method with state-of- the-art continuous SR methods, i.e. MetaSR (5] and LIIF (6) We also compare with the discrete SR methods, i.e., RDN (18) and RCAN 4 which are state-of-ther Since RDN and RCAN rely on specific up-sampling modules, they have to train different models for different upsampling scales and cannot be tested for the scales not in the training set. For fair comparison, we retrain all the compared methods using our training set with the recommended parameters and codes released by the authors. Note that, during test, we downsample the ground truth with different ratios to generate the LR inputs for different magnification ratios. For larger magnification ratios, the details are fewer in its corresponding LR input.
natural image SR methods.
7

---

Table 1: Quantitative comparison SCI1K and S K-ooppessioon test sets in terms of PSNR (dB). The best (second best) results are red (blue). RDN [8] and RCAN 4 use different models for different upsampling scales. MetaSR [5], LIIF [6] and (TSRN( use one model for all the upsampling scales, and the three models are trained with continuous random scales uniformly sampled from x x4.
Datasct: SCIK
Dataset: SCI K-ooppreeniiin
Method
In In-training-scal
Out-of-training -caale
In-training-sctt -calee
Our-of-trainngg
x2
x3
x4
x5
x7
x9
x2
x3
x4
x5
x7
x9
Bicubic 18
28.81
25.15
23.18
22.02
20.72
19.96
28.28
24.87
22,99
21.84
20.58
19.84
RDN 18
38.45
33.59
29.81
35.16
30.60
27.17
RCAN 4
38.61
33.91
30.80
35.25
31.15
27.78
MetaSR RDN S
38.57
33.67
30.12
27.52
23.91
22.02
35.20
30.96
27.63
25.31
22.57
21.30
LIIF-DN 6
38.65
33.97
30.55
27.77
23.99
22.18
35.43
31.07
27.69
25.27
22.59
21.36
ITSRN-RDN(Oun)
38.74
34.32
30.82
28.15
24.36
22.36
35.53
31.31
28.02
25.62
22.79
21.45
Table 2: Quantitative evaluation on SCI quality assessment datasets in terms of PSNR (dB). The best (second best) results in red (blue). RDN [18] and RCAN [4] train different models for different upsampling scales. The rest methods train one model for all the upsampling scales. All the models trained the SCIIK training set.
In-training-seee
Out-on-training-- scale
Dataset
Method
x2
x3
x4
x5
x6
x7
x8
x9
x 10
RDN
34.00
28.34
25.74
RCAN
33,90
28.98
26.02
SCID 12
MetaSR-RDN
33.84
29.08
25.76
23.62
22.38
21.59
21.07
20.71
20.41
LIIF-RDN 66
34.24
29.10
25.89
23.77
22.53
21.73
21.21
20.84
20.54
ITSRN-RDN
34.19
29.46
26.22
23.96
22.64
21.80
21.26
20.87
20.56
RDN [18)
33.53
26.89
23.38
RCAN
32.87
27.27
23.69
SIQAD 13
MetaSR ReDaSR-RDN A
34.12
28.40
23.55
21.18
20.18
19.63
19.25
18.94
18.65
LIIF RIF-
34.31
28.27
23.44
21.16
20.25
19.70
19.36
19.02
18.70
ITSRN-RDY
34.68
29.07
24.03
21.44
20.38
19.77
19.40
19.09
18.79
Table and Table present the quantitative comparison results on different datasets. Table |lists the average SR results for 200 screen content images in our SCIIK and SCIJ KCoopppmprrssionn test sets. All the methods are retrained on the corresponding training sets of SCIIK and SCI SCIIK-compression. can be observed that our method consistently outperforms all the compared methods. Compared with RDN, which is our backbone, our method achieves nearly 1 dB gain at x4 SR on SCIIK test set. Compared with the competitive RCAN, our method still achieves nearly 0.4 dB gain at x3 upsampling for the uncompressed dataset. Note that at x4 SR, the result of our method is similar as that of RCAN. The main reason that our backbone RDN generates much worse results than RCAN at x4 SR. If we change our backbone to more powerful structures, our results could also be further improved. For the scales that are not used in the training process (denoted by out-of-trainioe method outperforms the continuous
scales), RDN and RCAN are not applicable. Meanwhile, SR methods (i.e., MetaSR and LIIF), which demonstrates that the proposed implicit transformer scheme is superior in modeling both coordinates and features. Table 2 two SCI quality assessment datasets. Since the images in the two datasets are not compressed, we directly utilize the models trained SCIIK training set to test. It can be observed that our method still outperforms the compared methods.
presents the SR results on
Besides visual results in Fig- Fig. presents an example of the qualitative comparison results on SCIIK test sef] It can observed that our method recovers more realistic edges of characters than the compared methods. Fig. 4 presents the visual results for x x10 SR. It can be observed that our method reconstructs the thin edges better than the compared methods at large magnification ratios.
Moor visual comparison results presented in the supplementary file.

---

Table Ablation study for scale token and implicit position encoding. The PSNR/SSIM results are the averaging results on all the SCIIK test images.
Scale token
X
X
implicit position encoding
X
v
PSNR
30.43
30.60
30.76
30.82
(n-tranning-scal(x44
SSIM
0,9329
0.9351
0.9353
0,9364
PSNR
25.94
25.95
26.05
26.00
Out-of-trainnnng-sclee(x0
SSIM 0.8686
0.8715
0.8725
0.8746
5.4 Ablation Study Results
In this section, we perform ablation study to demonstrate the effectiveness of the proposed modules. Table 3 llists quantitative comparison results on SCI1K test set. It can be observed that the scale token and implicit position encoding totally contributes 0.39 dB to the final SR result at SR. Even for the sitt-of-training-see
the two modules also contribute 0.06 dB gain. Note that, the gain is lower for larger magnification factor because that the thin edges in screen contents are hardly be distinguished after x downsampling. Therefore, it is difficult to bring gains with scale token and position encoding.
Table 4: Comparison of using different realizations of the weight w in IPE. "Fixed" means w calculated based on the spatial distance between the query and its neighbors. "Lerredd refers to learning w via an MIP, which is used in this work.
In-training-scanl
Out-of-training-scal scale
Weight w in IPE
x2
x3
x4
x5
x7
x9
x24
x40
Fixed
35.27
31.03
27.78
25.47
22.78
21.45
18.42
17.31
Learned
35.53
31.31
28.02
25.62
22.79
21.45
18.40
17.29
Table 5: Comparison of SR performance with different training scales.
Scale
x4
x6
x8
x10
Training scale x2-x4
30.82
26.00
23.16
21.77
Training scale x2-x8
30.83
26.37
23.59
21.86
We also conduct ablation study on the w in IPE by changing it to different realizations. Table lists the SR results on coopppessono test set. From the table. we can observe that our proposed learnable scheme is superior to the fixed scheme when the input LR images suffer from JPEG compression. This because that, with JPEG compression, the neighboring information can help alleviate the compression artifacts and the fixed distance based weighting strategy is not suitable in this case. Note that, for very large magnification ratios, the learned w is slightly inferior to the fixed by 0.02 dB since the heavy distortions in the LR input may confuse the learning process. Besides, we conduct ablation study on the training scales. As shown in Table the SR results at x6 and x8 upsampling with training scales x2 x8 are better than that with training scale x2 x4. The main reason that the scales of x6 and x8 are in the range of the second training scales. In addition, the second strategy is also beneficial for x 10 upsampling. This indicates us that using wide distributed training scales is better than using narrow distributed training scales.
5.5 Discussion
The most related work to our proposed ITSRN is LIIF 6 which utilizes implicit function for continuous SR. However, it directly concatenates the image coordinate and its CNN feature, and then map them to the RGB value via an MLP. Different from it, we reformulate the SR process as an implicit transformer. There are two MLPs in our scheme. The first MLP is utilized to model the relationship between pixel coordinates. Then the "relationship" is multiplied by the pixel features to generate the pixel value, which can also termed as an attention strategy. The second MI.P utilized for implicit position encoding, which is useful keep the continuous of the reconstructed

---

pixels. The experiment results also demonstrate that the proposed two strategies make our method consistently outperform LIIF on all the four test sets.
Conclusion
In this paper, we propose novel arbitrary-scale SR method for screen content images. With the proposed implicit transformer and implicit position encoding modules, the proposed method achieves the best results on four datasets various magnification ratios. Due to the continuous magnification ability, our method enables users to display the received screen contents on screens with various sizes. In addition, we construct the first SCI SR dataset, which will facilitate more research on this topic.
Note that, although our method outperforms state-of the-arts on screen content images, it may not work the best for natural images a fixed magnification ratio. The main reason is that our method is designed for SCIs with high contrast and dense edges, which are suitable to be modeled by point-to- point mapping. Besides, we simulate the degradation process with bicubic and JPEG compression, which may be different from the actual degradation in transmission and acquisition. Thus, there will be limitation in practical applications. In the future, would like to develop blind distortion based SCI SR to make our model adapt to real scenarios better.
Acknowledgments
The authors would like to thank the anonymous reviewers for their valuable comments. This research was supported in part by the National Natural Science Foundation of China under Grant 62072331, Grant 62171317, and Grant 61771339.
Broader Impact
This work is an exploratory work on screen content images super-resolution with arbitrary-scal magnification. The construeted dataset can facilitate research this topic. In addition, the proposed method can be combined with image (video) compression technology to enable screen contents transmission with limited bandwidth. As for societal influence, this work will improve the quality of pictures displayed on the screen of any resolution. However, We would like to point out that SR is actually predicting (hallucinating) new pixels, which may make the image deviate from the ground truth. Therefore, image SR has a weak link with deep fakes. is worth noting that the positive social impact of this technology far exceeds the potential problems. We call on people use this technology and its derivative applications without harming the personal interests of the public.
References
[1] Meng Wang, Jizheng Xu, Li Zhang, Junru Li, and Shiqi Wang. Super resolution for compressed screen content video. In 2021 Data Compression Conference (DCC), pages 173-182, 2021.
[2] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), 2017.
[3] J. Kim, J. K. Lee, and K. M. Lee. Accurate image super-resolution using very deep convolutional networks. In IEEE Conference on Computer Vision Pattern Recognition, 2016.
[4] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 294-310, 2018.
[5] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang. Zilei Wang, Tieniu Tan, and Jian Sun. Meta-sr: magnificaaiinnarbiraay network for super-resolution. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1575-1584, 2019.
[6] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. arXiv preprint arXhv:2012.09161, 2020.
10

---

[7] K. O. Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines, 8(2):111-12, 2007.
[8] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[9] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and N. Ren. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
[10] Huan Yang, Yuming Fang, and Weisi Lin. Perceptual quality assessment of screen content images. IEEE Transactions on Image Processing, 24(1 11:4408-4424 2015.
[11] Ke Gu, Guangtao Zhai, Weisi Lin, Xiaokang Yang, and Wenjun Zhang. Learning a blind quality evaluation engine of screen content images. Neurocomputing, 196:140-149, 2016.
[12] Zhangkai Ni, Lin Ma, Huanqiang Zeng, Jing Chen, Canhui Cai, and Kai-Kuang Ma. Esim: Edge similarity for screen content image quality assessment. JEEE Transactions on Image Processing, (6(10):4818-483) 2017.
[13] Huan Yang. Yuming Fang, and Weisi Lin. Perceptual quality assessment of screen content images. IEEE Transactions on Image Processing, 24(1 1):4408-4421, 2015.
[14] Shiqi Wang, Xinfeng Zhang, Xianming Liu, Jian Zhang, Siwei Ma, and Wen Gao. Utility-driven adaptive preprocessing for screen content video compression. IEEE Transactions on Multimedia, 19(3):660-667, 2017.
[15] Wen-Hsiao Peng, Frederick G. Walls, Robert A. Cohen, Jizheng Xu, Jorn Ostermann, Alexander MacInnis, and Tao Lin. Overview of screen content video coding: Technologies, standards, and beyond. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 6(4):393 2016.
[16] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184-199, 2014.
[17] W. S. Lai, J. B. Huang, N. Ahuja, and M. H. Yang. Deep laplacian pyramid networks for fast and accurate super-resolution In IEEE Conference Computer Vision & Pattern Recognition, pages 5835-5843, 2017.
[18] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2472-2481, 2018.
[19] S. Anwar and N. Barnes. Densely residual laplacian super-resolution. IEEE Transactions on Software Engineering, PP(99), 2020.
[20] Y. Mei, Y. Fan, Y. Zhou, L. Huang, T. S. Huang, and H. Shi. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[21] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939-5948, 2019.
[22] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In 2020 IEEF/CV Conference Computer Vision and Pattern Recognition (CVPR), pages 2565-2574 2020.
[23] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for shape. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4857-4866, 2020.

---

[24] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 165-174, 2019.
[25] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi Derf: Decomposed radiance fields. arXiv preprint arXiv:2011.12490 2020.
[26] Songyou Peng, Michael Niemeyer, Lars M. Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In ECCV (3), pages 523-540, 2020.
[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, volume 30, pages 5998- 6008, 2017.
[28] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 4171-4186, 2018.
[29] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243- 1252. PMLR, 2017.
[30] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[31] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Processing Systems, volume 33, pages 7537-7547, 2020.
[32] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[33] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? arXiv preprint arXiv:2102.10880 2021.
[34] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. International Conference on Leurning Representations, 12 2014.
12


Reviews:
[{"review_id": "qVEStyDxcio", "reviewer": "Reviewer_2ePa", "summary": "This work addresses the SCISR problem by combining transformers to the network architecture. The backbone of the architecture utilizes a CNN to extract feature maps, to enlarge the receptive field, an implicit transformer is used. After that the implicit position encoding is conducted via MLP. Natural images are suitable for SR but there is a lack of SR techniques that are suitable for screen content, which this paper tries to address. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is described well. The experiments show that the method can perform well in terms of quantitative analysis and qualitative comparison. The qualitative comparison shows that the method is well suited for screen content compared to other SOTA methods. \nWith the help of transformers, the SR seems to improve in quality, but there is no analysis on computational complexity. Normally, when the transformer is used, it can be utilized to reduce the computational load while also improving quality. Does this work also convey such characteristics? \nThe latest SOTA for SR using transformers is the CVPR 2020 paper TTSR which is not mentioned in the main paper. There is no comparison with the method either nor in the related works section. The latest SOTA is essential to show in the paper so that it can be compared to the latest SOTA. ALthough this work is focussed on SCISR, it is still very related to SR and further uses transformers as a component to the network architecture. The most similar work is arguably the TTSR which need s to be addressed.\nAlso, the authors conduct experiments on RDN, RCAN, MetaSR etc. However, these are somewhat outdated SR models. Thus, it is not entirely surprising that the current model outperforms them. The latest SR models that the authors should consider is definitely TTSR, but also EDVR, EDSR, IGNN, SRGAN etc. \nI would say the experimental section is not as extensive as I would have expected. Although there are not many methods combining transformers with CNNs yet for SR, it is much more relevant to compare with CNN only methods to show the effectiveness for combining an additional module; the transformer. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "SR has the potential to be a piece of technology that can impact the society perhaps in a negative way. SR in the end is trying to construct new pixels or even hallucinate details that were not seen or non existent. This means that the picture may not be entirely true to the true visual. This has a weak link with deep fakes. Some thought on this is required. However, the authors do not address such societal or ethical implications in this paper. \n", "review_text": "", "extra_scores": {"ethical_concerns": "See above", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "mU_tCp25pFX", "reviewer": "Reviewer_vvmM", "summary": "This work presents a continuous image super-resolution based on an implicit transformer. Authors extend LIIF [6] with some modifications including transformer-like formulation in (2), scale token in (5), and implicit position encoding (7). The proposed implicit transformer learns the transformation weight from query (high-resolution) coordinate, key (low-resolution) coordinate, and scale token. A pixel intensity is then computed by transforming the pixel features (obtained from CNN backbone) with the transformation weights. The pixel intensity is further refined with the implicit position encoding, which is an adaptive weighting scheme with learned weights. Experimental results on screen content images demonstrate superior performance over existing continuous super-resolution methods [5,6].", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1) The implicit transformer with scale tokens seems to be an appealing solution for continuous super-resolution.\n2) The results on screen content images including sharp and thin edges are excellent.\n3) Intensive ablation study is provided.\n\nCons\n1) A proper motivation of the proposed methods is needed.\n- The transformer-like formulation in (2) is presented to avoid the issue by a direct concatenation of pixel features and relative coordinates in LIIF [6]. However, it seems that solid justification about (2) is rather insufficient.\n- L151-152: Why did you choose the subtraction operation?\n- (7) is an adaptive weight scheme in which the weight $w$ is learned from MLP. 'Implicit position encoding' does not seem like the right term.\n\n2) RDN [18] and RCAN [4] are too out-of-date to be compared.\n\n3) The comparison was conducted with only screen content images containing sharp and thin edges (e.g., text) yet less textures. However, some screen contents may also have diverse textures like natural images.\nHow is the performance in the natural image? It would be better to provide the performance evaluation in the natural images in order to clarify the limitation of this work, if the performance on the natural image is not as good as that of state-of-the-art SR methods (including both discrete and continuous SR approaches).\n\n4) Some minor issues\n- L145: $I_q$ is computed with the multiplication of $\\phi$ and $v$, not the weighted average of $v$.\n- L171: 'implicit means' is a wrong expression.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not address the potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "cuohhlswE4p", "reviewer": "Reviewer_zDGM", "summary": "This paper presents a novel method called implicit transformer for the task of super-resolution of screen content images. The proposed method is able to integrate the advantage of transformer in capturing relation between pixels and that of implicit neural representation in producing continuous and sharp results. The effectiveness of the proposed method is validated on both newly collected datasets w/o and w/ compression degradation, and datasets originally created for image quality assessment. Ablation study also demonstrates the role of the proposed modules.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the idea of integrating implicit neural representation into a transformer is interesting and its effectiveness has been validated in the extensive ablation study and comparison with SOTA methods. However, there are still some issues to be clarified.\n1.\tIt is not clear what is the meaning of computing relation matrix using coordinates in transformer. Since coordinates are content-independent, it means that for images of the same size, they share the same $\\phi(q, k)$. \n2.\tIn the comparison, the proposed method shows better performance on both qualitative and quantitative comparison. However, it is not clear whether comparison is fair since number of parameters and flops are not mentioned.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have addressed the limitation and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "Jocy8S8wgEa", "reviewer": "Reviewer_bkFN", "summary": "The paper proposes a transformer-based image super-resolution method for screen content images which have many thin edges. The transformer is designed to learn the mapping from coordinates to rgb values. It additionally has a scale token representing the magnification factor. Due to targeting a new application, the paper also constructs suitable training and testing datatests.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*\n\n\\+ Addresses the problem of super-resolution of screen content images, which have different characteristics than natural images (e.g., many thin edges). Seem to be the first one looking at this specific application.\n\n\\+ Allows for continuous super resolution and is not limited to fixed, discrete magnification factors.\n\n\\+ Introduces an implicit transformer based super-resolution network.\n\n\\+ Construct two kinds of benchmark datasets, once with bicubic downscaling and once adding JPEG compression.\n\n*Quality*\n\n\\+ The superiority of the method for the targeted domain of screen images has been shown on its own dataset as well as related datasets constructed for image quality assessment and in comparison, to other SR methods.\n\n\\+ Method also works for scale factors which are not used during training.\n\n\\+/- Ablation study has been performed to analyze the benefit of the different modules. The scale token, however, seems to be mainly beneficial for in-training scales. \n\n\\+/- For completeness, it might be interesting to add the performance of the proposed methods on natural images.\n\n*Clarity*\n\n\\+ Code submitted.\n\n\\- If available, ground truth could be added to Figure 1.\n\n\\+/- It mentions that the graphic memory needs to be sufficient large at test time. What does that mean? What is the inference time?\n\n\\- I am wondering how the performance would be if in-training scale are distributed more (e.g., 2,4,8)? Is the method also working for fractional scale factors?\n\n*Significance*\n\n\\+ Constructs a benchmark dataset for the specific application domain of super resolution of screen content images.\n\n\\+ Shows superior performance on the targeted application domain.\n\n\\- Limited to the specific application. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the relevant limitations have been discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FGAxiFaBTz0", "reviewer": "Reviewer_cjuE", "summary": "This paper addresses the problem of arbitrary screen content image super-resolution, which is quite different from natural image super-resolution.  The author proposed two datasets specifically designed for this scenario, which can greatly motivate research  effort into this field. Besides, the authors proposed Implicit Transformer Super-Resolution Network (ITSRN) to achieve arbitrary scaling for screen content images. The ITSRN is largely inspired by LIIF, the main novelty of ITSRN includes the proposed  implicit transformer and the implicit position encoding, these schemes are empirically evaluated to show its effectiveness. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The problem of arbitrary scale super-resulution of screen content image is addressed in this work, which is a specific subarea of image super-resolution and is rarely studied by researchers. The proposed ITSRN is inspired by LIIF and its performance is improved by implicit transformer and implicit positional coding. In my opinion, the task is interesting and the method is novel. The newly proposed datasets can benchmark this subarea and is helpful for the developing of the subarea.\n\nQuality: the quality of this paper is above the bar. Technical details are clear and the evaluation is convincing, extensive results are provided to show the effectiveness of implicit transformer and implicit position encoding, some design choices are also empirically verified. However, the following two issues should be addressed to further improve this paper:\n1) what is the complexity of the proposed method, compared to existing state-of-the-arts. This information can help readers know better about the strength of the paper as well as its overhead.\n2) as stated in line201-202, \"the 9c-dimensional output is mulitplied by the corresponding feature value v, generating the coarse results I_q\". It is a little bit confusing how the mulitiplication is conducted, because  v is also a 9c-dimensional vector, i am guessing that I_q is a scalor or not? If yes, how to generate the 3-channel output RGB pixel value at q. Please make this more clear to readers. \n\nClarity: This paper is well organized and is easy to follow. The writting of this paper is good. Though small ambiguity exists as stated above, codes and models are currently available for verification, which is a plus.  \n\nSignificance: This work makes a different to the community. Firstly, the datasets can benchmark the subarea; secondly, the problem is interesting; lastly, the method is new and inspiring. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new model for image super-resolution, which builds on an implicit transformer, to address the resolution enhancement problem in the specific settings of screen-content images. The performance of the model has been extensively studied by the other and improved through the active discussion with the reviewers. The results are quite convincing for the specific SR task under consideration, which has not been commonly studied so far.  
Decision: Accept (Poster)