Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a transformer-based image super-resolution method for screen content images which have many thin edges. The transformer is designed to learn the mapping from coordinates to rgb values. It additionally has a scale token representing the magnification factor. Due to targeting a new application, the paper also constructs suitable training and testing datatests.
main_review: *Originality*

\+ Addresses the problem of super-resolution of screen content images, which have different characteristics than natural images (e.g., many thin edges). Seem to be the first one looking at this specific application.

\+ Allows for continuous super resolution and is not limited to fixed, discrete magnification factors.

\+ Introduces an implicit transformer based super-resolution network.

\+ Construct two kinds of benchmark datasets, once with bicubic downscaling and once adding JPEG compression.

*Quality*

\+ The superiority of the method for the targeted domain of screen images has been shown on its own dataset as well as related datasets constructed for image quality assessment and in comparison, to other SR methods.

\+ Method also works for scale factors which are not used during training.

\+/- Ablation study has been performed to analyze the benefit of the different modules. The scale token, however, seems to be mainly beneficial for in-training scales. 

\+/- For completeness, it might be interesting to add the performance of the proposed methods on natural images.

*Clarity*

\+ Code submitted.

\- If available, ground truth could be added to Figure 1.

\+/- It mentions that the graphic memory needs to be sufficient large at test time. What does that mean? What is the inference time?

\- I am wondering how the performance would be if in-training scale are distributed more (e.g., 2,4,8)? Is the method also working for fractional scale factors?

*Significance*

\+ Constructs a benchmark dataset for the specific application domain of super resolution of screen content images.

\+ Shows superior performance on the targeted application domain.

\- Limited to the specific application. 

limitations_and_societal_impact: Yes, the relevant limitations have been discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addresses the problem of arbitrary screen content image super-resolution, which is quite different from natural image super-resolution.  The author proposed two datasets specifically designed for this scenario, which can greatly motivate research  effort into this field. Besides, the authors proposed Implicit Transformer Super-Resolution Network (ITSRN) to achieve arbitrary scaling for screen content images. The ITSRN is largely inspired by LIIF, the main novelty of ITSRN includes the proposed  implicit transformer and the implicit position encoding, these schemes are empirically evaluated to show its effectiveness. 
main_review: Originality: The problem of arbitrary scale super-resulution of screen content image is addressed in this work, which is a specific subarea of image super-resolution and is rarely studied by researchers. The proposed ITSRN is inspired by LIIF and its performance is improved by implicit transformer and implicit positional coding. In my opinion, the task is interesting and the method is novel. The newly proposed datasets can benchmark this subarea and is helpful for the developing of the subarea.

Quality: the quality of this paper is above the bar. Technical details are clear and the evaluation is convincing, extensive results are provided to show the effectiveness of implicit transformer and implicit position encoding, some design choices are also empirically verified. However, the following two issues should be addressed to further improve this paper:
1) what is the complexity of the proposed method, compared to existing state-of-the-arts. This information can help readers know better about the strength of the paper as well as its overhead.
2) as stated in line201-202, "the 9c-dimensional output is mulitplied by the corresponding feature value v, generating the coarse results I_q". It is a little bit confusing how the mulitiplication is conducted, because  v is also a 9c-dimensional vector, i am guessing that I_q is a scalor or not? If yes, how to generate the 3-channel output RGB pixel value at q. Please make this more clear to readers. 

Clarity: This paper is well organized and is easy to follow. The writting of this paper is good. Though small ambiguity exists as stated above, codes and models are currently available for verification, which is a plus.  

Significance: This work makes a different to the community. Firstly, the datasets can benchmark the subarea; secondly, the problem is interesting; lastly, the method is new and inspiring. 
limitations_and_societal_impact: None
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work addresses the SCISR problem by combining transformers to the network architecture. The backbone of the architecture utilizes a CNN to extract feature maps, to enlarge the receptive field, an implicit transformer is used. After that the implicit position encoding is conducted via MLP. Natural images are suitable for SR but there is a lack of SR techniques that are suitable for screen content, which this paper tries to address. 

main_review: The method is described well. The experiments show that the method can perform well in terms of quantitative analysis and qualitative comparison. The qualitative comparison shows that the method is well suited for screen content compared to other SOTA methods. 
With the help of transformers, the SR seems to improve in quality, but there is no analysis on computational complexity. Normally, when the transformer is used, it can be utilized to reduce the computational load while also improving quality. Does this work also convey such characteristics? 
The latest SOTA for SR using transformers is the CVPR 2020 paper TTSR which is not mentioned in the main paper. There is no comparison with the method either nor in the related works section. The latest SOTA is essential to show in the paper so that it can be compared to the latest SOTA. ALthough this work is focussed on SCISR, it is still very related to SR and further uses transformers as a component to the network architecture. The most similar work is arguably the TTSR which need s to be addressed.
Also, the authors conduct experiments on RDN, RCAN, MetaSR etc. However, these are somewhat outdated SR models. Thus, it is not entirely surprising that the current model outperforms them. The latest SR models that the authors should consider is definitely TTSR, but also EDVR, EDSR, IGNN, SRGAN etc. 
I would say the experimental section is not as extensive as I would have expected. Although there are not many methods combining transformers with CNNs yet for SR, it is much more relevant to compare with CNN only methods to show the effectiveness for combining an additional module; the transformer. 

limitations_and_societal_impact: SR has the potential to be a piece of technology that can impact the society perhaps in a negative way. SR in the end is trying to construct new pixels or even hallucinate details that were not seen or non existent. This means that the picture may not be entirely true to the true visual. This has a weak link with deep fakes. Some thought on this is required. However, the authors do not address such societal or ethical implications in this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work presents a continuous image super-resolution based on an implicit transformer. Authors extend LIIF [6] with some modifications including transformer-like formulation in (2), scale token in (5), and implicit position encoding (7). The proposed implicit transformer learns the transformation weight from query (high-resolution) coordinate, key (low-resolution) coordinate, and scale token. A pixel intensity is then computed by transforming the pixel features (obtained from CNN backbone) with the transformation weights. The pixel intensity is further refined with the implicit position encoding, which is an adaptive weighting scheme with learned weights. Experimental results on screen content images demonstrate superior performance over existing continuous super-resolution methods [5,6].
main_review: Pros
1) The implicit transformer with scale tokens seems to be an appealing solution for continuous super-resolution.
2) The results on screen content images including sharp and thin edges are excellent.
3) Intensive ablation study is provided.

Cons
1) A proper motivation of the proposed methods is needed.
- The transformer-like formulation in (2) is presented to avoid the issue by a direct concatenation of pixel features and relative coordinates in LIIF [6]. However, it seems that solid justification about (2) is rather insufficient.
- L151-152: Why did you choose the subtraction operation?
- (7) is an adaptive weight scheme in which the weight $w$ is learned from MLP. 'Implicit position encoding' does not seem like the right term.

2) RDN [18] and RCAN [4] are too out-of-date to be compared.

3) The comparison was conducted with only screen content images containing sharp and thin edges (e.g., text) yet less textures. However, some screen contents may also have diverse textures like natural images.
How is the performance in the natural image? It would be better to provide the performance evaluation in the natural images in order to clarify the limitation of this work, if the performance on the natural image is not as good as that of state-of-the-art SR methods (including both discrete and continuous SR approaches).

4) Some minor issues
- L145: $I_q$ is computed with the multiplication of $\phi$ and $v$, not the weighted average of $v$.
- L171: 'implicit means' is a wrong expression.

limitations_and_societal_impact: The authors did not address the potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a novel method called implicit transformer for the task of super-resolution of screen content images. The proposed method is able to integrate the advantage of transformer in capturing relation between pixels and that of implicit neural representation in producing continuous and sharp results. The effectiveness of the proposed method is validated on both newly collected datasets w/o and w/ compression degradation, and datasets originally created for image quality assessment. Ablation study also demonstrates the role of the proposed modules.
main_review: Overall, the idea of integrating implicit neural representation into a transformer is interesting and its effectiveness has been validated in the extensive ablation study and comparison with SOTA methods. However, there are still some issues to be clarified.
1.	It is not clear what is the meaning of computing relation matrix using coordinates in transformer. Since coordinates are content-independent, it means that for images of the same size, they share the same $\phi(q, k)$. 
2.	In the comparison, the proposed method shows better performance on both qualitative and quantitative comparison. However, it is not clear whether comparison is fair since number of parameters and flops are not mentioned.
limitations_and_societal_impact: Yes, they have addressed the limitation and potential negative societal impact.
review_title: null

Meta-review: The paper proposes a new model for image super-resolution, which builds on an implicit transformer, to address the resolution enhancement problem in the specific settings of screen-content images. The performance of the model has been extensively studied by the other and improved through the active discussion with the reviewers. The results are quite convincing for the specific SR task under consideration, which has not been commonly studied so far.  
Decision: Accept (Poster)