Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper modifies HRNet replacing standard convolutional layers with a transformer block which consists of a local multi-head attention operation (on non-overlapping windows) followed by a residual block with a 3x3 depth-wise convolution. The proposed model is benchmarked on ImageNet, COCO pose estimation, and a number of segmentation datasets.
main_review: Strengths:
- combining the advances in dense architecture design from HRNet with transformer blocks is a good step to take, taking advantage of approaches that work well for ConvNets makes a lot of sense as we see this wave of new transformer-based models.
- thorough experiments show that the proposed block is a good upgrade to HRNet, this is especially clear from the comparison made in Table 10
- experiments across many benchmarks including classification, pose, and segmentation are comprehensive showing the proposed model is competitive with recent work
- the proposed block is clear and straightforward

Weaknesses:
- it seems that a number of relevant results are left out of the benchmark comparisons - notably results that outperform the proposed method. For example, Swin [26] is absent from the ImageNet-1K results (Table 3). I also notice that [19] and [54] are cited but not included in the results in Table 5. While getting state-of-the-art results is nice, it is important to acknowledge recent work even if it has better performance
- use of a 3x3 depth-wise convolution is a straightforward enough change that it feels as though it gets an unnecessary amount of attention in the paper (for example, I appreciate what Figure 3 is communicating, but I am not sure it is needed). Given the simplicity of the module, it would have been nice to see ablations with other strategies to pass information across windows - like, how does 3x3 conv compare to consecutive local window attention blocks with shifted windows?

Overall I am currently borderline with the paper, I think it is a nice, simple idea that clearly improves a common architecture in the literature. That said, given the simplicity I think it would be nice to see more exploration of the design decisions of the model (e.g. different window sizes at different resolutions, alternatives to pass info across windows besides a conv layer). Also, I think it would be better to not leave out relevant recent work from the benchmark results.

**Update**

I was borderline on the paper, but the authors have put a good amount of work into their rebuttal to address my concerns. While not the most technically novel, it is well motivated and executed. Overall the results are in the ballpark of other work in this space, I personally find Table 10 the most compelling case for the paper as there is plenty of work building off of HRNet, and this proposed version improves it across all tasks while using less params and compute. I raise my score from a 5 to a 6.
limitations_and_societal_impact: currently no discussion is included in the submission
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a vision transformer like HRNet (a CNN-based visual architecture) for dense prediction. The architecture is tested on classification/segmentation and pose estimation.
main_review: Totally, transferring an idea (HRNet) from CNN to vision transformer is ok, but the authors should provide the rationality or effectiveness. Though the architecture is different from some recent vision transformer, the improvement is marginal.

Originality: The idea of this paper is highly like HRNet, and the attention mechanism in Figure 1 is not a new thing. Actually, most new vision transformers reduce the cost by divide-and-conquer. For example, SwinTransformer etc. all resort to self-attention in a local region.

Clarity: The paper is easy to follow.

Others (to be improved):
1. The authors cite some papers: PVT/Swin/CvT etc. in related works. So, I think the authors have read these papers before finishing their own work. Since there have been very few published vision transformers, I think it is neccessary to compare with these arxiv-ones. I think the design of PVT/Swin/CvT are simpler than this paper and they show better performance, which makes me doubt the effectiveness of HRNet in vision  transformers.

2. I think the results in Table 4 show the significance of vision transformer over CNN, but does not prove that HRT is a good choice for vision transformer.

3. A little advice: In table 3, all models take a 224x224 image as input, it is also ok to explain it in the caption rather than repeat it again and again in the table.


limitations_and_societal_impact: I do not see any potential negative societal impact of their work
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors proposed a new model framework (HRT) using transformers for dense prediction in computer vision tasks.

Comparing to conventional patch-based vision transformers for dense prediction (such as segmentation and human pose estimation), HRT has three key components: 1) pixel-based local-windowed multiheaded attention with reduced compute overhead; 2) Depth-wise convolution (inside FFN modules) to aggregate info across local-windowed attention 3) Multi-resolution framework to enhance the representation quality for dense predictions (instead of using deconv after patch-based  VIT transformer encoders).

Empirically on image classification, human pose estimation and segmentation tasks, the authors shows that HRT can attain higher accuracy at similar or lower param count / FLOPs than 1) conv / deconv baselines and 2) conventional patch-based ViT encoder + deconv baselines.
main_review: Originality: I think the techniques (including pixel-based attention, depth-with convolution in FFN and Multi-resolution backbone) are novel to be used in transformer based models for dense prediction in vision tasks. I don't think the fact that depth-width conv and multi-resolution backbone have been used in the convolutional models affects the originality of this paper. By leveraging these techniques in the paper for transformer models, it leads to a new design paradigm for attention based models for dense predictions. 

Quality: Given this is empirical-study-oriented work, I think the full model accuracy demonstration is impressive across tasks using various baselines but 1) the ablation study needs to be more systematic to fully discussing the merits of the techniques and 2) Additional results/info needs to be provided to achieve a full evaluation and reproducibility. Specifically:

1. Ablation: In order to ablate out the contribution of each of the proposed technique, it should demonstrate results when each of the technique is ablated out or added (in a technique-individual fashion or an incrementally building-up fashion). In the current ablation study, when comparing to patch-based VIT, it is unclear whether the major benefits is from the pixel-based-attention or from the multi-resolution design. These information are relevant to readers who are looking for the most critical technique in a trade-off between system and statistical performance.

2. Additional results: I suggest reporting of results over multiple random seed with both averaged accuracy and error bar. This will help fully evaluate whether the results are statistically meaningful or reliable. Besides this, key hyperparameters / tuning protocol needs to be revealed to get full reproducibility.

I am willing to raise the score if the above experiment concerns are resolved.


Clarity: I think the paper is meaningfully well written. Here are a few minor suggestions on further improvement:

1. Figure 1,2 and their captions does not very explicitly cover the fact that it is pixel based attention. This potentially triggers some confusion when people step in with the typical feeling that ViT is patch-based.

2. It might be helpful to visualize the accuracy as a function of param count or flops to compare the different models / methods; this could help better demonstrate the advantage across the spectrum. The current results are organized in tables which is okey but less straight-forward to fully extract the information.

Significance: Using transformer / attention based model is an active topic for computer vision, the techniques proposed in this paper (such as pixel based attention instead of patch-based) could create very positive impact on the model design trend for various type of vision tasks. 
limitations_and_societal_impact: I think the attached checklist answers the questions adequately. The only thing that worth mentioning is there is no technical limitation discussion (the author give N/A for the questions but I do not see this being a factor that downgrades the score for the review.) I suggest some technical discussion on the situations where the proposed methods might not be preferred etc.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a way to replace the main convolution layers in the HRNet with self-attention layers. A feedforward network with 3x3 depth-wise convolution is also proposed to replace the MLP in transformer. The proposed High-Resolution Transformer achieves performance improvements for dense prediction tasks such as pose estimation and semantic segmentation while saving memory and computation.
main_review: The method presented in this paper is not very novel, but technically sound.
Strong points:
- The proposed FFN completes the task of information exchange between different windows well.
- Experimental results on pose estimation and semantic segmentation show that the proposed High-Resolution Transformer is better than HRNet in terms of performance, memory cost and computation cost.
Weak points:
- Not very novel. The main innovation of this paper is to design a FFN with 3x3 depth-wise convolution to perform information exchange across different windows. The rest is simply replacing the main structure of HRNet with the local attention layer.
- There is no description of limitation in this paper. Considering that DETR has the disadvantage of slow convergence, I wonder how the convergence speed of the architecture proposed in this paper compares with HRNet?

limitations_and_societal_impact: There is no description of limitation in this paper.
review_title: null

Meta-review: The authors propose a "high-resolution Transformer" architecture for dense prediction tasks using a multiresolution hierarchy and self-attention over local windows. All reviewers agreed that the ideas were sensible and technically sound, and that the experiments demonstrated improvement. There were concerns about missing comparisons (reviewers Jvmd, G26c) and missing ablations (tHJt, Jvmd), but the authors seem to have addressed all of these in rebuttal. There were also concerns of limited novelty (reviewers 143m, G26c), but I don't think this alone should disqualify the paper. Therefore I recommend acceptance.
Decision: Accept (Poster)