Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces neural matching fields into semantic correspondence. To the best my knowledge, this approach should be the first method to do the task using implicit neural representation. There are two problems: the computation for 4D matching field and the inference efficiency. Authors provide effect method to address the two problems.
strengths_and_weaknesses: This paper employs implicit neural representation to do semantic correspondence. This should be the major contribution. According to the statement of authors, I can follow the idea easily and this idea should work.

The disadvantage of this work is the experiments. There are too many quantitative comparisons. According to the data, the performance of this method seems OK. However, authors should provide more visual experiments to convince readers.
questions: I only have one concern. Traditional Implicit Neural Representation method such as LIIF and NeRF records images into the weights of neural network. One neural network represents one image or one scene. Does NeMF take a neural network to represent a semantic correspondence or a matching cost.   If so, how much time will your method cost to train a network? If not so, what is the difference between your method and other semantic correspondence methods.
limitations: According to my understand, NeMF takes a network to represent a matching cost. In practice, people need a method to compute different matching cost for different image pairs. How does NeMF to deal with this situation.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Deep learning based image matching algorithms rely on creating a 4D tensor representing the cost of matching each pair of pixels. Due to large memory consumption this 4D tensor has to be kept at a coarse scale which affects the quality of matching at a finer scale (i.e., small localization errors are prominent). Inspired from the concept of implicit neural representations, this paper proposes to learn an MLP which can efficiently interpolate in the space of matching costs. To recover the matching during inference, the paper proposes two ways: sampling and gradient based optimization. Experiments show that this approach indeed can match images significantly better at a finer scale than competitive approaches. Overall this idea of implicit neural representation combined with test time optimization can also have applications in other tasks which also compute cost volumes e.g., depth estimation, optical flow etc. 
strengths_and_weaknesses: A. Strengths:
1. The idea of using MLP for recovering fine details from matching costs is novel and possibly useful in other applications as well.
2. The paper is mostly written well and easy to understand.
3. Extensive experiments do show improvement of this approach over competitive approaches on several datasets. 
4. Even though this approach incurs slightly more memory burden than best competitive approach (CATs[8]) but it allows to compute the matching costs at any arbitrary resolution foregoing the need for upsampling of the cost tensor.

B. Weakness:
1. The concept of implicit representation was motivated by the need for reducing memory consumption. However, this approach ended-up ultimately using a 5D cost tensor requiring 16x more memory than competitive approach of CATs[8] which uses 4D. 
2. Computation: Inference time of this approach is significantly more than competitive approaches. Although the authors do mention this limitation but memory consumption is not reported (for both training and testing).  

questions: C. Major questions and suggestions:
1. In equation 6, is $y$ initialized from $F_{pred}$ similar to PatchMatch? 
2. In line 272, CATs[8] are said to have a cost tensor of size $16^4$ which at first glance seems to be more than this approach of NMT. However in line 9 of Appendix it is mentioned that NMT needs even more ($16^5$). I would suggest to change the wording in line 272.
3. I did not find details about training time, batch size, compute hardware used for training etc. Also peak memory consumption during training and inference needs to be compared with CATs. I think NMF would need more due to 5D cost representation instead of 4D.
4. The details of cost embedding network 4.2 are fuzzy. In Table 2 please define the components concretely. See Table 1 in [R-MVSNet](https://openaccess.thecvf.com/content_CVPR_2019/supplemental/Yao_Recurrent_MVSNet_for_CVPR_2019_supplemental.pdf) for an example.

D. Minor comments and suggestions (do not need to be addressed in rebuttal):
1. It would be better to have a list of contributions at the end of introduction. This can help the reader in exactly understanding what is novel and what is not in this article.  
2. Continuing further on the point B1, have the authors tried making the cost tensor even smaller (e.g. $12^5$) to 'stress-test' the MLP? How worse are the results? It might decrease memory consumption and inference time. I understand if these results cannot be reported within the rebuttal period, but it would be good to have these for final version.
3. Relating to point C4 above, it would be good to know the exact details of architectural changes done on top of CATs possibly in Appendix. Lines 7-9 of Appendix are not detailed enough in my opinion.
3. I assume the concept of neural fields presented in this paper is also interesting for optical flow and depth estimation applications. Thus in broader impact these applications can also be added. 
4. The reference of [30] is missing the field of year.
 
limitations: Only limitation is more computational burden (compute time and memory) than competitive approaches. This is not a major limitation because the idea itself is novel. Moreover the increase in memory consumption is still linear instead of $O(M^4)$ for $M \times M$-sized images for conventional approaches. Nonetheless these limitations need to be made more clear in the final manuscript.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new matching field generation method called a neural matching field (NeMF). It uses a cost embedding network consisting of convolution and self-attention layers to process the coarse cost volume to obtain cost feature representation. Through such a mechanism, NeMF can realize high-resolution semantic matching. During the training, to reduce the computational cost of the 5-D matching field, NeMF combines the convolution operations with the self-attention operations to get initial matching tensors.  NeMF also incorporates random sampling when calculating the local matching fields. 
strengths_and_weaknesses: ***Strength***
1. This paper is well-organized and can be easily understood by readers. The technical details are introduced clearly.
2. The authors conducted extensive experiments on multiple benchmarks to investigate the effectiveness of different modules and designs in this paper.
3. NeMF aims to get matching fields in high resolutions which can resolve the matching ambiguity during semantic matching.

***Weakness***
1. Some core techniques of this paper have been used in some disparity or optical flow estimation papers. In Figures 3 and 4, the way to construct 5-D matching cost volumes has been widely used in disparity estimation. Besides, although the 5-D matching cost volumes can get richer matching information, it takes in a huge amount of computational cost. When the resolution of the matching field is very large, it's too expensive to process these 5-D matching volumes.
2. In lines 137-138, to get the semantic matching field in high resolution, NeMF uses interpolation methods (as that in MMNet, ICCV 2021) to upscale the matching fields. But such an interpretation will lead to matching ambiguity obviously. Can these ambiguities brought by interpolation methods be eliminated in subsequent modules?
3.  In section 4.3 (lines 180-182), if NeMF uses random sampling, the training will encounter the long-tail problem. Because most of the point pairs will be classified as negative matching, and only very few matched point pairs can be classified as positive matching.  In fact, the negative point pairs are much more than the positive pairs. If the random sampling strategy is adopted, there will be no positive samples. And this will lead to the crash of the model training. I feel a little confused about the random sampling.
4. In lines 185-190, how to generate dense matching flow through sparse annotations is not well-introduced.
5. In Figure 5, is the inference conducted on the validation or test set? Why there is back-propagation during the inference process? Does NeMF use the output of PatchMatch as pseudo labels to train the model?
6. It seems that NeMF uses the output of PatchMatch as the pseudo label. But there is no ablation study to support it. It will make readers confused about whether the performance gain comes from the model itself or its supervision.
7. It seems that the computational cost of NeMF is relatively high.
questions: 1. What is the biggest difference between NeMF and other semantic matching methods based on 5-D matching cost volume?
2. It seems that the output of PatchMatch is used as pseudo labels to guide the training. 
3. During the inference, it's unclear why NeMF needs back-propagation. 
limitations: The computational cost of NeMF is relatively high. I don't find any other negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a novel method based on Implicit neural representation(INR) for semantic correspondence, named neural matching field (NeMF). Also a cost embedding network containing convolution and self-attention layers is established in order to represent complicated and high-dimensional continuous field better. During the inference phase, this paper introduced the patchmatch-based sampling and coordinate optimization which make the NeMF performs better and more efficient. Experiments on several standard benchmarks and ablation study validate the proposed architecture with the full resolution input image pairs.
strengths_and_weaknesses: Originality: Comparing to existing methods, this paper firstly leverage the implicit neural representation in semantic correspondence.

Quality: Benefiting from INR which is not coupled to spatial resolution, images can be input to the proposed network with their original resolution and SOTA performance is attained with competitive inference time. 

Clarity: The paper is somewhat clear, but some important details are missing or unclear

Significance: The paper is likely to have moderate impact on semantic correspondence

questions: Figure 3 illustrates the proposed network architecture but it is not clear, especially the concrete architecture of the cost embedding network.

Figure 6 demonstrates the flow maps for different N iterations, the author should give more explanations on the meanings of different colors.

Figure 7 shows several semantic matches on PF-PASCAL, however, it seems that the results of the third pair in the first two columns and the second pair in the last two columns look almost the same while some correspondences predicted by CATs are marked red. Additionally, the correspondence marked red of the last image pair in the third columns looks different from that reported in the original paper(CATs)

qualitative results of CATs on PF-WILLOW are different from that reported in the original paper (i.e. 50.3 when α=0.05 and 79.2 when α=0.1 )

limitations: This paper formulates predicting semantic correspondence as a classification problem which lacks of generalization between datasets, Concretely, the network trained on one datasets with fixed number of categories may not performer well on other datasets with unseen categories
review_title: null

Meta-review: metareview: The paper concerns itself with computing high resolution matchings. The authors propose to use represent matchings as maxima of neural "matching" fields, which is a novel and interesting theoretical contribution that allows to obtain high resolution matchings with fixed representation size of the neural field. The matchings are extracted from the neural field via coordinate optimization. State of the art performance is attained on a variety of semantic correspondence benchmarks. Reviewers also acknowledge that the paper is well written and easy to follow. On the downside are larger computational costs. Also newer versions of CATS, i.e. CATS++ (which is concurrent work), outperform the presented paper. Overall, the interesting theoretical contribution that might be useful in other domains and strong empirical performance make the paper a good fit for NeurIPS.
In a final version the reviewer recommendations must be taken into account.
confidence: Less certain
Decision: Accept