Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Authors proposed a spatial mixture-of-expert layer for CNNs to take into account of spatial dependence in data or features.
strengths_and_weaknesses: Strengths: to incorporate local dependence or locality in data and features is always beneficial. Yes, most CNNs do not explicitly consider this or have mechanisms to embed this feature into the learning process.

Weaknesses: authors create an impression that the proposed scheme is the only one that consider locality in data. There are always many mechanisms to incorporate local dependences in CNNs such as graphs, CRF or MRF, in segmentation and recognition. The proposed scheme seems only marginally improve performances if any, while added computational costs may outweigh the benefit.

Title is misleading - word "Layer" should be added.
questions: Computational costs should be analyzed and compared.
limitations: Okay
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a method based on spatial MoE for location-dependent learning. Unlike the translation-equivariant network (i.e., CNN) that shares the expert (convolution kernel) locally, the proposed method assigns the convolution kernel conditional on the location. Specifically, the work proposes a novel gating function to select an expert for each location and use the expert to extract information for the location. The paper validates the effectiveness of their method in multiple tasks and shows the SOTA results on medium-range weather prediction.
strengths_and_weaknesses: Originality:
The paper presents a novel viewpoint on utilizing deep networks. For some tasks, it's beneficial to violate the translation-equivariance. This is very interesting to the research community. Furthermore, the proposed method is also novel. The proposed gating function and classification loss obviate the need for a variety of regularizers.

Quality:
The method supports the claims well with multiple experiments. The only concern I have is the missing analysis of mode collapse -- a typical pitfall of MoE. For completeness, it would be better to give more analysis -- for example, is there any mode collapse and any insight/discussion on why the proposed method can avoid it?

Clarity:
The paper is well-written in general. I only have minor comments.
- The motivation behind the proposed tricks is missing. The paper didn't show the logic behind the proposed tricks. For example, why is the expert error damping essential, and the logic behind its design?

Significance:
The method is novel. The results are also strong and very interesting -- by violating translation-equivariance, the method shows improved performance.
questions: Does the gating learn the translation-equivariance? It might be interesting to look at. The reason I ask is that, for location-dependent earth data, I still believe that the translation-equivariance could be helpful since patterns are repetitive. And as shown in Fig.3, I saw that there is a local pattern (e.g., repetitive local patches) in the expert selection.
limitations: The author addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new way of tailoring the processing of subparts of the input data based on their localisation, in the case where all samples from a given dataset share the same spatial structure. This can be of use in cases where treating the localisation as a mere input feature is not enough, but appeals for more drastic measures breaking the spatial invariance properties of some classic networks such as CNN.

The method presented relies on learning different experts, each being specialized in processing certain parts of the input data. At each localisation, the choice of experts to use is done by looking at a "routing table", scoring each expert for each localisation, regardless of the input sample. There is only one routing table for the whole dataset: the routing is independent of the input sample. It is learnt via a newly introduced method treating the routing as a classification task, with labels derived from the loss of the task at hand.

The method is shown to compare favorably to previous baselines on a task created for this paper where the "true parameters" of the model generating the sythetic data are known, so as to perform an in-depth ablation study highlighting the impact of each design decision. On a weather-prediction benchmark using real-world data, the method is shown to be state-of the art, and can also bring a slight improvement to a ResNet 50 on Imagenet.
strengths_and_weaknesses: $\textbf{Strengths}$:

The paper is clearly motivated and well written overall. The method introduced to train both the routing system and the experts at the same time is original and kept simple, the relevance of each component is thoroughly tested, and the robustness of the model to changes in variables unknown by the user is challenged in a controlled setting on synthetic data. The method is shown to work in practice on real-world data.

$\textbf{Weaknesses}$:

* The explanation of the procedure used to create the labels for the classification task to train the routing tensor is a bit unclear to me: the quantile of exactly which distribution is used to create the labels ? 
* In WeatherBench (Table 3), SMoE is only compared to one method whereas other models leveraging the position where tested and shown to be on par with SMoE performances on the heat diffusion data (Figure 3.a) and simple Image datasets (Table D.1 and D.2). It would be interesting to know how those previously introduced methods fare on a case where it seems breaking the spatial invariance makes sense.
questions: * On the heat diffusion dataset, did you try to compare yourselves to some baseline CNN architectures having the same number of parameters  as your SMoE model ? (In Figure 3, I only see a line writing 146 for the CNN vs 27+12288 for SMoE)
* Is it possible to compare SMoE to the other methods on WeatherBench ?
* The quantile of exactly which distribution is used to create the labels in the training of your routing tensor ? 
* In the practical implementation part, you said that all experts at all points are evaluated. If I am understanding right, this means that in the last layer of your model, instead of using the $\texttt{scatter}$ and $\texttt{gather}$ functions of Pytorch, you could directly evaluate the loss for all experts. Would it be useful to make use of this signal to train your routing system ?

* (there is a small typo line 311: "Each SMOE ~~has~~ selects")
limitations: Yes, the authors adressed the limitations and potential negative societal impact in their discussion paragraph.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a location-based mixture of experts for use in applications where modeling depends on the image grid location, such as weather forecasting.  Experts are selected using a map of size (E,H,W), where E is the size of experts pool, so that one set of top-k experts are learned for each grid location.  In order to encourage the system to select appropriate experts, an auxiliary loss is added that classifies whether the selected experts are "correct" or "incorrect" based on quantile of the gradient magnitude; experts with large error gradient are assigned a uniform classification target among the other experts, encouraging the system to find a new one.  The largest gradients are also scaled down in their bprop application, with the intuition that they will be applied to experts that may no longer be selected.  The system is verified on a toy dataset constructed from synthesized heat diffusion, and evaluated on WeatherBench and Imagenet, obtaining significant improvements in weather forecasting.

strengths_and_weaknesses: The idea of incorporating location dependence for some applications, like weather forecasting, is sound, and well-suggested by the results in this paper.  Recovery of near-exact diffusion kernels in the toy data is good to see.  However, there are some additional baseline comparisons I think are missing, in particular, LCN, LRLCN and resnet with wider layers to match the number of parameters of SMoE on WeatherBench.  SMoE with a fixed set of random 128/256 experts at each location selected, as in the init, without learning the gating map, would also be interesting, as improvement over this would show the effect of learning the map in WeatherBench.  Measuring on an additional relevant application, such as face landmarks or another of the ones mentioned in the intro, would also strengthen the demonstrated reach of the method.

There are also some details around the order of the channels (experts) selected, and if that matters.  Are the top ones per location selected in index order, or sorted gate score order?  When a filter changes, and an expert i is replaced with expert j, are experts with indices between i and j will shifted in the output representation happens?  This would look like a sudden shift permutation to the next layer.  And if experts are selected in score order, then as scores change, input channels to the next layer will swap when scores change during training.

As an aside, I think it's debatable whether the system described is a mixture of experts.  It seems more to be a channel subselection, since the outputs are concatenated instead of mixed, and subselection is not dependent on the same input as the layers.  This is a very minor point, though, it seems close enough to me to be an accurate description.

Overall, this is an interesting idea, and one that clearly seems effective at incorporating effects of spatial differences, such as land vs water in weather prediction.  A few comparisons appear to be missing in the experiments (described above), and while the toy heat diffusion experiment and existing ablations serve to profile the behavior of the system, I still would have liked to see more discussion on the effects of discontinuous subselection to downstream layers.

questions: Additional questions:

l.194:  The claim that sparse selection results in the same number of FLOPS as a standard conv layer with same number of filters, is not quite right.  For each location, the additional patch context must also be included.  In the worst case, each neighboring location selects different experts in a 3x3 checkerboard pattern, in which case FLOPS required would be that of a standard convnet with the same number of filters times the spatial kernel size (e.g. 9).  And even this depends on having a sparse implementation when there are more than 9 experts to select from.

Fig 3:  do you have any ideas on why LRLCN-ND performs worse than SMoE on the toy example? it's a very similar architecture for 1 output channel, the only difference is that it uses per-location softmax mixture of the 3 kernels, rather than top1 max.  will adding RC loss fix this?

If land and water are in different locations between images, is it possible to feed the land/water map in as a binary map and gate based on this, rather than location?

limitations: -
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a convolutional neural architecture that learns parameters specialized towards different locations in the input image. This can be seen as a middle way between a ConvNet, where all parameters are shared across spatial locations and fully-connected networks, where all parameters are specific for their location. Specifically, the authors propose a learnable matrix that assigns each spatial location to a subset of convolutional filters. To learn this matrix, the authors propose a routing classification loss that aims to change assignments for locations with large gradients. The method is then evaluated on a small set of specialized datasets, where it performs well.
strengths_and_weaknesses: Making neural models larger has been an effective way to improve model performance. Making those large models sparse has been shown to be a successful way to combine large capacity models with fast (theoretical) inference and training speeds. As such the paper at hand is of interest to the research community. 

Overall the paper is well presented and the method is clearly explained. 

One concern is that the authors make a very strong assumption in that SMOE relies on the spatial structure to be similar across all samples in a dataset. This leads to very limited applications and no generalization across datasets. Given this very large limitation, it would have been great if the authors could extend their motivation and highlight a series of key tasks where such a model could provide impact. 
questions: Would the authors be able to highlight a series of key tasks where such a model could provide impact?
limitations: As far as I can see, the authors have not addressed potential limitations. However, I do not see any clear issues.
review_title: null

Meta-review: metareview: This work introduces a novel layer in order to extract information better at a fine grained level. The basic idea is to introduce a spatial mixture of expert which will localized computations to a specific part of the image. Given the novelty the method, I recommend accepting this paper. However, please add the modifications proposed to reviewer vuZ5.
confidence: Less certain
Decision: Accept