Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed a semi-structured composition of free-form filters and structured Gaussian filters to learn the deep representations. Experiments demonstrate its effectiveness in semantic segmentation. The idea is interesting and somewhat reasonable but I still have several concerns. However, I still have several concerns:
1.	The authors proposed to compose the free-form filters and structured filters with a two-step convolution. The authors are expected to clarify why and how the decomposition can realized its purpose? The authors need to further justify the methods by providing more theoretical analysis, and comparing with alternative methods. 
2.	The experiments are rather insufficient, and the authors are expected to make more comprehensive evaluations, e.g., more comparisons with the traditional CNN models. 
3.	The improvement is rather incremental compared with the alternative methods. The authors actually introduce some prior to the learning process. It would be better if the authors could show some other advantages, e.g., whether it can train the model with smaller number of samples, and whether we can integrate other prior besides Gaussian filters for other structures since Gaussian is a good prior for blurring. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes semi-structured neural filter composed of structured Gaussian filters and the usual structure-agnostic free-form filters found in neural networks. They are optimized using end-to-end training. Effectively, this lead to increased receptive field size and shape with just few additional parameters. Further, this module is architecture agnostic and can also be integrated with any dynamic inference models. Specifically, when applied on deformable convolutional filters, the deformation at each input can be structured using gaussian filters. Empirical experiments suggest that when integrated with state-of-the-art semantic segmentation architectures, the absolute accuracy on Cityscapes improves by 2%. Large improvement in seen on naive / sub-optimal architectures for segmentation.

Given that this is first work which demonstrates the efficient composition of classic structured filters with neural layer filters, I believe that research community will benefit to good extent if this paper is accepted.

Clarification:
1. I note that single gaussian is shared across different free-form filters. Is same gaussian also shared across input channels ?
2. For dynamic inference, what is the sampling resolution used ? How is it related to diagonal elements of covariance ? 2\sigma ?
3. In case of blurring and resampling, does the model learn another filter for sampling ? To me, sampling seems similar to dynamic inference operation but with static parameters.
4. As noted in paper, blurring is fundamental hwen dilating. Does DRN-A and DLA-34 models used for comparison in Table 1 includes blurring prior to dilation ?

Additional experiment:
1. Does improved receptive field size and shape also lead to improvement in other downstream tasks such as classification, object detection, depth estimation etc. ?
2. Table 4 shows that the networks with reduced depth when integrated with composed filters can perform as well as large networks. Does this holds true when extended to above tasks ? 
3. I note that in all the presented results, the composed filters are only included at the last few layers. How the results prunes out when included at the lower as well as at the intermediate layers ? Please include a plot of accuracy vs depth (at which it is included).
4. I am glad to note that Gaussian deformable models performs as good as free-form deformable models with largely reduced parameters. Can you please add total network parameters comparison in Table 5 ? Further, are these also included only at the top few layers ?
5. In Table 1, DLA-34 + DoG ?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:
- key problem: improved visual representation learning with limited increase in parameters by leveraging Gaussian structure;
- contributions: 1) compose Gaussian blurs and free-form convolutional filters in an end-to-end differentiable fashion, 2) showing that learning the covariance enables a factorized parameter-efficient representation covering wide and flexibly structured filters, 3) experiments on CityScapes showing the proposed layers can help improve semantic segmentation performance for different architectures (DRN, DLA, and ResNet34).

Recommendation: weak reject

Key reason 1: mismatch between the generality of the claims and experiments.
- Learning to adapt and optimize receptive fields successfully would be a great fundamental improvement to CNN architectures. Experiments are done on a single dataset for a single task, which seems insufficient to support the generality of the approach and claims in the submission. I would recommend using other datasets (e.g., COCO) and tasks (e.g., object detection, instance segmentation, depth estimation/completion), where the benefits of the approach could be demonstrated more broadly and clearly (including its inherent trade-offs).
- The improved efficiency (one of the main claims) is only assessed on the number of parameters, which is a direct consequence of the parametrization. Is it significant at the scale of the evaluated architectures? Does it result in runtime performance benefits? If it is indeed a useful structural inductive bias, does it result in improved few-shot generalization performance or less overfitting? Does it enable learning deeper networks on the same amount of data?
- Why modifying only later layers in the architecture (end of 4.1)? It seems that early layers would make sense too, as it is where most of the downsampling happens.

Key reason 2: lack of clarity and details.
- Section 1 and the beginning of section 4 are repetitive and verbose; in particular, Sections 4.1 and 4.2 would benefit from less textual descriptions replaced by more concise mathematical formula (simpler in this case), especially in order to know the details behind the methods compared in Tables 1-2-3. 
- Overall, the paper could contain less text describing the hypothetical advantages of the method and the basic preliminaries (section 3), to focus more on the method itself, its details and evaluated benefits. In particular, the dynamic part (section 4.2) is unclear and the method is mostly described in one sentence: "To infer the local covariances we learn a convolutional regressor, which is simply a convolutional filter." Another example of the lack of details is "many" vs. "some" in the "params" column of Table 4.
- There is also a missed opportunity to provide compelling feature visualizations and qualitative experiments (beyond Fig. 7). For instance, what are the typical covariances learned? What are the failure modes that the proposed modifications address, in particular w.r.t. thin structures and boundaries that are typical hard cases for semantic segmentation and where blurring might be counterproductive?

Additional Feedback:
- missing reference: Learning Receptive Field Size by Learning Filter Size, Lee et al, WACV'19;
- missing reference (w.r.t. local filtering): Pixel-Adaptive Convolutional Neural Networks, Su et al, CVPR'19



review_title: Official Blind Review #3

Meta-review: The paper proposes an interesting idea of inserting Gaussian convolutions into ConvNet in order to increase and to adapt effective receptive fields of network units. The reviewers generally agree that the idea is interesting and that the results on CityScapes are promising. However, it is hard not to agree with Reviewer 3, that validation on a single dataset for a single task is not sufficient. This criticism is unaddressed. 
Decision: Reject