Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work introduces fixed grouping layers to deep learning models. Unlike convolutional layers, the fixed grouping layer only allows the output to be impacted by the specific inputs associated to it by its group. The paper stays firmly in the area of brain scans with the authors hoping to generalize to other applications later. Their experiments compare to what they consider to be the state-of-the-art approaches to the problem. 

I did find Figure S2A helpful to quickly understand some of the simulated data approach, maybe it should be in the paper and not supplementary. 

I argue for accepting this paper because the experiments, while focused solely in one discipline, are thorough. I would have liked clearer description of other areas that might benefit in the last sentence. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: "Towards a Deep Network Architecture for Structured Smoothness" proposes a new layer, dubbed FGL, specifically focused on structured smoothness.
The paper clearly discusses the proposed layer, introduces the necessary formulation, and demonstrates the effectiveness of this layer in a suite of practical applications in fMRI analysis. The writing is clear, the method is well formulated, and performance compared to relevant and recent benchmarks is strong. Thus most of the questions remaining are about the finer details of the proposed layer and method.

What is the sensitivity of FGL to non-optimal group proposals for A? Given the authors reference Aydore et. al., do randomized (or perhaps meta-optimized) groupings perform poorly? Are there any potential ways to jointly-or-iteratively learn the groupings? Is Ward clustering the primary practical method for getting the groupings?

Can the authors discuss a bit the relationship and potential tradeoffs between "early" segmentation architectures (such as FGL), and those often favored in semantic segmentation ("late" segmentation)? Is there a particular reason why "late" segmentation is particularly a poor choice in brain imaging?

My primary concerns come in relation to other methods for structured smoothness such as CRF/MRF, and the trade-offs and drawbacks of imperfect or downright wrong A, and alternative methods for practically finding good A matrices (potentially with an eye to applications outside fMRI). Addressing some or all of these in a section or two of the text body would potentially raise my score.

Some small errors:
Optimg -> Opting Introduction
Guassian -> Gaussian section 3.2

Potential useful references:
CRF as RNN https://arxiv.org/abs/1502.03240
Efficient Piecewise Training https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Lin_Efficient_Piecewise_Training_CVPR_2016_paper.html
Deep Learning Markov Random Field for Semantic Segmentation https://arxiv.org/abs/1606.07230
Pixel Adaptive Convolutional Neural Networks http://openaccess.thecvf.com/content_CVPR_2019/html/Su_Pixel-Adaptive_Convolutional_Neural_Networks_CVPR_2019_paper.html
review_title: Official Blind Review #1

Meta-review: The AC has carefully looked at the paper/comments/discussion in order to arrive at this meta-review.

Looking over the paper, the FGL layer is an interesting idea, but its utility is only evaluated in a limited setting (fMRI data), rather that other types of images/data. Also, the approach seems to work on some of the fMRI datasets, on others the performance is on par with the baselines. 

Overall, the paper is borderline but the AC believes the paper would be a good contribution to the conference.
Decision: Accept (Poster)