Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a DNN, called subspace network, for nonlinear multi-task censored regression problem. The topic is important. Experiments on real data show improvements compared to several traditional approaches.

My major concerns are as follows.

1. The paper is not self-contained. The authors claim that they establish both asymptotic and non-asymptotic convergence properties for Algorithm 1. However, for some key steps in the proof, they refer to other references. If this is due to space limitation in the main text, they may want to provide a complete proof in the appendix.

2. The experiments are unconvincing. They compare the proposed SN with other traditional approaches on a very small data  set with 670 samples and 138 features. A major merit of DNN is that it can automatically extract useful features. However, in this experiment, the features are handcrafted before they are fed into the models. Thus, I would like to see a comparison between SN with vanilla DNN. 
review_title: The authors propose a DNN, called subspace network, for nonlinear multi-task censored regression problem. The writing needs more elaboration. The experiments are unconvincing.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a new multi-task network architecture within which low-rank parameter spaces were found using matrix factorization. As carefully proved and tested, only one pass of the training data would help recover the parametric subspace, thus network could be easily trained layer-wise and expanded.

Some novel contributions:
1. Layer by layer feedforward training process, no back-prop.
2. On-line settings to train parameters ( guaranteed convergence in a single pass of the data)

Weakness :
1. The assumption that a low-rank parameter space exists among tasks rather than original feature spaces is not new and widely used in literature.
2. The proof part(Section 2.2) can be extended with more details in Appendix.
3. In synthetic data experiments (Table1), only small margins could be observed between SN, f-MLP and rf-MLP, and only Layer 1 of SN performs better above all others. 
4. Typo: In Table2,3,5, Multi-l_{2,1} (denotes the L2,1 norm) were written wrong.
5. In the synthetic data experiments on comparison with single-task and multi-task models, counter-intuitive results (with larger training data split, ANMSE raises instead of decreases) of multi-task models may need further explanation. 
6. Extra models like Deep Networks with/without matrix factorization could be added. ( As proposed model is a deep model, the lack of comparison with deep methods is dubious)
7. In Section 4.2, the real dataset is rather small thus the results on this small dataset were not convincing enough. SN model outperforms the state-of-the-art with only small margin. Extensive experiments could be added.
8. The performance on One-Layer Subspace Network (with only the input features) could be added. 

Conclusion:
Though with a quite novel idea on solving multi-task censored regression problem, the experiments conducted on synthetic data and real data are not convincing enough to ensure the contribution of the Subspace Network. 

review_title: This paper introduces a multi-task network architecture within which low-rank parameter spaces were found using matrix factorization. As carefully proved and tested, only one pass of the training data would help recover the parametric subspace, thus network could be easily trained layer-wise and expanded.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work proposes a multi task learning framework for the modeling of clinical data in neurodegenerative diseases. 
Differently from previous applications of machine learning in neurodegeneration modeling, the proposed approach models the clinical data accounting for the bounded nature of cognitive tests scores. The framework is represented by a feed-forward deep architecture analogous to a residual network. At each layer a low-rank constraint is enforced on the linear transformation, while the cost function is specified in order to differentially account for the bounds of the predicted variables.

The idea of explicitly accounting for the boundedness of clinical scores is interesting, although the assumption of the proposed model is still incorrect: clinical scores are defined on discrete scales. For this reason the Gaussian assumption for the cost function used in the method is still not appropriate for the proposed application. 
Furthermore, while being the main methodological drive of this work, the paper does not show evidence about improved predictive performance and generalisation when accounting for the boundedness of the regression targets. 
The proposed algorithm is also generally compared with respect to linear methods, and the authors could have provided a more rigorous benchmark including standard non-linear prediction approaches (e.g. random forests, NN, GP, …). 

Overall, the proposed methods seems to provide little added value to the large amount of predictive methods proposed so far for prediction in neurodegenerative disorders. Moreover, the proposed experimental paradigm appears flawed. What is the interest of predicting baseline (or 6 months at best) cognitive scores (relatively low-cost and part of any routine clinical assessment) from brain imaging data (high-cost and not routine)?

Other remarks. 

- In section 2.2 and 4 there is some confusion between iteration indices and samples indices “i”. 

- Contrarily to what is stated in the introduction, the loss functions proposed in page 3 (first two formulas) only accounts for the lower bound of the predicted variables.  

-  Figure 2, synthetic data. The scale of the improvement of the subspace difference is quite tiny, in the order of 1e-2 when compared to U, and of 1e-5 across iterations. The loss function of Figure 2.b also does not show a strong improvement across iterations, while indicating a rather large instability of the optimisation procedure. These aspects may be a sign of convergence issues. 

- The dimensionality of the subspace representation importantly depends on the choice of the rank R of U and V. This is a crucial parameters that is however not discussed nor analysed in the paper. 

- The synthetic example of page 7 is quite misleading and potentially biased towards the proposed model. The authors are generating the synthetic data according to the model, and it is thus not surprising that they managed to obtain the best performance.  In particular, due to the nonlinear nature of (1), all the competing linear models are expected to perform poorly in this kind of setting.

- The computation time for the linear model shown in Table 3 is quite surprising (~20 minutes for linear regression of 5k observations). Is there anything that I am missing?

review_title: Interesting idea which is however not clearly developed. Incremental results.

Meta-review: Authors present a method for modeling neurodegenerative diseases using a multitask learning framework that considers "censored regression" problems (to model where the outputs have discrete values and ranges). Given the pros/cons, the committee feels this paper is not ready for acceptance in its current state.


Pro:
- This approach to modeling discrete regression problems is interesting and may hold potential, but the evaluation is not in a state where strong meaningful conclusions can be made.

Con:
- Reviewers raise multiple concerns regarding evaluation and comparison standards for tasks. While authors have added some model comparisons in response, in other areas comparisons don't appear complete. For example, when using MRI data, networks compared all use features derived from images, rather than systems that may learn from images themselves. Authors claim dataset is too small to learn directly from pixels in this data (in comments), but transfer learning and data augmentation have been successfully applied to learn from datasets of this size. In addition, new multitask techniques in the imaging domain have also been presented that dynamically learn the network structure, rather than relying on a hand-crafted neural network design. How this approach would compare is not addressed.



Decision: Reject