Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 

This paper proposes a multi-pathway neural network for facial landmark detection with multitask learning. In particular, each pathway corresponds to one task, and the intermediate features are fused at multiple layers. The fused features are added to the task-specific pathway using a residual connection (the input of the residual connection are the concatenation of the task-specific features and the fuse features). The residual connection allows each pathway to selectively use the information from other pathways and focus on its own task.

This paper is well written. The proposed neural network architectures are reasonable. 

The residual connection can help each pathway to focus on its own task (suggested by Figure 8). This phenomenon is not guaranteed by the training objective but happens automatically due to the architecture, which is interesting. 

The proposed model outperforms several baseline models. On MTFL, when using the AlexNet, the improvement is significant; when using the ResNet18, the improvement is encouraging but not so significant. On AFLW (trained on MTFL), the improvements are significant in both cases. 

What is missing is the comparison with other methods (besides the baseline). For examples, it will be helpful to compare with existing non-multitask learning methods, like TCDCN (Zhang et al., 2014) (it seems to achieve 25% failure rate on AFLW, which is lower than the numbers in Figure 5), and  multi-task learning method, like MTCNN (Zhang et al., 2016). It is important to show that proposed multitask learning method is useful in practice. 
In addition, many papers take the average error as the performance metric. Providing results in the average error can make the experiments more comprehensive.

The proposed architecture is a bit huge. It scales linearly with the number of tasks, which is not quite preferable. It is also not straightforward to add new tasks to finetune a trained model. 

In Figure 5 (left), it is a bit weird that the pretrained model underperforms the nonpretrained one. 

I am likely to change the rating based on the comparison with other methods.



review_title: Interesting method; better to do comparison with previous methods

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
1. This paper proposed a new block which can aggregate features from different tasks. By doing this, it can take advantage of common information between related tasks and improve the generalization of target tasks.

2. The achievement in this paper seems good, which is 24.31%.

Cons:
1. The novelty of this submission seems a little limited.

2. The target task utilized in this paper is too simple, which only detects 5 facial landmarks. It is hard to say this proposed work can still work when facing more challenging tasks, for example, 60+ facial landmarks prediction.

3. " Also, one drawback of HyperFace is that the proposed feature fusion is specific to AlexNet," In the original submission, HyperFace is based on AlexNet, but does this mean it can only work on AlexNet?
review_title: This paper proposed a new block to combine domain-specific information from related tasks, in order to improve generalization of the target tasks. Although the relative improvement seems high (24.31%), its novelty is a little limited, and the target task in this submission(5 landmarks detection) is too simple to prove the effectiveness. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The collaborative block that authors propose is a generalized module that can be inserted in deep architectures for better multi-task learning. The problem is relevant as we are pushing deep networks to learn representation for multiple tasks. The proposed method while simple is novel. The few places where the paper needs improvement are:

1. The authors should test their collaborative block on multiple tasks where the tasks are less related. Ex: Scene and object classification. The current datasets where the model is evaluated is limited to Faces which is a constrained setting. It would be great if Authors provide more experiments beyond Faces to test the universality of the proposed approach.
2. The Face datasets are rather small. I wonder if the accuracy improvements hold on larger datasets and if authors can comment on any large scale experiments they have done using the proposed architecture. 

In it's current form I would say the experiment section and large scale experiments are two places where the paper falls short.  
review_title: The authors propose a collaborative block that can be inserted in any deep network for multi-task learning and evaluate the method on multiple tasks related to Faces

Meta-review: The experimental work in this paper leaves it just short of being suitable for acceptance.
The work needs more comparisons with prior work and other approaches.
The numerical ratings of the work by reviewers are just too low.

Decision: Reject