Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Despite the interesting results, the paper is a very empirical paper on mixing of experts. Mixing of experts is a very old issue. Authors should cite some references around mixing of experts that would help the reader to understand the problem at hand and to asses the contributions.

Honestly i do not understand all the nets proposed (Figure 4, doesn't help me). In general section 3 should be improved. 

Around the idea of mixing of experts i remember a paper that was published in Nature (i think so) where the authors proposed a very interesting idea for mixing experts beyond the typical weight associated to the expert reliability. The authors propose to ask an additional question to the experts about what they think the other experts are going to answer. And use this additional question to detect where an expert is a good expert. For instance, when one expert is sure about his decision but at the same time he knows that the problem is hard he thinks that the other experts (or some group) are going to fail and then he is going to claim that his answer is A but others experts answer is going to be B. 

In my opinion, these are the things that would be interesting to explore, beyond weighting opinion. Perhaps a NN could help to solve this problem without that additional question. Perhaps this paper is in that direction but sorry i couldn't understand it.



review_title: Empirical results on an unique dataset using mixing of experts

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work aims to improve classification accuracy in cases where there is high disagreement among labelers, some of which may be due to systemic differences between labelers.  The general approach is simple and interesting, making separate predictions for each labeler individually, and averaging at test time.  Weights to make this a weighted average are also learned, and two additional conditionings for the model are explored.  A single dataset, to classify diabetic retinopathy, is explored.

Overall I feel this is an interesting approach, though a few details could be better explained and justified, in my opinion:

- If a single doctor does more labeling than any other doctor, the majority vote may tend to favor this labeler (they have more chances to be in the majority).  Would the learned weights then mostly just favor the most frequent doctor, and how could this be detected, since they may have provided a disproportionate number of labels in the test set as well?  Is there any rebalancing between doctors (as opposed to classes)?

- The appendix mentions a step where the biases are adjusted to account for class frequencies in the test set.  IMO this is a slightly questionable step, assuming that the test distribution is known, but this indeed may be the case in many situations.  Also I'm supposing that all baselines in comparisons also used this adjustment -- is this the case?

- I feel the summary of the loss theta_ll' could be a bit clearer:  What is the final loss exactly?

review_title: null

Decision: Reject