Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a temporal probabilistic asymmetric multi-task learning model for sequential data. The asymmetric property lies in the feature level.

This paper criticize that the task loss used in previous works may be unreliable as a measure of the knowledge from the task. However, the feature uncertainty used in the proposed model can also be unreliable and I cannot see any reliable guarantee. 

The organization of Section 3 is not good. The logics in different subsections are not so clear. Some notations seem undefined.

Eq. (4) defines p(z_d|x,\omega). But later it defines z_d is from p(z_d|x,y_d,\omega). The difference between these two distributions lies in y_d. I don’t know which one is used. What is p_\theta(z_d|x,\omega)? The notations need to be properly defined.

Based on Section 3.3, the proposed model seems to require different tasks have the same total time step T and this requirement is a bit strong.

It seems that there is no Figure 2c.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors proposed an asymmetric multi-task learning method on temporal data, and applied it to clinical risk prediction.

Overall, the problem studied in this paper is interesting, the proposed method looks reasonable, and the application is also interesting. However, there are also some concerns.

1. The details on how to construct F_\theta to generate the hyper-parameter of a in (10) are missing. As the proposed MTL method is asymmetric, F_\theta(f, g) is supposed to be different from F_\theta(g, f), right? Otherwise, the transfer weight \alpha from f to g would be the same as the one from g to f. What is the design of F_\theta.  Moreover, the authors mentioned that "the network F can learn a meaningful distribution of the transfer weight, such that it sets the value of \alpha high when f and g are related, with low uncertainty on f and high uncertainty on g." However, only based on (8), (9) and (10), it is quite difficult to understand why the claim can be implemented. More details are needed.

2. The complexity analysis is only focused on the relation to the number of timestamps T, without taking the number of tasks D into consideration. As the proposed method is asymmetric across tasks and across time, when the number of tasks is large, the scalability may be an issue. In real-world applications, e.g., in a big hospital, the number of patients can be very large. In this case, is the proposed method practical? 

3. Though the proposed method looks reasonable, it contains many components or networks. I guess to train such a composition network precisely needs a lot of tricks in practice.
 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The submission proposes a probabilistic method for multitask learning for sequence data focusing on the use of uncertainty to control the amount of transfer between tasks. In comparison to prior work, the approach directly models uncertainty by parametrising Gaussian distributions and dropout variational inference. The approach additionally models transfer between features from different time-steps. The evaluation shows that the method outperforms recent state-of-the-art models on two medical benchmarks.

The paper is overall well written but includes some vagueness regarding the proposed method as well as the experimental setup. While describing techniques to model two kinds of uncertainty the aspect of the model which adapts transfer to the amount of uncertainty in source and target features is not described in detail (F_theta). It is described as if it only received samples for both features without any additional information regarding uncertainty. This makes it improbable that the model is able to take the feature’s uncertainty into account when modelling transfer.

The paper follows in section 3.3 with an argument for the lower computational complexity of only transferring from previous time-steps. I might misunderstand the description but having every time-step’s features be influenced only by previous timesteps should still result in complexity O(T^2) (even though the amount of computation is reduced by a constant factor). 

My background is not in machine learning with clinical data (so I do not know common datasets) but it is unclear to me why the datasets evaluated on only consist of a very small section of the overall dataset they’re taken from (in case 1). Looking at table 4 the main factor seems to be probabilistic modelling so additional ablations with either version of uncertainty would be interesting.

By introducing probabilistic modelling to control the amount of transfer between tasks the paper provides an interesting perspective and is able to show strong performance. However it is also quite vague on important aspects such as the exact modelling of transfer and limited regarding evaluation.

Minor aspects:
- V from the final equation on page 5 is never described.
- After describing modelling both kinds of uncertainty in the method section, the reader is uninformed which uncertainty is shown in plots ⅘
- Table 4 does not have the two highest results in bold (see AMTL samestep)
- Very limited multitask and transfer learning references pre 2016.
- Some descriptions are unclear including the mention of ‘attention allocation’

review_title: Official Blind Review #2

Meta-review: The authors propose a method for multi-task learning with time series data. The reviewers found the paper interesting, but the majority found the description of the method in the paper confusing and several technical details missing. Moreover, the reviewers were not convinced that the technique used for uncertainty quantification of the features at each stage of the time series is well founded.
Decision: Reject