Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary.** Authors present a methodology for performing multi-task learning from data with disjoint and heterogeneous input domains. Particularly, they introduce an embedding of the inputs, in order to project each pair of input-output observations in a common continuous manifold where the exploration is significantly easier. Results show that the approach is valid with both synthetic and real-world data and they also demonstrate that the model is flexible when increasing/decreasing the dimensionality of the latent manifold.

**Strengths.** The explanation of the multi-task learning scenario with disjoint input domains is particularly well-written. This description makes easier to understand the reasons behind the introduction of the embedding between every single input and latent vectors z. Additionally, authors did an effort for explaining point-by-point the structure of the deep NN transformation behind the embedding. This is valuable. I appreciated the design of experiments and (author-blind) video on youtube was impressive.

**Weaknesses, Questions & Recommendations.** 
The main weaknesses (to me) in the paper are: 
[W1]. There is likely a lack of references and analysis about similar works on multi-task learning with the particular problem of disjoint inputs. This makes the reader doubt about the potential novelty of the model, in particular about the embedding.
[W2]. The notation based on subsets V_t is a bit confusing, (I think that keeping the (x,y,z) notation all along the paper would be better). Particularly in the pp.3, this notation is a difficult to follow before the introduction of the TOM embedding.
[W3]. The TOM implementation may be better placed before the experiments, being a bit better connected with the main section of the manuscript, but this is just an opinion.
[W4]. More analysis on the dimensionality D of the manifold could be of interest for the reader. In the last experiment, this dimensionality is pretty high. [Q] Why is this? What is the principal consequence?
[W5]. Error metrics in the experiments do not include confidence intervals or variance values from several runs.
[W6]. Typically, one chooses Discussion or Conclusion. The content of the Conclusion is similar to the thing said in the previous section.

Recommendations: 
[Rec1]. Motivating even better the disjoint input problem from the very beginning would make the paper stronger.
[Rec2]. An input-output notation all along the paper and some diagram explaining the projection into a continuous manifold would help as well.
[Rec3]. Details about the implementation could be better placed in the appendix, or at least integrated with the model and the flow of explanations.
[Rec4]. Confidence intervals in the tables of error metrics as well as a bit more of motivation for the circle experiment would improve the presentation of experiments.

**Reasons for score.** I understood the idea that authors presented and the problem of disjoint input domains. However, I feel that the presentation of the model is a bit weak as well as the experiments could be improved with a few details. The last pp. of the manuscript with the duplicity Discussion+Conclusion is also a bit odd. For this reason, I cannot recommend an acceptance score for this venue.

**Post-rebuttal comments.** Thanks to the authors for their response. The updated version of the manuscript addressed my main concerns and recommendations. Now, it is clearly improved, figures and metrics updated and the proposed methodology is better presented. Authors even did major changes on the structure of the paper, what I recognize as an important revision. Having said this, I raised my score.
review_title: Embedding for multi-task learning problems with disjoint inputs

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tries to solve a multitask learning task by building a task embedding use the training data and use a shared decoder to predict new data. 

The paper is well structured and easy to understand the general idea. This idea that maps the observed input and output into a shared space as the task embedding which is good. However, I start to get confused when I start to fetch more details and intuition behind this. I list some of my puzzles below and hope to hear from the authors:

1. The tasks could be unrelated. Do they need to have the same output dimension? Otherwise, you cannot use the same decoder for all the tasks.
2. When you train the g, are you still using observable data? 
3. I am a little puzzled about the meaning of x,  y,  z in each task. In the toy example (Figure 1). Are all x, y, z the positions? Do we need to predict the value given z or to predict y give z? What is the value mean in Figure 1? Meanwhile, can you explicitly introduce what x, y, z represents in other experiments? 
4. What does this sentence mean "the z’s are not known a priori, but they can be learned alongside f and g by gradient descent"(10th line on page two), from my perspective, z should be the input or the transformation of input, right? Because in equation 4, x_j is shown on the left side but the right side only contains z_j. 

The name of the model is somehow misleading: TRAVELING OBSERVER MODEL. Travel has the meaning of time flow. But it seems that you can see all the training data (observation) at any time (e.g. the last experiment). 

It will be better if the authors explain each abbreviation before using them, for example
- VEs:  Does it mean variable embedding? 
- HW, LN, MS, SNN in Table 4. 

Can the proposed method compare with some meta-learning baseline? For example[1], this paper is targeted at multimodal learning. They also have task embedding for each task and borrow the idea from FILM. Their task embedding also depends on the observable data.  They use meta loss to train the model.

I am sorry if I missed any part which has been explained in the paper. Looking forward to your reply.


[1]Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation

review_title: Confused in the experimental setting. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Paper is very well written and addresses an important topic; using Traveling Observer Model (TOM) in multi-task learning for tasks that do not have no spatial organization unlike, for example, images. Although the paper is said to be a first implementation of TOM, it does thorough experimenting and result analysis of its preformance from various aspects and by comparing it to many sophisticated models. Future research for improving and testing the algorithm is clearly detailed. 

Related scientific literature is sufficiently addressed,  mathematical background and the method are clearly presented, extensive and relevant experiments are done and result analyzed.

 I didn't even find any typos. 
review_title: Paper is very well written and addresses an important topic; using Traveling Observer Model in multi-task learning for tasks that do not have no spatial organization unlike, for example, images. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper presents the traveling observer model (TOM), a general framework to learn multiple heterogenous supervized (input,output) tasks, which are indexed by a continuous "variable embedding" that is automatically learned by the system. The authors show on simple problems that the learned task embeddings can recover an intuitive organization of the problems' variables in space or time. They also show that the model simultaneously trained on 121 seemingly unrelated classification tasks can outperform state-of-the art supervized methods fine-tuned on single tasks.

The proposed model is novel, technically sound, of broad interest and very promising. The paper is clearly written and easy to follow.  The presented experiments convincingly demonstrate the sensibility and usefulness of the approach. The topic perfectly fits the scope of ICLR.

Minor suggestions for improvement:
- Section 2 (first paragraph) the notations are a bit confusing here. First, the sample indices s=1...S_t are denoted as superscripts while task indices t=1...T are denoted as subscript, but then the sample indices are dropped and never used again, while task indices become superscripts and variable dimensions are denoted as subscript. The definitions of sets V_t^In and V_t^out is also strange. I think they should denote the union of all the spaces that variables are living in, but instead they are defined as finite sets of specific variables. The definition of "the universe" V in section 3 is also a bit sketchy. Is that a set of sets? a category?
- I think that when using a pre-defined "oracle" variable embedding, the proposed model becomes very similar or even equivalent to conditional neural processes (Gamello et al. 2018). It would be interesting to comment on that.
- There is an unfortunate double use of the letter h for two different things in equation (3) and (4)
- Sec. 4.4 "after joint training the model is finetuned on each task with at least 5K samples" -> is the whole model fine-tuned or only the function g? or g and h? Please clarify.
review_title: A novel and promising framework for heterogenous multi-task learning

Meta-review: Post rebuttal, the reviewers all recommend acceptance.
Decision: Accept (Spotlight)