Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Short summary of paper:
The paper investigates representation learning  from trajectories in a framework which generalizes MDPs (NMRDPs), in which rewards are non-Markovian and follow a hidden process (this can be seen as a more structured, special case of a POMDP). The authors suggest learning a trajectory embedding by using a triplet loss, justified by a sufficient condition for learning an embedding which corresponds to the true task id.  

Overall I felt the paper fell a bit short of the standards for ICLR, based on subpar writing, lack of comparison to convincing baselines, and unclear applicability to more complex environments.

Main issues:
- The methodological sections of the paper seems conceptually accessible while being written in an overly mathematical, meandering fashion, using heavy, sometimes unclear notation, or referring to notation from other papers.

As an example, \mathcal{M}^* is first introduced by reference to another paper (the notation does not really need to be introduced until the mapping is made precise later in the paper); \phi(\theta)_i^j(-1) is an unnecessarily heavy, unpleasant notation for the concept used. 
It was unclear if T is meant to represent the total trajectory length or a time index of a given trajectory (the notation would imply the former, the way it used, the latter).  Similarly equation (6) is a very mathematical way to denote a relatively simple condition (which could be written more simply by comparing inf and sup distances between two trajectory embeddings corresponding to different or identical latent h). Again, equation (7) seems like a contrived way to write a relatively simple concept, and the simple proof that (6) implies (7) is not included.
I feel this section could be improved by using precise but as simple notation as possible, and a sequence of propositions explaining why the triplet loss allows for identifiability of the reward process, with a clear flow between statements.

- Lack of baselines:
No alternative or baseline seems considered in the paper. The setting is perhaps a bit unusual, but since the data considered is essentially sequence of observations, any model of sequences could be used to produce embeddings (for instance, recurrent VAE, autoregressive models, time-contrastive methods) and compare the different trajectory representations.

- Unclear generalization to more complex problems:
(6) only implies (7) as far as I can tell, is sufficient but not necessary, and may be too strong of a condition to enforce; similary, theoretically requiring injectivity of phi seems practically too strong of a condition, as it essentially requires hashing each trajectory into a different embedding, in spite of the fact that many aspects of observations may be irrelevant for the task at hand). A key difficulty to scale this algorithm will be to find how to distance between trajectories in high dimension - perhaps the most important question for representation learning of trajectories, and a point the paper does not address.


Positives:
The method is overall well principled and the theoretical justification of the triplet loss is interesting. The dataset used for experiments is also interesting.

Minor/Questions:
- IIUC, the equation (3) needs to hold for all times T, this needs to be clearly stated as often T is used to denote total episode length.
- Bottom of page 4, 'The construction of the task ... \phi_T=\phi(\mathcal T(o_T, \phi_{T-1}); does this not require injectivity of \phi, which is only stated afterwards?
- Calling the algorithm 'EM' is a bit of a stretch (there are no latent variables), it is better to call it a form of coordinate ascent.
- The way the EM-like algorithm is set up, it appears the gradient with respect to theta is only computed with respect to a single time step (similar to elman networks style truncated backprop) - why not use regular backpropagation through time to train though the entire sequence?
- typo on section 5.1: 'Grid-world problem'
- It is very hard to see on figure 3 what we are supposed to see (I don't see these as having the same destination, perhaps highlight more clearly?)

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: ** Summary
The paper studies a specific class of Non-Markovian Reward Decision Processes (NMRDPs). In general, in NMRDP the dynamics is Markovian but the reward function may depend on the entire trajectory. The authors consider a sub-case where the trajectory can be mapped to a specific "task" and the reward function can be formalized as a mapping between the state-task pair to the reward. This greatly simplifies the problem that can be mapped onto an augmented MDP and solved using standard RL tools. The authors focus on the representation learning problem of the mapping between trajectories to an embedding of the task itself. In particular, they consider contrastive learning to find a representation that discriminate between trajectories associated to the same task and trajectories coming from different tasks. The resulting LSTM-based architecture is then evaluated on a grid-world synthetic problem and on GPS trajectories from tourists in the city of Salzburg.

** Overall evaluation
My main concerns about the paper is that the exact objective of the setting and the representation learning is not that clear and the empirical validation is limited to a qualitative assessment of the representation learning with no down-stream task. More in detail.

1- The assumption of task-dependent reward functions is very sensible. Yet I wonder to which extent it overlaps with literature in multi-task and meta-learning, or more in general with embedding of time-series (the trajectory in your case). It would be useful to frame/compare at the conceptual level the similarities and differences of the proposed setting with those fields.
2- At training time, the PK dataset is somehow supervised, as it is possible to know which trajectories are associated to the same task. At test time, the learned f_theta recursively maps trajectories to tasks. As such, it seems like it could effectively detect changes into the task itself by tracking how the trajectory evolves (when the task function is unknown). This aspect is never really evaluated in the empirical section, but it would be one of the most interesting uses of the learned representation.
3- If the task function T is known, it means that standard RL techniques can be used to solve MDP N. The actual advantage of using a specific function to embed the task to a space in Re^d is never really explained in the paper. Does it make solving N simpler or more effective than using a simple encoding for the task? No evaluation is available in this sense.
4- The empirical evaluation is limited to qualitative analysis of the representation learned in the problem. Yet, there is no clear support that the representation is good/useful in a more quantitative way (e.g., by actually solving the RL part or in identifying quickly the current task).

Some more specific questions/comments

1- Some of the notation is a bit redundant. For instance, phi is mapping from H to Re^d, while f is a mapping from trajectories to Re^d, but in the end they are doing the same thing, as H itself can be obtained as a mapping from trajectories to tasks through T. 
2- "Using RL techniques, finding an optimal policy pi* in the equivalent MDP \hat M is possible and by extension pi* is optimal for N". This passage is not fully clear, if phi is introducing some form of approximation, then pi* may no longer be optimal for the original MDP N. On the other hand, if phi is not approximating but "just" changing the representation from H to Re^d, then it is not clear what is the interest of it.
3- The assumption that similar trajectories can be identified is somehow strong. It would be good to have a more thorough support for it.
4- It is not fully clear what L_BH^local is indeed a local loss. It seems like you are simply using the mapping from the whole trajectory. Is this why it is called local?
5- While I appreciate that the introduction is sketching many different scenarios to support the models studied in the paper, in the end they mostly lack of depth and they rather give a confusing impression instead of clarifying in a compelling way what is the problem studied in the paper. I suggest you rather pick one single scenario with a good level of detail to provide a more solid support to the paper.

** Minor comments
1- You often use "he" to refer to the agent. It would be better to use "it" or "she".
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary of the Paper:
  
        This paper proposes a method to address Non Markovian Reward Decision Processes using RL. For this, NMRPDs are transformed into Markovian Decision Processes. The idea is that the reward function can only depend on the last state of the trajectory and the task. A task representation is introduced. This is learned recursively. The proposed approach is evaluated in several experiments.

Detailed comments:

A point of criticism is that it seems that the authors do not compare with similar or related methods in the experiments. In any case, I think that the paper is interesting and will receive the attention of the community.

The paper is well written in general with only some typos: E.g.

"that separate"

"these proposition"

review_title: Official Blind Review #1

Meta-review: The paper considers a special case of decision making processes with                                                               
non-Markovian reward functions, where conditioned on an unobserved task-label                                                      
the reward function becomes Markovian.                                                                                             
A semi-supervised loss for learning trajectory embeddings is proposed.                                                             
The approach is tested on a multi-task grid-world environment and ablation                                                         
studies are performed.                                                                                                             
                                                                                                                                   
The reviewers mainly criticize the experiments in the paper. The environments                                                      
studied are quite simple, leaving it uncertain if the approach still works in                                                      
more complex settings.                                                                                                             
Apart from ablation results, no baselines were presented although the setting is                                                   
similar to continual learning / multi-task learning (with unobserved task label)                                                   
where prior work does exist.                                                                                                       
Furthermore, the writing was found to be partially lacking in clarity, although                                                    
the authors addressed this in the rebuttal.                                                                                        
                                                                                                                                   
The paper is somewhat below acceptance threshold, judging from reviews and my own                                                  
reading, mostly due to lack of convincing experiments. Furthermore, the general setting                                            
considered in this paper seems quite specific, and therefore of limited impact.
Decision: Reject