Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method to incorporate prior knowledge in an RL system to improve exploration. The authors propose learning a prior over actions conditioned on either the state, or just past actions and show that priors conditioned on just past actions generalize better than those conditioned on the full state when the observation space of offline data is different from the RL problem. 

main_review: The general premise of the paper is that learning priors on actions conditioned on past actions allow the agent to have a reasonable directed behaviour even when the observation space of the data used to learn the priors is different from the observation space of the RL problem. 

For the general RL problem, the premise of the paper is false --- one can design an offline dataset, or design a downstream task such that priors on actions conditioned on past actions from an offline dataset would hurt more than help. Similarly, one can design a problem for which priors conditioned on the state would outperform priors conditioned on just a sequence of actions.

For instance, imagine a grid-world in which the policy for the offline dataset is to go right with 97% probability, and the other directions with 1% probability, whereas the optimal policy on the downstream environment is to always go up. Action priors conditioned on past actions would only mislead the agent in such an enviroment. Similarly, an environment in which different parts of state space require very different action sequences for directed exploration would clearly benefit from priors conditioned on the state instead of past actions. 


As a result, I find the primary claim "However, an agent should ideally be able to produce efficient explorative behaviors even in unseen environments and unrelated tasks. Behavior priors don’t give us that, temporal priors must." unfounded. The priors can absolutely hurt when the downstream task is unrelated to the offline data, and a few empirical results are not sufficient to support the broad claim. The fact that a prior over actions only conditioned on the last action works only tells me that achieving directed exploration in the benchmarks the authors considered is not particularly challenging, and a simple prior enables directed exploration across the state space. 
summary_of_the_review: The main claim of the paper is not supported by the evidence. It's not hard to imagine scenarios for which the main claim can be demonstratively false. In my opinion, the paper must clearly mention when the proposed method would help and hurt, and tone down the main claim.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a novel method for encouraging exploration based on the idea of behavior priors. The proposed prior, called temporal prior, considers the history of the agent when determining the prior probability of an action. The authors then derive an variant of soft-actor critic (SAC) adapted to make use of these temporal priors. This is done by defining a behavior policy that is a mixture of the priors and the agent's learned policy based on the certainty of the agent's policy, as defined by its entropy in that state. Additionally, the authors propose using a custom loss meant to encourage uncertainty in the learned policy and therefore encourage actions sampled from the temporal priors. This approach is shown to encourage state coverage, to be capable of accelerating learning in unseen and different tasks.
main_review: The paper is well written and clear. The main contributions are novel, interesting, and well motivated. The experimental results seem informative and in support of the claims made throughout the paper, but the low choice of seeds leaves room for doubt in some cases.

The decision to only use 3 seeds greatly weakens the significance of these results, and is my only major concern with this work. For instance, the low seed count is likely why in the "reach" plot in Figure 4, the 2 and 5 action results appear lower than the 1 and 10 action. Similarly, it is very difficult to conclude anything from the TempoRL-simplified vs TempoRL comparison as a result of the low seed count. With so few seeds, a single bad run can drastically alter the mean. This issue of significance doesn't seem as bad in the case of Figure 6 which use 5 seeds and also shows some consistent patterns across the different domains so I don't think this necessarily invalidates all the empirical results.

Similarly, with so few seeds, e.g., 3 or even 5, the empirical standard deviation can be very noisy which makes it a very poor representation of the expected variance in the results. With so few seeds, plotting the min, max, and median or mean would provide a much more representative visualization of the learning behavior of each method. Alternatively, with few seeds, it isn't unreasonable to simply plot each learning curve with the addition of curve representing the mean/median (see [1, Figure 1] for an example of what I mean).

Overall, I think the main idea is good, and the work is interesting. If it weren't for the aforementioned issues, this would be a strong paper. That being said, given not all results are equally affected, I don't think it fair to reject. I feel like the paper would still be borderline if you were to remove the problematic results which is my recommendation for now.

# Questions:

- p. 6, Why do we need a new objective, i.e., eq. (4)? What is the motivation for wanting policies that are less certain to reach the goal?

- p. 6, "We finally note that the modified learning objective remains aligned with the original formulation.", it's not quite clear to me what aligned means here. Is this a formal statement or an observation based on empirical evidence? If it's the latter, then what is the evidence of their alignment?

- Appendix A, eq. (9) and (10) seem to differ by a missing $\Lambda (\mathcal{H}(\pi(\cdot|s_t)))$ term. What is the motivation for dropping this term?

- Appendix D.1, cumulative returns per episode?

- Appendix D.2, What were the scripted policies that generated data for the priors in "reach" and "room-maze"?


# Minor comments and nitpicks:

- p. 3, "the issue of temporal correlation is merely relocated in the hierarchy, as the high-level planner is not encouraged to produce correlated sequences of skills", I'm not sure this is a fair assessment. Although it is true that the high-level planner is not encouraged to produce correlated sequences of skills, the skills themselves will generate correlated sequences of actions. The issue of temporal correlation isn't relocated so much as implemented at the skills/options level and  abstracted away at the high-level. Doing so doesn't change the fact that when the high-level planner executes just a single skill, it will be producing a correlated sequences of primitive actions, by construction, regardless of any encouragement to do so.

- p. 3, definition of goal-conditioned MDP, the cited work doesn't seem to use the same definition, most notably it doesn't include a discount factor in its definition and the distribution over goals is part of the definition. They don't need to match but I would make it clear in the wording.


[1] Fujimoto S, Meger D, Precup D. Off-policy deep reinforcement learning without exploration. InInternational Conference on Machine Learning 2019 May 24 (pp. 2052-2062). PMLR.

Post-Rebuttal
=========

The increase in number of seeds and the corrections of the error in the one of the figures gives me significantly more faith in the empirical results. I've increased my scores to reflect this.
summary_of_the_review: The paper is well written and clear. The main contributions are novel, interesting, and well motivated. The experimental results seem informative and in support of the claims made throughout the paper, but the low choice of seeds leaves room for doubt in some cases.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper suggests a task-agnostic exploration priors that can be trained on policies that solve random tasks in a given environment (task-agnostic data). Given a new task, the induced exploration quickly finds how to solve it, compared to other methods that disposes of the same pre-training data. 

Temporal priors learns by imitation (of task-agnostic data) a policy conditioned by the whole history, rather than only the current states.
Here, they observe that only using a condition on the previous action is already sufficient to outperforms other methods. 

To use the prior, they use an $\epsilon$-greedy approach where $\epsilon$ ($\lambda_t$ in the paper) depends on the entropy of the exploitation policy: the more the exploitation policy is random, the more the agents refers to the exploration policy (the temporal prior). 

In order to favorise the utilisation of the exploration, they use SAC which enforces the entropy of the exploitation policy. 

main_review: The idea is novel, simple and shows good empirical results. They use a good choice for the baselines and show that their methods outperforms them on environment from the meta-world suite.

On the weaknesses side, I've found the description of SAC + TempRL quite unclear and not exactly correct:

First, given the objective that takes $\Lambda$ in the formulation (eq 4), the soft-policy target (soft-max of the Q-functions in SAC) may differ, but is not described. 

Then, the objective $J_\pi$ looks like the target of the value function $V_\psi$ in SAC, if the expectation was on the action instead of the states. In sac, the objective for the policy $J_\pi$ should be the KL divergence between the policy and the soft-max of the Q-function. Only the formulation of $J_\theta(Q) looks correct. 

The parameters $\psi$ and $\theta$ should appear in the objectives function to understand what 'update $\psi$ and $\theta$ means in Algo.1.

And since, at the end, these objectives are not used but they simply used the original SAC, why not just say, after equation 3:
"To directly encourage sampling from the exploration prior, we use an entropy-regularized RL algorithm, for instance SAC" ?

Regarding the experiment, I was expecting more type of environment, to show that this method does not only work on the meta-world suite. For example, using Mujoco or even tasks with discrete actions.


summary_of_the_review: +) novel and simple approach and nice results on meta-world.

-) Unclear and useless SAC + TempRL description in section 4.2 and appendix A. Needs more environments for experiments.

I am on the fence on this paper. Since the results are nice and the method is corrects (they finally used the original SAC), I would be more favorable for a weak accept, but I strongly suggest to remove the paragraph about the entropic regularisation in section 4.2 as well as the appendix A and re-write Algo1 to explicite that the used RL algorithm to learn the exploitation policy is SAC.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes the concept of temporal priors, extending previously proposed behavioral priors. Behavioral priors try to recover some prior (state conditioned) policy utilizing a set of trajectories obtained from multiple 'training' tasks, to be used to guide exploration in 'downstream' tasks. Instead, temporal priors take a more general non-Markovian approach by learning a prior policy that is potentially conditioned on the whole history of previous states and actions. This work proposes a new algorithm, TempoRL, that employs a generative flow policy to parameterize the temporal prior and a modified SAC agent that is incentivized to perform exploration with the temporal prior. In practice, the authors propose to limit the conditioning information of the temporal prior to, exclusively, the previous action in the trajectory, only learning the correlation between consecutive actions. This choice makes the resulting temporal prior less dependent on the diversity and the optimality of the trajectory data from the training tasks than traditional behavioral priors. Empirically, the proposed algorithm obtains promising results in learning temporal priors from a single training task in a few robotics environments from meta-world (Yu et al. 2020) and simple navigation environments with sparse rewards.
main_review: __Strengths__:

1. The paper is well-written and easy to follow.

2. The authors provide an appropriate and concise overview of the most related prior work and its limitations.

3. The proposed idea of capturing correlation between different actions is simple, intuitively useful, and general since it only requires the action space to match between the 'training' and 'downstream' environments.

4. The authors provide experimental results analyzing different aspects of temporal priors, such as the effects of the conditioning variables and the type of employed generative model.

__Main Concerns/Doubts__:

My major concerns with this work are the objective formulation and the experimental results. In order of relevance:

1. Both hindsight experience replay (HER) and n-step returns are prior techniques that are orthogonal to the author's contribution. Both applicability and effectiveness of these techniques are very much environment dependant (e.g. HER requires access to the true reward function). Hence, it would be important if the authors could clarify whether these are also adopted by the other baselines. Moreover, it would be very relevant to include an ablation study to test the contribution of these ancillary methods and the effectiveness/sample-efficiency of TempoRL without them.

2. The authors benchmark their algorithm on a very small, handpicked selection of adapted environments. For a more comprehensive and fair evaluation for robotics manipulation, I believe it would have been more appropriate to consider all of the environments in one of the pre-specified evaluation modes in meta-world (e.g., MT10). Thus, it would be important to either include additional experiments or, at least, provide a justification for the specific environment choices.

3. It is unclear what is the effect of the proposed optimization objectives (Equations 4-7) on the sampling distribution policy. The authors propose to use a sampling policy derived from a mixture of $\pi$ (the parametric downstream policy) and $\bar{\pi}$ (the temporal prior). Hence, to directly optimize w.r.t. the sampling distribution (as in SAC), both the policy's optimization objective and the targets for the Q-value would need to sample actions from the same mixture distribution. Yet, both of these objectives inconsistently consider sampling actions solely from $\pi$.  If possible, It would be important to justify this specific choice over its natural alternative through formal analysis of these objectives.

4. The baseline algorithms utilized have been proposed for different problem settings:
 - PolyRL (Amin et al. 2020) does not make use of prior trajectory data.
 - PARROT (Singh et al. 2020) assumes the prior trajectory data comes from a diverse set of multiple training environments from the same distribution as the downstream environment.
  -> It would be appropriate to compare TempoRL to additional methods/algorithms that have been designed for reinforcement learning transfer given experience/demonstrations from a single environment, since this problem setting more closely matches the examined one. For instance, the authors could consider algorithms that learn diverse low-level skills in a training environment. E.g. the work from Florensa et al. 2017 and/or Haarnoja et al. 2018 (both of which share their code) could be adapted to use offline data by treating each trajectory as representing a different skill. 
 -> This work asserts itself as removing the fundamental limitations of behavioral priors (Introduction, 3rd paragraph), without analyzing its own introduced limitations. Hence, for a more comprehensive comparison with behavioral prior algorithms, such as PARROT, it would be important to perform evaluation also on the behavioral priors problem setting. For instance, this could be done by adding an experiment treating all environments but one as training environments. This would assess the scalability of the proposed algorithm by evaluating how much useful information can be extrapolated from the correlation of consecutive actions in trajectories from multiple diverse training environments.

5. The authors introduce a conspicuous amount of additional hyperparameters without providing details on how they were selected or an ablation study that provides information about the algorithm's robustness to their choices (e.g., $\alpha$, $\beta_t$, $\beta_s$, hindsight replay ratio, n-step, ...).

__Minor concerns__:

1. Given the high standard deviations for some of the performance curves, I believe the authors should increase the number of utilized random seeds from the current 3/5 per experimental setting (Peter et al. 2018).

2. It is unclear to me how the policy's entropy quantifies the agent's confidence to reach the goal, as stated in Section 4.2 (if multiple actions lead to the same outcome, wouldn't the entropy still be high regardless of the agent's confidence?).

3. Figure 5 does not show the exploration results for Parrot.

4. which is capable -> that is capable (Introduction, 3rd paragraph).

__Further suggestions__:

1. It would be interesting to compare the proposed SAC integration with utilizing the temporal prior as a reparameterized action space, as done in many hierarchical RL methods and PARROT.

2. Since incorporating past states and actions provides information about the underlying task, a further connection could be drawn between the concept of temporal priors and implicitly performing task inference (e.g. as in Rakelly et al. 2019).


__Post-rebuttal__

While I appreciate the additional experiments performed by the authors, I believe they accentuate very severe issues w.r.t. the paper's empirical results. For instance, there is a large gap between the results in the original hand-picked meta-world tasks and the other tasks in MT-10. Moreover, there is no formal analysis to provide strong intuition about the algorithm's exploratory properties. This would be particularly important given some of the authors' algorithm design choices, which appear to be inconsistent with prior literature (not optimizing policy and Q-function w.r.t. exploratory policy).

While I like the simple idea introduced in the paper, after reading the rebuttal and the other reviews, I do not think that TempoRL is currently shown to make a significant contribution. Thus, I think this paper is not yet ready for publication.


__References__

Yu, Tianhe, et al. "Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning." Conference on Robot Learning. PMLR, 2020.

Amin, Susan, et al. "Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards." arXiv preprint arXiv:2012.13658 (2020).

Singh, Avi, et al. "Parrot: Data-driven behavioral priors for reinforcement learning." arXiv preprint arXiv:2011.10024 (2020).

Florensa, Carlos, Yan Duan, and Pieter Abbeel. "Stochastic neural networks for hierarchical reinforcement learning." arXiv preprint arXiv:1704.03012 (2017).

Haarnoja, Tuomas, et al. "Latent space policies for hierarchical reinforcement learning." International Conference on Machine Learning. PMLR, 2018.

Henderson, Peter, et al. "Deep reinforcement learning that matters." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

Rakelly, Kate, et al. "Efficient off-policy meta-reinforcement learning via probabilistic context variables." International conference on machine learning. PMLR, 2019.


summary_of_the_review: While this paper introduces some simple and interesting ideas, without a more comprehensive experimental evaluation and a clearer explanation of the properties of the algorithm, it is not yet ready for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the exploration problem in RL and proposes a new type of the behavioral prior, one only conditioned by recent actions. The paper empirically demonstrates the effectiveness of this approach in sparse-reward environments. It also shows that this type of the behavioral prior can be learned in a simple non-visual task and transferred to a visual task. 
main_review: Overall, the empirical results are impressive, but I feel that the potential weakness of the method is neither tested nor discussed.

# Strength
- S1. The empirical results of the proposed method are impressive.
- S2. It is nice that the proposed method is agnostic of the state space of the downstream task.

# Weakness

The detailed comment is given in the next section.

- W1. The paper seems to claim that previous methods suffer from the domain gap while the proposed method does not. However, I can think of some situations that this method could be problematic.
- W2. This is somehow related to W1. I feel that the paper tries to overly sell the proposed method.
- W3. The objective function for the proposed SAC-like algorithm contains a mistake.

# Detailed Comments

As for W1, it seems to me that the proposed method can suffer from the domain gap in some cases. For example, let us consider a maze 
(not a simple one like U-maze) with visual state information, i.e., the agent can observe the maze from bird's-eye view. Then, it is likely that without conditioning the prior by a state, the agent might simply hit the wall and continue to do so because of the state-agnostic prior. To claim that the proposed method does not suffer from the domain gap, would you add experiments in an environment like a general maze?

In addition, from Figure 5, I have an impression that the proposed method learns to simply repeats the same action, occasionally changing it. It seems that all downstream tasks are solvable with this simple strategy since the agent can occasionally hit the door or window knob, resulting in opening/closing it. Would you provide results of SAC+AR(n) with a higher n, like 5, 10, or so.

As for W2, for example, the paper says that "... we propose a principled manner of integrating priors..." What do you mean by "principled"? The proposed algorithm seems to use an ad-hoc way to adjust the ratio of a mixture policy. (Furthermore, the derivation of the proposed objective seems to contain a mistake.)

Also, the paper says "We show how state-independent temporal priors can be learned from few expert trajectories on simple tasks and used to improve exploration efficiency in new tasks, despite the presence of massive domain gaps". How do you measure the size of domain gaps? To me, a reaching task and a window opening task are not so different. In case of a reaching task, a robot needs to move its arm to the direction of a target, whereas in case of a window opening task, a robot similarly needs to move its arm to the direction of window opening while pushing the knob of the window. If a downstream task is grasping and stacking objects, I would say it is completely different, though.

There are many other claims which I feel is not well supported, and this is confusing to me.

As for W3, the maximization of the objective (Eq 4) cannot be achieved by the minimization of Eq 5. To see this, let us look at Eq 11 and Eq 13. If you take the expectation of Eq 13 (ignoring $\min$) wrt the next state, reward, and next action, what you will get is $\mathbb{E}[r + \gamma (1-d) Q(s', a') + \alpha \Lambda (-\log \pi (a'|s'))]$, which is not equal to Eq 11 unless $\Lambda$ is a linear function. If $\Lambda$ is a linear function, the proposed objective is the same as SAC's objective. In other words, Eq 13 is NOT an unbiased estimate of Eq 11. The same argument can be made about Eq 4. I think the biasedness is OK as long as the paper clarifies this, and the algorithm still works. But it needs to be clarified.

# Suggestions

In sum, I suggest the authors to improve the draft by removing some overly claimed sentences and adverbs, add some more experiments with downstream tasks which require completely different skills like grasping, and consider pointing out the biasedness of derived estimators.

# Post-discussion

After the author rebuttal, I am mostly satisfied with it and raised the score to accept. While I am still OK to accept the paper if the authors discuss the limitation of the method and correct some overselling wordings in the main text, I was convinced that my score should be based on what is in the current pdf not the paper's potential. Therefore, I lowered the score to weak accept. I strongly encourage the authors to resubmit the paper with discussion on the limitation of the method and correction of some overselling wordings in the main text.
summary_of_the_review: The paper proposes an interesting idea, and current experimental results are impressive. However, I think the proposed method is not well-tested or discussed deeply. Accordingly, I am more inclined to the rejection unless my concerns are resolved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed learned temporal action priors conditioned on past trajectories of the agent, and demonstrated using it in a soft actor critic agent, applied to long-horizon control problems.
main_review: Strengths:
1. The paper tackles a difficult but prevalent problem in RL: solving control environments with long horizons and sparse rewards.
2. The proposed learned priors showed good transfer properties on selected environments, some even with a wide gap.

Weaknesses:
1. The baselines being compared to are rather weak both in terms of their exploration capabilities, and that they were not designed with transfer tasks in mind. There seem to be many other baselines that could be considered in this space, such as intrinsic motivation based agents (e.g. Agent57), randomized value function / policy based exploration, or model-based exploration agents (e.g. Plan2Explore) etc.  
2. There is danger of overfit to selected control environments when the paper's statements for contribution seem much broader. After reading the paper, I'm not sure why such temporal priors would necessarily generalize onto other long horizon sparse reward environments. Is there a piece of theory, an explanation or an illustration that could help convince us of the general usefulness of these temporal priors?
summary_of_the_review: The paper presents a set of temporal priors for exploration, empirically showing good results for certain transfer tasks. I'm concerned with the generality of the approach beyond the selected environments and the lack of competitive baselines that the paper considered.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Temporal priors for exploration in off-policy reinforcement learning are proposed, priors for action sampling that are state-independent and depend only on the most recent past actions. In essence, an earlier trained temporal prior come in as a replacement, if a policy shows a great deal of uncertainty at a particular, possible yet unexplored, state. Exploration resorts to more generic task-agnostic trajectories instead of trusting the current policy. It is shown that sampling from such a mixture of policy and temporal priors can accelerate transfers to novel tasks that differ significantly from those originally trained for.
main_review: The proposed solution is benchmarked against a rather large set of related approaches and it is shown to perform very well, for tasks that differ significantly from the original tasks in particular. It also allows for the exploration of larger actions spaces without sacrificing the ability to achieve high rewards.

It is shown that conditioning the prior on the current state complicates transfer to tasks that differ too much from those of the source domains. What is clearly shown in the paper is that you often benefit from instead ignoring the state and only relying on the recent actions. Of particular interest is the fact that a prior only conditioned on the most recent action, rather than a longer sequence of actions, is as competitive as it is.

Unlike behavioural priors, the proposed temporal do not assume that the state space has yet been explored, which in turn would assume that the novel tasks need to be similar to those the prior was trained on. Another benefit of the proposed priors is that they can be trained off-line in many different ways, such as by exploiting policies from earlier tasks or from demonstrations.

However, the most important concern when it comes to the proposed approach is the inclusion of the mixing weights in (4). It is hard to see where this actually comes from and it would be good for the authors to sort this out. The addition would suggest the entropy component is suppressed for samples drawn from the policy and is instead dominated by those drawn from the prior, which should further encourage the policy to explore a wider space of actions, which might explain the good performance. Another concern is that since the samples are drawn from the mixture you no longer have an approximation of an entropy. It seems more like a cross-entropy between the mixture distribution and the policy distribution. 

The addition of the monotonically increasing function with parameters that have to be experimentally set also seems like an arbitrary solution that needs a better explanation. The reasoning behind the function is quite unclear in the paper. If the mixing weights in the objective are seen as a kind of importance sampling, then the expectation would not be over a distribution that sums up to 1, which might motivate the need for the monotonic function. It is true though that without the inclusion of the mixing weights in the objective, the proposed solution becomes very similar to Soft Actor-Critic, with the main difference being the sampling.

The notation a_t ~ pi(s_t) in (2) and (3) is a bit confusing since it gives the impression that the policy could be deterministic. It would be better to include a do, as in pi(.|s_t). Further, since policy parameters are used in (5) and (6), it would be good to include also the parameters for the Q-function. By the way, the parameters for the Q-function are called both phi and psi in the paper. The paper does not mention how alpha is controlled over time, which can be understood from earlier literature but would be good to include also here.

Finally, from the experiments, it is hard to deduce whether observed improvements are due to the mixed sampling or from the changed policy objective that includes the mixing weights. It would be good to test with and without the weights, assuming there is no good motivation why the weights need to be in the objective.

summary_of_the_review: Given the concerns raised in the core part of the paper that relates to the policy objective, the paper cannot be directly accepted, but not completely rejected either. There might be a good motivation behind the objective that the authors may be able to clarify, something that would lead to a minor modification of the paper.
review_title: null

Meta-review: This paper was close and also very polarizing with the reviewers. On the positive side, some reviewers found:
1. the results impressive
2. the proposed method to be novel, interesting, and produce good performance across several settings
3. the paper was well written

On the other hand, others found:
1. the motivation suspect
2. missing experiments to characterize the sensitivity to numerous hyper-parameters 
3. the baselines compared with weak and not representative
4. significant performance drop comparing the results in the original submission and the new ones added during discussion period
5. low number of seeds initially

In the end, multiple reviewers raised serious issues regarding the motivation for the approach and the quality and ultimately credibility of the results presented. One of the high-scoring reviewers agreed the paper was a bit misleading (limitations relegated to the appendix). Unfortunately, none of the high-scoring reviewers provided counters to this points.
Decision: Reject