Authors of this paper derived an efficient quantum-inspired learning algorithm based on a hierarchical representation that is known as tree tensor network, which is inspired by the multipartite entanglement renormalization ansatz approach where the tensors in the TN are kept to be unitary during training.

Some observations are: The limitation of learnability of TTN strongly depends on the physical indexes and the geometrical indexes determine how well the TTNs approximate the limit;

TTNs exhibit same increase level of abstractions as CNN or DBN;

Fidelity and entanglement entropy can be considered as some measurements of the network.

Authors introduced the two-dimensional hierarchical tensor networks for solving image recognition problems,

which suits more the 2-D nature of images.

In section 2, authors stated that the choice of feature function is arbitrary,

and a specific feature map was introduced in Section 4.

However, it is not straightforward to connect (10) to (1) or (2).

It is better to clarify this connection

because some important parameters such as the virtual bond and input bond are related to the complexity of the proposed algorithm as well as the limitation of learnability.

For example, the scaling of the complexity O(dN_T(b_v^5 + b_i^4)) is not easy to understand.

Is it related to specific feature map?

How about the complexity of eigen-decomposition for one tensor at each iterates.

And also, whether the tricks used to accelerate the computations will affect the convergence of the algorithm?

More details on these problems are required for readers’ better understanding.

From Fig 2, it is difficult to see the relationship between learnability and parameters such input bond and virtual bond

because it seems there are no clear trends in the Fig 2(a) and (b) to make any conclusion.

It is better to clarify these relationships with either clear explanation or better examples.

From Fig 3, authors claimed that TN obtained the same levels of abstractions as in deep learning.

However, from Fig 3 only, it is hard to make this conclusion.

First, there are not too many differences from Fig 3(a) to Fig 3(e).

Second, there is no visualization result reported from deep learning on the same data for comparison.

Hence, it is not convincing to draw this conclusion only from Fig 3.

In Section 4.2, what strategy is used to obtain these parameters in Table 1?

In Section 5, it is interesting to see more experiments in terms of fidelity and entanglement entropy.

fact fact fact evaluation fact evaluation fact fact evaluation request evaluation evaluation request request request request evaluation evaluation request fact evaluation evaluation fact evaluation non-arg request


*Paper summary* The paper considers GANs from a theoretical point of view.

The authors approach GANs from the 3-Wasserstein point of view and provide several insights for a very specific setting.

In my point of view, the main novel contribution of the paper is to notice the following fact: (*) It is well known that the 2-Wasserstein distance W2(PY,QY) between multivariate Gaussian PY and its empirical version QY scales as $n^{-2/d}$, i.e. converges very slow as the dimensionality of the space $d$ increases.

In other words, QY is not such a good way to estimate PY in this setting.

A somewhat better way is use a Gaussian distribution PZ with covariance matrix S computed as a sample covariance of QY.

In this case W2(PY, PZ) scales as $\sqrt{d/n}$.

The paper introduces this observation in a very strange way within the context of GANs.

Moreover, I think the final conclusion of the paper (Eq. 19) has a mistake,

which makes it hard to see why (*) has any relation to GANs at all.

There are several other results presented in the paper regarding relation between PCA and the 2-Wasserstein minimization for Gaussian distributions (Lemma 1 & Theorem 1).

This is indeed an interesting point,

however the proof is almost trivial

and I am not sure if this provides any significant contribution for the future research.

Overall, I think the paper contains several novel ideas,

but its structure requires a *significant* rework

and in the current form it is not ready for being published.

*Detailed comments* In the first part of the paper (Section 2) the authors propose to use the optimal transport distance Wc(PY, g(PX)) between the data distribution PY (or its empirical version QY) and the model as the objective for GAN optimization.

This idea is not novel:

WGAN [1] proposed (and successfully implemented) to minimize the particular case of W1 distance by going through the dual form,

[2] proposed to approach any Wc using auto-encoder reformulation of the primal (and also shoed that [5] is doing exactly W2 minimization),

and [3] proposed the same using Sinkhorn algorithm.

So this point does not seem to be novel.

The rest of the paper only considers 2-Wasserstein distance with Gaussian PY and Gaussian g(PX) (which I will abbreviate with R),

which looks like an extremely limited scenario (and certainly has almost no connection to the applications of GANs).

Section 3 first establishes a relation between PCA and minimizing 2-Wasserstein distance for Gaussian distributions (Lemma 1, Theorem 1).

Then the authors show that if R minimizes W2(PY, R) and QR minimizes W2(QY, QR) then the excess loss W2(PY, QR) - W2(PY, R) approaches zero at the rate $n^{-2/d}$ (both for linear and unconstrained generators).

This result basically provides an upper bound showing that GANs need exponentially many samples to minimize W2 distance.

I don't find these results novel,

as they already appeared in [4] with a matching lower bound for the case of Gaussians

(Theorem B.1 in Appendix can be modified easily to show this).

As the authors note in the conclusion of Section 3, these results have little to do with GANs,

as GANs are known to learn quite quickly

(which contradicts the theory of Section 3).

Finally, in Section 4 the authors approach the same W2 problem from its dual form and notice that for the LQG model the optimal discriminator is quadratic.

Based on this they reformulate the W2 minimization for LQG as the constrained optimization with respect to p.d. matrix A (Eq 16).

The same conclusion does not work unfortunately for W2(QY, R),

which is the real training objective of GANs.

Theorem 3 shows that nevertheless, if we still constrain discriminator in the dual form of W2(QY, R) to be quadratic, the resulting soliton QR* performs the empirical PCA of Pn.

This leads to the final conclusion of the paper,

which I think contains a mistake.

In Eq 19 the first equation, according to the definitions of the authors, reads \[W2(PY, QR) = W2(PY, PZ),   (**)\] where QR is trained to minimize min_R W2(QY, R) and PZ is as defined in (*) in the beginning of these notes.

However, PZ is not the solution of min_R W2(QY, R)

as the authors notice in the 2nd paragraph of page 8.

Thus (**) is not true

(at least, it is not proved in the current version of the text).

PZ is a solution of min_R W2(QY, R) *where the discriminator is constrained to be quadratic*.

This mismatch is especially strange,

given the authors emphasize in the introduction that they provide bounds on divergences which are the same as used during the training (see 2nd paragraph on page 2)

--- here the bound is on W2, but the empirical GAN actually does a regularized training (with constrained discriminator).

Finally, I don't think the experiments provide any convincing insights,

because the authors use W1-minimization to illustrate properties of the W2.

Essentially the authors say "we don't have a way to perform W2 minimization, so we rather do the W1 minimization and assume that these two are kind of similar".

* Other comments * (1) Discussion in Section 2.1 seems to never play a role in the paper.

(2) Page 4: in p-Wasserstein distance, ||.|| does not need to be a Euclidean metric.

It can be any metric.

(3) Lemma 2 seems to repeat the result from (Canas and Rosasco, 2012)

as later cited by authors on page 7?

(4) It is not obvious how does Theorem 2 translate to the excess loss?

(5) Section 4. I am wondering how exactly the authors are going to compute the conjugate of the discriminator, given the discriminator most likely is a deep neural network?

[1] Arjovsky et al., Wasserstein GAN, 2017

[2] Bousquet et al, From optimal transport to generative modeling: the VEGAN cookbook, 2017

[3] Genevay et al., Learning Generative Models with Sinkhorn Divergences, 2017

[4] Arora et al, Generalization and equilibrium in GANs, 2017

[5] Makhazani et al., Adversarial Autoencoders, 2015

fact fact evaluation fact fact fact evaluation fact evaluation fact evaluation evaluation evaluation evaluation evaluation evaluation fact evaluation fact fact fact evaluation fact evaluation fact fact fact evaluation fact fact fact evaluation fact fact fact fact fact fact fact fact fact fact fact fact fact fact evaluation fact evaluation evaluation fact evaluation fact evaluation fact fact fact evaluation evaluation reference reference reference reference reference


Summary This paper presents a dataset of mathematical equations and applies TreeLSTMs to two tasks: verifying and completing mathematical equations.

For these tasks, TreeLSTMs outperform TreeNNs and RNNs.

In my opinion, the main contribution of this paper is this potentially useful dataset, as well as an interesting way of representing fixed-precision floats.

However, the application of TreeNNs and TreeLSTMs is rather straight-forward,

so in my (subjective) view there are only a few insights salvageable for the ICLR community

and compared to Allamanis et al. (2017) this paper is a rather incremental extension.

Strengths The authors present a new datasets for mathematical identities.

The method for generating additional correct identities could be useful for future research in this area.

I find the representation of fixed-precision floats presented in this paper intriguing.

I believe this contribution should be emphasized more

as it allows the model to generalize to unseen numbers

and I am wondering whether the authors see some wider application of this representation for neural programming models.

I liked the categorization of the related work.

Weaknesses p2: It is mentioned that the framework is the first to combine symbolic expressions with black-box function evaluations,

but I would argue that Neural Programmer-Interpreters (NPI; Reed & De Freitas) are already doing that

(see Fig 1 in that paper where the execution trace is a symbolic expression and some expressions "Act(LEFT)" are black-box function applications directly changing the image).

The differences to Allamanis et al. (2017) are not worked out well.

For instance, the authors use the TreeNN model from that paper as a baseline

but the EqNet model is not mentioned at all.

The obvious question is whether EqNets can be applied to the two tasks (verifying and completing mathematical equations) and if so why this has not been done.

The contribution regarding black box function application is unclear to me.

On page 6, it is unclear to me what "handles […] function evaluation expressions".

As far as I understand, the TreeLSTM learns to the return value of function evaluation expressions in order to predict equality of equations,

but this should be clarified.

I find the connection of the proposed model and task to "neural programming" weak.

For instance, as far as I understand there is no support for stateful programs.

Furthermore, it would be interesting to hear how this work can be applied to existing programming languages such as Haskell.

What are the limitations of the architecture?

Could it learn to identify equality of two lists in Haskell?

p6: The paragraph on baseline models is rather uninformative.

TreeLSTMs have been shown to outperform Tree NN's in various prior work.

The statement that "LSTM cell […] helps the model to have a better understanding of the underlying functions in the domain" is vague.

LSTM cells compared to fully-connected layers in Tree NNs ameliorate vanishing and exploding gradients along paths in the tree.

Furthermore, I would like to see a qualitative analysis of the reasoning capabilities that are mentioned here.

Did you observe any systematic differences in the ~4% of equations where the TreeLSTM fails to generalize (Table 3; first column).

Minor Comments Abstract: "Our framework generalizes significantly better" I think it would be good to already mention in comparison to what this statement is.

p1: "aim to solve tasks such as learn mathematical" -> "aim to solve tasks such as learning mathematical"

p2: You could add a citation for Theano, Tensorflow and Mxnet.

p2: Could you elaborate how equation completion is used in Mathematical Q&A?

p3: Could you expand on "mathematical equation verification and completion […] has broader applicability" by maybe giving some concrete examples.

p3 Eq. 5: What precision do you consider?

Two digits?

p3: "division because that they can" -> "division because they can"

p4 Fig. 1: Is there a reason 1 is represented as 10^0 here?

Do you need the distinction between 1 (the integer) and 1.0 (the float)?

p5: "we include set of changes" -> "we include the set of changes"

p5: In my view there is enough space to move appendix A to section 2.

In addition, it would be great to see more examples of generated identities at this stage (including negative ones).

p5: "We generate all possible equations (with high probability)"

– what is probabilistic about this?

p5: I don't understand why function evaluation results in identities of depth 2 and 3.

Is it both or one of them?

p6: The modules "symbol" and "number" are not shown in the figure.

I assume they refer to projections using Wsymb and Wnum?

p6: "tree structures neural networks" -> "tree structured neural networks"

p6: A reference for the ADAM optimizer should be added.

p6: Which method was used for optimizing these hyperparameters?

If a grid search was used, what intervals were used?

p7: "the superiority of Tree LSTM to Tree NN shows that is important to incorporate cells that have memory" is not a novel insight.

p8: When you mention "you give this set of equations to the models look at the top k predictions" I assume you ranked the substituted equations by the probability that the respective model assigns to it?

p8: Do you have an intuition why prediction function evaluations for "cos" seem to plateau certain points?

Furthermore, it would be interesting to see what effect the choice of non-linearity on the output of the TreeLSTM has on how accurately it can learn to evaluate functions.

For instance, one could replace the tanh with cos and might expect that the model has now an easy time to learn to evaluate cos(x).

p8 Fig 4b; p9: Relating to the question regarding plateaus in the function evaluation: "in Figure 4b […] the top prediction (0.28) is the correct value for tan with precision 2, but even other predictions are quite close" – they are all the same and this bad, right?

p9: "of the state-of-the-art neural reasoning systems" is very broad and in my opinion misleading too.

First, there are other reasoning tasks (machine reading/Q&A, Visual Q&A, knowledge base inference etc.) too

and it is not obvious how ideas from this paper translate to these domains.

Second, for other tasks TreeLSTMs are likely not state-of-the-art

(see for example models on the SQuAD leaderboard: https://rajpurkar.github.io/SQuAD-explorer/) .

p9: "exploring recent neural models that explicitly use memory cells"

– I think what you mean is models with addressable differentiable memory.

fact fact fact evaluation evaluation evaluation fact evaluation evaluation request fact request evaluation fact fact fact evaluation fact fact request evaluation evaluation fact request evaluation fact request request request evaluation fact evaluation fact request request request request request request request request non-arg non-arg request request request request request quote request request request fact request request request request request evaluation non-arg non-arg request evaluation request evaluation fact evaluation evaluation reference quote fact


In this paper, the authors have proposed a GAN based method to conduct data augmentation.

The cross-class transformations are mapped to a low dimensional latent space using conditional GAN.

The paper is technically sound and the novelty is significant.

The motivation of the proposed methods is clearly illustrated.

Experiments on three datasets demonstrate the advantage of the proposed framework.

However, this paper still suffers from some drawbacks as below:

(1)	The illustration of the framework is not clear enough.

For example, in figure 3, it says the GAN is designed for “class c”, which is ambiguous whether the authors trained only one network for all class or trained multiple networks and each is trained on one class.

(2)	Some details is not clearly given, such as the dimension of the Gaussian distribution, the dimension of the projected  noise and .

(3)	The proposed method needs to sample image pairs in each class.

As far as I am concerned, in most cases sampling strategy will affect the performance to some extent.

The authors need to show the robustness to sampling strategy of the proposed method.

fact fact evaluation evaluation fact evaluation evaluation evaluation evaluation fact evaluation request


This paper introduces a generative approach for 3D point clouds.

More specifically, two Generative Adversarial approaches are introduced: Raw point cloud GAN, and Latent-space GAN (r-GAN and l-GAN as referred to in the paper).

In addition, a GMM sampling + GAN decoder approach to generation is also among the experimented variations.

The results look convincing for the generation experiments in the paper, both from class-specific (Figure 1) and multi-class generators (Figure 6).

The quantitative results also support the visuals.

One question that arises is whether the point cloud approaches to generation is any more valuable compared to voxel-grid based approaches.

Especially Octree based approaches [1-below] show very convincing and high-resolution shape generation results,

whereas the details seem to be washed out for the point cloud results presented in this paper.

I would like to see comparison experiments with voxel based approaches in the next update for the paper.

[1] @article{tatarchenko2017octree, title={Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs}, author={Tatarchenko, Maxim and Dosovitskiy, Alexey and Brox, Thomas}, journal={arXiv preprint arXiv:1703.09438}, year={2017} }

fact fact fact evaluation fact request evaluation fact request reference


Summary: This paper presents a derivation which links a DNN to recursive application of maximum entropy model fitting.

The mathematical notation is unclear,

and in one cases the lemmas are circular (i.e. two lemmas each assume the other is correct for their proof).

Additionally the main theorem requires complete independence,

but the second theorem provides pairwise independence,

and the two are not the same.

Major comments: - The second condition of the maximum entropy equivalence theorem requires that all T are conditionally independent of Y.

This statement is unclear,

as it could mean pairwise independence, or it could mean jointly independent (i.e. for all pairs of non-overlapping subsets A & B of T I(T_A;T_B|Y) = 0).

This is the same as saying the mapping X->T is making each dimension of T orthogonal, as otherwise it would introduce correlations.

The proof of the theorem assumes that pairwise independence induces joint independence

and this is not correct.

- Section 4.1 makes an analogy to EM,

but gradient descent is not like this process as all the parameters are updated at once, and only optimised by a single (noisy) step.

The optimisation with respect to a single layer is conditional on all the other layers remaining fixed,

but the gradient information is stale

(as it knows about the previous step of the parameters in the layer above).

This means that gradient descent does all 1..L steps in parallel,

and this is different to the definition given.

- The proofs in Appendix C which are used for the statement I(T_i;T_j) >=I(T_i;T_j|Y) are incomplete,

and in generate this statement is not true,

so requires proof.

- Lemma 1 appears to assume Lemma 2, and Lemma 2 appears to assume Lemma 1.

Either these lemmas are circular or the derivations of both of them are unclear.

- In Lemma 3 what is the minimum taken over for the left hand side?

Elsewhere the minimum is taken over T, but T does not appear on the left hand side.

Explicit minimums help the reader to follow the logic,

and implicit ones should only be used when it is obvious what the minimum is over.

- In Lemma 5, what does "T is only related to X" mean?

The proof states that Y -> T -> X forms a Markov chain,

but this implies that T is a function of Y, not X.

Minor comments:- I assume that the E_{P(X,Y)} notation is the expectation of that probability distribution,

but this notation is uncommon,

and should be replaced with a more explicit one.

- Markov is usually romanized with a "k" not a "c".

- The paper is missing numerous prepositions and articles,

and contains multiple spelling mistakes & typos.

fact evaluation fact fact fact fact fact evaluation fact fact fact fact fact fact fact fact fact fact fact evaluation fact request fact evaluation request fact evaluation evaluation request fact fact fact evaluation request evaluation evaluation fact


This paper proposes a deep learning (DL) approach (pre-trained CNNs) to the analysis of histopathological images for disease localization.

It correctly identifies the problem that DL usually requires large image databases to provide competitive results,

while annotated histopathological data repositories are costly to produce and not on that size scale.

It also correctly identifies that this is a daunting task for human medical experts

and therefore one that could surely benefit from the use of automated methods like the ones proposed.

The study seems sound from a technical viewpoint to me

and its contribution is incremental, as it builds on existing research,

which is correctly identified.

Results are not always too impressive,

but authors seem intent on making them useful for pathogists in practice

(an intention that is always worth the effort).

I think the paper would benefit from a more explicit statement of its original contributions (against contextual published research)

Minor issues: Revise typos (e.g. title of section 2)

Please revise list of references

(right now a mess in terms of format, typos, incompleteness

fact fact fact fact fact evaluation evaluation fact evaluation evaluation evaluation request request request evaluation


This paper creates adversarial images by imposing a flow field on an image such that the new spatially transformed image fools the classifier.

They minimize a total variation loss in addition to the adversarial loss to create perceptually plausible adversarial images,

this is claimed to be better than the normal L2 loss functions.

Experiments were done on MNIST, CIFAR-10, and ImageNet,

which is very useful to see that the attack works with high dimensional images.

However, some numbers on ImageNet would be helpful

as the high resolution of it make it potentially different than the low-resolution MNIST and CIFAR.

It is a bit concerning to see some parts of Fig. 2.

Some of Fig. 2 (especially (b)) became so dotted that it no longer seems an adversarial that a human eye cannot detect.

And model B in the appendix looks pretty much like a normal model.

It might needs some experiments, either human studies, or to test it against an adversarial detector, to ensure that the resulting adversarials are still indeed adversarials to the human eye.

Another good thing to run would be to try the 3x3 average pooling restoration mechanism in the following paper:

Xin Li, Fuxin Li. Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics . ICCV 2017.

to see whether this new type of adversarial example can still be restored by a 3x3 average pooling the image

(I suspect that this is harder to restore by such a simple method than the previous FGSM or OPT-type, but we need some numbers).

I also don't think FGSM and OPT are this bad in Fig. 4.

Are the authors sure that if more regularization are used these 2 methods no longer fool the corresponding classifiers?

I like the experiment showing the attention heat maps for different attacks.

This experiment shows that the spatial transforming attack (stAdv) changes the attention of the classifier for each target class, and is robust to adversarially trained Inception v3 unlike other attacks like FGSM and CW.

I would likely upgrade to a 7 if those concerns are addressed.

fact fact fact fact evaluation request fact evaluation fact evaluation request request reference fact evaluation evaluation non-arg evaluation fact non-arg


Paper proposes to use a convolutional network with 3 layers (convolutional + maxpoolong + fully connected layers) to embed time series in a new space such that an Euclidian distance is effective to perform a classification.

The algorithm is simple and experiments show that it is effective on a limited benchmark.

It would be interesting to enlarge the dataset to be able to compare statistically the results with state-of-the-art algorithms.

In addition, Authors compare themselves with time series metric learning and generalization of DTW algorithms.

It would also be interesting to compare with other types of time series classification algorithms (Bagnall 2016) .


This paper presents MAd-RL, a method for decomposition of a single-agent RL problem into a simple sub-problems, and aggregating them back together.

Specifically, the authors propose a novel local planner - emphatic, and analyze the newly proposed local planner along of two existing ones - egocentric and agnostic.

The MAd-RL, and theoretical analysis, is evaluated on the Pac-Boy task, and compared to DQN and Q-learning with function approximation.

Pros: 1. The paper is well written, and well-motivated.

2. The authors did an extraordinary job in building the intuition for the theoretical work, and giving appropriate examples where needed.

3. The theoretical analysis of the paper is extremely interesting.

The observation that a linearly weighted reward, implies linearly weighted Q function, analysis of different policies, and local minima that result is the strongest and the most interesting points of this paper.

Cons:1. The paper is too long.

14 pages total - 4 extra pages (in appendix) over the 8 page limit,

and 1 extra page of references.

That is 50% overrun in the context,

and 100% overrun in the references.

The most interesting parts and the most of the contributions are in the Appendix,

which makes it hard to assess the contributions of the paper.

There are two options:   1.1 If the paper is to be considered as a whole, the excessive overrun gives this paper unfair advantage over other ICLR papers.

The flavor and scope and quality of the problems that can be tackled with 50% more space is substantially different from what can be addressed within the set limit.

If the extra space is necessary, perhaps this paper is better suited for another publication?

1.2 If the paper is assessed only based on the main part without Appendix, then the only novelty is emphatic planner, and the theoretical claims with no proofs.

The results are interesting,

but are lacking implementation details.

Overall, a substandard paper.

2. Experiments are disjoint from the method’s section.

For example:2.1 Section 5.1 is completely unrelated with the material presented in Section 4.

2.2 The noise evaluation in Section 5.3 is nice,

but not related with the Section 4.

This is problematic because, it is not clear if the focus of the paper is on evaluating MAd-RL and performance on the Ms.PacMan task, or experimentally demonstrating claims in Section 4.

Recommendations:1. Shorten the paper to be within (or close to the recommended length) including Appendix.

2. Focus paper on the analysis of the advisors,

and Section 5. on demonstrating the claims.

3. Be more explicit about the contributions.

4. How does the negative reward influence the behavior the agent?

The agent receives negative reward when near ghosts.

5. Move the short (or all) proofs from Appendix into the main text.

6. Move implementation details of the experiments (in particular the short ones) into the main text.

7. Use the standard terminology (greedy and random policies vs. egoistic and agnostic) where possible.

The new terms for well-established make the paper needlessly more complex.

8. Focus the literature review on the most relevant work, and contrast the proposed work with existing peer reviewed methods.

9. Revise the literature to emphasize more recent peer reviewed references.

Only three references are recent (less than 5 years), peer reviewed references,

while there are 12 historic references.

Try to reduce dependencies on non-peer reviewed references (~10 of them).

10. Make a pass through the paper, and decouple it from the van Seijen et al., 2017a

11. Minor: Some claims need references:

11.1 Page 5: “egocentric sub-optimality  does not come from the actions that are equally good, nor from the determinism of the policy, since adding randomness…” -

Wouldn’t adding epsilon-greediness get the agent unstuck?

11.2 Page 1. “It is shown on the navigation task ….” -

This seems to be shown later in the results,

but in the intro it is not clear if some other work, or this one shows it.

12. Minor:12.1 Mix genders when talking about people.

Don’t assume all people that make “complex and important problems”, or who are “consulted for advice”, are male.

12.2 Typo: Page 5: a_0 sine die

12.3 Page 7 - omit results that are not shown

12.4 Make Figures larger - it is difficult, if not impossible to see

12.5 What is the difference between Pac-Boy and Ms. Pacman task? And why not use Ms. Packman?

fact fact fact evaluation evaluation evaluation evaluation evaluation fact fact fact fact evaluation evaluation evaluation evaluation request evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation request request request request non-arg fact request request request evaluation request request evaluation fact request request request quote non-arg quote fact evaluation fact request fact fact request non-arg


The paper develops an interesting approach for solving multi-class classification with softmax loss.

The key idea is to reformulate the problem as a convex minimization of a "double-sum" structure via a simple conjugation trick.

SGD is applied to the reformulation: in each step samples a subset of the training samples and labels, which appear both in the double sum.

The main contributions of this paper are: "U-max" idea (for numerical stability reasons) and an ""proposing an "implicit SGD" idea.

Unlike the first review, I see what the term "exact" in the title is supposed to mean.

I believe this was explained in the paper.

I agree with the second reviewer that the approach is interesting.

However, I also agree with the criticism

(double sum formulations exist in the literature;

comments about experiments);

and will not repeat it here.

I will stress though that the statement about Newton in the paper is not justified.

Newton method does not converge globally with linear rate.

Cubic regularisation is needed for global convergence.

Local rate is quadratic.

I believe the paper could warrant acceptance if all criticism raised by reviewer 2 is addressed.

I apologise for short and late review: I got access to the paper only after the original review deadline.

evaluation evaluation fact evaluation evaluation fact evaluation evaluation fact non-arg non-arg fact fact fact fact evaluation non-arg


In conventional boosting methods, one puts a weight on each sample.

The wrongly classified samples get large weights such that in the next round those samples will be more likely to get right.

Thus the learned weak learner at this round will make different mistakes.

This idea however is difficult to be applied to deep learning with a large amount of data.

This paper instead designed a new boosting method which puts large weights on the category with large error in this round.

In other words samples in the same category will have the same weight

Error bound is derived.

Experiments show its usefulness

though experiments are limited

fact fact fact evaluation fact fact fact fact evaluation


Summary ******* The paper provides a collection of existing results in statistics.

Comments ******** Page 1: references to Q-learning and Policy-gradients look awkwardly recent,

given that these have been around for several decades.

I dont get what is the novelty in this paper.

There is no doubt that all the tools that are detailed here are extremely useful and powerful results in mathematical statistics.

But they are all known.

The Gibbs variational principle is folklore,

Proposition 1,2 are available in all good text books on the topic,

and Proposition 4 is nothing but a transportation Lemma.

Now, Proposition 3 is about soft-Bellman operators.

This perhaps is less standard

because contraction property of soft-Bellman operator in infinite norm is more recent than for Bellman operators.

But as mentioned by the authors, this is not new either.

Also I don't really see the point of providing the proofs of these results in the main material, and not for instance in appendix,

as there is no novelty either in the proof techniques.

I don't get the sentence "we have restricted so far the proof in the bandit setting":

bandits are not even mentioned earlier.

Decision ******** I am sorry but unless I missed something (that then should be clarified) this seems to be an empty paper: Strong reject.

fact evaluation fact evaluation evaluation fact evaluation fact fact fact evaluation fact fact evaluation evaluation evaluation fact evaluation


The paper treats the interesting problem of long term video prediction in complex video streams.

I think the approach of adding more structure to their representation before making longer term prediction is also a reasonable one.

Their approach combines an RNN that predicts an encoding of scene and then generating an image prediction using a VAN (Reed et al.).

They show some results on the Human3.6M and the Robot Push dataset.

I find the submission lacking clarity in many places.

The main lack of clarity source I think is about what the contribution is.

There are sparse mentions in the introduction

but I think it would be much more forceful and clear if they would present VAN or Villegas et al method separately and then put the pieces together for their method in a separate section.

This would allow the author to clearly delineate their contribution and maybe why those choices were made.

Also the use of hierarchical is non-standard and leads to confusion I recommend maybe "semantical" or better "latent structured" instead.

Smaller ambiguities in wording are also in the paper :

e.g. related work -> long term prediction "in this work" refers to the work mentioned but could as well be the work that they are presenting.

I find some of the claims not clearly backed by a thorough evaluation and analysis.

Claiming to be able to produce encodings of scenes that work well at predicting many steps into the future is a very strong claim.

I find the few images provided very little evidence for that fact.

I think a toy example where this is clearly the case

because we know exactly the factors of variations and they are inferred by the algorithm automatically or some better ones are discovered by the algorithm,

that would make it a very strong submission.

Reed et al. have a few examples that could be adapted to this setting and the resulting representation, analyzed appropriately, would shed some light into whether this is the right approach for long term video prediction and what are the nobs that should be tweaked in this system.

In the current format, I think that the authors are on a good path

and I hope my suggestions will help them improve their submission,

but as it stands I recommend rejection from this conference.

fact evaluation fact fact evaluation evaluation fact request evaluation request evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation


The paper makes a striking connection between two apparently unrelated problems: the problem of designing neural networks to handle a certain type of correlation and the problem of designing a structure to represent wave-function with quantum entanglement.

In the wave-function context, the Schmidt decomposition of the wave function is an inner product of tensors.

Thus, the mathematical glue connecting the neural networks and quantum entanglement is shown to be tensor networks,

which can represent higher order tensors through inner product of lower-order tensors.

The main technical contribution in the paper is to map convolutional networks with product pooling function (called ConvACs) to a tensor network.

Given this mapping, the authors exploit results in tensor networks (in particular the quantum max-flow min-cut theorem) to calculate the rank of the matricized tensor between a pair of vertex sets using the (appropriately defined) min-cut.

The connection has potential to yield fruitful new results,

however, the potential is not manifested (yet) in the paper.

The main application in deep convolutional networks proposed by the paper is to model how much correlation between certain partition of input variables can be captured by a given convolutional network design.

However, it is unclear how to use Theorem 1 to design neural networks that capture a certain correlation.

A simple example is given in the experiment where the wider layers can be either early in the the neural network or at the later stages; demonstrating that one does better than the other in a certain regime.

It seems that there is an obvious intuition that explains this phenomenon: wider base networks with large filters are better suited to the global task and narrow base networks that have more parameters later down have more local early filters suited to the local task.

The experiments do not quite reveal the power of the proposed approach,

and it is unclear how, if at all, the proposed approach can be applied to more complicated networks.

In summary, this paper is of high theoretical interest and has potential for future applications.

fact fact fact fact evaluation fact evaluation evaluation fact evaluation fact evaluation evaluation evaluation evaluation


This paper extends and speeds up PROSE, a programming by example system, by posing the selection of the next production rule in the grammar as a supervised learning problem.

This paper requires a large amount of background knowledge

as it depends on understanding program synthesis as it is done in the programming languages community.

Moreover the work mentions a neurally-guided search,

but little time is spent on that portion of their contribution.

I am not even clear how their system is trained.

The experimental results do show the programs can be faster but only if the user is willing to suffer a loss in accuracy.

It is difficult to conclude overall if the technique helps in synthesis.

fact evaluation fact fact evaluation evaluation fact evaluation


Review for Deformation of Bregman divergence and its application

Summary:This paper considers parameter estimation for discrete probability models.

The authors propose an estimator that is computed by minimizing a deformed Bregman divergence.

The authors prove that the proposed estimator is more computationally efficient and more robust than the maximal likelihood estimator (MLE), both in theory and simulation.

Major Comments:1. After the definition 1, the likelihood $L(\theta)$ is defined to be the sum of $\log \bar{q}_{\theta}(x_i).

$ Why the gradient of $L(\theta)$ is a related to $\tilde{p}$.

2. After the equation (4), when the authors say $f=(U’)^{-1}$ the authors assume that the first order derivative of $U$ should be a strictly increasing function

(otherwise the inverse function is not well defined, at least in classic notations).

I would like to know whether we only need assume the convexity of $U$.

Are there other assumptions?

3. In Proposition 1, I think the “Fisher consistent” means that (6) holds for any reasonable $U$ and $f$ just as the authors said before Proposition 1.

It is better to add this in the statement of Proposition 1 too.

4. The “Proof 1” is better to be replaced with “Proof of Proposition 1”

(same issues for “Proof 2”, “Proof 3”, etc).

5. In the statement of Theorem 1, do the authors have any constraint for $U$?

6. $\xi_{U,f}$ appears in Theorem 2 without a clear definition.

Even if it seems to be defined in (17), it is better to be defined again.

7. Why Theorem 2 indicates that “the estimator (5) is not influenced so much by the outlier”?

8. How to solve (5)?

Is it trivial?

I expect to see something like “We use … algorithm or toolbox to solve (5)“.

Minor Comments:1. In Example 2, I suggest use some more beautiful symbol like $\top$ to denote the transpose instead of $T$.

2. The length of the equations should not exceed the line-width (e.g., (4) and (7)).

3. In page 5, “We find some examples satisfying 25 in Theorem 2”.

The “25” should be “(25)”.

non-arg fact fact fact fact non-arg fact evaluation non-arg non-arg fact request request request request evaluation request evaluation request request request request request quote request


The authors present two autoregressive models for sampling action probabilities from a factorized discrete action space.

On a multi-agent gridworld task and a multi-agent multi-armed bandit task, the proposed method seems to benefit from their lower-variance entropy estimator for exploration bonus.

A few key citations were missing - notably the LSTM model they propose is a clear instance of an autoregressive density estimator, as in PixelCNN, WaveNet and other recently popular deep architectures.

In that context, this work can be viewed as applying deep autoregressive density estimators to policy gradient methods.

At least one of those papers ought to be cited.

It also seems like a simple, obvious baseline is missing from their experiments - simply independently outputting D independent softmaxes from the policy network.

Without that baseline it's not clear that any actual benefit is gained by modeling the joint distribution between actions, especially since the optimal policy for an MDP is provably deterministic anyway.

The method could even be made to capture dependencies between different actions by adding a latent probabilistic layer in the middle of the policy network, inducing marginal dependencies between different actions.

A direct comparison against one of the related methods in the discussion section would help better contextualize the paper as well.

A final point on clarity of presentation - in keeping with the convention in the field, the readability of the tables could be improved by putting the top-performing models in bold, and Table 2 should almost certainly be replaced by a boxplot.

fact fact evaluation evaluation request evaluation evaluation evaluation request request


The paper proposes an online distillation method, called co-distillation, where the two different models are trained to match the predictions of other model in addition to minimizing its own loss.

The proposed method is applied to two large-scale datasets

and showed to perform better than other baselines such as label smoothing, and the standard ensemble.

The paper is clearly written and was easy to understand.

My major concern is the significance and originality of the proposed method.

As written by the authors, the main contribution of the paper is to apply the codistillation method, which is pretty similar to Zhang et. al (2017), at scale.

But, because from Zhang's method, I don't see any significant difficulty in applying to large-scale problems,

I'm not sure that this can be a significant contribution.

Rather, I think, it would have been better for the authors to apply the proposed methods to a smaller scale problems as well in order to explore more various aspects of the proposed methods including the effects of number of different models.

In this sense, it is also a limitation that the authors showing experiments where only two models are codistillated.

Usually, ensemble becomes stronger as the number of model increases.

fact fact fact evaluation evaluation evaluation evaluation evaluation request request evaluation


Summary ======== The authors present a new regularization term, inspired from game theory, which encourages the discriminator's gradient to have a norm equal to one.

This leads to reduce the number of local minima,

so that the behavior of the optimization scheme gets closer to the optimization of a zero-sum games with convex-concave functions.

Clarity ====== Overall, the paper is clear and well-written.

However, the authors should motivate better the regularization introduced in  section 2.3.

Originality ========= The idea is novel and interesting.

In addition, it is easy to implement it for any GANs since it requires only an additional regularization term.

Moreover, the numerical experiments are in favor of the proposed method.

Comments ========= - Why should the norm of the gradient should to be equal to 1 and not another value?

Is this possible to improve the performance if we put an additional hyper-parameter instead?

- Are the performances greatly impacted by other value of lambda and c (the suggested parameter values are lambda = c = 10)?

- As mentioned in the paper, the regularization affects the modeling performance.

Maybe the authors should add a comparison between different regularization parameters to illustrate the real impact of lambda and c on the performance.

- GANs performance is usually worse on very big dataset such as Imagenet.

Does this regularization trick makes their performance better?

fact fact fact evaluation request evaluation evaluation fact request request request fact request fact request


Summary ------- This paper proposes a generative model of symbolic (MIDI) melody in western popular music.

The model uses an LSTM architecture to map sequences of chord symbols and structural identifiers (e.g., verse or chorus) to predicted note sequences which constitute the melody line.

The key innovation proposed in this work is to jointly encode note symbols along with timing and duration information to form musical "words" from which melodies are composed.

The proposed model compares compared favorably to prior work in listener preference and Turing-test studies, and performs.

Quality ------- Overall, I found the paper interesting, and the provided examples generated by the model sound relatively good.

The quantitative evaluations seem promising,

though difficult to interpret fully due to a lack of provided detail (see below).

Apart from clarification issues enumerated below, where the paper could be most substantially improved is in the evaluation of the various ingredients of the model.

Many ideas are presented with some abstract motivation,

but there is no comparative evaluation to demonstrate what happens when any one piece is removed from the system.

Some examples:- How important is the "song part" contextual input?

- What happens if the duration or timing information is not encoded with the note?

- How important is the pitch range regularization?

Since the authors claim these ideas as novel,

I would expect to see more evaluation of their independent impact on the resulting system.

Without such an evaluation, it is difficult to take any general lessons away from this paper.

Clarity ------- While the main ideas of this paper are presented clearly,

I found the details difficult to follow.

Specifically, the following points need to be substantially clarified: - The note encoding described in Section 3.1: "w_i = (p_i, t_i, l_i)" describes the pitch, timing, and duration of the i'th chord, but it is not explained how time and duration are represented.

Since these are derived from MIDI, I would expect either ticks or seconds -- or maybe a tempo-normalized variant --

but Figure 2 suggests staff notation, which is not explicitly coded in MIDI.

Please explain precisely how the data is represented.

- Also in Section 3, several references are made to a "previous" model, but no citation is given.

Was this a specific published work?

- Equation 1 is missing a variable (j) for the range of the summation.

It took a few passes for me to parse what was going on here.

One could easily mistake it for summing over i to describe partial subsequences,

but I don't think this is what is intended.

- "... our model does not have to consider intervals that do not contain notes" --

this contradicts the implication of Figure 2b, where a rest is explicitly notated in the generated sequence.

Since MIDI does not explicitly encode rests

(they must be inferred from the absence of note events),

I'd suggest wording this more carefully, and being more explicit about what is produced by the model and what is notational embellishment for expository purposes.

- Equation 2 describes the LSTM gate equations,

but there is no concrete description of the model architecture used in this paper.

How are the hidden states mapped to note predictions?

What is the loss function and optimizer?

These details are necessary to facilitate replication.

- Equation 3 and the accompanying text implies that song part states (x_i) are conditionally independent given the current chord state (z_i).

Is that correct?

If so, it seems like a strange choice,

since I would expect a part state to persist across multiple chord state transitions.

Please explain this part in more detail.

Also a typo in the second factor: p(z_N | z_{N-1}) should be p(z_n | z_{n-1}); likewise  p(x_n | z_N).

- The regularization penalty (Alg. 1) is also difficult to follow.

Is S derived from  P by viterbi decoding, or independent (point-wise argmax) decoding?

What exactly is the "E" that results in the derivative at step 8, and why does the derivative for p_i depend on the total sum C?

This all seems non-obvious, and worth describing in more detail since it seems critical to the performance of the model.

- Table 2: what does "# samples" mean in this context?

Why is it different from "# songs"?

- Section 4.2: the description of the evaluation suggests that the proposed model's output was always played before the baseline.

Is that correct?

If so, does that  bias the results?

- Section 4.2: are the examples provided to listeners just the melody lines, or full mixes on top of the input chord sequence?

It's unclear from the text,

and it seems like a relevant detail to correctly assess the fairness of the comparison to the  baselines.

- Section 4.2: how many generated examples were included in this evaluation?

Should this instead be in or out of key, since the tuning is presumably fixed by MIDI synthesis?

Originality ----------- As far as I know, the proposed method is novel, though strongly related to (cited) prior work.

The key idea seems to be encoding of notes and properties as analogous to "words".

I find this analogy a little bit of a stretch,

since even with timing and duration included, it's hard to argue that a single note event has semantic content in the way that a word does.

A little more development of this idea, and some more concrete motivation for the specific choices of which properties to include, would go a long way in strengthening the paper.

Significance ------------ The significance of this work is difficult to assess without independent evaluation of the proposed novel components.

fact fact evaluation fact evaluation evaluation evaluation evaluation fact fact request request request fact request evaluation evaluation evaluation request request fact request request non-arg fact non-arg evaluation evaluation quote fact fact fact request fact fact request request evaluation fact evaluation evaluation evaluation request request evaluation request request evaluation request request fact request request request evaluation fact request request evaluation fact evaluation evaluation evaluation evaluation


This paper proposes leveraging labelled controlled data to accelerate reinforcement-based learning of a control policy.

It provides two main contributions: pre-training the policy network of a DDPG agent in a supervised manner so that it begins in reasonable state-action distribution and regalurizing the Q-updates of the q-network to be biased towards existing actions.

The authors use the TORCS enviroment to demonstrate the performance of their method both in final cumulative return of the policy and speed of learning.

This paper is easy to understand but has a couple shortcomings and some fatal (but reparable) flaws:.

1) When using RL please try to standardize your notation to that used by the community,

it makes things much easier to read.

I would strongly suggest avoiding your notation a(x|\Theta) and using \pi(x)

(subscripting theta or making conditional is somewhat less important).

Your a(.) function seems to be the policy here,

which is invariable denoted \pi in the RL literature.

There has been recent effort to clean up RL notation which is presented here:

You have no obligation to use this notation but it does make reading of your paper much easier on others in the community.

This is more of a shortcoming than a fundamental issue.

2) More fatally, you have failed to compare your algorithm's performance against benchline implementations of similar algorithms.

It is almost trivial to run DDPG on Torcs using the openAI baselines package

I would have loved, for example, to see the effects of simply pre-training the DDPG actor on supervised data, vs. adding your mixture loss on the critic.

Using the baselines would have (maybe) made a very compelling graph showing DDPG, DDPG + actor pre-training, and then your complete method.

3) And finally, perhaps complementary to point 2), you really need to provide examples on more than one environment.

Each of these simulated environments has its own pathologies linked to determenism, reward structure, and other environment particularities.

Almost every algorithm I've seen published will often beat baselines on one environment and then fail to improve or even be wors on others,

so it is important to at least run on a series of these.

Mujoco + AI Gym should make this really easy to do

(for reference, I have no relatinship with OpenAI).

Running at least cartpole (which is a very well understood control task), and then perhaps reacher, swimmer, half-cheetah etc. using a known contoller as your behavior policy (behavior policy is a good term for your data-generating policy.)

4) In terms of state of the art you are very close to Todd Hester et. al's paper on imitation learning,

and although you cite it, you should contrast your approach more clearly with the one in that paper.

Please also have a look at some more recent work my Matej Vecerik, Todd Hester & Jon Scholz: 'Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards' for an approach that is pretty similar to yours.

Overall I think your intuitions and ideas are good,

but the paper does not do a good enough job justifying empirically that your approach provides any advantages over existing methods.

The idea of pre-training the policy net has been tried before

(although I can't find a published reference)

and in my experience will help on certain problems, and hinder on others,

primarily because the policy network is already 'overfit' somewhat to the expert, and may have a hard time moving to a more optimal space.

Because of this experience I would need more supporting evidence that your method actually generalizes to more than one RL environment.

fact evaluation fact evaluation request evaluation request evaluation evaluation fact fact reference request evaluation evaluation evaluation reference request evaluation request fact evaluation request evaluation reference request evaluation request request evaluation evaluation fact evaluation non-arg evaluation request


This paper proposes training binary and ternary weight distribution networks through the local reparametrization trick and continuous optimization.

The argument is that due to the central limit theorem (CLT) the distribution on the neuron pre-activations is approximately Gaussian, with a mean given by the inner product between the input and the mean of the weight distribution and a variance given by the inner product between the squared input and the variance of the weight distribution.

As a result, the parameters of the underlying discrete distribution can be optimized via backpropagation by sampling the neuron pre-activations with the reparametrization trick.

The authors further propose appropriate initialisation schemes and regularization techniques to either prevent the violation of the CLT or to prevent underfitting.

The method is evaluated on multiple experiments.

This paper proposed a relatively simple idea for training networks with discrete weights that seems to work in practice.

My main issue is that while the authors argue about novelty,

the first application of CLT for sampling neuron pre-activations at neural networks with discrete r.v.s is performed at [1].

While [1] was only interested in faster convergence and not on optimization of the parameters of the underlying distribution,

the extension was very straightforward.

I would thus suggest that the authors update the paper accordingly.

Other than that, I have some other comments: - The L2 regularization on the distribution parameters for the ternary weights is a bit ad-hoc;

why not penalise according to the entropy of the distribution which is exactly what you are trying to achieve?

- For the binary setting you mentioned that you had to reduce the entropy thus added a “beta density regulariser”.

Did you add R(p) or log R(p) to the objective function?

Also, with alpha, beta = 2 the beta density is unimodal with a peak at p=0.5;

essentially this will force the probabilities to be close to 0.5, i.e. exactly what you are trying to avoid.

To force the probability near the endpoints you have to use alpha, beta < 1 which results into a “bowl” shaped Beta distribution.

I thus wonder whether any gains you observed from this regulariser are just an artifact of optimization.

- I think that a baseline (at least for the binary case) where you learn the weights with a continuous relaxation, such as the concrete distribution, and not via CLT would be helpful.

Maybe for the network to properly converge the entropy for some of the weights needs to become small (hence break the CLT).

[1] Wang & Manning, Fast Dropout Training.

fact fact fact fact fact evaluation fact fact fact evaluation request evaluation request fact non-arg fact fact fact fact request fact reference


This paper addresses the problem of one class classification.

The authors suggest a few techniques to learn how to classify samples as negative (out of class) based on tweaking the GAN learning process to explore large areas of the input space which are out of the objective class.

The suggested techniques are nice and show promising results.

But I feel a lot can still be done to justify them, even just one of them.

For instance, the authors manipulate the objective of G using a new parameter alpha_new and divide heuristically the range of its values.

But, in the experimental section results are shown only for a  single value, alpha_new=0.9

The authors also suggest early stopping

but again (as far as I understand) only a single value for the number of iterations was tested.

The writing of the paper is also very unclear, with several repetitions and many typos e.g.:

'we first introduce you a'

'architexture'

'future work remain to'

'it self'

I believe there is a lot of potential in the approach(es) presented in the paper.

In my view a much stronger experimental section together with a clearer presentation and discussion could overcome the lack of theoretical discussion.

fact fact evaluation evaluation fact fact fact fact evaluation quote quote quote quote evaluation request


Summary The paper is well-written

but does not make deep technical contributions and does not present a comprehensive evaluation or highly insightful empirical results.

Abstract / Intro I get the entire focus of the paper is some variant of Pac-Man which has received attention in the RL literature for Atari games,

but for the most part the impressive advances of previous Atari/RL papers are in the setting that the raw video is provided as input,

which is much different than solving the underlying clean mathematically abstracted problem (as a grid world with obstacles) as done here and evident in the videos.

Further it is honestly hard for me to be strongly motivated about a paper that focuses on the need to decompose Pac-man into sub-agents/advisor value functions.

Section 2 Another historically well-cited paper for MDP decomposition:

Flexible Decomposition Algorithms for Weakly Coupled Markov Decision Problems, Ronald Parr. UAI 98. https://dslpitt.org/uai/papers/98/p422-parr.pdf

Section 3 Is the additive reward decomposition a required part of the problem specification?

It seems so, i.e., there is no obvious method for automatically decomposing a monolithic reward function over advisors.

Section 4 * Egocentric: Definition 1: Sure, the problem will have local optima (attractors) when decomposed suboptimally

-- I'm not sure what new insight we've gained from this analysis...

it is a general problem with any function approximation scheme that does not guarantee that the rank ordering of actions for a state is preserved.

* Agnostic Other than approximating some type of myopic rollout, I really don't see why this approach would be reasonable?

I am surprised it works at all

though my guess is that this could simply be an artifact of evaluating on a single domain with a specific structure.

* Empathic This appears to be the key contribution

though related work certainly infringes on its novelty.

Is this paper then an empirical evaluation of previous methods in a single Pac-man grid world variant?

I wonder if the theory of DEC-MDPs would have any relevance for novel analysis here?

Section 5 I'm disappointed that the authors only evaluate on a single domain;

presumably the empathic approach has applications beyond Pac-Man?

The fact that empathic generally performs better is not at all surprising.

The fact that a modified discount factor for egocentric can also perform well is not surprising given that lower discount factors have often been shown to improve approximated MDP solutions, e.g.,

Biasing Approximate Dynamic Programming with a Lower Discount Factor Marek Petrik, Bruno Scherrer (NIPS-08). http://marek.petrik.us/pub/Petrik2009a.pdf

***Side note: The following part is somewhat orthogonal to the review above in that I would not expect the authors to address this on revision, *but* at the same time I think it provides a connection to the special case of concurrent action decomposition into advisors,

which could potentially provide a high impact direction of application for this work

(i.e., concurrent problems are hard and show up in numerous operations research problems covering inventory control, logistics, epidemic response).

For the special case that each advisor is assigned to one action in a factored space of concurrent actions, the egocentric algorithm would be very close to the Hindsight approximation in Section 6 of this paper (including an additive decomposition of rewards):

Planning in Factored Action Spaces with Symbolic Dynamic Programming Aswin Nadamuni Raghavan, Alan Fern, Prasad Tadepalli, Roni Khardon, and Saket Joshi (AAAI-12). https://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/download/5012/5336

This simple algorithm is hard to beat

for the following reason that connects some details of your egocentric and empathic settings: rather than decomposing a concurrent MDP into independent problems per concurrent action, the optimization of each action (by each advisor) is done in sequence (advisors are ordered) and gets to condition on the previously selected advisor actions.

So it provides an alternate paradigm where advisors actually get to see and condition their policy on what other advisors are doing.

In my own work comparing optimal concurrent solutions to this approach, I have found this approach to be near-optimal and much more efficient to solve since it exploits decomposition.

Why is this relevant to this work?

Because (a) it suggests another variant of the advisor decomposition that at least makes sense in the case of concurrent actions (and perhaps shared actions though this would require some extension)

and (b) it suggests there are more options than just the full egocentric and empathic settings in this important class of concurrent action problems that are necessarily solved in practice for large action spaces by some form of decomposition.

This could be an interesting direction for future exploration of the ideas in this work, where there might be additional technical novelty and more space for empirical contributions and observations.

evaluation evaluation evaluation fact evaluation evaluation evaluation reference request evaluation fact evaluation fact evaluation evaluation evaluation evaluation evaluation evaluation request evaluation evaluation evaluation evaluation reference non-arg evaluation evaluation evaluation reference evaluation fact fact non-arg non-arg fact fact evaluation


The authors propose an objective whose Lagrangian dual admits a variety of modern objectives from variational auto-encoders and generative adversarial networks.

They describe tradeoffs between flexibility and computation in this objective leading to different approaches.

Unfortunately, I'm not sure what specific contributions come out,

and the paper seems to meander in derivations and remarks that I didn't understand what the point was.

First, it's not clear what this proposed generalization offers.

It's a very nuanced and not insightful construction (eq. 3) and with a specific choice of a weighted sum of mutual informations subject to a combinatorial number of divergence measure constraints, each possibly held in expectation (eq. 5) to satisfy the chosen subclass of VAEs and GANs; and with or without likelihoods (eq. 7).

What specific insights come from this that isn't possible without the proposed generalization?

It's also not clear with many GAN algorithms that reasoning with their divergence measure in the limit of infinite capacity discriminators is even meaningful

(e.g., Arora et al., 2017;

Fedus et al., 2017).

It's only true for consistent objectives such as MMD-GANs.

Section 4 seems most pointed in explaining potential insights.

However, it only introduces hyperparameters and possible combinatorial choices with no particular guidance in mind.

For example, there are no experiments demonstrating the usefulness of this approach except for a toy mixture of Gaussians and binarized MNIST, explaining what is already known with the beta-VAE and infoGAN.

It would be useful if the authors could make the paper overall more coherent and targeted to answer specific problems in the literature rather than try to encompass all of them.

Misc + The "feature marginal" is also known as the aggregate posterior (Makhzani et al., 2015) and average encoding distribution (Hoffman and Johnson, 2016); also see Tomczak and Welling (2017).

fact fact evaluation evaluation evaluation evaluation fact evaluation reference reference fact evaluation fact fact request fact


This paper proposes to use a hybrid of convolutional and recurrent networks to predict the DSL specification of a GUI given a screenshot of the GUI.

Pros:The paper is clear

and the proposed problem is novel and well-defined.

The training data is synthetic, allowing for arbitrarily large training sets to be generated.

The authors have made their synthetic dataset publicly available.

The method seems to work well based on the samples and ROC curves presented.

Cons: This is mostly an application of an existing method to a new domain -- as stated in the related work section, effectively the same convnet+RNN architecture has been in common use for image captioning and other vision applications.

The UIs that are represented in the dataset seem quite simple;

it’s not clear that this will transfer to arbitrarily complex and multi-page UIs.

The main motivation for the proposed system seems to be for non-technical designers to be able to implement UIs just by drawing a mockup screenshot.

However, the paper hasn’t shown that this is necessarily possible assuming the hand-designed mockups aren’t pixel-for-pixel matches with a screenshot that could be generated by the “DSL code -> screenshot” mapping that this system learns to invert.

There exist a number of “drag and drop” style UI design products (at least for HTML) that would seem to accomplish the same basic goal as the proposed system in a more reliable way.

(Though the proposed system does have the advantage of only requiring a screenshot created using any software, rather than being restricted to a particular piece of software.)

Overall, the paper is well-written but the novelty and applicability seems a bit limited.

fact evaluation evaluation fact fact evaluation fact evaluation evaluation fact fact fact fact evaluation


This paper proposes a new method for reverse curriculum generation by gradually reseting the environment in phases and classifying states that tend to lead to success.

It additionally proposes a mechanism for learning from human-provided "key states".

The ideas in this paper are quite nice,

but the paper has significant issues with regard to clarity and applicability to real-world problems:

First, it is unclear is the proposed method requires access only high-dimensional observations (e.g. images) during training or if it additionally requires low-dimensional states (e.g. sufficient information to reset the environment).

In most compelling problems settings where a low-dimensional representation that sufficiently explains the current state of the world is available during training, then it is also likely that one can write down a nicely shaped reward function using that state information during training, in which case, it makes sense to use such a reward function.

This paper seems to require access to low-dimensional states, and specifically considers the sparse-reward setting,

which seems contrived.

Second, the paper states that the assumption "when resetting, the agent can be reset to any state" can be satisfied in problems such as real-world robotic manipulation.

This is not correct.

If the robot could autonomously reset to any state, then we would have largely solved robotic manipulation.

Further, it is not always realistic to assume access to low-dimensional state information during training on a real robotic system (e.g. knowing the poses of all of the objects in the world).

Third, the experiments section lacks crucial information needed to understand the experiments.

What is the state, observation, and action space for each problem setting?

What is the reward function for each problem setting?

What reinforcement learning algorithm is used in combination with the curriculum and tendency rewards?

Are the states and actions continuous or discrete?

Without this information, it is difficult to judge the merit of the experimental setting.

Fourth, the proposed method seems to lack motivation, making the proposed scheme seem a bit ad hoc.

Could each of the components be motivated further through more discussion and/or ablative studies?

Finally, the main text of the paper is substantially longer than the recommended page limit.

It should be shortened by making the writing more concise.

Beyond my feedback on clarity and significance, here are further pieces of feedback with regard to the technical content, experiments, and related work:I'm wondering -- can the reward shaping in Equation 2 be made to satisfy the property of not affecting the final policy?

(see Ng et al. '09)

If so, such a reward shaping would make the method even more appealing.

How do the experiments in section 5.4 compare to prior methods and ablations?

Without such a comparison, it is impossible to judge the performance of the proposed method and the level of difficulty of these tasks.

At the very least, the paper should compare the performance of the proposed method to the performance a random policy.

The paper is missing some highly relevant references.

First, how does the proposed method compare to hindsight experience replay?

[1] Second, learning from keyframes (rather than demonstrations) has been explored in the past [1].

It would be preferable to use the standard terminology of "keyframe".

[1] Andrychowicz et al. Hindsight Experience Replay. 2017

[2] Akgun et al. Keyframe-based Learning from Demonstration. 2012

In summary, I think this paper has a number of promising ideas and experimental results,

but given the significant issues in clarity and significance to real world problems, I don't think that the current version of this paper is suitable for publication in ICLR.

More minor feedback on clarity and correctness:- Abstract: "Deep RL algorithms have proven successful in a vast variety of domains" -- This is an overstatement.

- The introduction should be more clear with regard to the assumptions.

In particular, it would be helpful to see discussion of requiring human-provided keyframes.

As is, it is unclear what is meant by "checkpoint scheme",

which is not commonly used terminology.

- "This kind of spare reward, goal-oriented tasks are considered the most difficult challenges" -- This is also an overstatement.

Long-horizon tasks and high-dimensional observations are also very difficult.

Also, the sentence is not grammatically correct.

- "That is, environment" -> "That is, the environment"

- In the last paragraph of the intro, it would be helpful to more clearly state what the experiments can accomplish.

Can they handle raw pixel inputs?

- "diverse domains" -> "diverse simulated domains"

- "a robotic grasping task" -> "a simulated robotic grasping task"

- There are a number of issues and errors in citations, e.g. missing the year, including the first name, incorrect reference

- Assumption 1: \mathcal{P} has not yet been defined.

- The last two paragraphs of section 3.2 are very difficult to understand without reading the method yet

- "conventional RL solver tend" -> "conventional RL tend",

also should mention sparse reward in this sentence.

- Algorithm 1 and Figure 1 are not referenced in the text anywhere, and should be

- The text in Figure 1 and Figure 3 is extremely small

- The text in Figure 3 is extremely small

fact fact evaluation evaluation evaluation evaluation fact evaluation fact fact fact evaluation evaluation request request request request evaluation evaluation request fact request request reference evaluation request fact request evaluation request fact request reference reference evaluation evaluation evaluation request request evaluation evaluation evaluation evaluation fact request request request request request fact fact evaluation request request request request request


The paper provides methods for training deep networks using half-precision floating point numbers without losing model accuracy or changing the model hyper-parameters.

The main ideas are to use a master copy of weights when updating the weights, scaling the loss before back-prop and using full precision variables to store products.

Experiments are performed on a large number of state-of-art deep networks, tasks and datasets

which show that the proposed mixed precision training does provide the same accuracy at half the memory.

Positives - The experimental evaluation is fairly exhaustive on a large number of deep networks, tasks and datasets

and the proposed training preserves the accuracy of all the tested networks at half the memory cost.

Negatives - The overall technical contribution is fairly small and are ideas that are regularly implemented when optimizing systems.

- The overall advantage is only a 2x reduction in memory which can be gained by using smaller batches at the cost of extra compute.

fact evaluation evaluation fact evaluation fact evaluation fact


The authors try to use continuous time generalizations of normalizing flows for improving upon VAE-like models or for standard density estimation problems.

Clarity: the text is mathematically very sloppy / hand-wavy.

1. I do not understand proposition (1).

I do not think that the proof is correct

(e.g. the generator L needs to be applied to a function

-- the notation L(x) does not make too much sense):

indeed, in the case when the volatility is zero (or very small), this proposition would imply that any vector field induces a volume preserving transformation, which is indeed false.

2. I do not really see how the sequence of minimization Eq(5) helps in practice.

The Wasserstein term is difficult to hand.

3. in Equation (6), I do not really understand what $\log(\bar{\rho})$ is if $\bar{\rho}$ is an empirical distribution.

One really needs $\bar{\rho}$ to be a probability density to make sense of that.

fact evaluation evaluation fact fact evaluation fact evaluation evaluation evaluation evaluation


The authors propose an extension to the Neural Statistician which can model contexts with multiple partially overlapping features.

This model can explain datasets by taking into account covariate structure needed to explain away factors of variation and it can also share this structure partially between datasets.

A particularly interesting aspect of this model is the fact that it can learn these context c as features conditioned on meta-context a, which leads to a disentangled representation.

This is also not dissimilar to ideas used in 'Bayesian Representation Learning With Oracle Constraints' Karaletsos et al 2016

where similar contextual features c are learned to disentangle representations over observations and implicit supervision.

The authors provide a clean variational inference algorithm to learn their model.

However, a key problem is the following: the nature of the discrete variables being used makes them hard to be inferred with variational inference.

The authors mention categorical reparametrization as their trick of choice, but do not go into empirical details int heir experiments regarding the success of this approach.

In fact, it would be interesting to study which level of these variables could be analytically collapsed (such as done in the Semi-Supervised learning work by Kingma et al 2014) and which ones can be sampled effectively using a form of reparametrization.

This also touches on the main criticism of the paper: While the model technically makes sense and is cleanly described and derived,  the empirical evaluation is on the weak side and the rich properties of the model are not really shown off.

It would be interesting if the authors could consider adding a more illustrative experiment and some more empirical results regarding inference in this model and the marginal structures that can be learned with this model in controlled toy settings.

Can the model recover richer structure that was imposed during data generation?

How limiting is the learning of a?

How does the likelihood of the model behave under the circumstances?

The experiments do not really convey how well this all will work in practice.

fact fact evaluation evaluation evaluation evaluation evaluation fact request evaluation request request request request fact


Summary of paper: The paper proposes an RNN-based neural network architecture for embedding programs, focusing on the semantics of the program rather than the syntax.

The application is to predict errors made by students on programming tasks.

This is achieved by creating training data based on program traces obtained by instrumenting the program by adding print statements.

The neural network is trained using this program traces with an objective for classifying the student error pattern (e.g. list indexing, branching conditions, looping bounds).

---Quality: The experiments compare the three proposed neural network architectures with two syntax-based architectures.

It would be good to see a comparison with some techniques from Reed & De Freitas (2015)

as this work also focuses on semantics-based embeddings.

Clarity: The paper is clearly written.

Originality: This work doesn't seem that original from an algorithmic point of view

since Reed & De Freitas (2015) and Cai et. al (2017) among others have considered using execution traces.

However the application to program repair is novel (as far as I know).

Significance: This work can be very useful for an educational platform

though a limitation is the need for adding instrumentation print statements by hand.

--- Some questions/comments: - Do we need to add the print statements for any new programs that the students submit?

What if the structure of the submitted program doesn't match the structure of the intended solution and hence adding print statements cannot be automated?

---References  Cai, J., Shin, R., & Song, D. (2017). Making Neural Programming Architectures Generalize via Recursion. In International Conference on Learning Representations (ICLR).

fact fact fact fact fact request fact evaluation evaluation fact evaluation evaluation request request request reference


This paper provides an overview of the Deep Voice 3 text-to-speech system.

It describes the system in a fair amount of detail and discusses some trade-offs w.r.t. audio quality and computational constraints.

Some experimental validation of certain architectural choices is also provided.

My main concern with this work is that it reads more like a tech report:

it describes the workings and design choices behind one particular system in great detail,

but often these choices are simply stated as fact and not really motivated, or compared to alternatives.

This makes it difficult to tell which of these aspects are crucial to get good performance, and which are just arbitrary choices that happen to work okay.

As this system was clearly developed with actual deployment in mind (and not purely as an academic pursuit),

all of these choices must have been well-deliberated.

It is unfortunate that the paper doesn't demonstrate this.

I think this makes the work less interesting overall to an ICLR audience.

That said, it is perhaps useful to get some insight into what types of models are actually used in practice.

An exception to this is the comparison of "converters", model components that convert the model's internal representation of speech into waveforms.

This comparison is particularly interesting

because some of the results are remarkable, i.e. Griffin-Lim spectrogram inversion and the WORLD vocoder achieving very similar MOS scores in some cases (Table 2).

I wish there would be more of that kind of thing in the paper.

The comparison of attention mechanisms is also useful.

I'm on the fence as I think it is nice to get some insight into a practical pipeline which benefits from many current trends in deep learning research (autoregressive models, monotonic attention, ...),

but I also feel that the paper is a bit meager when it comes to motivating all the architectural aspects.

I think the paper is well written

so I've tentatively recommended acceptance.

Other comments: - The separation of the "decoder" and "converter" stage is not entirely clear to me.

It seems that the decoder is trained to predict spectrograms autoregressively, but its final layer is then discarded and its hidden representation is then used as input to the converter stage instead?

The motivation for doing this is unclear to me,

surely it would be better to train everything end-to-end, including the converter?

This seems like an unnecessary detour,

what's the reasoning behind this?

- At the bottom of page 2 it is said that "the whole model is trained end-to-end, excluding the vocoder",

which I think is an unfortunate turn of phrase.

It's either end-to-end, or it isn't.

- In Section 3.3, the point of mixing of h_k and h_e is unclear to me.

Why is this done?

- The gated linear unit in Figure 2a shows that speaker embedding information is only injected in the linear part.

Has this been experimentally validated to work better than simpler mechanisms such as adding conditioning-dependent biases/gains?

- When the decoder is trained to do autoregressive prediction of spectrograms, is it autoregressive only in time, or also in frequency?

I'm guessing it's the former,

but this means there is an implicit independence assumption

(the intensities in different frequency bins are conditionally independent, given all past timesteps).

Has this been taken into consideration?

Maybe it doesn't matter because the decoder is never used directly anyway, and this is only a "feature learning" stage of sorts?

- Why use the L1 loss on spectrograms?

- The recent work on Parallel WaveNet may allow for speeding up WaveNet when used as a vocoder,

this could be worth looking into seeing as inference speed is used as an argument to choose different vocoder strategies (with poorer audio quality as a result).

- The title heavily emphasizes that this model can do multi-speaker TTS with many (2000) speakers,

but that seems to be only a minor aspect that is only discussed briefly in the paper.

And it is also something that preceding systems were already capable of

(although maybe it hasn't been tested with a dataset of this size before).

It might make sense to rethink the title to emphasize some of the more relevant and novel aspects of this work.

fact evaluation fact evaluation evaluation evaluation evaluation evaluation request evaluation evaluation request fact evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation fact evaluation request evaluation request fact evaluation fact evaluation request fact request request evaluation fact fact request non-arg request fact evaluation evaluation evaluation fact fact request


The paper proposes a VAE inference network for a non-parametric topic model.

The model on page 4 is confusing to me

since this is a topic model,

so document-specific topic distributions are required,

but what is shown is only stick-breaking for a mixture model.

From what I can tell, the model itself is not new, only the fact that a VAE is used to approximate the posterior.

In this case, if the model is nonparametric, then comparing with Wang, et al (2011) seems the most relevant non-deep approach.

Given the factorization used in that paper, the q distributions are provably optimal by the standard method.

Therefore, something must be gained by the VAE due to a non-factorized q.

This would be best shown by comparing with the corresponding non-deep version of the model rather than LDA and other deep models.

fact evaluation fact fact fact evaluation evaluation evaluation evaluation request


The paper proposes a method which jointly learns the label embedding (in the form of class similarity) and a classification model.

While the motivation of the paper makes sense,

the model is not properly justified,

and I learned very little after reading the paper.

There are 5 terms in the proposed objective function.

There are also several other parameters associated with them: for example, the label temperature of z_2’’ and and parameter alpha in the second last term etc.

For all the experiments, the same set of parameters are used,

and it is claimed that “the method is robust in our experiment and simply works without fine tuning”.

While I agree that a robust and fine-tuning-free model is ideal

1) this has to be justified by experiment.

2) showing the experiment with different parameters will help us understand the role each component plays.

This is perhaps more important than improving the baseline method by a few point,

especially given that the goal of this work is not to beat the state-of-the-art.

fact evaluation evaluation evaluation fact fact fact quote evaluation request evaluation evaluation evaluation


Proposal is to restrict the feasible parameters to ones that have produce a function with small variance over pre-defined groups of images that should be classified the same.

As authors note, this constraint can be converted into a KKT style penalty with KKT multiplier lambda.

Thus this is very  similar to other regularizers that increase smoothness of the function, such as total variation or a graph Laplacian defined with graph edges connecting the examples in each group, as well as manifold regularization (see e.g. Belkin, Niyogi et al. JMLR).

Heck, in practie ridge regularization will also do something similar for many function classes.

Experiments didn't compare to any similar smoothness regularization

(and my preferred would have been a comparison to graph Laplacian or total variation on graphs formed by the same clustered examples).

It's also not clear either how important it is that they hand-define the groups over which to minimize variance or if just generally adding smoothness regularization would have achieved the same results.

That made it hard to get excited about the results in a vacuum.

Would this proposed strategy have thwarted the Russian tank legend problem?

Would it have fixed the Google gorilla problem?

Why or why not?

Overall, I found the writing a bit bombastic for a strategy that seems to require the user to hand-define groups/clusters of examples.

Page 2: calling additional instances of the same person “counterfactual observations” didn’t seem consistent with the usual definition of that term…

maybe I am just missing the semantic link here,

but this isn't how we usually use the term counterfactual in my corner of the field.

Re: “one creates additional samples by modifying…”

be nice to quote more of the early work doing this,

I believe the first work of this sort was Scholkopf’s, he called it “virtual examples”

and I’m pretty sure he specifically did it for rotation MNIST images (and if not exactly that, it was implied).

I think the right citation is “Incorporating invariances in support vector learning machines“ Scholkopf, Burges, Vapnik 1996, but also see Decoste * Scholkopf 2002 “Training invariant support vector machines.”

fact fact evaluation evaluation fact request evaluation evaluation request request request evaluation evaluation evaluation evaluation quote request fact evaluation request


The authors ask when the hidden layer units of a multi-layer feed-forward neural network will display selectivity to object categories.

They train 3-layer ANNs to categorize binary patterns,

and find that typically at least some of the hidden layer units are category selective.

The number of category selective ("localist") units varies depending on the size of the hidden layer, the structure of the outputs the network is trained to return (i.e., one-hot vs distributed), the neurons' activation functions, and the level of dropout-induced noise in the training procedure.

Overall, I find the work to hint at an interesting phenomenon.

However, the paper as presented uses an overly-simplistic task for the ANNs,

and the work is sloppily presented.

These factors detract from my enthusiasm.

My specific criticisms are as follows: 1) The binary pattern classification seems overly simplistic a task for this study.

If you want to compare to the medial temporal lobe's Jennifer Aniston cells (i.e., the Quiroga result), then an object recognition task seems much more meaningful, as does a deeper network structure.

Likewise, to inform the representations we see in deep object recognition networks, it is better to just study those networks, instead of simple shallow binary classification networks.

Or, at least show that the findings apply to those richer settings, where the networks do "real" tasks.

2) The paper is somewhat sloppy, and could use a thorough proofreading.

For example, what are "figures 3, ?? and 6"?

And which is Figure 3.3.1?

3) What formula is used to quantify the selectivity?

And do the results depend on the cut-off used to label units as "selective" or not (i.e., using a higher or lower cutoff than 0.05)?

Given that the 0.05 number is somewhat arbitrary, this seems worth checking.

4) I don't think that very many people would argue that the presence of distributed representations strictly excludes the possibility of some of the units having some category selectivity.

Consequently, I find the abstract and introduction to be a bit off-putting, coming off almost as a rant against PDP.

This is a minor stylistic thing, but I'd encourage the authors to tone it down a bit.

5) The finding that more of the selective units arise in the hidden layer in the presence of higher levels of noise is interesting,

and the authors provide some nice intuition for this phenomenon (i.e., getting redundant local representations makes the system robust to the dropout).

This seems interesting in light of the Quiroga findings of Jennifer Aniston cells: the fact that the (small number of) units they happened to record from showed such selectivity suggests that many neurons in the brain would have this selectivity, so there must be a large number of category selective units.

Does that finding, coupled with the result from Fig. 6, imply that those "grandmother cell" observations might reflect an adaptation to increase robustness to noise?

fact fact fact fact evaluation fact evaluation evaluation evaluation request request request evaluation request request request request request evaluation evaluation request evaluation fact evaluation non-arg


The authors present a new RL algorithm for sparse reward tasks.

The work is fairly novel in its approach,

combining a learned reward estimator with a contextual bandit algorithm for exploration/exploitation.

The paper was mostly clear in its exposition,

however some additional information of the motivation for why the said reduction is better than simpler alternatives would help.

\n\nPros\n1. The results on bandit structured prediction problems are pretty good\n

2. The idea of a learnt credit assignment function, and using that to separate credit assignment from the exploration/exploitation tradeoff is good.

\n\nCons: \n1. The method seems fairly more complicated than PPO / A2C,

yet those methods seem to perform equally well on the RL problems (Figure 2.).

It also seems to be designed only for discrete action spaces.\n

2. Reslope Boltzmann performs much worse than Reslope Bootstrap,

thus having a bag of policies helps.

However, in the comparison in Figures 2 and 3, the policy gradient methods dont have the advantage of using a bag of policies.

A fairer comparison would be to compare with methods that use ensembles of Q-functions.

(like this https://arxiv.org/abs/1706.01502 by Chen et al.).

The Q learning methods in general would also have better sample efficiency than the policy gradient methods.\n

3. The method claims to learn an internal representation of a denser reward function for the sparse reward problem,

however the experimental analysis of this is pretty limited (Section 5.3).

It would be useful to do a more thorough investigation of whether it learnt a good credit assignment function in the games.

One way to do this would be to check the qualitative aspects of the function in a well understood game, like Blackjack.\n\n

Suggestions:\n1. What is the advantage of the method over a simple RL method that predicts a reward at every step (such that the dense rewards add up to match the sparse reward for the episode), and uses this predicted dense reward to perform RL?

This, and also a bigger discussion on prior bandit learning methods like LOLS will help under the context for why we\u2019re performing the reduction stated in the paper.

\n\nSignificance: While the method is novel and interesting, the experimental analysis and the explanations in the paper leave it unclear as to whether its significant compared to prior work.

fact evaluation fact evaluation request evaluation evaluation evaluation fact fact fact fact fact request reference fact fact evaluation request request non-arg request evaluation