Neal (1994) showed that a one hidden layer Bayesian neural network, under certain conditions, converges to a Gaussian process as the number of hidden units approaches infinity.

Neal (1994) and Williams (1997) derive the resulting kernel functions for such Gaussian processes when the neural networks have certain transfer functions.

Similarly, the authors show an analogous result for deep neural networks with multiple hidden layers and an infinite number of hidden units per layer, and show the form of the resulting kernel functions.

For certain transfer functions, the authors perform a numerical integration to compute the resulting kernels.

They perform experiments on MNIST and CIFAR-10, doing classification by scaled regression.

Overall, the work is an interesting read, and a nice follow-up to Neal’s earlier observations about 1 hidden layer neural networks.

It combines several insights into a nice narrative about infinite Bayesian deep networks.

However, the practical utility, significance, and novelty of this work -- in its current form -- are questionable,

and the related work sections, analysis, and experiments should be significantly extended.

In detail:(1) This paper misses some obvious connections and references, such as

* Krauth et. al (2017): “Exploring the capabilities and limitations of Gaussian process models” for recursive kernels with GPs.

* Hazzan & Jakkola (2015): “Steps Toward Deep Kernel Methods from Infinite Neural Networks” for GPs corresponding to NNs with more than one hidden layer.

* The growing body of work on deep kernel learning, which “combines the inductive biases and representation learning abilities of deep neural networks with the non-parametric flexibility of Gaussian processes”.

E.g.: (i) “Deep Kernel Learning” (AISTATS 2016);

(ii) “Stochastic Variational Deep Kernel Learning” (NIPS 2016);

(iii) “Learning Scalable Deep Kernels with Recurrent Structure” (JMLR 2017).

These works should be discussed in the text.

(2) Moreover, as the authors rightly point out, covariance functions of the form used in (4) have already been proposed.

It seems the novelty here is mainly the empirical exploration (will return to this later), and numerical integration for various activation functions.

That is perfectly fine

-- and this work is still valuable.

However, the statement “recently, kernel functions for multi-layer random neural networks have been developed, but only outside of a Bayesian framework” is incorrect.

For example, Hazzan & Jakkola (2015) in “Steps Toward Deep Kernel Methods from Infinite Neural Networks” consider GP constructions with more than one hidden layer.

Thus the novelty of this aspect of the paper is overstated.

See also comment [*] later on the presentation.

In any case, the derivation for computing the covariance function (4) of a multi-layer network is a very simple reapplication of the procedure in Neal (1994).

What is less trivial is estimating (4) for various activations,

and that seems to the major methodological contribution.

Also note that multidimensional CLT here is glossed over.

It’s actually really unclear whether the final limit will converge to a multidimensional Gaussian with that kernel without stronger conditions.

This derivation should be treated more thoroughly and carefully.

(3) Most importantly, in this derivation, we see that the kernels lose the interesting representations that come from depth in deep neural networks.

Indeed, Neal himself says that in the multi-output settings, all the outputs become uncorrelated.

Multi-layer representations are mostly interesting

because each layer shares hidden basis functions.

Here, the sharing is essentially meaningless,

because the variance of the weights in this derivation shrinks to zero.

In Neal’s case, the method was explored for single output regression,

where the fact that we lose this sharing of basis functions may not be so restrictive.

However, these assumptions are very constraining for multi-output classification and also interesting multi-output regressions.

[*]: Generally, in reading the abstract and introduction, we get the impression that this work somehow allows us to use really deep and infinitely wide neural networks as Gaussian processes, and even without the pain of training these networks.

“Deep neural networks without training deep networks”.

This is not an accurate portrayal.

The very title “Deep neural networks as Gaussian processes” is misleading,

since it’s not really the deep neural networks that we know and love.

In fact, you lose valuable structure when you take these limits,

and what you get is very different than a standard deep neural network.

In this sense, the presentation should be re-worked.

(4) Moreover, neural networks are mostly interesting because they learn the representation.

To do something similar with GPs, we would need to learn the kernel.

But here, essentially no kernel learning is happening.

The kernel is fixed.

(5) Given the above considerations, there is great importance in understanding the practical utility of the proposed approach through a detailed empirical evaluation.

In other words, how structured is this prior and does it really give us some of the interesting properties of deep neural networks, or is it mostly a cute mathematical trick?

Unfortunately, the empirical evaluation is very preliminary, and provides no reassurance that this approach will have any practical relevance:

(i) Directly performing regression on classification problems is very heuristic and unnecessary.

(ii) Given the loss of dependence between neurons in this approach, it makes sense to first explore this method on single output regression, where we will likely get the best idea of its useful properties and advantages.

(iii) The results on CIFAR10 are very poor.

We don’t need to see SOTA performance to get some useful insights in comparing for example parametric vs non-parametric,

but 40% more error than SOTA makes it very hard to say whether any of the observed patterns hold weight for more competitive architectural choices.

A few more minor comments:(i) How are you training a GP exactly on 50k training points?

Even storing a 50k x 50k matrix requires about 20GB of RAM.

Even with the best hardware, computing the marginal likelihood dozens of times to learn hyperparameters would be near impossible.

What are the runtimes?

(ii) "One benefit in using the GP is due to its Bayesian nature, so that predictions have uncertainty estimates (Equation (9)).”

The main benefit of the GP is not the uncertainty in the predictions, but the marginal likelihood which is useful for kernel learning.

fact fact fact fact fact evaluation evaluation evaluation request evaluation reference reference reference reference reference reference request fact evaluation evaluation evaluation fact fact evaluation non-arg evaluation evaluation evaluation evaluation evaluation request fact fact evaluation fact evaluation fact fact evaluation evaluation evaluation quote evaluation evaluation fact fact evaluation request evaluation evaluation fact fact evaluation request evaluation evaluation request evaluation evaluation evaluation evaluation fact fact request quote fact


This paper describes how to use a set of sentences in the source language mapped to another set of sentences in the target sentences instead of using single sentence to sentence samples.

The paper claims superior results using the described method.

Overall, there are a few problems with the paper.

1) The arguments for using clusters instead of single sentences are questionable.

The paper claims several times that MLE training for NMT faces over-training (or data sparsity)

-- while that can be true depending on the corpus and model used, there are well-known remedies for that, for example regularization via dropout (almost everybody uses that).

It is not clear why that is not used or at least compared to the method presented.

2) The writing of the paper is often unclear (and sometimes grammatically wrong, typos etc. but that aside),

there are some made up words/concepts (What is 'Golden Centroid Augmentation" or "Model Centroid Augmentation"?

The reason for attention is not to better memorize input information,

it is to be able to attend to certain regions in the input.

The reason to use RL is to focus on optimizing directly for BLEU score or other metrics instead of likelihood

but not for improving on the train/test loss discrepancy.

There are lots more examples of unclear statements in this paper

-- it should be heavily improved.

3) Section 3 and 4 are very hard/impossible to understand,

it is not clear how the formulas help the reader to better understand the concept in any way.

5) The results presented in this paper given the complexity of the method are just not great

-- for example, WMT en-de is 21.3 BLEU reported by you while much older papers report for example 24.67 BLEU (Google's Neural Machine Translation System)

-- why not first try to get to state-of-the-art with already published methods and then try to improve on top of that? .

6) Finally, what is missing most is simply why a much simpler method (just generate some data using a trained system and use that as additional training data, with details on how much etc.) -- is not directly compared to this very complicated looking method.

fact fact evaluation evaluation fact fact evaluation evaluation fact fact fact fact fact evaluation request evaluation evaluation evaluation fact request request


This paper collects a cloze-style fill-in-the-missing-word dataset constructed manually by English teachers to test English proficiency.

Experiments are given which are claimed to show that  this dataset is difficult for machines relative to human performance.

The dataset seems interesting

but I find the empirical evaluations unconvincing.

The models used to evaluate machine difficulty are basic language models.

The problems are multiple choice with at most four choices per question.

This allows multiple choice reading comprehension architectures to be used.

A window of words around the blank could be used as the "question".

A simple reading comprehension baseline is to encode the question (a window around the blank) and use the question vector to compute an attention over the passage.

One can then compute a question-specific representation of the passage and score each candidate answer by the inner product of the question-specific sentence representation and the vector representation of the candidate answer.

See "A thorough examination of the CNN/Daily Mail reading comprehension task" by Chen, Bolton and Manning.

fact fact evaluation evaluation fact fact fact fact evaluation fact reference


This is an interesting paper that builds a parameterized network to select actions for a robot in a simulated environment, with the objective of quickly reaching an internal belief state that is predictive of the true state.

This is an interesting idea

and it works much better than I would have expected.

In more careful examination it is clear that the authors have done a good job of designing a network that is partly pre-specified and partly free, in a way that makes the learning effective.

In particular- the transition model is known and fixed (in the way it is used in the belief update process)

- the belief state representation is known and fixed (in the way it is used to decide whether the agent should be rewarded)

- the reward function is known and fixed (as above)

- the mechanics of belief update

But we learn - the observation model

- the control policy

I'm not sure that global localization is still an open problem with known models.

Or, at least, it's not one of our worst.

Early work by Cassandra, Kurien, et al used POMDP models and solvers for active localization with known transition and observation models.

It was computationally slow but effective.

Similarly, although the online speed of your learned method is much better than for active Markov localization, the offline training cost is dramatically higher;

it's important to remember to be clear on this point.

It is not obvious to me that it is sensible to take the cosine similarity between the feature representation of the observation and the feature representation of the state to get the entry in the likelihood map.

It would be good to make it clear this is the right measure.

How is exploration done during the RL phase?

These domains are still not huge.

Please explain in more detail what the memory images are doing.

In general, the experiments seem to be well designed and well carried out, with several interesting extensions.

I have one more major concern:  it is not the job of a localizer to arrive at a belief state with high probability mass on the true state

---it is the job of a localizer to have an accurate approximation of the true posterior under the prior and observations.

There are situations (in which, for example, the robot has gotten an unusual string of observations) in which it is correct for the robot to have more probability mass on a "wrong" state.

Or, it seems that this model may earn rewards for learning to make its beliefs overconfident.

It would be very interesting to see if you could find an objective that would actually cause the model to learn to compute the appropriate posterior.

In the end, I have trouble making a recommendation:

Con:  I'm not convinced that an end-to-end approach to this problem is the best one

Pro: It's actually a nice idea that seems to have worked out well

Con: I remain concerned that the objective is not the right one

My rating would really be something like 6.5 if that were possible.

fact evaluation evaluation evaluation fact fact fact fact fact fact fact fact fact fact evaluation evaluation evaluation request request evaluation request evaluation fact fact fact fact request evaluation evaluation evaluation evaluation non-arg


SUMMARY The paper considers the problem of using cycle GANs to decipher text encrypted with historical ciphers.

Also it presents some theory to address the problem that discriminating between the discrete data and continuous prediction is too simple.

The model proposed is a variant of the cycle GAN in which in addition embeddings helping the Generator are learned for all the values of the discrete variables.

The log loss of the GAN is replaced by a quadratic loss and a regularization of the Jacobian of the discriminator.

Experiments show that the method is very effective.

REVIEW The paper considers an interesting and fairly original problem

and the overall discussion of ciphers is quite nice.

Unfortunately, my understanding is that the theory proposed in section 2 does not correspond to the scheme used in the experiments

(contrarily to what the conclusion suggest and contrarily to what the discussion of the end of section 3, which says that using embedding is assumed to have an equivalent effect to using the methodology considered in the theoretical part).

Another important concern is with the proof: there seems to be an unmotivated additional assumption that appears in the middle of the proof of Proposition 1

+ some steps need to be clarified (see comment 16 below).

The experiments do not have any simple baseline, which is somewhat unfortunate.

DETAILED COMMENTS: 1- The paper makes a few bold and debatable statements:

line 9 of section 1 "Such hand-crafted features have fallen out of favor (Goodfellow et al., 2016) as a result of their demonstrated inferiority to features learned directly from data in end-to-end learning frameworks such as neural networks"

This is certainly an overstatement

and although it might be true for specific types of inputs it is not universally true,

most deep architectures rely on a human-in-the-loop

and there are number of areas where human crafted feature are arguably still relevant, if only to specify what is the input of a deep network:

there are many domains where the notion of raw data does not make sense, and, when it does, it is usually associated with a sensing device that has been designed by a human and which implicitly imposes what the data is based on human expertise.

2- In the last paragraph of the introduction, the paper says that previous work has only worked on vocabularies of 26 characters while the current paper tackles word level ciphers with 200 words.

But, isn't this just a matter of scalability and only possible with very large amounts of text?

Is it really because of an intrinsic limitation or lack of scalability of previous approaches or just because the authors of the corresponding papers did not care to present larger scale experiments?

3- The discussion at the top of page 5 is difficult to follow.

What do you mean when you say "this motivates the benefits of having strong curvature globally, as opposed to linearly between etc"

Which curvature are we talking about?

and what how does the "as opposed to linearly" mean?

Should we understand "as opposed to having curvature linearly interpolated between etc" or "as opposed to having a linear function"?

Please clarify.

4- In the same paragraph: what does "a region that has not seen the Jacobian norm applied to it" mean?

How is a norm applied to a region?

I guess that what you mean is that the generator G might creates samples in a part of the space where the function F has not yet been learned and is essentially close to 0.

Is this what you mean?

5- I do not understand why the paper introduces WGAN

since in the end it does not use them but uses a quadratic loss, introduced in the first display of section 4.3.

6- The paper makes a theoretical contribution which supports replacing the sample y by a sample drawn from a region around y.

But it seems that this is not used in the experiment

and that the authors consider that the introduction of the embedding is a substitution for this.

Indeed, in the last paragraph of section 3.1, the paper says "we make the assumption that the training of the embedding vectors approximates random sampling similar to what is described in Proposition 1".

This does not make any sense to me

because the embedding vectors map each y deterministically to a single point,

and so the distribution on the corresponding vectors is still a fixed discrete distribution.

This gives me this impression that the proposed theory does not match what is used in the experiments.

(The last sentence of section 3.1, which is commenting on this and could perhaps clarify the situation is ill formed with two verbs.)

7- In the definitions: "A discriminator is said to perform uninformative discrimination" etc.

-> It seems that the choice of the word uninformative would be misleading:

an uninformative discrimination would be a discrimination that completely fails, while what the condition is saying it that it cannot perform perfect discrimination.

I would thus suggest to call this "imperfect discrimination".

8- It seems that the same embedding is used in X space and in Y space (from equations 6 and 7).

Is there any reason for that?

I would seem more natural to me to introduce two different embeddings

since the objects are a priori different...

Actually I don't understand how the embeddings can be the same in the Vignere code case

since time taken into account one one side.

9- On the 5th line after equation (7), the paper says "the embeddings... are trained to minimize L_GAN and L_cyc, meaning... and are easy to discriminate"

-> This last part of the sentence seems wrong to me.

The discriminator is trying to maximize L_GAN

and so minimizing w.r.t. to the embedding is precisely trying to prevent to the discriminator to tell apart too easily the true elements from the estimated ones.

In fact the regularization of the Jacobian that will be preventing the discriminator to vary too quickly in space is more likely to explain the fact that the discrimination is not too easy to do between the true and mapped embeddings.

This might be connected to the discussion at the top of page 5.

Since there are no experiments with alpha different than the default value = 10,

this is difficult to assess.

10-The Vigenere cipher is explained again at the end of section 4.2 when it has already been presented in section 1.1

11- Concerning results in Table 2: I do not see why it would not be possible to compare the performance of the method with classical frequency analysis, at least for the character case.

12- At the beginning of section 4.3, the text says that the log loss was replaced with the quadratic loss, but without giving any reason.

Could you explain why.

13- The only comparison of results with and without embeddings is presented in the curves of figure 3, for Brown-W with a vocabulary of 200 words.

In that case it helps.

Could the authors report systematically results about all cases?

(I guess this might however be the only hard case...)

14- It would be useful to have a brief reminder of the architecture of the neural network

(right now the reader is just refered to Zhu et al., 2017):

how many layers, how many convolution layers etc.

The same comment applies for the way the position of the letter/word in the text appear is in encoded in a feature that is provided as input to the neural network: it would be nice if the paper could provide a few details here and be more self contained.

(The fact that the engineering of the time feature can "dramatically" improve the performance of the network should be an argument to convince the authors that hand-crafted feature have not fallen out of favor completely yet...)

15- I disagree with the statement made in the conclusion that the proposed work "empirically confirms [...] that the use of continuous relaxation of discrete variable facilitates [...] and prevents [...]"

because for me the proposed implementation does not use at all the theoretical idea of continuous relaxation proposed in the paper, unless there is a major point that I am missing.

16- I have two issues with the proof in the appendix

a) after the first display of the last page the paper makes an additional assumption which is not announced in the statement of the theorem, which is that two specific inequality hold...

Unless I am mistaken this assumption is never proven (later or earlier).

Given that this inequality is just "the right inequality to get the proof go through"

and given that there are no explanation for why this assumption is reasonable, to me this invalidates the proof.

The step of going from G(S_y) to S_(G(y)) seems delicate...

b) If we accept these inequalities, the determinant of the Jacobian (the notation is not defined) of F at (x_bar) disappears from the equations, as if it could be assumed to be greater than one.

If this is indeed the case, please provide a justification of this step.

17- A way to address the issue of trivial discrimination in GANs with discrete data has been proposed in Luc, P., Couprie, C., Chintala, S., & Verbeek, J. (2016). Semantic segmentation using adversarial networks. arXiv preprint arXiv:1611.08408.

The authors should probably reference this paper.

18- Clarification of the Jacobian regularization: in equation (3), the Jacobian computed seems to be w.r.t D composed with F

while in equation (8) it is only the Jacobian of D.

Which equation is the correct one?

TYPOS: Proposition 1: the if-then statement is broken into two sentences separated by a full point and a carriage return.

sec. 4.3 line 10 we use a cycle loss *with a regularization coefficient* lambda=1 (a piece of the sentence is missing)

sec. 4.3 lines 12-13 the learning rates given are the same at startup and after "warming up"...

In the appendix: 3rd line of proof of prop 1: I don' understand "countably infinite finite sequences of vectors lying in the vertices of the simplex"

-> what is countable infinite here?

The vertices?

fact fact fact fact evaluation evaluation evaluation evaluation fact evaluation request evaluation evaluation quote evaluation evaluation fact evaluation evaluation fact fact evaluation evaluation request request request request request request request evaluation request evaluation fact fact fact fact quote evaluation fact fact evaluation evaluation quote evaluation fact request fact request evaluation fact evaluation fact quote evaluation fact fact fact evaluation fact evaluation fact request fact request fact evaluation request evaluation request fact request request evaluation evaluation evaluation evaluation fact fact evaluation evaluation evaluation fact request fact request fact fact request fact fact fact evaluation request request


This paper proposes an importance-weighted estimator of the MMD, in order to estimate the MMD between distributions based on samples biased according to a known scheme.

It then discusses how to estimate the scheme when it is unknown, and further proposes using it in either the MMD-based generative models of Y. Li et al. (2015) / Dziugaite et al. (2015), or in the MMD GAN of C.-L. Li et al. (2017).

The estimator itself is natural (and relatively obvious),

though it has some drawbacks that aren't fully discussed (below).

The application to GAN-type learning is reasonable, and topical.

The first, univariate, experiment shows that the scheme is at least plausible.

But the second experiment, involving a simple T ratio based on whether an MNIST digit is a 0 or a 1, doesn't even really work!

(The best model only gets the underrepresented class from 20% up to less than 40%, rather than the desired 50%, and the "more realistic" setting only to 33%.)

It would be helpful to debug whether this is due to the classifier being incorrect, estimator inaccuracies, or what.

In particular, I would try using T based on a pretrained convnet independent of the autoencoder representation in the MMD GAN, to help diagnose where the failure mode comes from.

Without at least a working should-be-easy example like this, and with the rest of the paper's technical contribution so small, I just don't think this paper is ready for ICLR.

It's also worth noting that the equivalent algorithm for either vanilla GANs or Wasserstein GANs would be equally obvious.

Estimator: In the discussion about (2): where does the 1/m bias come from?

This doesn't seem to be in Robert and Casella section 3.3.2, which is the part of the book that I assume you're referring to

(incidentally, you should specify that rather than just citing a 600-page textbook).

Moreover, it is worth noting that Robert and Cassela emphasize that if E[1 / \tilde T] is infinite, the importance sampling estimator can be quite bad (for example, the estimator may have infinite variance).

This happens when \tilde T puts mass in a neighborhood around 0, i.e. when the thinned distribution doesn't have support at any place that P does.

In the biased-observations case, this is in some sense unsurprising:

if we don't see *any* data in a particular class of inputs, then our estimates can be quite bad

(since we know nothing about a group of inputs that might strongly affect the results).

In the modulating case, the equivalent situation is when F(x) lacks a mean,

which seems less likely.

Thus although this is probably not a huge problem for your case,

it's worth at least mentioning.

(See also the following relevant blog posts:

and https://xianblog.wordpress.com/2012/03/12/is-vs-self-normalised-is/ .)

The paper might be improved by stating (and proving) a theorem with expressions for the rate of convergence of the estimator, and how they depend on T.

Minor: Another piece of somewhat-related work is Xiong and Schneider, Learning from Point Sets with Observational Bias, UAI 2014.

Sutherland et al. 2016 and 2017, often referenced in the same block of citations, are the same paper.

On page 3, above (1): "Since we have projected the distributons into an infinite-dimensional space, the distance between the two distributions is zero if and only if all their moments are the same."

An infinite-dimensional space isn't enough;

the kernel must further be characteristic, as you mention.

See e.g. Sriperumbuder et al. (AISTATS 2010) for more details.

Figure 1(b) seems to be plotting only the first term of \tilde T, without the + 0.5.

fact fact evaluation evaluation evaluation evaluation evaluation fact request request evaluation evaluation request fact request evaluation fact evaluation evaluation fact fact evaluation evaluation request evaluation reference reference request evaluation fact quote fact fact reference evaluation


This paper discusses using neural networks for super-resolution.

The positive aspects of this work is that the use of two neural networks in tandem for this task may be interesting,

and the authors attempt to discuss the network's behavior by drawing relations to successful sparsity-based super-resolution.

Unfortunately I cannot see any novelty in the relationship the authors draw to LASSO style super-resolution and dictionary learning beyond what is already in the literature (see references below), including in one reference that the authors cite.

In addition, there are a number of sloppy mistakes (e.g. Equation 10 as a clear copy-paste error) in the manuscript.

Given that much of the main result seems to already be known,

I feel that this work is not novel enough at this time.

Some other minor points for the authors to consider for future iterations of this work: - The authors mention the computational burden of solving L1-regularized optimizations.

A lat of work has been done to create fast, efficient solvers in many settings (e.g. homotopy, message passing etc.).

Are these methods still insufficient in some applications?

If so, which applications of interest are the authors considering?

- In figure 1, it seems that under "superresolution problem": 'f' should be 'High res data' and 'g' should be 'Low res data' instead of what is there.

I'm also not sure how this figure adds to the information already in the text.

- In the results, the authors mention how some network features represented by certain neurons resemble the training data.

This seems like over-training and not a good quality for generalization.

The authors should clarify if, and why, this might be a good thing for their application.

- Overall a heavy editing pass is needed to fix a number of typos throughout.

References: [1] K. Gregor and Y. LeCun , “Learning fast approximations of sparse coding,” in Proc. Int. Conf. Mach. Learn., 2010, pp. 399–406.

[2] P. Sprechmann, P. Bronstein, and G. Sapiro, “Learning efficient structured sparse models,” in Proc. Int. Conf. Mach. Learn., 2012, pp. 615–622.

[3] M. Borgerding, P. Schniter, and S. Rangan, ``AMP-Inspired Deep Networks for Sparse Linear Inverse Problems [pdf] [arxiv]," IEEE Transactions on Signal Processing, vol. 65, no. 16, pp. 4293-4308, Aug. 2017.

[4] V. Papyan*, Y. Romano* and M. Elad, Convolutional Neural Networks Analyzed via Convolutional Sparse Coding, accepted to Journal of Machine Learning Research, 2016.

fact evaluation fact evaluation evaluation evaluation evaluation fact evaluation request request request evaluation fact evaluation request request reference reference reference reference


This paper discusses a text-to-speech system which is based on a convolutional attentive seq2seq architecture.

It covers experiments on a few datasets, testing the model's ability to handle increasing numbers of speakers.

By and large, this is a "system" paper

- it mostly describes the successful application of many different existing ideas to an important problem (with some exceptions, e.g. the novel method of enforcing monotonic alignments during inference).

In this type of paper, I typically am most interested in hearing about *why* a particular design choice was made, what alternatives were tried, and how different ideas worked.

This paper is lacking in this regard

- I frequently was left looking for more insight into the particular system that was designed.

Beyond that, I think more detailed description of the system would be necessary in order to reimplement it suitably

(another important potential takeaway for a "system" paper).

Separately, I the thousands-of-speakers results are just not that impressive

- a MOS of 2 is not really useable in the real-world.

For that reason, I think it's a bit disingenuous to sell this system as "2000-Speaker Neural Text-to-Speech".

For the above reasons, I'm giving the paper a "marginally above" rating.

If the authors provide improved insight, discussion of system specifics, and experiments, I'd be open to raising my review.

Below, I give some specific questions and suggestions that could be addressed in future drafts.

- It might be worth giving a sentence or two defining the TTS problem

- the paper is written assuming background knowledge about the problem setting, including different possible input sources, what a vocoder is, etc.

The ICLR community at large may not have this domain-specific knowledge.

- Why "softsign" and not tanh?

Seems like an unusual choice.

- What do the "c" and "2c" in Figure 2a denote?

- Why scale (h_k + h_e) by \sqrt{0.5} when computing the attention value vectors?

- "An L1 loss is computed using the output spectrograms"

I assume you mean the predicted and target spectrograms are compared via an L1 loss.

Why L1?

- In Vaswani et al., it was shown that a learned positional encoding worked about as well as the sinusoidal position encodings despite being potentially more flexible/less "hand-designed" for machine translation.

Did you also try this for TTS?

Any insight?

- Some questions about monotonic attention: Did you use the training-time "soft" monotonic attention algorithm from Raffel et al. during training and inference, or did you use the "hard" monotonic attention at inference time?

IIUC the "soft" algorithm doesn't actually force strict monotonicity.

You wrote "monotonic attention results in the model frequently mumbling words",

can you provide evidence/examples of this?

Why do you think this happens?

The monotonic attention approach seems more principled than post-hoc limiting softmax attention to be monotonic,

why do you think it didn't work as well?

- I can't find an actual reference to what you mean by a "wavenet vocoder".

The original wavenet paper describes an autoregressive model for waveform generation.

In order to use it as a vocoder, you'd have to do conditioning in some way.

How?

What was the structure of the wavenet you used?

Why?

These details appear to be missing.

All you write is the sentence (which seems to end without a period)

"In the WaveNet vocoder, we use mel-scale spectrograms from the decoder to condition a Wavenet, which was trained separated".

- Can you provide examples of the mispronunciations etc. which were measured for Table 1?

Was the evaluation of each attention mechanism done blindly?

- The 2.07 MOS figure produced for tacotron seems extremely low,

and seems to indicate that something went wrong or that insufficient care was taken to report this baseline.

How did you adapt tacotron (which as I understand is a single-speaker model) to the multi-speaker setting?

- Table 3 begs the question of whether Deep Voice 3 can outperform Deep Voice 2 when using a wavenet vocoder on VCTK (or improve upon the poor 2.09 MOS score reported).

Why wasn't this experiment run?

- The paragraph and appendix about deploying at scale is interesting and impressive,

but seems a bit out of place

- it probably makes more sense to include this information in a separate "systems" paper.

fact fact evaluation evaluation evaluation evaluation evaluation request evaluation evaluation evaluation evaluation evaluation request non-arg request evaluation evaluation request evaluation request request quote evaluation request fact request request request fact quote request request evaluation request evaluation fact fact request request request fact fact quote request request evaluation evaluation request request request evaluation evaluation request


This work proposes DIVE (distributional inclusion vector embeddings), an unsupervised method for hypernymy discovery that preserves the inclusion property of old fashioned sparse BOW representations (i.e., that hypernyms tend to have higher counts in more varies contexts than hyponyms).

The learned representations are evaluated on a large set of datasets and are shown to work well.

The work is very thorough, and almost reads as a systematic study in places.

Unfortunately, this makes it hard to follow sometimes,

and although the work is generally interesting,

I am left to wonder about its novelty and general impact on the field.

The main proposition essentially comes down to a slight tweak in word2vec's SGNS method:

instead of shifting the PMI value of word co-occurrences by 1/k, we shift by #w/(k*|D|/|V|) and add a non-negativity constraint, i.e., we explicitly force Eq. 2 to be true in perfect reconstructions.

This makes sense and I am surprised nobody has done it before.

It feels like a minor tweak,

however, and any major improvement seems to be largely dependent on the scoring function, rather than on this slightly different objective.

Consequently, I feel like the paper is in two minds about what it wants to be: a systematic study of hypernymy detection methods, or an introduction of a novel algorithm for learning distributional inclusion embeddings.

I like the main ideas of this work and I think it's great that the study is so thorough,

but I don't think it should be accepted in its current form.

Main concerns:- Presentation: the experiments show that (1) DIVE outperforms GE, HyperScore and H-Feature baselines;

(2) which scoring function works best for DIVE;

(3) DIVE outperforms SBOW in many cases, but not always, though better on average;

and (4) we can use DIVE for WSD (only shown qualitatively).

The paper spans 13 pages, excluding appendices,

which is rather long.

I feel that (2), (3) and (4) are part of a systematic study/review paper,

while (1) could be interesting in and of itself, if it included a full comparison against alternative methods.

The way results are presented in the tables is confusing,

and it's unclear why only one baseline is included in each case.

- Comparison: it appears that there are methods missing from the results tables.

On HyperLex, for example, results as high as 0.512 (Poincare embeddings), 0.540 (HyperVec) and 0.686 (LEAR) have been reported,

while the highest result in the paper is 34.5.

That's a big difference.

On Weeds' BLESS, it's 68.6 versus e.g. 0.75 for Kiela et al. (unsupervised, using images) and 0.850 for HyperVec.

If these results were omitted due to space, I think experiment (2) and (4) can safely be moved to the appendix.

Especially for something that hinges on being a review paper, such as this work, it is important to be complete.

In short, I think the presentation doesn't help;

I am left to wonder what the main contribution is of the work;

and I think the comparison to previous work is incomplete.

Questions:- Why use WaCkypedia?

It's old and, by now, small.

It would be interesting to try all scoring functions and types of model (PPMI, PPMI with shift, PPMI with inclusion shift) trained on the exact same corpus and same negatives, and showing that it works best there.

- General question: Is Vendrov's test set only yes/no, e.g. if I set the threshold really low, I get 100% accuracy, or does it contain negatives?

Same for BLESS.

If so how valuable is this evaluation?

- To what extent are the baselines tuned in the semi-supervised case?

If the numbers are from papers, that should be mentioned.

It would be better to use the same corpus, and the same amount of attention to tuning the results, for both cases.

Minor/Typos:- Medicl

- "From the recent review (Santus et al) ... suggested by the review study (Vulic et al)" this way of citing reviews isn't very pretty

fact evaluation evaluation evaluation evaluation evaluation evaluation fact evaluation evaluation evaluation evaluation evaluation evaluation fact request fact fact fact evaluation evaluation evaluation evaluation evaluation fact fact fact evaluation fact evaluation request evaluation evaluation evaluation non-arg evaluation request non-arg non-arg evaluation request request request request evaluation


This paper introduces an algorithm for learning Wasserstein GANs for discrete distributions.

Taking the dual form of the discrete Wasserstein distance introduced by [Evans 1997] (which produces a constrained optimization problem) and using this as a basis of a GAN training algorithm.

The key algorithmic distinction from conventional GAN approaches is that the critic takes pairs of real and simulated datapoints as inputs and returns a measure of which it thinks is the real datapoint, namely f(x_r)–f(g(z)) where x_r is the data point and g(z) the generator output, rather than the critic corresponding to f directly.

An architecture is proposed for this framework that guarantees that the constraints from the formulation of [Evans 1997] as satisfied.

The topic of this paper is timely and of clear interest to the ICLR community.

The underlying idea is interesting

and the architecture seems appropriate for the chosen target.

Further, the paper is relatively clear and easy to follow.

However, the experimental evaluation is not sufficiently strenuous and produces very underwhelming results.

Relatedly, I think the motivation behind using the discrete Wasserstein distance, though seemingly reasonable, needs more careful consideration.

Unfortunately, the serious shortfalls in the experiments mean that the paper, in my opinion, falls noticeably below the acceptance threshold for ICLR.

Having said that, my stance might change substantially if more impressive experimental results can be obtained;

without this, I am unconvinced the method actually behaves as intended.

Regretfully, this is probably beyond the scope of a revision during the rebuttal period

though as it will most likely require significant algorithmic changes.

%%%% Shortcomings with Experiments %%%%% Put simply, I do not think that the experiments demonstrate that the approach works and actually suggest the opposite.

The so-called “mode collapse” issue is effectively a sugar-coated way of saying that the method has learned to return a single output rather than learning a generative model.

This is a huge issue and needs sorting before the paper can be seriously considered for publication.

The attempted at a fix at the end of the paper might be a step in the right direction but is not evaluated sufficiently

and the preliminary results provided are not particularly promising.

Going past this issue, there are still a lot of problems with the experimental evaluation.

More numerous and more difficult problems need to be considered.

For example, an NLP problem would fit well with the motivation for the approach given in the intro.

It is also necessary to demonstrate that the GAN is doing something different to just memorizing previous examples.

For example, this is a problem where one can actually reasonably compare to just sampling from the data by checking that more unseen correct samples are generated then incorrect samples.

It is worrying that the convergence plots show a single line that ostensibly comes from the “best result”.

This is not a reasonable scientific comparison method

and should be replaced by mean (or median) performance with uncertainty estimates (i.e. +/- some numbers of standard deviations or quantiles).

The small number of samples from the MNIST problem really doesn’t convey anything meaningful

– even if this looked exceptional, a small number of unqualified samples is hardly a serious evaluation metric.

More iterations for Figure 4a or needed to see if the WGAN keeps improving beyond on the DWGAN, noting that this has not converged at 100%.

All four methods should be added to Figure 5.

%%%% Is the Discrete Metric Always Better for Training the Generative Model? %%%%%

Though I think it is probably true that the discrete metric should more beneficial from the point of the view of the critic,

I am not completely convinced it is always beneficial from the point of view of training the generator, which is at the end of the day what really matters.

Note that I am not trying to argue that discrete metric is worse,

just that you haven’t done enough to convince me that it is always better.

If indeed it is always better, please tear apart my following argument,

which will hopefully provide stimulation for improving the motivation of the metric in the paper.

If it is not always better, the paper should be updated to outline some of the potential pathologies and the cases were you expect the approach to work well and when you do not.

To demonstrate my argument, consider the training sample [0, 0, 1; 0, 1, 0] and the following four example generated outputs (1) [0.333, 0.333, 0.334; 0.333, 0.334, 0.333] (2) [0, 0, 1; 0, 1, 0] (3) [0, 0, 1; 0.334, 0.333, 0.333] (4) [0.333, 0.334, 0.333; 0.334, 0.333, 0.333] giving respective discrete distances to of 0, 0, 1, and 2; and continuous distances of 1.330668, 0, 0.667334, and 1.334668.

Now imagine we are training a GAN with the one example datapoint [0, 0, 1; 0, 1, 0].

Even though (1) will lead to the target sample [0, 0, 1; 0, 1, 0] after passing through the argmax function and (3) will not,

(1) is also very close to (4) which has the maximum possible discrete distance.

Consequently, the generator for (1) is most likely not at a stable optimum and very close in the space of neural net weights to some very poor generators.

During training, we would thus like to guide our network towards generating (2),

as this is a stable solution, particularly given we are using stochastic optimization methods.

From this perspective, then (3) is perhaps a better generated output than (1), a fact conveyed by the continuous metric but not the discrete metric.

In other words, even though (1) is arguably a better final solution than (3), from the perspective of effective training, it may be favorable to use a target function that prefers (3) to (1) to better guide the training to a stable solution.

Once we consider the fact that our aim is not to replicate a single datapoint but learn a generator that in some way interpolates between datapoints,

it becomes even less clear if the discrete metric is better.

For example, small changes to the input z for (1) are likely to lead to generating samples that are very different to anything seen in the training data (which is, in this case, a single point).

Though I do not think this argument undermines the suggested approach,

I do think it highlights why the supplied motivation for the approach is insufficient and needs more explicitly linking back to the training procedure.

At the very least, I think the above argument shows why it is not immediately clear cut that the suggested approach will perform better

and so needs backing up with strong empirical evidence,

which unfortunately the paper does not currently have, and/or a more convincing argument for why the discrete metric is better.

%%%% Other points %%%% - Tables 1, 2, and 3 are not worth spending more than a few lines on, let alone nearly a page.

They should be substantially compressed or preferably just cut (particularly Tables 2 and 3).

The point they convey is obvious and actually somewhat tangential to the key questions.

- It worries me that the method learns an h that explicitly takes y as input, not x.

This is a notational issue in the exposition, but also, more importantly, raises questions about whether the original linear programming problem is actually being solved

because the inputs from the generative method are not discrete variables.

- It should be made clear that the Appendices are directly following the derivation of Evans 1997, rather than being a new derivation.

- At times the paper is little sloppy at making distinguishing between y ∈ [0, 1] and y ∈ {0, 1}.

This makes it a bit confusing at times what is output from the generator, particularly when you talk about it being one-hot encoded.

As I understand it, the output is always the former,

while data lives in the latter

but this is not always clear.

I think you should also make a bigger deal of this in terms of the problem with previous methods that ignore that the critic might be able to distinguish based on the fact that the training and generated data points have an explicitly different type (discrete and continuous respectively).

- On the fourth line of the abstract, both instances of GAN should be GANs.

- What do you mean “explained in the sequel”?

- Does k need to be the same for each variable?

%%%% References %%%% Lawrence C Evans. Partial differential equations and Monge-Kantorovich mass transfer. Current developments in mathematics, 1997(1):65–126, 1997.

fact fact fact fact evaluation evaluation evaluation evaluation evaluation request evaluation evaluation evaluation evaluation evaluation evaluation evaluation request evaluation evaluation evaluation request evaluation request evaluation evaluation evaluation request evaluation evaluation request request request evaluation evaluation evaluation evaluation request evaluation request request request fact evaluation evaluation evaluation fact fact evaluation fact evaluation evaluation evaluation evaluation evaluation request evaluation request request evaluation evaluation evaluation fact request evaluation evaluation fact fact evaluation request request request request reference


Motivated via Talor approximation of the Residual network on a local minima, this paper proposed a warp operator that can replace a block of a consecutive number of residual layers.

While having the same number of parameters as the original residual network, the new operator has the property that the computation can be parallelized.

As demonstrated in the paper, this improves the training time with multi-GPU parallelization, while maintaining similar performance on CIFAR-10 and CIFAR-100.

One thing that is currently not very clear to me is about the rotational symmetry.

The paper mentioned rotated filters,

but continue to talk about the rotation in the sense of an orthogonal matrix applying to the weight matrix of a convolution layer.

The rotation of the filters (as 2D images or images with depth) seem to be quite different from "rotating" a general N-dim vectors in an abstract Euclidean space.

It would be helpful to make the description here more explicit and clear.

fact fact fact evaluation fact fact evaluation request


The authors provide a novel, interesting, and simple algorithm capable of training with limited memory.

The algorithm is well-motivated and clearly explained,

and empirical evidence suggests that the algorithm works well.

However, the paper needs additional examination in how the algorithm can deal with larger data inputs and outputs.

Second, the relationship to existing work needs to be explained better.

Pro: The algorithm is clearly explained, well-motivated, and empirically supported.

Con: The relationship to stochastic gradient markov chain monte carlo needs to be explained better.

In particular, the update form was first introduced in [1],

the annealing scheme was analyzed in [2],

and the reflection step was introduced in [3].

These relationships need to be explained clearly.

The evidence is presented on very small input data.

With something like natural images, the parameterization is much larger and with more data, the number of total parameters is much larger.

Is there any evidence that the proposed algorithm could continue performing comparatively as the total number of parameters in state-of-the-art networks increases?

This would require a smaller ratio of included parameters.

[1] Welling, M. and Teh, Y.W., 2011. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11)(pp. 681-688).

[2] Chen, C., Carlson, D., Gan, Z., Li, C. and Carin, L., 2016, May. Bridging the gap between stochastic gradient MCMC and stochastic optimization. In Artificial Intelligence and Statistics(pp. 1051-1060).

[3] Patterson, S. and Teh, Y.W., 2013. Stochastic gradient Riemannian Langevin dynamics on the probability simplex. In Advances in Neural Information Processing Systems (pp. 3102-3110).

evaluation evaluation evaluation request request evaluation request fact fact fact request evaluation evaluation request evaluation reference reference reference


This paper proposes a method to generate adversarial examples for text classification problems.

They do this by iteratively replacing words in a sentence with words that are close in its embedding space and which cause a change in the predicted class of the text.

To preserve correct grammar, they only change words that don't significantly change the probability of the sentence under a language model.

The approach seems incremental and very similar to existing work such as Papernot et. al.

The paper also states in the discussion in section 5.1 that they generate adversarial examples in state-of-the-art models,

however, they ignore some state of the art models entirely such as Miyato et. al.

The experiments are solely missing comparisons to existing text adversarial generation approaches such as Papernot et. al and a comparison to adversarial training for text classification in Miyato et. al which might already mitigate this attack.

The experimental section also fails to describe what kind of language model is used,

(what kind of trigram LM is used?

A traditional (non-neural) LM?

Does it use backoff?).

Finally, algorithm 1 does not seem to enforce the semantic constraints in Eq. 4 despite it being mentioned in the text.

This can be seen in section 4.5 where the algorithm is described as choosing words that were far in word vector space.

The last sentence in section 6 is also unfounded.

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z.Berkay Celik, and Ananthram Swami Practical Black-Box Attacks against Machine Learning. Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security

Takeru Miyato, Andrew M. Dai and Ian Goodfellow Adversarial Training Methods for Semi-Supervised Text Classification. International Conference on Learning Representation (ICLR), 2017

fact fact fact evaluation fact fact fact fact request request request fact fact fact reference reference


The authors present an end to end training of a CNN architecture that combines CT image signal processing and image analysis.

This is an interesting paper.

Time will tell whether a disease specific signal processing will be the future of medical image analysis,

but - to the best of my knowledge - this is one of the first attempts to do this in CT image analysis,

a field that is of significance both to researchers dealing with image reconstruction (denoising, etc.) and image analysis (lesion detection).

As such I would be positive about the topic of the paper and the overall innovation it promises both in image acquisition and image processing,

although I would share the technical concerns pointed out by Reviewer2,

and the authors would need good answers to them before this study would be ready to be presented.

fact evaluation evaluation fact evaluation evaluation evaluation evaluation


The authors present confidence-based autodidactic returns, a Deep learning RL method to adjust the weights of an eligibility vector in TD(lambda)-like value estimation to favour more stable estimates of the state.

The key to being able to learn these confidence values is to not allow the error of the confidence estimates propagate back though the deep learning architecture.

However, the method by which these confidence estimates are refined could be better described.

The authors describe these confidences variously as: "some notion of confidence that the agent has in the value function estimate" and "weighing the returns based on a notion of confidence has been explored earlier (White & White, 2016; Thomas et al., 2015)".

But the exact method is difficult to piece together from what is written.

I believe that the confidence estimates are considered to be part of the critic

and the w vector to be part of the theta_c parameters.

This would then be captured by the critic gradient for the CAR method that appears towards the end of page 5.

If so, this should be stated explicitly.

There is another theoretical point that could be clearer.

The variation in an autodidactic update of a value function (Equation (4)) depends on a few things,

the in variation future value function estimates themselves being just one factor.

Another two sources of variation are: the uncertainty over how likely each path is to be taken, and the uncertainty in immediate rewards accumulated as part of some n-step return.

In my opinion, the quality of the paper would be much improved by a brief discussion of this,

and some reflection on what aspects of these variation contribute to the confidence vectors and what isn't captured.

Nonetheless, I believe that the paper represents an interesting and worthy submission to the conference.

I would strongly urge the authors to improve the method description in the camera read version though.

A few additional comments are as follows:  • The plot in Figure 3 is the leading collection of results to demonstrate the dominance of the authors' adaptive weight approach (CAR) over the A3C (TD(0) estimates) and LRA3C (truncated TD(lambda) estimates) approaches.

However, the way the results are presented/plotted, namely the linear plot of the (shifted) relative performance of CAR (and LRA3C) versus A3C, visually inflates the importance of tasks on which CAR (and LRA3C) perform better than A3C, and diminishes the importance of those tasks on which A3C performs better.

It would be better kept as a relative value and plotted on a log-scale so that positive and negative improvements can be viewed on an equal setting.

• On page 3, when Gt is first mentioned, Gt should really be described first, before the reader is told what it is often replaced with.

• On page 3, where delta_t is defined (the j step return TD error, I think the middle term should be $gamma^j V(S_{t+j})$

• On page 4 and 5, when describing the gradient for the actor and critic, it would be better if these were given their own terminology, but if not, then use of the word respectively in each case would help.

fact evaluation request fact evaluation evaluation evaluation fact request evaluation fact fact fact request request evaluation request evaluation evaluation request request request request


Thank you for your contribution to ICLR.

The paper covers a very interesting topic and presents some though-provoking ideas.

The paper introduces "covariant compositional networks" with the purpose of learning graph representations.

An example application also covered in the experimental section is graph classification.

Given a finite set S, a compositional network is simply a partially ordered set P where each element of P is a subset of S and where P contains all sets of cardinality 1 and the set S itself.

Unfortunately, the presentation of the approach is extremely verbose and introduces old concepts (e.g., partially ordered set) under new names.

The basic idea (which is not new) of this work is that we need to impose some sort of hierarchical order on the nodes of the graph so as to learn hierarchical feature representations.

Moreover, the hierarchical order of the nodes should be invariant to valid permutations of the nodes, that is, two isomorphic graphs should have the same hierarchical order on their nodes and the same feature representations.

Since this is the case for graph embedding methods that collect feature representations from their neighbors in the graph (and where the feature aggregation functions are symmetric)

it makes sense that "compositional networks" generalize graph convolutional networks (and the more general message passing neural networks framework).

The most challenging problem, however, namely the problem of finding a concrete and suitable permutation invariant hierarchical decomposition of the nodes plus some aggregation/pooling functions to compute the feature representations is not addressed in sufficient detail.

The paper spends a lot of time on some theoretical definitions and (trivial) proofs

but then fails to make the connection to an approach that works in practice.

The description of the experiments and which compositional network is chosen and how it is chosen seems to be missing.

The only part hinting at the model that was actually used in the experiments is the second paragraph of the section 'Experimental Setup', consisting of one long sentence that is incomprehensible to me.

Instead of spending a lot of effort on the definitions and (somewhat trivial) propositions in the first half of the paper, the authors should spend much more time on detailing the experiments and the actual model that they used.

In an effort to make the framework as general as possible, you ended up making the paper highly verbose and difficult to follow.

Please address the following points or clarify in your rebuttal if I misunderstood something:- what precisely is the novel contribution of your work

(it cannot be "compositional networks" and the propositions concerning those

because these are just old concepts under new names)?

- explain precisely (and/or more directly/less convoluted) how your model used in the experiments looks like;

why do you think it is better than the other methods?

- given that compositional network is a very general concept (partially ordered set imposed on subsets of the graph vertices),

what is the principled set of steps one has to follow to arrive at such a compositional network tailored to a particular graph collection?

isn't (or shouldn't) that be the contribution of this work?

Am I missing something?

In general, you should write the paper much more to the point and leave out unnecessary math (or move to an appendix).

The paper is currently highly inaccessible.

non-arg evaluation fact fact fact evaluation fact fact fact evaluation evaluation evaluation fact fact evaluation request evaluation request fact evaluation request request fact request fact non-arg request evaluation


re. Introduction, page 2: Briefly explain here how SAB is different from regular Attention?

Good paper.

There's not that much discussion of the proposed SAB compared to regular Attention,

perhaps that could be expanded.

Also, I suggest summarizing the experimental findings in the Conclusion.

request evaluation evaluation request request


The paper ‘Deep learning for Physical Process: incorporating prior physical knowledge’ proposes to question the use of data-intensive strategies such as deep learning in solving physical inverse problems that are traditionally solved through assimilation strategies.

They notably show how physical priors on a given phenomenon can be incorporated in the learning process and propose an application on the problem of estimating sea surface temperature directly from a given collection of satellite images.

All in all the paper is very clear and interesting.

The results obtained on the considered problem are clearly of great interest, especially when compared to state-of-the-art assimilation strategies such as the one of Béréziat.

While the learning architecture is not original in itself,

it is shown that a proper physical regularization greatly improves the performance.

For these reasons I believe the paper has sufficient merits to be published at ICLR.

That being said, I believe that some discussions could strengthen the paper:

- Most classical variational assimilation schemes are stochastic in nature, notably by incorporating uncertainties in the observation or physical evolution models.

It is still unclear how those uncertainties can be integrated in the model;

- Assimilation methods are usually independent of the type of data at hand.

It is not clear how the model learnt on one particular type of data transpose to other data sequences.

Notably, the question of transfer and generalization is of high relevance here.

Does the learnt model performs well on other dataset (for instance, acquired on a different region or at a distant time).

I believe this type of issue has to be examinated for this type of approach to be widely use in inverse physical problems.

fact fact evaluation evaluation evaluation evaluation evaluation evaluation fact evaluation fact evaluation evaluation request request


This paper studies how the variance of the discriminator affect the gradient signal provided to the generator and therefore how it might limit its ability to learn the true data distribution.

The approach suggested in this paper models the output of the discriminator using a mixture of two Gaussians (one for “fake” and the other for “not fake”).

This seems like a rather crude approximation as the distribution of each “class” is likely to be multimodal.

Can the authors comment on this?

Could they extend their approach to use a mixture of multimodal distributions?

The paper mentions that fixing the means of the distribution can be “problematic during optimization as the discriminator’s goal is to maximize the difference between these two means.“.

This relates to my previous comment where the distribution might not be unimodal.

In this case, shifting the mean doesn’t seem to be a good solution and might just yield to oscillations between different modes.

Can you please comment on this?

Mode collapse: Can you comment on the behavior of your approach w.r.t. to mode collapse?

Implementation details: How is the mean of the two Gaussians initialized?

Relation to instance noise and regularization techniques: Instance noise is a common trick being used to train GANs,

see e.g. http://www.inference.vc/instance-noise-a-trick-for-stabilising-gan-training/

This also relates to some regularization techniques, e.g. Roth et al., 2017 that provides a regularizer that amounts to convolving the densities with white Gaussian noise.

Can you please elaborate on the potential advantages of the proposed solution over these existing techniques?

Comparison to existing baselines: Given that the paper addresses the stability problem, I would expect some empirical comparison to at least one or two of the stability methods cited in the introduction, e.g. Gulrajani et al., 2017 or Roth et al., 2017.

Relation to Kernel MMD: Can the authors elaborate on how their method relates to approaches that replace the discriminator with MMD nets. e.g.

- Training generative neural networks via Maximum Mean Discrepancy optimization, Dziugaite et al

- Generative models and model criticism via optimized maximum mean discrepancy, Sutherland et al

More explicitly, the variance in these methods can be controlled via the bandwidth of the kernel

and I therefore wonder what would one use a simple mixture of Gaussians instead?

fact fact evaluation non-arg request fact fact evaluation non-arg request request fact reference evaluation request request request reference reference fact evaluation


The claimed results of  "combining transformations" in the context of RC was done in the works of Herbert Jaeger on conceptors [1],

which also should be cited here.

The argument of biological plausibility is not justified.

The authors  use an echo-state neural network with standard tanh activations, which is as far away from real neuronal signal processing than  ordinary RNNs used in the field, with the difference that the recurrent weights are not trained.

If the authors want to make the case of biological plausibility, they should use spiking neural networks.

The experiment on MNIST seems artificial, in particular transforming the image into a time-series and thereby imposing an artificial temporal structure.

The assumption that column_i is obtained  by information  of column_{i-k},..,column_{i-1} is not true for images.

To make a point, the authors should use a datasets with related sets of time-series data, e.g EEG or NLP data.

In total this paper does not have enough novelty for acceptance

and the experiments are not well chosen for this kind of work.

Also the authors overstate the claim of biological plausibility

(just because we don't train the recurrent weights does not make a method biologically plausible).

[1] H. Jaeger (2014): Controlling Recurrent Neural Networks by Conceptors. Jacobs University technical report Nr 31 (195 pages)

fact request evaluation fact request evaluation fact request evaluation evaluation evaluation evaluation reference


Let me first note that I am not very familiar with the literature on program generation, molecule design or compiler theory,

which this paper draws heavily from,

so my review is an educated guess.

This paper proposes to include additional constraints into a VAE which generates discrete sequences, namely constraints enforcing both semantic and syntactic validity.

This is an extension to the Grammar VAE of Kusner et. al, which includes syntactic constraints but not semantic ones.

These semantic constraints are formalized in the form of an attribute grammar, which is provided in addition to the context-free grammar.

The authors evaluate their methods on two tasks, program generation and molecule generation.

Their method makes use of additional prior knowledge of semantics,

which seems task-specific and limits the generality of their model.

They report that their method outperforms the Character VAE (CVAE) and Grammar VAE (GVAE) of Kusner et. al.

However, it isn't clear whether the comparison is appropriate:

the authors report in the appendix that they use the kekulised version of the Zinc dataset of Kusner et. al,

whereas Kusner et. al do not make any mention of this.

The baselines they compare against for CVAE and GVAE in Table 1 are taken directly from Kusner et. al though.

Can the authors clarify whether the different methods they compare in Table 1 are all run on the same dataset format?

Typos: - Page 5: "while in sampling procedure" -> "while in the sampling procedure"

- Page 6: "a deep convolution neural networks" -> "a deep convolutional neural network"

- Page 6: "KL-divergence that proposed in" -> "KL-divergence that was proposed in"

- Page 6: "since in training time" -> "since at training time"

- Page 6: "can effectively computed" -> "can effectively be computed"

- Page 7: "reset for training" -> "rest for training"

non-arg evaluation non-arg fact fact fact fact fact evaluation fact evaluation fact fact fact request request request request request request request


This paper investigates the effect of replacing identity skip connections with trainable convolutional skip connections in ResNet.

The authors find that in their experiments, performance improves.

Therefore, the power of skip connections is due to their linearity rather than due to the fact that they represent the identity.

Overall, the paper has a clear and simple message and is very readable.

The paper contains a good amount of experiments,

but in my opinion not quite enough to conclude that identity skip connections are inherently worse.

The question is then: how non-trivial is it that tandem networks work?

For someone who understands and has worked with ResNet and similar architectures, this is not a surprise.

Therefore, the paper is somewhat marginal but, I think, still worth accepting.

Why did you choose a single learning rate for all architectures and datasets instead of choosing the optimal one for each archtitecture and dataset?

Was it a question of computational resources?

Using custom step sizes would strenghten your experimental results significantly.

In the absence of this, I would still ask that you create an appendix where you specify exactly how hyperparameters were chosen.

Other comments:- "and that it’s easier for a layer to learn from a starting point of keeping things the same (the identity map) than from the zero map" I don't understand this comment.

Networks without skip connections are not initialized to the zero map but have nonzero, usually Gaussian, weights.

- in section 2, reason (ii), you seem to imply that it is a good thing if a network behaves as an ensemble of shallower networks.

In general, this is a bad thing.

Therefore, the fact that ResNet with tandom networks is an ensemble of shallower networks is a reason for why it might perform badly, not well.

I would suggest removing reason (ii).

- in section 3, reason (iii), you state that removing nonlinearities from the skip path can improve performance.

However, using tandom blocks instead of identity skip connections does not change the number of nonlinearity layers.

Therefore, I do not see how reason (iii) applies to tandem networks.

- "The best blocks in each challenge were competitive with the best published results for their numbers of parameters; see Table 2 for the breakdown."

What are the best published results?

I do not see them in table 2.

fact fact fact evaluation evaluation evaluation evaluation evaluation evaluation non-arg non-arg request request evaluation fact fact evaluation fact request fact fact evaluation quote request fact


This paper presents a very thorough empirical exploration of the qualities and limitations of very simple word-embedding based models.

Average and/or max pooling over word embeddings (which are initialized from pretrained embeddings) is used to obtain a fixed-length representation for natural language sequences, which is then fed through a single layer MLP classifier.

In many of the 9 evaluation tasks, this approach is found to match or outperform single-layer CNNs or RNNs.

The varied findings are very clearly presented and helpfully summarized,

and for each task setting the authors perform an insightful analysis.

My only criticism would be the fact that the study is limited to English,

even though the conclusions are explicitly scoped in light of this.

Moreover, I wonder how well the findings would hold in a setting with a more severe OOV problem than is perhaps present in the studied datasets.

Besides concluding from the presented results that these SWEMs should be considered a strong baseline in future work,

one might also conclude that we need more challenging datasets!

Minor things:- It wasn't entirely clear how the text matching tasks are encoded.

Are the two sequences combined into a single sequence before applying the model, or something else?

I might have missed this detail.

- Given the two ways of using the Glove embeddings for initialization (direct update vs mapping them with an MLP into the task space), it would be helpful to know which one ended up being used (i.e. optimal) in each setting.

- Something went wrong with the font size for the remainder of the text near Figure 1.

fact fact fact evaluation evaluation evaluation fact request evaluation evaluation evaluation request evaluation request fact


This paper proposes an optimization problem whose Lagrangian duals contain many existing objective functions for generative models.

Using this framework, the paper tries to generalize the optimization problems by defining computationally-tractable family which can be expressed in terms of existing objective functions.

The paper has interesting elements

and the results are original.

The main issue is that the significance is unclear.

The writing in Section 3 is unclear for me,

which further made it challenging to understand the consequences of the theorems presented in that section.

Here is a big-picture question that I would like to know answer for.

Do the results of sec 3 help us identify a more useful/computationally tractable model than exiting approaches?

Clarification on this will help me evaluate the significance of the paper.

I have three main clarification points.

First, what is the importance of T1, T2, and T3 classes defined in Def. 7, i.e., why are these classes useful in solving some problems?

Second, is the opposite relationship in Theorem 1, 2, and 3 true as well, e.g., is every linear combination of beta-ELBO and VMI is equivalent to a likelihood-based computable-objective of KL info-encoding family?

Is the same true for other theorems?

Third, the objective of section 3 is to show that "only some choices of lambda lead to a dual with a tractable equivalent form".

Could you rewrite the theorems so that they truly reflect this, rather than stating something which only indirectly imply the main claim of the paper.

Some small comments:- Eq. 4. It might help to define MI to remind readers.

- After Eq. 7, please add a proof (may be in the Appendix).

It is not that straightforward to see this.

Also, I suppose you are saying Eq. 3 but with f from Eq. 4.

- Line after Eq. 8, D_i is "one" of the following...

Is it always the same D_i for all i or it could be different?

Make this more clear to avoid confusion.

- Last line in Para after Eq. 15, "This neutrality corresponds to the observations made in.."

It might be useful to add a line explaining that particular "observation"

- Def. 7, the names did not make much sense to me.

You can add a line explaining why this name is chosen.

- Def. 8, the last equation is unclear.

Does the first equivalence impy the next one?

- Writing in Sec. 3.3 can be improved.

e.g., "all linear operations on log prob." is very unclear,

"stated computational constraints" which constraints?

fact fact evaluation evaluation evaluation evaluation evaluation non-arg request evaluation non-arg request request request fact request request request evaluation evaluation fact request request quote request evaluation request evaluation request request evaluation request


Overall: I really enjoyed reading this paper and think the question is super important.

I have some reservations about the execution of the experiments as well as some of the conclusions drawn.

For this reason I am currently a weak reject

(weak because I believe the question is very interesting).

However, I believe that many of my criticisms can be assuaged during the rebuttal period.

Paper Summary: For RL to play video games, it has to play many many many many times.

In fact, many more times than a human

where prior knowledge lets us learn quite fast in new (but related) environments.

The authors study, using experiments, what aspects of human priors are the important parts.

The authors’ Main Claim appears to be: “While common wisdom might suggest that prior knowledge about game semantics such as ladders are to be climbed, jumping on spikes is dangerous or the agent must fetch the key before reaching the door are crucial to human performance, we find that instead more general and high-level priors such as the world is composed of objects, object like entities are used as subgoals for exploration, and things that look the same, act the same are more critical.”

Overall, I find this interesting.

However, I am not completely convinced by some of the experimental demonstrations.

Issue 0: The experiments seem underpowered / not that well analyzed.

There are only 30 participants per condition

and so it’s hard to tell whether the large differences in conditions are due to noise and what a stable ranking of conditions actually looks like.

I would recommend that the authors triple the sample size and be more clear about reporting the outcomes in each of the conditions.

It’s not clear what the error bars in figure 1 represent,

are they standard deviations of the mean?

Are they standard deviations of the data?

Are they confidence intervals for the mean effect?

Did you collect any extra data about participants?

One potentially helpful example is asking how familiar participants are with platformer video games.

This would give at least some proxy to study the importance of priors about “how video games are generally constructed” rather than priors like “objects are special”.

Issue 1: What do you mean by “objects”?

The authors interpret the fact that performance falls so much between conditions b and c to mean that human priors about “objects are special” are very important.

However, an alternative explanation is that people explore things which look “different” (ie. Orange when everything else is black).

The problem here comes from an unclear definition of what the authors mean by an “object”

so in revision I would like authors to clarify what precisely they mean by a prior about “the world is composed of objects” and how this particular experiment differentiates “object” from a more general prior about “video games have clearly defined goals, there are 4 clearly defined boxes here, let me try touching them.”

This is important

because a clear definition will give us an idea for how to actually build this prior into AI systems.

Issue 2: Are the results here really about “high level” priors?

There are two ways to interpret the authors’ main claim:

the strong version would maintain that semantic priors aren’t important at all.

There is no real evidence here for the strong version of the claim.

A real test would be to reverse some of the expected game semantics and see if people perform just as well as in the “masked semantics” condition.

For example, suppose we had exactly the same game and N different types of objects in various places of the game where N-1 of them caused death but 1 of them opened the door (but it wasn’t the object that looked like a key).

My hypothesis would be that performance would fall drastically as semantic priors would quickly lead people in that direction.

Thus, we could consider a weaker version of the claim: semantic priors are important but even in the absence of explicit semantic cues (note, this is different from having the wrong semantic cues as above) people can do a good job on the game.

This is much more supported by the data,

but still I think very particular to this situation.

Imagine a slight twist on the game: There is a sword (with a lock on it), a key, a slime and the door (and maybe some spikes).

The player must do things in exactly this order: first the player must get the key, then they must touch the sword, then they must kill the slime, then they go to the door.

Here without semantic priors I would hypothesize that human performance would fall quite far (whereas with semantics people would be able to figure it out quite well).

Thus, I think the authors’ claim needs to be qualified quite a bit.

It’s also important to take into account how much work general priors about video game playing (games have goals, up jumps, there is basic physics) are doing here

(the authors do this when they discuss versions of the game with different physics).

evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation fact fact evaluation evaluation evaluation evaluation evaluation request evaluation request request request request request evaluation request fact fact evaluation request evaluation evaluation request fact evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation request evaluation fact


This paper tackles the overfitting problem when training neural networks based on regularization technique.

More precisely, the authors propose new regularization terms that are related to the underlying virtual geometrical transformations (shift, rotation and scale) of the input data (signal, image and video).

By formalizing the geometrical transformation process of a given image, the authors deduce constraints on the objective function which depend on the magnitude of the applied transformation.

The proposed method is compared to three methods: one baseline and two methods of the literature (AT and VAT).

The comparison is done on three datasets (synthetic data, MNIST and CIFAR10) in terms of test errors (for classification problems) and running time.

The paper is well formalized

and the idea is interesting.

The regularization approach is novel compared to the methods of the literature.

Main concerns: 1)	The experimental validation of the proposed approach is not consistent:

The description of the baseline method is not detailed in the paper.

A priori, the baseline should naturally be the method without your regularization terms.

But, this seems to be contrary with what you displayed in Figure 3.

Indeed, in Figure 3, there is three different graphs for the baseline method (i.e., one for each regularization term).

It seems that the baseline method depends on the different kinds of regularization term,

why?

Same question for AT and VAT methods.

In practice, what is the magnitude of the perturbations?

Please, explain the axis of all the figures.

Please, explain how do you mix your different regularization terms in your method that you call VMT-all?

All the following points are related to the experiment for which you presented the results in Table 2:

Please, provide the results of all your methods on the synthetic dataset (only VMT-shift is provided).

What is VMF?

Do you mean VMT?

For the evaluations, it would be more rigorous to re-implement also the state-of-the-art methods for which you only give the results that they report in their paper.

Especially, because you re-implemented AT with L-2 constraint,

so, it seems straightforward to re-implement also AT with L-infinite constraint.

Same remark for the dropout regularization technique,

which is easy to re-implement on the dense layers of your neural networks, within the Tensorflow framework.

As you mentioned, your main contribution is related to running time,

thus, you should give the running time in all experiments.

2)	The method seems to be a tradeoff between accuracy and running time:

The VAT method performs better than all your methods in all the datasets.

The baseline method is faster than all the methods (Table 3).

This being said, the proposed method should be clearly presented in the paper as a tradeoff between accuracy and running time.

3)	The positioning of the proposed approach is not so clear:

As mentioned above, your method is a tradeoff between accuracy and running time.

But you also mentioned (top of page 2) that the contribution of your paper is also related to the interpretability in terms of ‘’Human perception’’.

Indeed, you clearly mentioned that the methods of the literature lacks interpretability.

You also mentioned that your method is more ‘’geometrically’’ interpretable than methods of the literature.

The link between interpretability in terms of “human perception” and “geometry” is not obvious.

Anyway, the interpretability point is not sufficiently demonstrated, or at least, discussed in the paper.

4)	Many typos in the paper :

Section 1: “farward-backward”

Section 2.1: “we define the movement field V of as a n+1…”

Section 2.2: “lable” - “the another” - “of how it are generated”

– Sentence “Since V is normalized.” seems incomplete…

- \mathcal{L} not defined

- Please, precise the simplifications like \mathcal{L}_{\theta} to \mathcal{L}

Section 3: “DISCUSSTION”

Section 4.1: “negtive”

Figure 2: “negetive”

Table 2: “VMF”

Section 4.2: “Tab 2.3” does not exist

Section 4.3: “consists 9 convolutional” – “nerual networks”…

Please, always use the \eqref latex command to refer to equations.

There is many others typos in the paper,

so, please proofread the paper…

fact fact fact fact fact evaluation evaluation fact evaluation evaluation evaluation fact fact fact request request request request request fact request request request request fact request request evaluation fact request evaluation fact fact request evaluation evaluation fact fact fact evaluation evaluation evaluation request request request request request request request request request request fact request request evaluation request


The authors tackle the problem of estimating risk in a survival analysis setting with competing risks.

They propose directly optimizing the time-dependent discrimination index using a siamese survival network.

Experiments on several real-world dataset reveal modest gains in comparison with the state of the art.

- The authors should clearly highlight what is their main technical contribution.

For example, Eqs. 1-6 appear to be background material

since the time-dependent discrimination index is taken from the literature, as the authors point out earlier.

However, this is unclear from the writing.

- One of the main motivations of the authors is to propose a model that is specially design to avoid the nonidentifiability issue in an scenario with competing risks.

It is unclear why the authors solution is able to solve such an issue, specially given the modest reported gains in comparison with several competitive baselines.

In other words, the authors oversell their own work, specially in comparison with the state of the art.

- The authors use off-the-shelf siamese networks for their settting

and thus it is questionable there is any novelty there.

The application/setting may be novel,

but not the architecture of choice.

- From Eq. 4 to Eq. 5, the authors argue that the denominator does not depend on the model parameters and can be ignored.

However, afterwards the objective does combine time-dependent discrimination indices of several competing risks, with different denominator values.

This could be problematic if the risks are unbalanced.

- The competitive gain of the authors method in comparison with other competing methods is minor.

- The authors introduce F(t, D | x) as cumulative incidence function (CDF) at the beginning of section 2,

however, afterwards they use R^m(t, x), which they define as risk of the subject experiencing event m before t.

Is the latter a proxy for the former?

How are they related?

fact fact evaluation request fact fact evaluation fact evaluation evaluation fact evaluation evaluation evaluation fact fact evaluation evaluation fact fact request request


This paper addresses multiple issues arising from the fact that commonly reported best model performance numbers are a single sample from a performance distribution.

These problems are very real,

and they deserve significant attention from the ML community.

However, I feel that the proposed solution may actually compound the issues highlighted.

Firstly, the proposed metric requires calculation of multiple test set experiments for every evaluation.

In the paper up to 100 experiments were used.

This may be reasonable in scenarios where the test set is hidden, and individual test numbers are never revealed.

It also may be reasonable if we cynically assume that researchers are already running many test-set evaluations.

But I am very opposed to any suggestion that we should relax the maxim that the test set should be used only once, or as close to once as is possible.

Even the idea of researchers knowing their test set variance makes me very uneasy.

Secondly, this paper tries to account for variation in results due to different degrees of hyper-parameter tuning.

This is certainly an admirable aim,

since different research groups have access to very different types of resources.

However, the suggested approach relies on randomly picking hyper-parameters from "a range that we previously found to work reasonably well".

This randomization does not account for the many experiments that were required to find this range.

And the randomization is also not extended to parameters controlling the model architecture

(I suspect that a number of experiments went into picking the 32 layers in the ResNet used by this paper).

Without a solid and consistent basis for these hyper-parameter perturbations, I worry that this approach will fail to normalize the effect of experiment numbers while also giving researchers an excuse to avoid reporting their experimental process.

I think this is a nice idea

and the metric does merge the stability and low variance of mean score with the aspirations of best score.

The metric may be very useful at development time in helping researchers build a reasonable expectation of test time performance in cases where the dev and test sets are strongly correlated.

However, for the reasons outlined above, I don't think the proposed approach solves the problems that it addresses.

Ultimately, the decision about this paper is a subjective one.

Are we willing to increase the risk of inadvertent hyper-parameter tuning on the test set for the sake of a more stable metric?

fact evaluation evaluation evaluation fact fact evaluation evaluation evaluation evaluation fact evaluation evaluation fact fact fact evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation


This paper misses the point of what VAEs (or GANs, in general) are used for.

The idea of using VAEs is not to encode and decode images (or in general any input), but to recover the generating process that created those images so we have an unlimited source of samples.

The use of these techniques for compressing is still unclear

and their quality today is too low.

So the attack that the authors are proposing does not make sense

and my take is that we should see significant changes before they can make sense.

But let’s assume that at some point they can be used as the authors propose.

In which one person encodes an image, send the latent variable to a friend, but a foe intercepts it on the way and tampers with it so the receiver recovers the wrong image without knowing.

Now if the sender believes the sample can be tampered with, if the sender codes z with his private key would not make the attack useless?

I think this will make the first attack useless.

The other two attacks require that the foe is inserted in the middle of the training of the VAE.

This is even less doable,

because the encoder and decoder are not train remotely.

They are train of the same machine or cluster in a controlled manner by the person that would use the system.

Once it is train it will give away the decoder and keep the encoder for sending information.

evaluation evaluation evaluation evaluation evaluation evaluation evaluation fact evaluation evaluation fact evaluation fact fact fact