Active learning for deep learning is an interesting topic

and there is few useful tool available in the literature.

It is happy to see such paper in the field.

This paper proposes a batch mode active learning algorithm for CNN as a core-set problem.

The authors provide an upper bound of the core-set loss, which is the gap between the training loss on the whole set and the core-set.

By minimizing this upper bound, the problem becomes a K-center problem which can be solved by using a greedy approximation method, 2-OPT.

The experiments are performed on image classification problem (CIFAR, CALTECH, SVHN datasets), under either supervised setting or weakly-supervised setting.

Results show that the proposed method outperforms the random sampling and uncertainty sampling by a large margin.

Moreover, the authors show that 2-OPT can save tractable amount of time in practice with a small accuracy drop.

The proposed algorithm is new

and writing is clear.

However, the paper is not flawless.

The proposed active learning framework is under ERM and cover-set, which are currently not supported by deep learning.

To validate such theoretical result, a non-deep-learning model should be adopted.

The ERM for active learning has been investigated in the literature, such as "Querying discriminative and representative samples for batch mode active learning" in KDD 2013, which also provided an upper bound loss of the batch mode active learning

and seems applicable for the problem in this paper.

Another interesting question is most of the competing algorithm is myoptic active learning algorithms.

The comparison is not fair enough.

The authors should provide more competing algorithms in batch mode active learning.

evaluation evaluation evaluation fact fact fact fact evaluation evaluation evaluation evaluation evaluation fact request fact evaluation evaluation evaluation request


The paper considers the problem of training neural networks in mixed precision (MP), using both 16-bit floating point (FP16) and 32-bit floating point (FP32).

The paper proposes three techniques for training networks in mixed precision: first, keep a master copy of network parameters in FP32; second, use loss scaling to ensure that gradients are representable using the limited range of FP16; third, compute dot products and reductions with FP32 accumulation.

Using these techniques allows the authors to match the results of traditional FP32 training on a wide variety of tasks without modifying any training hyperparameters.

The authors show results on ImageNet classification (with AlexNet, VGG, GoogLeNet, Inception-v1, Inception-v3, and ResNet-50), VOC object detection (with Faster R-CNN and Multibox SSD), speech recognition in English and Mandarin (with CNN+GRU), English to French machine translation (with multilayer LSTMs), language modeling on the 1 Billion Words dataset (with a bigLSTM), and generative adversarial networks on CelebFaces (with DCGAN).

Pros: - Three simple techniques to use for mixed-precision training

- Matches performance of traditional FP32 training without modifying any hyperparameters

- Very extensive experiments on a wide variety of tasks

Cons: - Experiments do not validate the necessity of FP32 accumulation

- No comparison of training time speedup from mixed precision

With new hardware (such as NVIDIA’s Volta architecture) providing large computational speedups for MP computation, I expect that MP training will become standard practice in deep learning in the near future.

Naively porting FP32 training recipes can fail due to the reduced numeric range of FP16 arithmetic;

however by adopting the techniques of this paper, practitioners will be able to migrate their existing FP32 training pipelines to MP without modifying any hyperparameters.

I expect these techniques to be hugely impactful as more people begin migrating to new MP hardware.

The experiments in this paper are very exhaustive, covering nearly every major application of deep learning.

Matching state-of-the-art results on so many tasks increases my confidence that I will be able to apply these techniques to my own tasks and architectures to achieve stable MP training.

My first concern with the paper is that there are no experiments to demonstrate the necessity of FP32 accumulation.

With an FP32 master copy of the weights and loss scaling, can all arithmetic be performed solely in FP16, or are there some tasks where training will still diverge?

My second concern is that there is no comparison of training-time speedup using MP.

The main reason that MP is interesting is because new hardware promises to accelerate it.

If people are willing to endure the extra engineering overhead of implementing the techniques from this paper, what kind of practical speedups can they expect to see from their workloads?

NVIDIA’s marketing material claims that the Tensor Cores in the V100 offer an 8x speedup over its general-purpose CUDA cores

Since in this paper some operations are performed in FP32 (weight updates, batch normalization) and other operations are bound by memory and not compute bandwidth,

what kinds of speedups do you see in practice when moving from FP32 to MP on V100?

My other concerns are minor.

Mandarin speech recognition results are reported on “our internal test set”.

Is there any previously published work on this dataset, or any publicly available test set for this task?

The notation around the Inception architectures should be clarified.

According to [3] and [4], “Inception-v1” and “GoogLeNet” both refer to the architecture used in [1].

The architecture used in [2] is referred to as “BN-Inception” by [3] and “Inception-v2” by [4].

“Inception-v3” is the architecture from [3], which is not currently cited.

To improve clarity in Table 1, I suggest renaming “GoogLeNet” to “Inception-v1”, changing “Inception-v1” to “Inception-v2”, and adding explicit citations to all rows of the table.

In Section 4.3 the authors note that “half-precision storage format may act as a regularizer during training”.

Though the effect is most obvious from the speech recognition experiments in Section 4.3,

MP also achieves slightly higher performance than baseline for all ImageNet models but Inception-v1 and for both object detection models;

these results add support to the idea of FP16 as a regularizer.

Minor typos: Section 3.3, Paragraph 3: “either FP16 or FP16 math” -> “either FP16 or FP32 math”

Section 4.1, Paragraph 4: “ pre-ativation” -> “pre-activation”

Overall this is a strong paper,

and I believe that it will be impactful as MP hardware becomes more widely used.

References [1] Szegedy et al, “Going Deeper with Convolutions”, CVPR 2015

[2] Ioffe and Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, ICML 2015

[3] Szegedy et al, “Rethinking the Inception Architecture for Computer Vision”, CVPR 2016

[4] Szegedy et al, “Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning”, ICLR 2016 Workshop

fact fact fact fact evaluation fact evaluation evaluation fact evaluation fact fact evaluation evaluation evaluation fact request fact evaluation request fact reference fact request evaluation fact request request fact fact fact request fact evaluation fact evaluation request request evaluation evaluation reference reference reference reference


This paper is based on the theory of group equivariant CNNs (G-CNNs), proposed by Cohen and Welling ICML'16.

Regular convolutions are translation-equivariant, meaning that if an image is translated, its convolution by any filter is also translated.

They are however not rotation-invariant for example.

G-CNN introduces G-convolutions, which are equivariant to a given transformation group G.

This paper proposes an efficient implementation of G-convolutions for 6-fold rotations (rotations of multiple of 60 degrees), using a hexagonal lattice.

The approach is evaluated on CIFAR-10 and AID, a dataset of aerial scene classification.

The approach outperforms G-convolutions implemented on a squared lattice, which allows only 4-fold rotations on AID by a short margin.

On CIFAR-10, the difference does not seem significative (according to Tables 1 and 2).

I guess this can be explained by the fact that rotation equivariance makes sense for aerial images, where the scene is mostly fronto-parallel, but less for CIFAR (especially in the upper layers), which exhibits 3D objects.

I like the general approach of explicitly putting desired equivariance in the convolutional networks.

Using a hexagonal lattice is elegant, even if it is not new in computer vision

(as written in the paper).

However, as the transformation group is limited to rotations,

this is interesting in practice mostly for fronto-parallel scenes, as the experiences seem to show.

It is not clear how the method can be extended to other groups than 2D rotations.

Moreover, I feel like the paper sometimes tries to mask the fact that the proposed method is limited to rotations.

It is admittedly clearly stated in the abstract and introduction, but much less in the rest of the paper.

The second paragraph of Section 5.1 is difficult to keep in a paper.

It says that "From a qualitative inspection of these hexagonal interpolations we conclude that no information is lost during the sampling procedure."

"No information is lost" is a strong statement from a qualitative inspection, especially of a hexagonal image.

This statement should probably be removed.

One way to evaluate the information lost could be to iterate interpolation between hexagonal and squared lattices to see if the image starts degrading at some point.

fact fact fact fact evaluation fact fact evaluation fact evaluation evaluation fact fact evaluation evaluation evaluation evaluation evaluation quote evaluation request request


This paper studies the adjustment of dropout rates

which is a useful tool to prevent the overfitting of deep neural networks.

The authors derive a generalization error bound in terms of dropout rates.

Based on this, the authors propose a regularization framework to adaptively select dropout rates.

Experimental results are also given to verify the theory.

Major comments: (1) The Empirical Rademacher complexity is not defined.

For completeness, it would be better to define it at least in the appendix.

(2) I can not follow the inequality (5).

Especially, according to the main text, f^L is a vector-valued function .

Therefore, it is not clear to me the meaning of \sum\sigma_if^L(x_i,w) in (5).

(3) I can also not see clearly the third equality in (9).

Note that f^l is a vector-valued function.

It is not clear to me how it is related to a summation over j there.

(4) There is a linear dependency on the number of classes in Theorem 3.1.

Is it possible to further improve this dependency?

Minor comments:(1) Section 4: 1e-3,1e-4,1e-5 is not consistent with 1e^{-3}, 1e^{-4},1e^{-5}

(2) Abstract: there should be a space before "Experiments".

(3) It would be better to give more details (e.g., page, section) in citing a book in the proof of Theorem 3.1

Summary: The mathematical analysis in the present version is not rigorous.

The authors should improve the mathematical analysis.

fact evaluation fact fact fact fact request evaluation fact evaluation evaluation fact evaluation fact non-arg fact request request evaluation request


This paper proposes another entropic regularization term for deep neural nets.

The key idea can be stated as follows: Let X denote the observed input, C the hidden class label taking values in a finite set, and Y the representation computed by a neural net.

Then C -> X -> Y is a Markov chain.

Moreover, assuming that the mapping X -> Y is deterministic (as is the case with neural nets or any other deterministic representations), we can write down the mutual information between X and Y as

I(X;Y) = H(Y) - H(Y|X) = H(Y).

A simple manipulation shows that H(Y) = I(C;Y) + H(Y|C).

The authors interpret the first term, I(C;Y), as a data fit term that quantifies the statistical correlations between the class label C and the representation Y,

whereas the second term, H(Y|C), is the amount by which the representation Y can be compressed knowing the class label C.

The authors then propose to 'explicitly decouple' the data-fit term I(C;Y) from the regularization penalty and focus on minimizing H(Y|C).

In fact, they replace this term by the sum of conditional entropies of the form H(Y_{i,k}|C), where Y_{i,k} is the activation of the ith neuron in the kth layer of the neural net.

The final step is to recognize that the conditional entropy may not admit a scalable and differentiable estimator,

so they use the relation between a quantity called entropy power and second moments to replace the entropic penalty with the conditional variance penalty Var[Y_{i,k}|C].

Since the class-conditional distributions are unknown,

a surrogate model Q_{Y|C} is used.

The authors present some experimental results as well.

However, this approach has a number of serious flaws.

First of all, if the distribution of X is nonatomic and the mapping X -> Y is continuous (in the case of neural nets, it is even Lipschitz), then the mutual information I(X;Y) is infinite.

In that case, the representation of I(X;Y) in terms of entropies is not valid

-- indeed, one can write the mutual information between two jointly distributed random variables X and Y in terms of differential entropies as I(X;Y) = h(Y) - h(Y|X),

but this is possible only if both terms on the right-hand side exist.

This is not the case here,

so, in particular, one cannot relate I(X;Y) to I(C;Y).

Ironically, I(C;Y) is finite,

because C takes values in a finite set,

so I(C;Y) is at most the log cardinality of the set of labels.

One can start, then, simply with I(C;Y) and express it as H(C) - H(C|Y).

Both terms are well-defined Shannon entropies, where the first one does not depend on the representation,

whereas the second one involves the representation.

But then, if the goal is to _minimize_ the mutual information between I(C;Y), it makes sense to _maximize_ the conditional entropy H(C|Y).

In short, the line of reasoning that leads to minimizing H(Y|C) is not convincing.

Moreover, why is it a good idea to _minimize_ I(C;Y) in the first place?

Shouldn't one aim to maximize it subject to structural constraints on the representation, along the lines of InfoMax?

The next issue is the chain of reasoning that leads to replacing H(Y|C) with Var[Y|C].

One could start with that instead without changing the essence of the approach, but then the magic words "Shannon decay" would have to disappear altogether, and the proposed method would lose all of its appeal.

fact fact fact fact fact fact fact fact fact fact fact fact fact fact fact evaluation fact fact fact fact fact evaluation fact fact fact fact fact fact evaluation evaluation evaluation evaluation evaluation evaluation


This paper is about modifications to the skip-thought framework for learning sentence embeddings.

The results show performance comparable to or better than skip-thought while decreasing training time.

I think the overall approach makes sense:

use an RNN encoder

because we know it works well,

but improve training efficiency by changing the decoder to a combination of feed-forward and convolutional layers.

I think it may be the case that this works well because the decoder is not auto-regressive but merely predicts each word independently.

This is possible because the decoder will not be used after training.

So all the words can be predicted all at once with a fixed maximum sentence length.

In typical encoder-decoder applications, the decoder is used at test time to get predictions,

so it is natural to make it auto-regressive.

But in this case, the decoder is thrown away after training,

so it makes more sense to make the decoder non-auto-regressive.

I think this point should be made in the paper.

Also, I think it's worth noting that an RNN decoder could be used in a non-auto-regressive architecture as well.

That is, the sentence encoding could be mapped to a sequence of length 30 as is done with the CNN decoder currently;

then a (multi-layer) BiLSTM could be run over that sequence,

and then a softmax classifier can be attached to each hidden vector to predict the word at that position.

It would be interesting to compare that BiLSTM decoder with the proposed CNN decoder and also to compare it to a skip-thought-style auto-regressive RNN decoder.

This would let us understand whether the benefit is coming more from the non-auto-regressive nature of the decoder or from the CNN vs RNN differences.

That is, it would make sense to factor the decision of decoder design along multiple axes.

One axis could be auto-regressive vs predict-all-words.

Another axis could be using a CNN over the sequence of word positions or an RNN over the sequence of word positions.

For auto-regressive models, another axis could be train using previous ground-truth word vs train using previous predicted word.

Skip-thought corresponds to an auto-regressive RNN (using the previous ground-truth word IIRC).

The proposed decoder is a predict-all-words CNN.

It would be natural to also experiment with an auto-regressive CNN and a predict-all-words RNN (like what I described in the paragraph above).

The paper is choosing a single point in the space and referring to it as a "CNN decoder"

whereas there are many possible architectures that can be described this way

and I think it would strengthen the paper to increase the precision in discussing the architecture and possible alternatives.

Overall, I think the architectural choices and results are strong enough to merit publication.

Adding any of the above empirical comparisons would further strengthen the paper.

However, I did have quibbles with some of the exposition and some of the claims made throughout the paper.

They are detailed below:Sec. 2:In the "Decoder" paragraph: please add more details about how the words are predicted.

Are there final softmax layers that provide distributions over output words?

I couldn't find this detail in the paper.

What loss is minimized during training?

Is it the sum of log losses over all words being predicted?

Sec. 3:Section 3 does not add much to the paper.

The motivations there are mostly suggestive rather than evidence-based.

Section 3 could be condensed by about 80% or so without losing much information.

Overall, the paper has more than 10 pages of content,

and the use of 2 extra pages beyond the desired submission length of 8 should be better justified.

I would recommend adding a few more details to Section 2 and removing most of Section 3.

I'll mention below some problematic passages in Section 3 that should be removed.

Sec. 3.2:"...this same constraint (if using RNN as the decoder) could be an inappropriate constraint in the decoding process."

What is the justification or evidence for this claim?

I think the claim should be supported by an argument or some evidence or else should be removed.

If the authors intend the subsequent paragraphs to justify the claim, then see my next comments.

Sec. 3.2:"The existence of the ground-truth current word embedding potentially decreases the tendency for the decoder to exploit other information from the sentence representation."

But this is not necessarily an inherent limitation of RNN decoders

since it could be addressed by using the embedding of the previously-predicted word rather than the ground-truth word.

This is a standard technique in sequence-to-sequence learning; cf. scheduled sampling (Bengio et al., 2015).

Sec. 3.2: "Although the word order information is implicitly encoded in the CNN decoder, it is not emphasized as it is explicitly in the RNN decoder. The CNN decoder cares about the quality of generated sequences globally instead of the quality of the next generated word. Relaxing the emphasis on the next word, may help the CNN decoder model to explore the contribution of context in a larger space."

Again, I don't see any evidence or justification for these arguments.

Also see my discussion above about decoder variations;

these are not properties of RNNs vs CNNs but rather properties of auto-regressive vs predict-all-words decoders.

Sec. 5.2-5.3:There are a few high-level decisions being tuned on the test sets for some of the tasks, e.g., the length of target sequences in Section 5.2 and the number of layers and channel size in Section 5.3.

Sec. 5.4:When trying to explain why an RNN encoder works better than a CNN encoder, the paper includes the following:

"We stated above that, in our belief, explicit usage of the word order information will augment the transferability of the encoder, and constrain the search space of the parameters in the encoder. The results match our belief."

I don't think these beliefs are concrete enough to be upheld or contradicted.

Both encoders explicitly use word order information.

Can you provide some formal or theoretical statement about how the two encoders treat word order differently?

I fear that it's only impressions and suppositions that lead to this difference, rather than necessarily something formal.

Sec. 5.4:In Table 1, it is unclear why the "future predictor" model is the one selected to be reported from Gan et al (2017).

Gan et al has many settings and the "future predictor" setting is the worst.

An explanation is needed for this choice.

Sec. 6.1: In the "BYTE m-LSTM" paragraph:"Our large RNN-CNN model trained on Amazon Book Review (the largest subset of Amazon Review) performs on par with BYTE m-LSTM model, and ours works better than theirs on semantic relatedness and entailment tasks."

I'm not sure this "on par" assessment is warranted by the results in Table 2.

BYTE m-LSTM is better on MR by 1.6 points and better on CR by 4.7 points.

The authors' method is better on SUBJ by 0.7 and better on MPQA by 0.5.

So on sentiment tasks, BYTE m-LSTM is clearly better,

and on the other tasks the RNN-CNN is typically better, especially on SICK-r.

More minor things are below:Sec. 1:The paper contains this: "The idea of learning from the context information was first successfully applied to vector representation learning for words in Mikolov et al. (2013b)"

I don't think this is accurate.

When restricting attention to neural network methods, it would be more correct to give credit to Collobert et al. (2011).

But moving beyond neural methods, there were decades of previous work in using context information (counts of context words) to produce vector representations of words.

typo: "which d reduces" --> "which reduces"

Sec. 2:The notation in the text doesn't match that in Figure 1: w_i^1 vs. w_1 and h_i^1 vs h_1.

Instead of writing "non-parametric composition function", describe it as "parameter-free".

"Non-parametric" means that the number of parameters grows with the data, not that there are no parameters.

In the "Representation" paragraph: how do you compute a max over vectors?

Is it a separate max for each dimension?

This is not clear from the notation used.

Sec. 3.1:inappropriate word choice: the use of "great" in "a great and efficient encoding model"

Sec. 3.2:inappropriate word choice: the use of "unveiled" in "is still to be unveiled"

Sec. 3.4:Tying input and output embeddings can be justified with a single sentence and the relevant citations (which are present here).

There is no need for speculation about what may be going on, e.g.: "the model learns to explore the non-linear compositionality of the input words and the uncertain contribution of the target words in the same space".

Sec. 4:I think STS14 should be defined and cited where the other tasks are described.

Sec. 5.3:typo in Figure 2 caption: "and and"

Sec. 6.1: In the "Skip-thought" paragraph:inappropriate word choice: "kindly"

The description that says "we cut off a branch for decoding" is not clear to me.

What is a "branch for decoding" in this context?

Please modify it to make it more clear.

References:Bengio S, Vinyals O, Jaitly N, Shazeer N. Scheduled sampling for sequence prediction with recurrent neural networks. NIPS 2015.

Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P. Natural language processing (almost) from scratch. Journal of Machine Learning Research 2011.

fact fact evaluation fact evaluation fact fact fact fact fact evaluation fact evaluation request fact fact fact fact request evaluation evaluation request request request request request request fact evaluation request evaluation request evaluation request request fact request request evaluation evaluation evaluation fact request request non-arg quote fact request non-arg quote evaluation fact evaluation quote fact non-arg fact fact fact quote evaluation fact request evaluation evaluation fact request quote evaluation fact fact fact fact fact evaluation request fact request fact request fact request request evaluation evaluation evaluation evaluation evaluation request fact evaluation evaluation request request reference reference


I quite liked the revival of the dual memory system ideas and the cognitive (neuro) science inspiration.

The paper is overall well written and tackles serious modern datasets, which was impressive,

even though it relies on a pre-trained, fixed ResNet (see point below).

My only complaint is that I felt I couldn’t understand why the model worked so well.

A better motivation for some of the modelling decisions would be helpful.

For instance, how much the existence (and training) of a BLA network really help

— which is a central new part of the paper, and wasn’t in my view well motivated.

It would be nice to compare with a simpler baseline, such as a HC classifier network with reject option.

I also don’t really understand why the proposed pseudorehearsal works so well.

Some formal reasoning, even if approximate, would be appreciated.

Some additional comments below: - Although the paper is in general well written,

it falls on the lengthy side

and I found it difficult at first to understand the flow of the algorithm.

I think it would be helpful to have a high-level pseudocode presentation of the main steps.

- It was somewhat buried in the details that the model actually starts with a fixed, advanced feature pre-processing stage (the ResNet, trained on a distinct dataset, as it should).

I’m fine with that,

but this should be discussed.

Note that there is evidence that the neuronal responses in areas as early as V1 change as monkeys learn to solve discrimination tasks.

It should be stressed that the model does not yet model end-to-end learning in the incremental setting.

- p. 4, Eq. 4, is it really necessary to add a loss for the intermediate layers, and not only for the input layer?

I think it would be clearer to define the \mathcal{L} explictily somewhere.

Also, shouldn’t the sum start at j=0?

evaluation evaluation fact evaluation request request evaluation request evaluation request evaluation evaluation evaluation request fact evaluation request fact request request request request


The authors present an estimator for the mutual information (MI) based on the Donsker-Varadhan representation for the KL divergence and its generalization to arbitrary f-divergences by Ruderman et al.

While that last work introduced an estimator based on optimization over the unit ball in an RKHS,

the current work propose to use a parametric function class given by a neural network

(I'd suggest that the authors make this point more explicit,

as currently it's not totally clear what their actual contribution is and how their work compares to the prior art they cite).

The authors show that such an estimator can be used to train models with less mode-dropping in adversarial models.

The work is quite straightforward,

but improves over similar work in the GAN space by Nowozin et al. by using Ruderman's tighter variational representation instead of Nguyen's one.

The paper contains many typos and grammatical errors

and the authors should do an exhaustive proof-reading.

More problematic is that, right after eq. 10, the authors mention "We show in the Appendix that OMIE has the desirable strong consistency and convergence properties".

However, the appendix doesn't contain such a proof.

Is it missing from the submitted version?

I don't think that such a consistency proof is strictly necessary for a paper like this,

but for the review to be accurate I need to see the proof.

Since I can't find it,

I assume it does not exist.

In that case, the authors should give less emphasis to the MI estimator itself and more to the empirical properties and applications.

The authors present some experiments comparing different estimators of MI applied to synthetic data.

Figure 1 is hard to read,

I suggest the authors try to come up with a more legible plot.

Figure 2 is also a bit surprising,

why show error for 50 dimensions but estimates for 2 dimensions?

Since these experiments are quick to run,

it would be helpful to get more information on how the gap between the methods change as the dimensionality increases (e.g. a surface plot with d and # of iterations on the x and y axes).

Also it would be highly beneficial to compare with the method in Ruderman at al.,

so that people interested in MI estimation but who don't plan on using the estimator as part of a neural net architecture can get some idea on how the inductive bias of NNs compare to RKHS.

In the caption to Fig. 3 the authors state "The OMIEGAN generator learns a distribution with a high amount of structured noise", which I find hard to understand.

Probably the authors can be a bit more precise than saying "structured noise".

I would recommend dropping the Information Bottleneck section to focus on showing more convincingly the impact of OMIE in GANs.

The experiments section currently looks rushed and lacking in depth.

In summary, this work provides value by introducing a (previously known) superior f-divergence variational representation to the GAN community.

The mode-collapse prevention via MI maximisation is also interesting and deserves more experimental attention to make the paper stronger.

fact fact fact request evaluation fact evaluation fact fact request evaluation fact non-arg evaluation evaluation evaluation evaluation request fact evaluation request evaluation request evaluation request request evaluation evaluation request request evaluation fact evaluation


The authors study the problem of distributed routing in a network, where the goal is to minimize the maximal load (i.e. the load of the link with the highest utilization).

The authors advocate to use multi-agent reinforcement learning.

The main idea put forward by the authors is that by designing artificial rewards (to guide the agents), one can achieve faster exploration, in order to reduce convergence time.

While the authors put forward several interesting ideas,

there are some shortcomings to the present version of the paper, including:

- The design objective seems flawed from the networking point of view:

while minimizing the maximal load of a link is certainly a good starting point (to avoid instable queues)

one typically wants to minimize delay (or maximize flow throughput).

Indeed, it is possible to have a larger maximal load while reducing delay in many cases.

- Furthermore, the authors do not provide a baseline to which the outcome of the learning algorithms they propose:

for instance how does their approach compare to simple policies (those are commonplace in networking) such as MaxWeight, Backpressure and so on ?

- The authors argue that using multi-agent learning is more desirable than single agent (i.e. with a single reward signal which is common to all agents).

However, is multi-agent guaranteed to converge in such a setting ?

If some versions of the problem (for some particular reward signal) are not guaranteed to converge, it is difficult to understand whether "convergence"  is slow due to an inefficient exploration, or simply because convergence cannot occur in the first place.

- The learning algorithms used are not clearly explained:

the authors simply state that they use "ACCNet" (from some unpublished prior work),

but to readers unfamiliar with this algorithm, it is difficult to judge the contents of the paper.

- In the numerical experiments, what is the "convergence rate" ?

is it the ratio between the mean reward of the learnt policy and that of the optimal ?

For how many time steps are the learning algorithm run before evaluating their outcome ?

What are the meaning of the various input parameter of ACCnet,

and is the performance sensitive to those parameters ?

fact fact fact evaluation evaluation evaluation evaluation evaluation evaluation fact request fact request evaluation evaluation fact evaluation request request request request request


This paper present the application of the memory buffer concept to speech synthesis,

and additionally learns a "speaker vector" that makes the system adaptive and work reasonably well on "in-the-wild" speech data.

This is a relevant problem, and a novel solution,

but synthesis is a wicked problem to evaluate,

so I am not sure if ICLR is the best venue for this paper.

I see two competing goals:

- If the focus is on showing that the presented approach outperforms other approaches under given conditions, a different task would be better (for example recognition, or some sort of trajectory reconstruction)

- If the focus is on showing that the system outperforms other synthesis systems, then a speech oriented venue might be best

(and it is unfortunate that optimized hyper-parameters for the other systems are not available for a fair comparsion)

- If fair comparisons with the other appraoches cannot be made, my sense is that the multi-speaker (post-training fitting) option is really the most interesting and novel contribution here,

which could be discussed in mroe detail

Still, the approach is creative and interesting and deserves to be presented.

I have a few questions/ suggestions:Introduction - The link to Baddeley's "phonological loop" concept seems weak at best.

There is nothing phonological about the features that this model stores and retrieves,

and no evidence that the model behaves in a way consistent with "phonologcial" (or articulatory) assumptions or models

- maybe best to avoid distracting the reader with this concept and strengthen the speaker adaptation aspect?

- The memory model is not an RNN,

but it is a recurrently called structure (as the name "phonological loop" also implies)

- so I would also not highlight this point much

- Why would the four properties of the proposed method (mid of p. 2, end of introduction: memory buffer, shared memory, shallow fully connected networks, and simple reader mechanism) lead to better robustness and improve performance on noisy and limited training data?

Maybe the proposed approach works better for any speech synthesis task?

Why specifically for "in-the-wild" data?

The results in Table 2 show that the proposed system outperforms other systems on Blizzard 2013, but not Blizzard 2011

- does this support the previous argument?

- Why not also evaluate MCD scores?

This should be a quick and automatic way to diagnose what the system is doing?

Or is this not meaningful with the noisy training data?

Previous work- Please introduce abbreviations the first time they are used ("CBHG" for example)

- There is other work on using "in-the-wild" speech as well:

Pallavi Baljekar and Alan W Black. Utterance Selection Techniques for TTS Systems using Found Speech, SSW 2016, Sunnyvale, USA Sept 2016

The architecture- Please explain the "GMM" (Gaussian Mixture Model?) attention mechanism in a bit more detail, how does back-propagation work in this case?

- Why was this approach chosen?

Does it promise to be robust or good for low data situations specifically?

- The fonts in Figure 2 are very small,

please make them bigger,

and the Figure may not print well in b/w.

Why does the mean of the absolute weights go up for high buffer positions?

Is there some "leaking" from even longer contexts?

- I don't understand "However, human speech is not deterministic and one cannot expect [...] truth".

You are saying that the model cannot be excepted to reproduce the input exactly?

Or does this apply only to the temporal distribution of the sequence (but not the spectral characteristics)?

The previous sentence implies that it does.

And how does teacher-forcing help in this case?

- what type of speed is "x5"?

Five times slower or faster than real-time?

Experiments- Table 2: maybe mention how these results were computed, i.e. which systems use optimized hyper parameters, and which don't?

How do these results support the interpretation of hte results in the introruction re in-the-wild data and found data?

- I am not sure how to read Figure 4.

Maybe it would be easier to plot the different phone sequences against each other and show how the timings are off, i.e. plot the time of the center of panel one vs the time of the center of panel 2 for the corresponding phone, and show how this is different from a straight line.

Or maybe plot phones as rectangles that get deformed from square shape as durations get learned?

- Figure 5: maybe provide spectrograms and add pitch contours to better show the effect of the dfifferent intonations?

- Figure 4 uses a lot of space, could be reduced, if needed

Discussion- I think the first claim is a bit to broad

- nowhere is it shown that the method is inherently more robust to clapping and laughs, and variable prosody.

The authors will know the relevant data-sets better than I do,

maybe they can simply extend the discussion to show that this is what happens.

- Efficiency: I think Wavenet has also gotten much faster and runs in less than real-time now

- can you expand that discussion a bit, or maybe give estimates in times of FLOPS required, rather than anecdotal evidence for systems that may or may not be comparable?

Conclusion- Now the advantage of the proposed model is with the number of parameters, rather than the computation required.

Can you clarify?

Are your models smaller than competing models?

fact fact evaluation evaluation evaluation fact request request evaluation request request evaluation evaluation fact fact request fact fact request request request request fact fact request evaluation request request fact reference request request request evaluation request evaluation evaluation evaluation evaluation evaluation fact fact request request request request request evaluation evaluation evaluation request request evaluation fact evaluation request evaluation request fact request request


This paper presents a new method for detecting hypernymy by extending the distributional inclusion hypothesis to low-rank embeddings.

The first half of the paper is written superbly, providing a sober account of the state-of-the-art in hypernymy detection via distributional semantics, and a clear, well-motivated explanation of the main algorithm (DIVE).

The reformulation of PMI is interesting;

the authors essentially replace P(w) with the uniform distribution

(this should be stated explicitly in the paper, BTW).

They then augment SGNS to reflect this change by dynamically selecting the number of negatively-sampled contexts according to the target word (w).

This is a clever trick.

I was also very happy to see that the authors evaluated on virtually every lexical inference dataset available.

However, the remainder of the paper describing the experiments and their results was extremely difficult to parse.

First of all, the line between the authors' contribution and prior art is blurry,

because they seem to be introducing new metrics for measuring hypernymy as part of the experiments.

There are also too many moving parts: datasets, evaluation metrics, detection metrics, embedding methods, hyperparameters, etc.

These need to be organized, controlled for, and clearly explained.

I have two major concerns regarding the experiments, beyond intelligibility.

First, the authors used a specific corpus (with specific preprocessing) to train their vectors, but compared their results to those reported in other papers, which are not based on the same data.

This invalidates these comparisons.

Instead, the other methods should be replicated and rerun on exactly the same corpus.

I would also recommend using a much larger corpus,

since 50M tokens is considered quite small when training word embeddings.

My second concern is that summation (dS) is doing all the heavy lifting.

In Table 2, we can see that the difference is only 0.1 between dS and W * dS (where W is also not trained using DIVE).

Since dS is basically a proxy for difference in word frequency,

could it be that the proposed method is just computing which word is more general?

This looks awfully familiar to Levy et al's prototypical hypernym result.

Miscellaneous Comments: - There's a very recent paper on using vector norms for detecting hypernymy that might be worth contrasting with:

- Micro-averaging these datasets is problematic,

because some of the datasets based on WordNet are much larger than the hand-annotated ones, and will likely drown them out.

Because these datasets are so different,

I think it is critical to look at the details and not only at the averages.

- PMI filtering needs to be controlled/ablated in the experiments.

- While explaining equation 6, the authors say that the gradients for x and y are similar;

this is not true,

because k is a function of x (or y), and if one appears more often in general (not necessarily with c), then the gradients will be different as well.

fact evaluation evaluation fact request fact evaluation evaluation evaluation evaluation fact evaluation request evaluation fact evaluation request request fact evaluation fact fact evaluation evaluation request reference evaluation evaluation evaluation request request fact fact fact


This paper leverages how deep Bayesian NNs, in the limit of infinite width, are Gaussian processes (GPs).

After characterizing the kernel function, this allows us to use the GP framework for prediction, model selection, uncertainty estimation, etc.

- Pros of this work The paper provides a specific method to efficiently compute the covariance matrix of the equivalent GP and shows experimentally on CIFAR and MNIST the benefits of using the this GP as opposed to a finite-width non-Bayesian NN.

The provided phase analysis and its relation to the depth of the network is also very interesting.

Both are useful contributions as long as deep wide Bayesian NNs are concerned.

A different question is whether that regime is actually useful.

- Cons of this work Although this work introduces a new GP covariance function inspired by deep wide NNs,

I am unconvinced of the usefulness of this regime for the cases in which deep learning is useful.

For instance, looking at the experiments, we can see that on MNIST-50k (the one with most data, and therefore, the one that best informs about the "true" underlying NN structure) the inferred depth is 1 for the GP and 2 for the NN, i.e., not deep.

Similarly for CIFAR, where only up to depth 3 is used.

None of these results beat state-of-the-art deep NNs.

Also, the results about the phase structure show how increased depth makes the parameter regime in which these networks work more and more constrained.

In [1], it is argued that kernel machines with fixed kernels do not learn a hierarchical representation.

And such representation is generally regarded as essential for the success of deep learning.

My impression is that the present line of work will not be relevant for deep learning and will not beat state-of-the-art results

because of the lack of a structured prior.

In that sense, to me this work is more of a negative result informing that to be successful, deep Bayesian NNs should not be wide and should have more structure to avoid reaching the GP regime.

- Other comments:In Fig. 5, use a consistent naming for the axes (bias and variances).

In Fig. 1, I didn't find the meaning of the acronym NN with no specified width.

Does the unit norm normalization used to construct the covariance disallow ARD input selection?

[1] Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The Curse of Dimensionality for Local Kernel Machines. 2005.

fact fact fact evaluation evaluation evaluation fact evaluation fact fact fact fact fact fact evaluation fact evaluation request request request reference


To speed up RL algorithms, the authors propose a simple method based on utilizing expert demonstrations.

The proposed method consists in explicitly learning a prediction function that maps each time-step into a state.

This function is learned from expert demonstrations.

The cost of visiting a state is then defined as the distance between that state and the predicted state according to the learned function.

This reward is then used in standard RL algorithms to learn to stick close to the expert's demonstrations.

An on-loop variante of this method consists of learning a function that maps each state into a next state according to the expert, instead of the off-loop function that maps time-steps into states.

While the experiments clearly show the advantage of this method,

this is hardly surprising or novel.

The concept of encoding the demonstration explicitly in the form of a reward has been around for over a decade.

This is the most basic form of teaching by demonstration.

Previous works had used other models for generalizing demonstrations (GMMs, GPs, Kernel methods, neural nets etc..).

This paper uses a three layered fully connected auto-encoder (which is not that deep of a model, btw) for the same purpose.

The idea of using this model as a reward instead of directly cloning the demonstrations is pretty straightforward.

Other comments: - Most IRL methods would work just fine by defining rewards on states only and ignoring actions all together.

If you know the transition function, you can choose actions that lead to highly rewarding states,

so you don't need to know the expert's executed actions.

- "We assume that maximizing likelihood of next step prediction in equation 1 will be globally optimized in RL".

Could you elaborate more on this assumption?

Your model finds rewards based on local state features, where a greedy (one-step planning) policy would reproduce the expert's demonstrations (if the system is deterministic).

It does not compare the global performance of the expert to alternative policies (as is typically done in IRL).

- Related to the previous point: a reward function that makes every step of the expert optimal may not be always exist.

The expert may choose to go to terrible states with the hope of getting to a highly rewarding state in the future.

Therefore, the objective functions set in this paper may not be the right ones, unless your state description contains features related to future states so that you can incorporate future rewards in the current state (like in the reacher task, where a single image contains all the information about the problem).

What you need is actually features that can capture the value function (like in DQN) and not just the immediate reward (as is done in IRL methods).

- What if in two different trajectories, the expert chooses opposite actions for the same state appearing in both trajectories?

For example, there are two shortest paths to a goal, one starts with going left and another starts with going right.

If you try to generate a state that minimizes the sum of distances to the two states (left and right ones), then you may choose to remain in the middle, which is suboptimal.

You wouldn't have this issue with regular IRL techniques,

because you can explain both behaviors with future rewards instead of trying to explain every action of the expert using only local state description.

fact fact fact fact fact fact evaluation evaluation fact evaluation fact fact evaluation evaluation evaluation fact quote request fact fact fact fact evaluation fact request fact fact fact fact


The authors provide an algorithm-agnostic active learning algorithm for multi-class classification.

The core technique is to construct a coreset of points whose labels inform the labels of other points.

The coreset construction requires one to construct a set of  points which can cover the entire dataset.

While this is NP-hard problem in general,

the greedy algorithm is 2-approximate.

The authors use a variant of the greedy algorithm along with bisection search to solve a series of feasibility problems to obtain a good cover of the dataset each time.

This cover tells us which points are to be queried.

The reason why choosing the cover is a good idea is

because under suitable Lipschitz continuity assumption the generalization error can be controlled via an appropriate value of the covering radius in the data space.

The authors use the coreset construction with a CNN to demonstrate an active learning algorithm for multi-class classification.

The experimental results are convincing enough to show that it outperforms other active learning algorithms.

However, I have a few major and minor comments.

Major comments: 1. The proof of Lemma 1 is incomplete.

We need the Lipschitz constant of the loss function.

The loss function is a function of the CNN function and the true label.

The proof of lemma 1 only establishes the Lipschitz constant of the CNN function.

Some more extra work is needed to derive the lipschitz constant of the loss function from the CNN function.

2. The statement of Prop 1 seems a bit confusing to me.

the hypothsis says that the loss on the coreset = 0.

But the equation in proposition 1 also includes the loss on the coreset.

Why is this term included.

Is this term not equal to 0?

3. Some important works are missing.

Especially works related to pool based active learning, and landmark results on labell complexity of agnostic active learning.

UPAL: Unbiased Pool based active learning by Ganti & Gray. http://proceedings.mlr.press/v22/ganti12/ganti12.pdf

Efficient active learning of half-spaces by Gonen et al. http://www.jmlr.org/papers/volume14/gonen13a/gonen13a.pdf

A bound on the label complexity of agnostic active learning. http://www.machinelearning.org/proceedings/icml2007/papers/375.pdf

4.  The authors use L_2 loss as their objective function.

This is a bit of a weird choice

given that they are dealing with multi-class classification and the output layer is a sigmoid layer, making it a natural fit to work with something like a cross-entropy loss function.

I guess the theoretical results do not extend to cross-entropy loss,

but the authors do not mention these points anywhere in the paper.

For example, the ladder network, which is one of the networks used by the authors is a network that uses cross-entropy for training.

Minor-comment: 1. The feasibility program in (6) is an MILP.

However, the way it is written it does not look like an MILP.

It would have been great had the authors mentioned that u_j \in {0,1}.

2. The authors write on page 4, "Moreover, zero training error can be enforced by converting average loss into maximal loss".

It is not clear to me what the authors mean here.

For example, can I replace the average error in proposition 1, by maximal loss?

Why can I do that?

Why would that result in zero training error?

On the whole this is interesting work and the results are very nice.

But, the proof for Lemma 1 seems incomplete to me,

and some choices (such as choice of loss function) are unjustified.

Also, important references in active learning literature are missing.

fact evaluation fact fact fact fact fact evaluation fact fact evaluation evaluation evaluation evaluation fact fact request evaluation fact fact request request evaluation evaluation reference reference reference fact evaluation fact evaluation fact fact fact evaluation request quote evaluation request request request evaluation evaluation evaluation evaluation


This paper presents a generic unbiased low-rank stochastic approximation to full rank matrices that makes it possible to do online RNN training without the O(n^3) overhead of real-time recurrent learning (RTRL).

This is an important and long-sought-after goal of connectionist learning

and this paper presents a clear and concise description of why their method is a natural way of achieving that goal, along with experiments on classic toy RNN tasks with medium-range time dependencies for which other low-memory-overhead RNN training heuristics fail.

My only major complaint with the paper is that it does not extend the method to large-scale problems on real data, for instance work from the last decade on sequence generation, speech recognition or any of the other RNN success stories that have led to their wide adoption

(eg Graves 2013, Sutskever, Martens and Hinton 2011 or Graves, Mohamed and Hinton 2013).

However, if the paper does achieve what it claims to achieve, I am sure that many people will soon try out UORO to see if the results are in any way comparable.

fact evaluation evaluation evaluation reference evaluation


The paper presents an off-policy actor-critic method for learning a stochastic policy with entropy regularization.

It is a direct extension of maximum entropy reinforcement learning for Q-learning (recently called soft-Q learning), and named soft actor-critic (SAC).

Empirically SAC is shown to outperform DDPG significantly in terms of stability and sample efficiency, and can solve relatively difficult tasks that previously only on-policy (or hybrid on-policy/off-policy) method such as TRPO/PPO can solve stably.

Besides entropy regularization, it also introduces multi-modal policy parameterization through mixture of Gaussians that enables diverse, on-policy exploration.

The main appeal of the paper is the strong empirical performance of this new off-policy method in continuous action benchmarks.

Several design choices could be the key,

so it is encouraged to provide more ablation studies on these, which would be highly valuable for the community.

In particular, - Amortization of Q and \pi through fitting state value function

- On-policy exploration vs OU process based off-policy exploration

- Mixture vs non-mixture-based stochastic policy

- SAC vs soft Q-learning

Another valuable discussion to be had is the stability of off-policy algorithm comparing Q-learning versus actor-critic method.

Pros: - Simple off-policy algorithm that achieves significantly better performance than existing off-policy baseline algorithms

- It allows on-policy exploration in off-policy learning, partially thanks to entropy regularization that prevents variance from shrinking to 0.

It could be considered a major success of off-policy algorithm that removes heuristic exploration noise.

Cons: - Method is relatively simple extension from existing work in maximum entropy reinforcement learning.

It is unclear what aspects lead to significant improvements in performance due to insufficient ablation studies.

Other question: - Above Eq. 7 it discusses that fitting a state value function wrt Q and \pi is shown to improve the stability significantly.

Is this comparison with directly estimating state value using finite samples?

If so, is the primary instability due to variance of the estimate, which can be avoided by drawing a lot of samples or do full integration (still reasonably tractable for finite mixture model)?

Or, is the instability from elsewhere?

By having SGD-based fitting of state value function, it appears to simulate slowly changing target values (similar role as target networks).

If so, could a similar technique be used with DDPG and get more stable performance?

fact fact fact fact fact fact request request request request request request evaluation fact evaluation evaluation evaluation fact request request request evaluation request


This paper describes an attempt of improving information flow in deep networks (but is used and tested here with seq2seq models although it is reality unrelated to seq2seq models per se).

Slightly different from Resnet the information flow is improved by not just adding the outputs from previous layers but instead concatenating the outputs from previous layers with the current outputs.

The authors claim better convergence speed and better results for a similar number of parameters although the differences seems to be in the noise.

Overall this is an OK technique but in my opinion not really novel enough to justify a whole paper about it

as it seems more like a relatively minor architecture tweak.

The results seem to indicate that there were some problems with getting deeper networks to work for the baseline (why is in Table 3 baseline-6L worse than baseline-4L?) for which the reason could be a multitude of issues probably related to hyper-parameter tuning.

What is also missing is a an analysis of the negative consequences of this technique -- for example, doesn't the number of parameters increase with the depth of the network because of the concatenation?

Also, it would have been good to see more experiments with smaller baseline networks as well to match the smaller DenseNet networks in Table 1 and 2.

Finally, the writing of the paper could be improved a lot:

The basic idea is not well described (however, many times repeated)

and the grammar is often wrong

and also there are some typos.

fact evaluation fact evaluation evaluation fact request request request evaluation evaluation evaluation


This paper proposes a new method to train residual networks in which one starts by training shallow ResNets, doubling the depth and warm starting from the previous smaller model in a certain way, and iterating.

The authors relate this idea to a recent dynamical systems view of ResNets in which residual blocks are viewed as taking steps in an Euler discretization of a certain differential equation.

This interpretation plays a role in the proposed training method by informing how the “step sizes” in the Euler discretization should change when doubling the depth of the network.

The punchline of the paper is that the authors are able to achieve similar performance as “full ResNet training” but with significantly reduced training time.

Overall, the proposed method is novel

— even though this idea of going from shallow to deep is natural for residual networks,

tying the idea to the dynamical systems perspective is elegant.

Moreover the paper is clearly written.

Experimental results are decent

— there are clear speedups to be had based on the authors' experiments.

However it is unclear if these gains in training speed are significant enough for people to flock to using this (more complicated) method of training.

I only have a few small questions/comments: * A more naive way to do multi-level training would be to again iteratively double the depth, but perhaps not halve the step size.

This might be a good baseline to compare against to demonstrate the value of the dynamical systems viewpoint.

* One thing I’m unclear on is how convergence was assessed…

my understanding is that the training proceeds for a fixed number of epochs (?)

- but shouldn’t this also depend on the depth in some way?

* Would the speedups be more dramatic for a larger dataset like Imagenet?

* Finally, not being very familiar with multigrid methods from the numerical methods literature

— I would have liked to hear about whether there are deeper connections to these methods.

fact fact evaluation evaluation evaluation evaluation evaluation evaluation evaluation fact evaluation evaluation request evaluation evaluation request request non-arg request


This paper proposed a new parametrization scheme for weight matrices in neural network based on the Householder  reflectors to solve the gradient vanishing and exploding problems in training.

The proposed method improved two previous papers:

1) stronger expressive power than Mahammedi et al. (2017),

2) faster gradient update than Vorontsov et al. (2017).

The proposed parametrization scheme is natrual from numerical linear algebra point of view and authors did a good job in Section 3 in explaining the corresponding expressive power.

The experimental results also look promising.

It would be nice if the authors can analyze the spectral properties of the saddle points in linear RNN (nonlinear is better but it's too difficult I believe).

If the authors can show the strict saddle properties then as a corollary, (stochastic) gradient descent finds a global minimum.

Overall this is a strong paper

and I recommend to accept.

fact fact reference reference evaluation evaluation request request evaluation evaluation


The paper considers multi-task setting of machine learning.

The first contribution of the paper is a novel PAC-Bayesian risk bound.

This risk bound serves as an objective function for multi-task machine learning.

A second contribution is an algorithm, called LAP, for minimizing a simplified version of this objective function.

LAP algorithm uses several training tasks to learn a prior distribution P over hypothesis space.

This prior distribution P is then used to find a posterior distribution Q that minimizes the same objective function over the test task.

The third contribution is an empirical evaluation of LAP over toy dataset of two clusters and over MNIST.

While the paper has the title of "life-long learning",

the authors admit that all experiments are in multi-task setting, where the training is done over all tasks simultaneously.

The novel risk bound and LAP algorithm can definitely be applied to life-long setting, where training tasks are available sequentially.

But since there is no empirical evaluation in this setting,

I suggest to adjust the title of the paper.

The novel risk bound of the paper is an extension of the bound from [Pentina & Lampert, ICML 2014].

The extension seems to be quite significant.

Unlike the bound of [Pentina & Lampert, ICML 2014], the new bound allows to re-use many different PAC-Bayesian complexity terms that were published previously.

I liked risk bound and optimization sections of the paper.

But I was less convinced by the empirical experiments.

Since the paper improves the risk bound of [Pentina & Lampert, ICML 2014],

I expected to see an empirical comparison of LAP and optimization  algorithm from the latter paper.

To make such comparison fair, both optimization algorithms should use the same base algorithm, e.g. ridge regression, as in [Pentina & Lampert, ICML 2014].

Also I suggest to use the datasets from the latter paper.

The experiment with multi-task learning over MNIST dataset looks interesting,

but it is still a toy experiment.

This experiment  will be more convincing with more sophisticated datasets (CIFAR-10, ImageNet) and architectures (e.g. Inception-V4, ResNet).

Minor remarks: Section 6, line 4: "Combing" -> "Combining"

Page 14, first equation: There should be "=" before the second expectation.

fact evaluation fact evaluation fact fact evaluation fact fact fact fact request evaluation evaluation fact evaluation evaluation fact evaluation request request evaluation evaluation request request request


This paper addresses the question of unsupervised clustering with high classification performance.

They propose a deep variational autoencoder architecture with categorical latent variables at the deepest layer and propose to train it with modifications of the standard variational approach with reparameterization gradients.

The model is tested on a medical imagining dataset where the task is to distinguish healthy from pathological lymphocytes from blood samples.

I am not an expert on this particular dataset,

but to my eye the results look impressive.

They show high sensitivity and high specificity.

This paper may be an important contribution to the medical imaging community.

My primary concern with the paper is the lack of novelty and relatively little in the way of contributions to the ICLR community.

The proposed model is a simple variant on the standard VAE models

(see for example the Ladder VAE https://arxiv.org/abs/1602.02282 for deep models with multiple stochastic layers).

This would be OK if a thorough evaluation on at least two other datasets showed similar improvements as the lymphocytes dataset.

As it stands, it is difficulty for me to assess the value of this model.

Minor questions / concerns: - The authors claim in the first paragraph of 3.2 that deterministic mappings lack expressiveness.

Would be great to see the paper take this claim seriously and investigate it.

- In equation (13) it isn't clear whether you use q_phi to be the discrete mass or the concrete density.

The distinction is discussed in https://arxiv.org/abs/1611.00712

- Would be nice to report the MCC in experimental results.

fact fact fact non-arg evaluation evaluation evaluation evaluation evaluation fact evaluation evaluation fact request evaluation fact request


The authors demonstrate experimentally a problem with the way common latent space operations such as linear interpolation are performed for GANs and VAEs.

They propose a solution based on matching distributions using optimal transport.

Quite heavy machinery to solve a fairly simple problem,

but their approach is practical and effective experimentally

(though the gain over the simple SLERP heuristic is often marginal).

The problem they describe (and so the solution) deserves to be more widely known.

Major comments: The paper is quite verbose, probably unnecessarily so.

Firstly, the authors devote over 2 pages to examples that distribution mismatches can arise in synthetic cases (section 2).

This point is well made by a single example (e.g. section 2.2)

and the interesting part is that this is also an issue in practice (experimental section).

Secondly, the authors spend a lot of space on the precise derivation of the optimal transport map for the uniform distribution.

The fact that the optimal transport computation decomposes across dimensions for pointwise operations is very relevant, and the matching of CDFs,

but I think a lot of the mathematical detail could be relegated to an appendix, especially the detailed derivation of the particular CDFs.

Minor comments: It seems worth highlighting that in practice, for the common case of a Gaussian, the proposed method for linear interpolation is just a very simple procedure that might be called "projected linear interpolation", where the generated vector is multiplied by a constant.

All the optimal transport theory is nice,

but it's helpful to know that this is simple to apply in practice.

Might I suggest a very simple approach to fixing the distribution mismatch issue?

Train with a spherical uniform prior.

When interpolating, project the linear interpolation back to the sphere.

This matches distribution, and has the attractive property that the entire geodesic between two points lies in a region with typical probability density.

This would also work for vicinity sampling.

In section 1, overfitting concerns seem like a strange way to motivate the desire for smoothness.

Overfitting is relatively easy to compensate for,

and investigating the latent space is interesting regardless.

When discussing sampling from VAEs as opposed to GANs, it would be good to mention that one has to sample from p(x | z) not just p(z).

Lots of math typos such as t - 1 should be 1 - t in (2), "V times a times r" instead of "Var" in (3) and "s times i times n" instead of "sin", etc, sqrt(1) * 2 instead of sqrt(12), inconsistent bolding of vectors.

Also strange use of blackboard bold Z to mean a vector of random variables instead of the integers.

Could cite an existing source for the fact that most mass for a Gaussian is concentrated on a thin shell (section 2.2),

e.g. David MacKay Information Theory, Inference and Learning Algorithms.

At the end of section 2.4, a plot of the final 1D-to-1D optimal transport function (for a few different values of t) for the uniform case would be incredibly helpful.

Section 3 should be a subsection of section 2.

For both SLERP and the proposed method, there's quite a sudden change around the midpoint of the interpolation in Figure 2.

It would be interesting to plot more points around the midpoint to see the transition in more detail.

(A small inkling that samples from the proposed approach might change fastest qualitatively near the midpoint of the interpolation perhaps maybe be seen in Figure 1,

since the angle is changing fastest there??)

fact fact evaluation evaluation evaluation evaluation evaluation fact evaluation evaluation evaluation evaluation request request evaluation evaluation evaluation request request fact fact evaluation evaluation evaluation request request evaluation request reference request request evaluation request evaluation evaluation


The authors present a variant of the adversarial feature learning (AFL) approach by Edwards & Storkey.

AFL aims to find a data representation that allows to construct a predictive model for target variable Y,

and at the same time prevents to build a predictor for sensitive variable S.

The key idea is to solve a minimax problem where the log-likelihood of a model predicting Y is maximized, and the log-likelihood of an adversarial model predicting S is minimized.

The authors suggest the use of multiple adversarial models, which can be interpreted as using an ensemble model instead of a single model.

The way the log-likelihoods of the multiple adversarial models are aggregated does not yield a probability distribution as stated in Eq. 2.

While there is no requirement to have a distribution here

- a simple loss term is sufficient

- the scale of this term differs compared to calibrated log-likelihoods coming from a single adversary.

Hence, lambda in Eq. 3 may need to be chosen differently depending on the adversarial model.

Without tuning lambda for each method, the empirical experiments seem unfair.

This may also explain why, for example, the baseline method with one adversary effectively fails for Opp-L.

A better comparison would be to plot the performance of the predictor of S against the performance of Y for varying lambdas.

The area under this curve allows much better to compare the various methods.

There are little theoretical contributions.

Basically, instead of a single adversarial model - e.g., a single-layer NN or a multi-layer NN - the authors propose to train multiple adversarial models on different views of the data.

An alternative interpretation is to use an ensemble learner where each learner is trained on a different (overlapping) feature set.

Though, there is no theoretical justification why ensemble learning is expected to better trade-off model capacity and robustness against an adversary.

Tuning the architecture of the single multi-layer NN adversary might be as good?

In short, in the current experiments, the trade-off of the predictive performance and the effectiveness of obtaining anonymized representations effectively differs between the compared methods.

This renders the comparison unfair.

Given that there is also no theoretical argument why an ensemble approach is expected to perform better,

I recommend to reject the paper.

fact fact fact fact fact fact fact evaluation fact fact evaluation fact request evaluation evaluation fact fact fact evaluation fact evaluation fact evaluation


Summary: Based on ideas within the context of kernel theory, the authors consider post-training of NNs as an extra training step, which only optimizes the last layer of the network.

This additional step makes sure that the embedding, or representation, of the data is used in the best possible way for the considered task

(which is also reflected in the experiments).

According to the authors, the contributions are the following: 1. Post-training step: keeping the rest of the NN frozen (after training), the method trains the last layer in order to "make sure" that the representation learned is used in the most efficient way.

2. Highlighting connections with kernel techniques and RKHS optimization (like kernel ridge regression).

3. Experimental results.

Clarity: The paper is well-written, the main ideas well-clarified.

Importance: While the majority of papers nowadays focuses on the representation part (i.e., how we get to \Phi_{L-1}(x)),

this paper assumes this is given and proposes how to optimize the weights in the final step of the algorithm.

This by itself is not enough to boost the performance universally (e.g., if \Phi_{L-1} is not well-trained, the problem is deeper than training the last layer);

however, it proposes an additional step that can be used in most NN architectures.

From that front (i.e., proposing to do something different than simply training a NN), I find the paper interesting, that might attract some attention at the conference.

On the other hand, to my humble opinion, the experimental results do not show a significant gain in the performances of all networks (esp. Figure 3 and Table 1 are within the range of statistical error).

In order to state something like this universally, either one needs to perform experiments with more than just MNIST/CIFAR datasets, or even more preferably, prove that the algorithm performs better.

Originality: It would be great to have some more theory (if any) for the post-training step, or investigate more cases, rather than optimizing only the last layer.

Comments: 1. I assume the authors focused in the last layer of the NN for simplicity,

but is there a reason why one might want to focus only on the last layer?

One reason is convexity in W of the problem (2).

Any other?

2. Have the authors considered (even in practice only) to include training of the last 2 layers of the NN?

The authors state this question in the future direction,

but it would make the paper more complete to consider it here.

fact evaluation evaluation fact fact fact evaluation evaluation fact evaluation fact evaluation evaluation request request evaluation request evaluation request non-arg fact request


This article aims at understanding the role played by the different words in a sentence, taking into account their order in the sentence.

In sentiment analysis for instance, this capacity is critical to model properly negation.

As state-of-the-art approaches rely on LSTM,

the authors want to understand which information comes from which gate.

After a short remainder regarding LSTM, the authors propose a framework to disambiguate interactions between gates.

In order to obtain an analytic formulation of the decomposition, the authors propose to linearize activation functions in the network.

In the experiment section, authors compare themselves to a standard logistic regression (based on a bag of words representation).

They also check the unigram sentiment scores (without context).

The main issue consists in modeling the dynamics inside a sentence (when a negation or a 'used to be' reverses the sentiment).

The proposed approach works fine on selected samples.

The related work section is entirely focused on deep learning

while the experiment section is dedicated to sentiment analysis.

This section should be rebalanced.

Even if the authors claim that their approach is general,

they also show that it fits well the sentiment analysis task in particular.

On top of that, a lot of fine-grained sentiment analysis tools has been developed outside deep-learning:

the authors should refer to those works.

Finally, authors should provide some quantitative analysis on sentiment classification:

a lot of standard benchmarks are widely use in the literature

and we need to see how the proposed method performs with respect to the state-of-the-art.

Given the chosen tasks, this work should be compared to the beermind system:

and the associated publication

fact evaluation fact fact fact fact fact fact fact evaluation evaluation evaluation request fact fact evaluation request request evaluation request request reference non-arg reference


This should be the first work which introduces in the causal structure into the GAN, to solve the label dependency problem.

The idea is interesting and insightful.

The proposed method is theoretically analyzed and experimentally tested.

Two minor concerns are 1) what is the relationship between the anti-labeler and and discriminator?

2) how the tune related weight of the different objective functions.


Strengths: * Very simple approach, amounting to coupled training of "e" identical copies  of a chosen net architecture, whose predictions are fused during training.

This forces the different model instances to become more complementary.

* Perhaps counterintuitively, experiments also show that coupled ensembling leads to individual nets that perform better than those produced by separate training.

* The practical advantages of the proposed approach are twofold: 1. Given a fixed parameter budget, coupled ensembling leads to better accuracy than a single net or an ensemble of disjointly-trained nets.

2. For the same accuracy, coupled ensembling yields significant parameter savings.

Weaknesses: * Although results are very strong,

the proposed models do not outperform the state-of-the-art, except for the models reported in Table 4,

which however were obtained by *traditional* ensembling of coupled ensembles.

* Coupled ensembling requires joint training of all nets in the ensemble

and thus is limited by the size of the model that can be fit in memory.

Conversely, traditional ensembling involves separate training of the different instances

and this enables the learning of an arbitrary number of individual nets.

* I am surprised by the results in Table 2,

which suggest that the optimal number of nets in the ensemble is remarkably low (only 3!).

It'd be valuable to understand whether this kind of result holds for other network architectures or whether it is specific to this choice of net.

* Strictly speaking it is correct to refer to the individual nets in the ensembles as "branches" and "basic blocks."

Nevertheless, I find the use of these terms confusing in the context of the proposed approach,

since they are commonly used to denote concepts different from those represented here.

I would recommend refraining from using these terms here.

Overall, the paper provides limited technical novelty.

Yet, it reveals some interesting empirical findings about the benefits of coordinated training of models in an ensemble.

evaluation fact fact fact evaluation evaluation evaluation fact fact fact fact fact evaluation fact request fact evaluation evaluation request evaluation evaluation


The authors combine an ensemble of DNNs as model for the dynamics with TRPO.

The ensemble is used in two steps: First to collect imaginary roll-outs for TRPO and secondly to estimate convergence of the algorithm.

The experiments indicate superior performance over the baselines.

The paper is well-written

and the experiments indicate good results.

However, idea of using ensembles in the context of (model-based) RL  is not novel,

and it comes at the cost of time complexity.

Therefore, the method should utilize the advantage an ensemble provides to its full extent.

The main strength of an ensemble is to provide lower test error, but also some from of uncertainty estimate given by the spread of the predictions.

The authors mainly utilize the first, but to a lesser extent the second advantage (the imaginary roll-outs will  utilize the spread to generate possible outcomes).

Ideally the exploration should also be guided by the uncertainty (such as VIME).

Related, what where the arguments in favor of an ensemble compared to Bayesian neural networks (possibly even as simple as using MH-dropout)?

BNNs provide a stronger theoretical justification that the predictive uncertainty is meaningful.

Can the authors comment on the time-complexity of the proposed methods compared to the baselines?

In Fig. 2  the x-axis is the time step of the  real data.

But I assume it took a different amount of time for each method to reach step t.

The same argument can be made for Fig. 4.

It seems here that in snake the larger ensembles reach convergence the quickest,

but I expect this effect to be reversed when considering actual training time.

In total I think this paper can provide a useful addition to the literature.

However, the proposed approach does not have strong novelty

and I am not fully convinced if the additional burden on time complexity outweighs the improved performance.

Minor:  In Sec. 2: "Both of these approaches assume a fixed dataset of samples which are collected before the algorithm starts operating."

This is incorrect,

while these methods consider the domain of fixed datasets,

the algorithms themselves are not limited to this context.

fact fact evaluation evaluation evaluation evaluation fact evaluation evaluation evaluation request request evaluation request fact evaluation fact fact evaluation evaluation evaluation evaluation quote fact fact fact


Summary: The paper focuses on the characterization of the landscape of deep neural networks; i.e., when and why local minima are global, what are the conditions for saddle critical points, etc.

The paper covers a somewhat wide range of deep nets (from shallow with linear activation to deeper with non-linear activation);

it focuces only on feed forward neural networks.

As the authors state, this paper provides a unifying perspective to the subject

(it justifies the results of others through this unifying theory,

but also provides new results; e.g., there are results that do not depend on assumptions on the target data matrix Y).

Originality: The paper provides similar results to previous work, while removing some of the assumptions made in previous work.

In that sense, the originality of the results is weak,

but definitely there is some novelty in the methodology used to get to these results.

Thus, I would say original.

Importance: The paper deals with the important problem of when and why training algorithms might get to global/local/saddle critical points.

While there are no direct connections with generalization properties,

characterizing the landscape of neural networks is an important topic to make further steps into better understanding of deep learning.

It will attract some attention at the conference.

Clarity: The paper is well-written

- some parts need improvement,

but overall I'm satisfied with the current version.

Comments: 1. If problem (4) is not considered at all in this paper (in its full generality that considers matrix completion and matrix sensing as special cases), then the authors could just start with the model in (5).

2. Remark 1 has a nice example

- could this example be shown with Y not being the all-zeros vector?

3. In section 5, the authors make a connection with the work of Ge et al. 2016.

They state that the problems in (10)-(11) constitute generalizations of the symmetric matrix completion case, considered in Ge et al. 2016.

However, in that work, the main difficulty of proving global optimality comes from the randomness of the sampling mask operator (which introduces the notion of incoherence and requires results in expectation).

It is not clear, and maybe it is an overstatement, that the results in section 5 generalize that work.

If that is the case, could the authors describe this a bit further?

fact evaluation fact fact evaluation fact evaluation evaluation evaluation evaluation evaluation fact evaluation evaluation evaluation evaluation evaluation request evaluation request fact fact fact evaluation request


The paper proposed a new regularization approach that simultaneously encourages the weight vectors (W) to be sparse and orthogonal to each other.

The argument is that the sparsity helps to eliminate the irrelevant feature vectors by making the corresponding weights zero.

Nearly orthogonal sparse vectors will have zeros at different indexes

and hence, encourages the weight vectors to have small overlap in terms of indices of nonzero entries (called support).

Small overlap in support of weight vectors, aids interpretability

as each weight vector is associated with a unique subset of feature vectors.

For example, in the topic model, small overlap encourages, each topic to have unique set of representation words.

The proposed approach used L1 regularizer for enforcing sparsity in W.

For enforcing orthogonality between different weight vectors (wi, wj), the log-determinant divergence (LDD) regularization term encourages the Gram Matrix G (Gij = wiTwj) to be close to an identity matrix I.

The authors applied and tested the performance of proposed approach on Neural Network and Sparse Coding (SC) machine learning models.

The authors validated the need for their proposed regularizer through experiments on 4 datasets (3 text and 1 images).

Major * The novelty of the paper is not clear.

Neither L1 no logdet() are novel regularizers (see the literature of Determinatal Point Process).

With the presence of the auto-differentiator, one cannot claim the making derivative a novelty.

* L1 is also encourages diversity although as explicit as logdet.

This is also obvious from Fig 2.

Perhaps the advantage of diversity is in interpretability

but that is hard to quantify and the authors did not put enough effort to do that;

we only have small anecdotal results in section 4.3.

* The Table 1 is not convincing

because one can argue, for example, gun (vec 1) and weapon (vec 4) are colinear.

* In section 4.2, the authors experimented with SC on text dataset.

The overlap score decreases as the strength of regularization increases.

The authors didn’t show the effect of increasing the regularization strength on the model accuracy and convergence time.

This analysis is important to make sure, the decrease in overlap score is not coming at the expense of model accuracy and performance.

* In section 4.4, increase in test set accuracy and difference between test and train set accuracy is used to validate the claim, that the proposed regularizer helps reducing over fitting.

In Table-2, , the test accuracy increases between SC and LDD-L1 SC

while the train accuracy remains almost the same.

Also, the authors didn’t do any cross validation to support their claim.

The difference is numbers is too small to support the claim.

* In section on LSTM for Language Modeling, the perplexity score of LDD-L1 regularization on PytorchLM received perplexity score of 1.2 lower than without regularization.

Although, the author mentions it as a significant reduction,

the lowest perplexity score in Table 3 is significantly lower than this result.

It’s not clear how 1.2 reduction in perplexity is significant and why the method should be preferred while much better models already exists.

* Results of the best perplexity model, Neural Architecture Search + WT V2, with proposed regularization would also help, validating the generalizability claims of the new approach.

* In CNN for Image Classification section, details of increase interpretability of the model, in terms of classification decision, is missing.

* In Table-4, the proposed LDD-L1 WideResNet is not the best results.

Results of adding the proposed regularization, to the best know method (Pyramid Sep Drop) would be interesting.

* The proposed regularization claims to provide more interpretable representation and less overfit model.

The given experiments are inadequate to validate the claims.

* A more extensive experimentation is required to validate the applicability of the method.

* In SC, aj are the linear coefficients or the coefficient vector of the j-th sample.

If A ∈ Rm×n then aj ∈ Rm×1 and j ranges between [1,n] as in equation 6.

The notation in section 2.2, Sparse Coding section is misleading

as j ranges between [1,m].

* In Related works, the authors mention previous work done on interpreting the results of the machine learning models.

Related works on enhancing interpretability and reducing overfitting by using regularization is missing.

fact fact fact fact fact fact fact fact fact fact fact evaluation evaluation evaluation fact evaluation evaluation evaluation evaluation evaluation fact fact fact fact evaluation fact fact fact fact evaluation fact fact evaluation evaluation request fact fact request fact evaluation request fact fact evaluation fact fact fact


The authors propose a new regularization term modifying the VAE (Kingma et al 2013) objective to encourage learning disentangling representations.

Specifically, the authors suggest to add penalization to ELBO in the form of -KL(q(z)||p(z)) , which encourages a more global criterion than the local ELBOs.

In practice, the authors decide that the objective they want to optimize is unwieldy

and resort to moment matching of covariances of q(z) and p(z) via gradient descent.

The final objective uses a persistent estimate of the covariance matrix of q and upgrades it at each mini-batch to perform learning.

The authors use this objective function to perform experiments measuring disentanglement

and find minor benefits compared to other objectives in quantitative terms.

Comments: 1. The originally proposed modification in Equation (4) appears to be rigorous

and as far as I can tell still poses a lower bound to log(p(x)).

The proof could use the result posed earlier: KL(q(z)||p(z)) is smaller than E_x KL(q(z|x)||p(z|x)).

2. The proposed moment matching scheme performing decorrelation resembles approaches for variational PCA and especially independent component analysis.

The relationship to these techniques is not discussed adequately.

In addition, this paper could really benefit from an empirical figure of the marginal statistics of z under the different regularizers in order to establish what type of structure is being imposed here and what it results in.

3. The resulting regularizer with the decorrelation terms could be studied as a modeling choice.

In the probabilistic sense, regularizers can be seen as structural and prior assumptions on variables.

As it stands, it is unnecessarily vague which assumptions this extra regularizer is making on variables.

4. Why is using the objective in Equation (4) not tried and tested and compared to?

It could be thought that subsampling would be enough to evaluate this extra KL term without any need for additional variational parameters \psi.

The reason for switching to the moment matching scheme seems not well motivated here without showing explicitly that Eq (4) has problems.

5. The model seems to be making on minor progress in its stated goal, disentanglement.

It would be more convincing to clarify the structural properties of this regularizer in a statistical sense more clearly given that experimentally it seems to only have a minor effect.

6. Is there a relationship to NICE (Laurent Dinh et al)?

7. The infogan is also an obvious point of reference and comparison here.

8. The authors claim that there are no models which can combine GANs with inference in a satisfactory way,

which is obviously not accurate nowadays given the progress on literature combining GANs and variational inference.

All in all I find this paper interesting

but would hope that a more careful technical justification and derivation of the model would be presented given that it seems to not be an empirically overwhelming change.

fact fact fact fact fact fact evaluation evaluation fact fact evaluation evaluation request request evaluation evaluation request evaluation evaluation evaluation request non-arg request fact evaluation evaluation request


This is a well-written paper with sound experiments.

However, the research outcome is not very surprising.

- Only macro-average F-scores are reported.

Please present micro-average scores as well.

- The detailed procedure of relation extraction should be described.

How do you use entity type information?

(Probably, you did not use entity types.)

- Table 3: The SotA score of EU-ATR target-disease (i.e. 84.6) should be in bold face.

- Section 5.3: Your system scorers in Table 3 are not consistent with Table 2 scores.

- Page 8. "Our approach outperforms ..."

The improvement is clear only for SNPPhenA and EU-ADR durg-disease.

Minor comments: - TreeLSTM --> Tree-LSTM

- Page 7. connexion --> connection

- Page 8. four EU-ADR subtasks --> three ...

- I suggest to conduct transfer learning studies in the similar settings.

evaluation evaluation fact request request request evaluation request evaluation quote fact request request request request


Overall strength: In this paper, the authors proposed target-aware memory networks to model sentiment interactions between target aspects and the context words with attentions.

This work has a well-established motivation:

traditional attention for target-dependent sentiment classification cannot model the interaction between target term and context words when making predictions.

To solve this problem, the authors proposed five formulations in the final prediction layer.

The illustration about the problem is clear, as well as the explanation for the formulations.

Major concerns: 1.	This work brings some modifications to the prediction layer,

which is a bit trivial.

Although the effect has been shown,

the model is too specific to a narrow area, and is not general to be applied in a broad sense.

It could have more contribution if the authors model the interactions within the attention model itself, instead of a simple prediction layer, which is problem-dependent.

2.	The experiments are insufficient to show the effectiveness.

It would be better to provide some statistics showing how the target-context interaction model outperforms the traditional ones in the special cases like the one shown in Table 4.

Only two examples are not convincing.

3.	In section 3, the authors claimed that (5) models the target and context independently.

However, in section 4, in (7), the authors claimed the target vector v_t will affect the context shifting their representation to c’_i.

This should also work for (5).

4.	There are too many typos in the paper, e.g., \alpha is replaced by a, etc.

Other concerns: 1.	It seems that one needs to train at least three embedding matrices: A, C, D which represent input embeddings, output embeddings, and interactive embeddings, respectively.

I wonder if this brings redundant parameters that do not guarantee convergence.

Why not use one matrix instead?

Did the authors try experiments with less embedding matrices?

2.	There is another work that also considers the target-context interaction using interactive attention model.

Please refer to this paper “Interactive Attention Networks for Aspect-Level Sentiment Classification”.

A comparison is needed.

3.	It is better to provide results in terms of accuracy for both datasets,

as previous methods usually use accuracy for comparison.

How’s the score of the proposed model compared with the above paper as well as [Tang et al. 2016]?

fact evaluation fact fact evaluation fact evaluation fact evaluation request evaluation request evaluation fact fact fact evaluation fact evaluation request request fact request request request fact request


This paper is in some sense a "position paper,"

giving a framework for thinking about the loss functions implicitly used by the generator of GAN-type models.

It advocates thinking about the loss in a way similar to how it is considered in structured prediction.

It also proposes that approximating the dual formulation of various divergences with functions from a parametric class, as is typically done in GAN-type setups, is not only more tractable (computationally and in sample complexity) than the full nonparametric estimation, but also gives a better actual loss.

Overall, I like the argument here, and think that it is a useful framework for thinking about these things.

My main concern is that the practical contribution on top of Liu et al. (2017) might be somewhat limited.

A few small points: - f-divergences can actually be nonparametrically estimated purely from samples, e.g. with the k-nearest neighbor estimator of https://arxiv.org/abs/1411.2045, or (for certain f-divergences) the kernel density based estimator of https://arxiv.org/abs/1402.2966.

These are unlikely to lead to a practical learning algorithm,

but could be mentioned in Table 1.

- The discussion of MMD in the end of section 3.1 is a little off.

MMD is fundamentally defined by the kernel choice;

Dziugaite et al. (2015) only demonstrated that the Gaussian RBF kernel is a poor choice for MNIST modeling,

while the samples of Li et al. (2015) simply by using a mixture of Gaussian kernels were much better.

No reasonable fixed kernel is likely to yield good results on a harder image modeling problem,

but that is a slightly different message than the one this paragraph conveys.

- It would be interesting to replicate the analysis of Danihelka et al. (2017) on the Thin-8 dataset.

This might help clarify which of the undesirable effects observed in the VAE model here are due to likelihood, and which due to other aspects of VAEs (like the use of the lower bound).

evaluation fact evaluation fact evaluation evaluation fact evaluation request evaluation fact evaluation evaluation evaluation evaluation request evaluation