launch/ampere huggingface dataset:
The author unveils some properties of the resnets, for example, the cosine loss and l2 ratio of the layers.  I think the author should place more focus to study "real" iterative inference with shared parameters rather than analyzing original resnets. In resnet without sharing parameters, it is quite ambiguous to say whether it is doing representation learning or iterative refinement. 1. The cosine loss is not meaningful in the sense that the classification layer is trained on the output of the last residual block and fixed.  Moving the classification layer to early layers will definitely result in accuracy loss.  Even in non-residual network, we can always say that the vector h_{i+1} - h_i is refining h_i towards the negative gradient direction.  The motivation of iterative inference would be to generate a feature that is easier to classify rather than to match the current fixed classifier.  Thus the final classification layer should be retrained for every addition or removal of residual blocks. 2. The l2 ratio. The l2 ratio is small for higher residual layers, I'm not sure how much this phenomenon can prove that resnet is actually doing iterative inference. 3. In section 4.4 it is shown that unrolling the layers can improve the performance of the network.  However, the same can be achieved by adding more unshared layers.  I think the study should focus more on whether shared or unshared is better. 4. Section 4.5 is a bit weak in experiments,  my conclusion is that currently it is still limited by batch normalization and optimization,  the evidence is still not strong enough to show that iterative inference is advantageous / disadvantageous. The the above said, I think the more important thing is how we can benefit from iterative inference interpretation, which is relatively weak in this paper.
This paper draws an interesting connection between deep neural networks and theories of quantum entanglement. They leveraged the tool for analyzing quantum entanglement to deep neural networks, and proposed a graph theoretical analysis for neural networks. They demonstrated how their theory can help designing neural network architectures on the MNIST dataset. I think the theoretical findings are novel and may contribute to the important problem on understanding neural networks theoretically. I am not familiar with the theory for quantum entanglement though.
Pros: 1. It provided theoretic analysis why larger feature norm is preferred in feature representation learning. 2. A new regularization method (feature incay) is proposed. Cons: It seems there is not much comparison between this proposed method and the concurrent work "COCO(Liu et al. (2017c))".
This paper describes a method for computing representations for out-of-vocabulary words, e.g. based on their spelling or dictionary definitions.  The main difference from previous approaches is that the model is that the embeddings are trained end-to-end for a specific task, rather than trying to produce generically useful embeddings.  The method leads to better performance than using no external resources, but not as high performance as using Glove embeddings.  The paper is clearly written, and has useful ablation experiments.  However, I have a couple of questions/concerns: - Most of the gains seem to come from using the spelling of the word.  As the authors note, this kind of character level modelling has been used in many previous works.  - I would be slightly surprised if no previous work has used external resources for training word representations using an end-task loss,  but I don’t know the area well enough to make specific suggestions  - I’m a little skeptical about how often this method would really be useful in practice.  It seems to assume that you don’t have much unlabelled text (or you’d use Glove),  but you probably need a large labelled dataset to learn how to read dictionary definitions well.  All the experiments use large tasks  - it would be helpful to have an experiment showing an improvement over character-level modelling on a smaller task. - The results on SQUAD seem pretty weak - 52-64%, compared to the SOTA of 81.  It seems like the proposed method is quite generic,  so why not apply it to a stronger baseline?
This very well written paper covers the span between W-GAN and VAE. For a reviewer who is not an expert in the domain, it reads very well, and would have been of tutorial quality if space had allowed for more detailed explanations. The appendix are very useful, and tutorial paper material (especially A). While I am not sure description would be enough to reproduce and no code is provided, every aspect of the architecture, if not described, if referred as similar to some previous work. There are also some notation shortcuts (not explained) in the proof of theorems that can lead to initial confusion, but they turn out to be non-ambiguous. One that could be improved is P(P_X, P_G) where one loses the fact that the second random variable is Y. This work contains plenty of novel material, which is clearly compared to previous work: - The main consequence of the use of Wasserstein distance is the surprisingly simple and useful Theorem 1. I could not verify its novelty, but this seems to be a great contribution. - Blending GAN and auto-encoders has been tried in the past, but the authors claim better theoretical foundations that lead to solutions that do not rquire min-max - The use of MMD in the context of GANs has also been tried. The authors claim that their use in the latent space makes it more practival The experiments are very convincing, both numerically and visually. Source of confusion: in algorithm 1 and 2, \tilde{z} is "sampled" from Q_TH(Z|xi), some one is lead to believe that this is the sampling process as in VAEs, while in reality Q_TH(Z|xi) is deterministic in the experiments.
This paper presents an alternative approach to constructing variational lower bounds on data log likelihood in deep, directed generative models with latent variables. Specifically, the authors propose using approximate posteriors shared across groups of examples, rather than posteriors which treat examples independently. The group-wise posteriors allow amortization of the information cost KL(group posterior || prior) across all examples in the group, which the authors liken to the "KL annealing" tricks that are sometimes used to avoid posterior collapse when training models with strong decoders p(x|z) using current techniques for approximate variational inference in deep nets. The presentation of the core idea is solid, though it did take two read-throughs before the equations really clicked for me. I think the paper could be improved by spending more time on a detailed description of the model for the Omniglot experiments (as illustrated in Figure 3). E.g., explicitly describing how group-wise and per-example posteriors are composed in this model, using Equations and pseudo-code for the main training loop, would have saved me some time. For readers less familiar with amortized variational inference in deep nets, the benefit would be larger. I appreciate that the authors developed extensions of the core method to more complex group structures, though I didn't find the related experiments particularly convincing. Overall, I like this paper and think the underlying group-wise posterior construction trick is worth exploring further. Of course, the elephant in the room is how to determine the groups across which the posteriors can be shared and their information costs amortized.
The paper studies the problem of DNN loss function design for reducing intra-class variance in the output feature space.  The key contribution is proposing an isotropic variant of the softmax loss that can balance the accuracy of classification and compactness of individual class.  The proposed loss has been compared extensively against a number of closely related approaches in methodology.  Numerical results on benchmark datasets show some improvement of the proposed loss over softmax loss and center loss (Wen et al., 2016), when applied to distance-based classifiers such as k-NN and k-means.  Pros: - The idea of isotropic normalization for enhancing compactness of class is well motivated - The paper is mostly clearly organized and presented. - Numerical study shows some promise of the proposed method. Cons: -  The novelty of method is mostly incremental given the prior work of (Wen et al., 2016) which has provided a slightly different isotropic variant of softmax loss. - The training procedure of the proposed method remains unclear in this paper.
This paper presents Defense-GAN: a GAN that used at test time to map the input generate an image (G(z)) close (in MSE(G(z), x)) to the input image (x), by applying several steps of gradient descent of this MSE.  The GAN is a WGAN trained on the train set (only to keep the generator).  The goal of the whole approach is to be robust to adversarial examples, without having to change the (downstream task) classifier, only swapping in the G(z) for the x. + The paper is easy to follow. + It seems (but I am not an expert in adversarial examples) to cite the relevant litterature (that I know of) and compare to reasonably established attacks and defenses. + Simple/directly applicable approach that seems to work experimentally,  but - A missing baseline is to take the nearest neighbour of the (perturbed) x from the training set. - Only MNIST-sized images, and MNIST-like (60k train set, 10 labels) datasets: MNIST and F-MNIST. - Between 0.043sec and 0.825 sec to reconstruct an MNIST-sized image. ? MagNet results were very often worse than no defense in Table 4,  could you comment on that? - In white-box attacks, it seems to me like L steps of gradient descent on MSE(G(z), x) should be directly extended to L steps of (at least) FGSM-based attacks, at least as a control.
The paper is not anonymized. In page 2, the first line, the authors revealed [15] is a self-citation and [15] is not anonumized in the reference list.
This paper introduces a new exploration policy for Reinforcement Learning for agents on the web called "Workflow Guided Exploration". Workflows are defined through a DSL unique to the domain. The paper is clear, very well written, and well-motivated. Exploration is still a challenging problem for RL. The workflows remind me of options though in this paper they appear to be hand-crafted. In that sense, I wonder if this has been done before in another domain. The results suggest that WGE sometimes helps but not consistently. While the experiments show that DOMNET improves over Shi et al, that could be explained as not having to train on raw pixels or not enough episodes.
This paper studies the question: Why does SGD on deep network is often successful, despite the fact that the objective induces bad local minima? The approach in this paper is to study a standard MNN with one hidden layer.  They show that in an overparametrized regime, where the number of parameters is logarithmically larger than the number of parameters in the input, the ratio between the number of (bad) local minima to the number of global minima decays exponentially.  They show this for a piecewise linear activation function, and input drawn from a standard Normal distribution.  Their improvement over previous work is that the required overparameterization is fairly moderate, and that the network that they considered is similar to ones used in practice.  This result seems interesting,  although it is clearly not sufficient to explain even the success on the setting studied in this paper,  since the number of minima of a certain type does not correspond to the probability of the SGD ending in one:  to estimate the latter, the size of each basin of attraction should be taken into account.  The authors are aware of this point and mention it as a disadvantage.  However, since this question in general is a difficult one,  any progress might be considered interesting.  Hopefully, in future work it would be possible to also bound the probability of starting in one of the basins of attraction of bad local minima. The paper is well written and well presented,  and the limitations of the approach, as well as its advantages over previous work, are clearly explained.  As I am not an expert on the previous works in this field, my judgment relies mostly on this work and its representation of previous work.  I did not verify the proofs in the appendix.
The paper takes an interesting approach to solve the existing problems of GAN training, using Coulomb potential for addressing the learning problem.  It is also well written with a clear presentation of the motivation of the problems it is trying to address, the background and proves the optimality of the suggested solution.  My understanding and validity of the proof is still an educated guess.  I have been through section A.2 , but I'm unfamiliar with the earlier literature on the similar topics so I would not be able to comment on it.  Overall, I think this is a good paper that provides a novel way of looking at and solving problems in GANs.  I just had a couple of points in the paper that I would like some clarification on :  * In section 2.2.1 : The notion of the generated a_i not disappearing is something I did not follow.  What does it mean for a generated sample to "not disappear" ?  and this directly extends to the continuity equation in (2).  * In section 1 : in the explanation of the 3rd problem that GANs exhibit i.e.  the generator not being able to generalize the distribution of the input samples, I was hoping if you could give a bit more motivation as to why this happens.  I don't think this needs to be included in the paper,  but would like to have it for a personal clarification.
Authors of this paper derived an efficient quantum-inspired learning algorithm based on a hierarchical representation that is known as tree tensor network, which is inspired by the multipartite entanglement renormalization ansatz approach where the tensors in the TN are kept to be unitary during training.  Some observations are: The limitation of learnability of TTN strongly depends on the physical indexes and the geometrical indexes determine how well the TTNs approximate the limit;  TTNs exhibit same increase level of abstractions as CNN or DBN;  Fidelity and entanglement entropy can be considered as some measurements of the network. Authors introduced the two-dimensional hierarchical tensor networks for solving image recognition problems,  which suits more the 2-D nature of images.  In section 2, authors stated that the choice of feature function is arbitrary,  and a specific feature map was introduced in Section 4.  However, it is not straightforward to connect (10) to (1) or (2).  It is better to clarify this connection  because some important parameters such as the virtual bond and input bond are related to the complexity of the proposed algorithm as well as the limitation of learnability.  For example, the scaling of the complexity O(dN_T(b_v^5 + b_i^4)) is not easy to understand.  Is it related to specific feature map?  How about the complexity of eigen-decomposition for one tensor at each iterates.  And also, whether the tricks used to accelerate the computations will affect the convergence of the algorithm?  More details on these problems are required for readers’ better understanding. From Fig 2, it is difficult to see the relationship between learnability and parameters such input bond and virtual bond  because it seems there are no clear trends in the Fig 2(a) and (b) to make any conclusion.  It is better to clarify these relationships with either clear explanation or better examples. From Fig 3, authors claimed that TN obtained the same levels of abstractions as in deep learning.  However, from Fig 3 only, it is hard to make this conclusion.  First, there are not too many differences from Fig 3(a) to Fig 3(e).   Second, there is no visualization result reported from deep learning on the same data for comparison.  Hence, it is not convincing to draw this conclusion only from Fig 3.  In Section 4.2, what strategy is used to obtain these parameters in Table 1? In Section 5, it is interesting to see more experiments in terms of fidelity and entanglement entropy.
In this paper, the authors have proposed a GAN based method to conduct data augmentation.  The cross-class transformations are mapped to a low dimensional latent space using conditional GAN.  The paper is technically sound and the novelty is significant.  The motivation of the proposed methods is clearly illustrated.  Experiments on three datasets demonstrate the advantage of the proposed framework.  However, this paper still suffers from some drawbacks as below: (1)	The illustration of the framework is not clear enough.  For example, in figure 3, it says the GAN is designed for “class c”, which is ambiguous whether the authors trained only one network for all class or trained multiple networks and each is trained on one class. (2)	Some details is not clearly given, such as the dimension of the Gaussian distribution, the dimension of the projected  noise and . (3)	The proposed method needs to sample image pairs in each class.  As far as I am concerned, in most cases sampling strategy will affect the performance to some extent.  The authors need to show the robustness to sampling strategy of the proposed method.
This paper introduces a generative approach for 3D point clouds. More specifically, two Generative Adversarial approaches are introduced: Raw point cloud GAN, and Latent-space GAN (r-GAN and l-GAN as referred to in the paper). In addition, a GMM sampling + GAN decoder approach to generation is also among the experimented variations. The results look convincing for the generation experiments in the paper, both from class-specific (Figure 1) and multi-class generators (Figure 6). The quantitative results also support the visuals. One question that arises is whether the point cloud approaches to generation is any more valuable compared to voxel-grid based approaches. Especially Octree based approaches [1-below] show very convincing and high-resolution shape generation results, whereas the details seem to be washed out for the point cloud results presented in this paper. I would like to see comparison experiments with voxel based approaches in the next update for the paper. [1] @article{tatarchenko2017octree, title={Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs}, author={Tatarchenko, Maxim and Dosovitskiy, Alexey and Brox, Thomas}, journal={arXiv preprint arXiv:1703.09438}, year={2017} }
Summary: This paper presents a derivation which links a DNN to recursive application of maximum entropy model fitting.  The mathematical notation is unclear,  and in one cases the lemmas are circular (i.e. two lemmas each assume the other is correct for their proof).  Additionally the main theorem requires complete independence,  but the second theorem provides pairwise independence,  and the two are not the same. Major comments: - The second condition of the maximum entropy equivalence theorem requires that all T are conditionally independent of Y.  This statement is unclear,  as it could mean pairwise independence, or it could mean jointly independent (i.e. for all pairs of non-overlapping subsets A & B of T I(T_A;T_B|Y) = 0). This is the same as saying the mapping X->T is making each dimension of T orthogonal, as otherwise it would introduce correlations.  The proof of the theorem assumes that pairwise independence induces joint independence  and this is not correct. - Section 4.1 makes an analogy to EM,  but gradient descent is not like this process as all the parameters are updated at once, and only optimised by a single (noisy) step.  The optimisation with respect to a single layer is conditional on all the other layers remaining fixed,  but the gradient information is stale  (as it knows about the previous step of the parameters in the layer above).  This means that gradient descent does all 1..L steps in parallel,  and this is different to the definition given. - The proofs in Appendix C which are used for the statement I(T_i;T_j) >=I(T_i;T_j|Y) are incomplete,  and in generate this statement is not true,  so requires proof. - Lemma 1 appears to assume Lemma 2, and Lemma 2 appears to assume Lemma 1. Either these lemmas are circular or the derivations of both of them are unclear. - In Lemma 3 what is the minimum taken over for the left hand side?  Elsewhere the minimum is taken over T, but T does not appear on the left hand side. Explicit minimums help the reader to follow the logic,  and implicit ones should only be used when it is obvious what the minimum is over. - In Lemma 5, what does "T is only related to X" mean?  The proof states that Y -> T -> X forms a Markov chain,  but this implies that T is a function of Y, not X. Minor comments:- I assume that the E_{P(X,Y)} notation is the expectation of that probability distribution,  but this notation is uncommon, and should be replaced with a more explicit one. - Markov is usually romanized with a "k" not a "c". - The paper is missing numerous prepositions and articles,  and contains multiple spelling mistakes & typos.
This paper proposes a deep learning (DL) approach (pre-trained CNNs) to the analysis of histopathological images for disease localization. It correctly identifies the problem that DL usually requires large image databases to provide competitive results, while annotated histopathological data repositories are costly to produce and not on that size scale. It also correctly identifies that this is a daunting task for human medical experts and therefore one that could surely benefit from the use of automated methods like the ones proposed. The study seems sound from a technical viewpoint to me and its contribution is incremental, as it builds on existing research, which is correctly identified. Results are not always too impressive, but authors seem intent on making them useful for pathogists in practice (an intention that is always worth the effort). I think the paper would benefit from a more explicit statement of its original contributions (against contextual published research) Minor issues: Revise typos (e.g. title of section 2) Please revise list of references (right now a mess in terms of format, typos, incompleteness