launch/ampere huggingface dataset:
Quality The method description, particularly about reference ambiguity, I found difficult to follow. The experiments and analysis look solid, although it would be nice to see experiments on more challenging natural image datasets. Clarity “In general this is not possible… “ - you are saying it is not possible to learn an encoder that recovers disentangled factors of variation? But that seems to be one of the main goals of the paper. It is not clear at all what is meant here or what the key problem is, which detracts from the paper’s motivation. What is the purpose of R_v and R_c in eq 2? Why can these not be collapsed into the encoders N_v and N_c? What does “different common factor” mean? What is f_c in proof of proposition 1? Previously f (no subscript) was referred to as a rendering engine. T(v,c) ~ p_v and c ~ p_c are said to be independent. But T(v,c) is explicitly defined in terms of c (equation 6). So which is correct? Overall the argument seems plausible - pairs of images in which a single factor of variation changes have a reference ambiguity - but the details are unclear. Originality The model is very similar to Mathieu et al, although using image pairs rather than category labels directly. The idea of weakly-supervised disentangling has also been explored in many other papers, e.g. “Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis”, Yang et al. The description of reference ambiguity seems new and potentially valuable, but I did not find it easy to follow. Significance Disentangling factors of variation with weak supervision is an important problem, and this paper makes a modest advance in terms of the model and potentially in terms of the theory. The analysis in figure 3 I found particularly interesting - illustrating that the encoder embedding dimension can have a drastic effect on the shortcut problem. Overall I think this can be a significant contribution if the exposition can be improved. Pros- Proposed method allows disentangling two factors of variation given a training set of image pairs with one factor of variation matching and the other non-matching. - A challenge inherent to weakly supervised disentangling called reference ambiguity is described. Cons- Only two factors of variation are studied, and the datasets are fairly simple. - The method description and the description of reference ambiguity are unclear.
This paper proposes a recurrent neural network for visual question answering.  The recurrent neural network is equipped with a carefully designed recurrent unit called MAC (Memory, Attention and Control) cell, which encourages sequential reasoning by restraining interaction between inputs and its hidden states.  The proposed model shows the state-of-the-art performance on CLEVR and CLEVR-Humans dataset, which are standard benchmarks for visual reasoning problem.  Additional experiments with limited training data shows the data efficiency of the model, which supports its strong generalization ability. The proposed model in this paper is designed with reasonable motivations and shows strong experimental results in terms of overall accuracy and the data efficiency.  However, an issue in the writing, usage of external component and lack of experimental justification of the design choices hinder the clear understanding of the proposed model. An issue in the writing Overall, the paper is well written and easy to understand,  but Section 3.2.3 (The Write Unit) has contradictory statements about their implementation.  Specifically, they proposed three different ways to update the memory (simple update, self attention and memory gate),  but it is not clear which method is used in the end. Usage of external component The proposed model uses pretrained word vectors called GloVE, which has boosted the performance on visual question answering.  This experimental setting makes fair comparison with the previous works difficult  as the pre-trained word vectors are not used for the previous works.  To isolate the strength of the proposed reasoning module, I ask to provide experiments without pretrained word vectors. Lack of experimental justification of the design choices The proposed recurrent unit contains various design choices such as separation of three different units (control unit, read unit and memory unit), attention based input processing and different memory updates stem from different motivations.  However, these design choices are not justified well  because there is neither ablation study nor visualization of internal states.  Any analysis or empirical study on these design choices is necessary to understand the characteristics of the model.  Here, I suggest to provide few visualizations of attention weights and ablation study that could support indispensability of the design choices.
This paper tried to analyze the subspaces of the adversarial examples neighborhood.  More specifically, the authors used Local Intrinsic Dimensionality to analyze the intrinsic dimensional property of the subspaces.  The characteristics and theoretical analysis of the proposed method are discussed and explained.  This paper helps others to better understand the vulnerabilities of DNNs.
This paper presents an empirical study of whether data augmentation can be a substitute for explicit regularization of weight decay and dropout. It is a well written and well organized paper. However, overall I do not find the authors’ premises and conclusions to be well supported by the results and would suggest further investigations. In particular: a) Data augmentation is a very domain specific process and limits of augmentation are often not clear. For example, in financial data or medical imaging data it is often not clear how data augmentation should be carried out and how much is too much. On the other hand model regularization is domain agnostic (has to be tuned for each task, but the methodology is consistent and well known). Thus advocating that data augmentation can universally replace explicit regularization does not seem correct. b) I find the results to be somewhat inconsistent. For example, on CIFAR-10, for 100% data regularization+augmentation is better than augmentation alone for both models, whereas for 80% data augmentation alone seems to be better. Similarly on CIFAR-100 the WRN model shows mixed trends, and this model is significantly better than the All-CNN model in performance. These results also seem inconsistent with authors statement “…and conclude that data augmentation alone - without any other explicit regularization techniques - can achieve the same performance to higher as regularized models…”
The paper consider a method for "weight normalization" of layers of a neural network.   The weight matrix is maintained normalized, which helps accuracy.   However, the simplest way to normalize on a fully connected layer is quadratic (adding squares of weights and taking square root). The paper proposes "FastNorm", which is a way to implicitly maintain the normalized weight matrix using much less computation.   Essentially, a normalization vector is maintained an updated separately. Pros:   Natural method to do weight normalization efficeintly Cons:   A very natural and simple solution that is fairly obvious. Limited experiments
Paper presents an interesting attention mechanism for fine-grained image classification. Introduction states that the method is simple and easy to understand. However, the presentation of the method is bit harder to follow. It is not clear to me if the attention modules are applied over all  pooling layers. How they are combined? Why use cross -correlation as the regulariser? Why not much stronger constraint such as orthogonality over elements of M in equation 1? What is the impact of this regularisation? Why use soft-max in equation 1? One may use a Sigmoid as well? Is it better to use soft-max? Equation 9 is not entirely clear to me. Undefined notations. In Table 2, why stop from AD= 2 and AW=2? What is the performance of AD=1, AW=1 with G? Why not perform this experiment over all 5 datasets? Is this performances, dataset specific? The method is compared against 5 datasets. Obtained results are quite good.
This paper considers the problem of improving sequence generation by learning better metrics.  Specifically, it focuses on addressing the exposure bias problem, where traditional methods such as SeqGAN uses GAN framework and reinforcement learning.  Different from these work, this paper does not use GAN framework.  Instead, it proposed an expert-based reward function training, which trains the reward function (the discriminator) from data that are generated by randomly modifying parts of the expert trajectories.  Furthermore, it also introduces partial reward function that measures the quality of the subsequences of different lengths in the generated data.  This is similar to the idea of hierarchical RL, which divide the problem into potential subtasks, which could alleviate the difficulty of reinforcement learning from sparse rewards.  The idea of the paper is novel.  However, there are a few points to be clarified. In Section 3.2 and in (4) and (5), the authors explains how the action value Q_{D_i} is modeled and estimated for the partial reward function D_i of length L_{D_i}.  But the authors do not explain how the rewards (or action value functions) of different lengths are aggregated together to update the model using policy gradient.  Is it a simple sum of all of them? It is not clear why the future subsequences that do not contain y_{t+1} are ignored for estimating the action value function Q in (4) and (5).  The authors stated that it is for reducing the computation complexity.  But it is not clear why specifically dropping the sequences that do not contain y_{t+1}.  Please clarify more on this point.
The authors propose an approach to dynamically generating filters in a CNN based on the input image.  The filters are generated as linear combinations of a basis set of filters, based on features extracted by an auto-encoder.  The authors test the approach on recognition tasks on three datasets: MNIST, MTFL (facial landmarks) and CIFAR10, and show a small improvement over baselines without dynamic filters. Pros: 1) I have not seen this exact approach proposed before. 2) There method is evaluated on three datasets and two tasks: classification and facial landmark detection. Cons: 1) The authors are not the first to propose dynamically generating filters,  and they clearly mention that the work of De Brabandere et al. is closely related.  Yet, there is no comparison to other methods for dynamic weight generation.  2) Related to that, there is no ablation study,  so it is unclear if the authors’ contributions are useful.  I appreciate the analysis in Tables 1 and 2,  but this is not sufficient.  Why the need for the autoencoder - why can’t the whole network be trained end-to-end on the goal task?  Why generate filters as linear combination - is this just for computational reasons, or also accuracy?  This should be analyzed empirically. 3) The experiments are somewhat substandard: - On MNIST the authors use a tiny poorly-performance network,  and it is no surprise that one can beat it with a bigger dynamic filter network. - The MTFL experiments look most convincing  (although this might be because I am not familiar with SoTA on the dataset),  but still there is no control for the number of parameters,  and the performance improvements are not huge - On CIFAR10 - there is a marginal improvement in performance,  which, as the authors admit, can also be reached by using a deeper model.  The baseline models are far from SoTA  - the authors should look at more modern architecture such as AllCNN (not particularly new or good, but very simple), ResNet, wide ResNet, DenseNet, etc. As a comment, I don’t think classification is a good task for showcasing such an architecture  - classification is already working extremely well.  Many other tasks - for instance, detection, tracking, few-shot learning - seem much more promising. To conclude, the authors propose a new approach to learning convolutional networks with dynamic input-conditioned filters.  Unfortunately, the authors fail to demonstrate the value of the proposed method.  I therefore recommend rejection.
This work proposes an approach for transcription factor binding site prediction using a multi-label classification formulation.  It is a very interesting problem  and application and the approach is interesting.  Novelty: The method is quite similar to matching networks (Vinyals, 2016) with a few changes in the matching approach.  As such, in order to establish its broader applicability there should be additional evaluation on other benchmark datasets.  The MNIST performance comparison is inadequate  and there are other papers that do better on it.  They should clearly list what the contributions are w.r.t to the work by Vinyals et al 2016. They should also cite works that learn embeddings in a multi-label setting such as StarSpace. Impact: In its current form the paper seems to be most relevant to the computational biology / TFBS community.  However, there is no comparison to the exact networks used in the prior works DeepBind/DeepSea/DanQ/Basset/DeepLift or bidirectional LSTMs.  Further there is no comparison to existing one-shot learning techniques either.  This greatly limits the impact of the work. For biological impact, a comparison to any of the motif learning approaches that are popular in the biology/comp-bio community will help (for instance, HOMER, FIMO). Cons: The authors claim they can learn TF-TF interactions and it is one of the main biological contributions,  but there is no evidence of why  (beyond very preliminary evaluation using the Trrust database).  Their examples are 200-bp long which does not mean that all TFs binding in that window are involved in cooperative binding.  The prototype loss is too simplistic to capture co-binding tendencies  and the combinationLSTM is not well motivated.  One interesting source of information they could tap into for TF-TF interactions is CAP-SELEX (Jolma et al, Nature 2015). One of the main drawbacks is the lack of interpretability of their model where approaches like DanQ/DeepLift etc benefit.  The PWM-like filters in some of the prior works help understand what type of sequence properties contribute to binding events.  Can their model lead to an understanding of this sort? Evaluation: The empirical evaluation itself is not very strong  as there are only modest improvements over simple baselines.  Further there are no error-bars etc to indicate the variance in their performance numbers. It will be useful to have a TF-level performance split-up to get an idea of which TFs benefit most. Clarity: The paper can benefit from more clarity in the technical aspects.  It is hard to follow for anyone not already familiar with matching networks.  The objective function, parameters need to be clearly introduced in one place.  For instance, what is y_i in their multi-label framework? Various choices are not well motivated; for instance cosine similarity, the value of hyperparameter epsilon. The prototype vectors are not motif-like at all --  can the authors motivate this aspect better?
The authors are motivated by two problems: Inputting non-Euclidean data (such as graphs) into deep CNNs, and analyzing optimization properties of deep networks. In particular, they look at the problem of maze testing, where, given a grid of black and white pixels, the goal is to answer whether there is a path from a designated starting point to an ending point. They choose to analyze mazes because they have many nice statistical properties from percolation theory. For one, the problem is solvable with breadth first search in O(L^2) time, for an L x L maze. They show that a CNN can essentially encode a BFS, so theoretically a CNN should be able to solve the problem. Their architecture is a deep feedforward network where each layer takes as input two images: one corresponding to the original maze (a skip connection), and the output of the previous layer. Layers alternate between convolutional and sigmoidal. The authors discuss how this architecture can solve the problem exactly. The pictorial explanation for how the CNN can mimic BFS is interesting but I got a little lost in the 3 cases on page 4. For example, what is r? And what is the relation of the black/white and orange squares? I thought this could use a little more clarity. Though experiments, they show that there are two kinds of minima, depending on whether we allow negative initializations in the convolution kernels. When positive initializations are enforced, the network can more or less mimic the BFS behavior, but never when initializations can be negative. They offer a rigorous analysis into the behavior of optimization in each of these cases, concluding that there is an essential singularity in the cost function around the exact solution, yet learning succumbs to poor optima due to poor initial predictions in training. I thought this was an impressive paper that looked at theoretical properties of CNNs. The problem was very well-motivated, and the analysis was sharp and offered interesting insights into the problem of maze solving. What I thought was especially interesting is how their analysis can be extended to other graph problems; while their analysis was specific to the problem of maze solving, they offer an approach -- e.g. that of finding "bugs" when dealing with graph objects -- that can extend to other problems. I would be excited to see similar analysis of other toy problems involving graphs. One complaint I had was inconsistent clarity: while a lot was well-motivated and straightforward to understand, I got lost in some of the details (as an example, the figure on page 4 did not initially make much sense to me). Also, in the experiments, the authors mention multiple attempt with the same settings -- are these experiments differentiated only by their initialization? Finally, there were various typos throughout (one example is "neglect minimua" on page 2 should be "neglect minima"). Pros: Rigorous analysis, well motivated problem, generalizable results to deep learning theory Cons: Clarity
This paper describes an extension to the recently introduced Transformer networks which shows better convergence properties and also improves results on standard machine translation benchmarks.  This is a great paper  -- it introduces a relatively simple extension of Transformer networks which only adds very few parameters and speeds up convergence and achieves better results.  It would have been good to also add a motivation for doing this  (for example, this idea can be interpreted as having a variable number of attention heads which can be blended in and out with a single learned parameter, hence making it easier to use the parameters where they are needed).  Also, it would be interesting to see how important the concatenation weight and the addition weight are relative to each other --  do you possibly get the same results even without the concatenation weight?  A suggested improvement: Please check the references in the introduction and see if you can find earlier ones --  for example, language modeling with RNNs has been done for a very long time, not just since 2017 which are the ones you list;  similar for speech recognition etc. (which probably has been done since 1993!).
The paper proposes an analysis on different adaptive regularization techniques for deep transfer learning.  Specifically it focuses on the use of an L2-SP condition that constraints the new parameters to be close to the ones previously learned when solving a source task.  + The paper is easy to read and well organized + The advantage of the proposed regularization against the more standard L2 regularization is clearly visible from the experiments - The idea per se is not new:  there is a list of shallow learning methods for transfer learning based on the same L2 regularization choice [Cross-Domain Video Concept Detection using Adaptive SVMs, ACM Multimedia 2007] [Learning categories from few examples with multi model knowledge transfer, PAMI 2014] [From n to n+ 1: Multiclass transfer incremental learning, CVPR 2013] I believe this literature should be discussed in the related work section - It is true that the L2-SP-Fisher regularization was designed for life-long learning cases with a fixed task,  however, this solution seems to work quite well in the proposed experimental settings.  From my understanding L2-SP-Fisher can be considered the best competitor of L2-SP  so I think the paper should dedicate more space to the analysis of their difference and similarities both from the theoretical and experimental point of view.  For instance: --  adding the L2-SP-Fisher results in table 2 --  repeating the experiments of figure 2 and figure 3 with L2-SP-Fisher
This paper essentially uses CycleGANs for Domain Adaptation. My biggest concern is that it doesn't adequately compare to similar papers that perform adaptation at the pixel level (eg. Shrivastava et al-'Learning from Simulated and Unsupervised Images through Adversarial Training' and Bousmalis et al - 'Unsupervised Pixel-level Domain Adaptation with GANs', two similar papers published in CVPR 2017 -the first one was even a best paper- and available on arXiv since December 2016-before CycleGANs). I believe the authors should have at least done an ablation study to see if they cycle-consistency loss truly makes a difference on top of these works-that would be the biggest selling point of this paper. The experimental section had many experiments, which is great. However I think for semantic segmentation it would be very interesting to see whether using the adapted synthetic GTA5 samples would improve the SOTA on Cityscapes. It wouldn't be unsupervised domain adaptation, but it would be very impactful. Finally I'm not sure the oracle (train on target) mIoU on Table 2 is SOTA, and I believe the proposed model's performance is really far from SOTA. Pros: * CycleGANs for domain adaptation! Great idea! * I really like the work on semantic segmentation, I think this is a very important direction Cons: * I don't think Domain separation networks is a pixel-level transformation- that's a feature-level transformation, you probably mean to use Bousmalis et al. 2017. Also Shrivastava et al is missing from the image-level papers. * the authors claim that Bousmalis et al, Liu & Tuzel and Shrivastava et al ahve only been shown to work for small image sizes. There's a recent work by Bousmalis et al. (Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping) that shows these methods working well (w/o cycle-consistency) for settings similar to semantic segmentation at a relatively high resolution. Also it was mentioned that these methods do not necessarily preserve content, when pixel-da explicitly accounts for that with a task loss (identical to the semantic loss used in this submission) * The authors talk about the content similarity loss on the foreground in Bousmalis et al. 2017, but they could compare to this method w/o using the content similarity or using a different content similarity tailored to the semantic segmentation tasks, which would be trivial. * Math seems wrong in (4) and (6). (4) should be probably have a minus instead of a plus. (6) has an argmin of a min, not sure what is being optimized here. In fact, I'm not sure if eg you use the gradients of f_T for training the generators? * The authors mention that the pixel-da approach cross validates with some labeled data. Although I agree that is not an ideal validation, I'm not sure if it's equivalent or not the authors' validation setting, as they don't describe what that is. * The authors present the semantic loss as novel, however this is the task loss proposed by the pixel-da paper. * I didn't understand what pixel-only and feat-only meant in tables 2, 3, 4. I couldn't find an explanation in captions or in text