launch/ampere huggingface dataset:
This paper presents, and analyzes, a method for learning word relationships based on co-occurrence.   In the method, relationships between pairs of words (A, B) are represented by the terms that tend to occur around co-mentions of A and B in text.   The paper shows the start of some interesting ideas,  but needs revisions and much more extensive experiments. On the plus side, the method proposed here does perform relatively well (Table 1) and probably merits further investigation.   The experiments in Table 1 can only be considered preliminary, however.   They only evaluate over a small number of relationships (three)  -- looking at 20 or so different relationships would greatly improve confidence in the conclusions. Beyond Table 1 the paper makes a number of claims that are not supported or weakly supported (the paper uses only a handful of examples as evidence).   An attempt to explain what Word2Vec is doing should be made with careful experiments over many relations and hundreds of examples,  whereas this paper presents only a handful of examples for most of its claims.   Further, whether the behavior of the proposed algorithm actually reflects what word2vec is doing is left as a significant open question. I appreciate the clarity of Assumption 1 and Proposition 1,  but ultimately this formalism is not used  and because Assumption 1 about which nouns are "semantically related" to which other nouns attempts to trivialize a complex notion (semantics) and is clearly way too strong  -- the paper would be better off without it.  Also Assumption 1 does not actually claim what the text says it claims  (the text says words outside the window are *not* semantically related, but the assumption does not actually say this)  and furthermore is soon discarded and only the frequency of noun occurrences around co-mentions is used.   I think the description of the algorithm could be retained without including Assumption 1. minor: References to numbered algorithms or assumptions should be capitalized in the text. what the introduction means about the "dynamics" of the vector equation is a little unclear A submission shouldn't have acknowledgments, and in particular with names that undermine anonymity MLE has a particular technical meaning that is not utilized here,  I would just refer to the most frequent words as "most related nouns" or similar In Table 1, are the "same dataset" results with w2v for the nouns-only corpus, or with all the other words? The argument made assuming a perfect Zipf distribution (with exponent equal to one) should be made with data. will likely by observed -> will likely be observed lions:dolphins probably ends up that way because of "sea lions" Table 4 caption: frequencies -> currencies Table 2 -- claim is that improvements from k=10 to k=20 are 'nominal' but they look non-negligible to me I did not understand how POS lying in the same subspace means that Vec(D) has to be in the span of Vecs A-C.
The authors use deep learning to learn a surrogate model for the motion vector in the advection-diffusion equation that they use to forecast sea surface temperature. In particular, they use a CNN encoder-decoder to learn a motion field, and a warping function from the last component to provide forecasting. I like the idea of using deep learning for physical equations. I would like to see a description of the algorithm with the pseudo-code in order to understand the flow of the method. I got confused at several points because it was not clear what was exactly being estimated with the CNN. Having an algorithmic environment would make the description easier. I know that authors are going to publish the code, but this is not enough at this point of the revision. Physical processes in Machine learning have been studied from the perspective of Gaussian processes. Just to mention a couple of references “Linear latent force models using Gaussian processes” and "Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations" In Theorem 2, do you need to care about boundary conditions for your equation? I didn’t see any mention to those in the definition for I(x,t). You only mention initial conditions. How do you estimate the diffusion parameter D? Are you assuming isotropic diffusion? Is that realistic? Can you provide more details about how you run the data assimilation model in the experiments? Did you use your own code?
The approach solves an important problem  as getting labelled data is hard.  The focus is on the key aspect, which is generalisation across heteregeneous data.  The novel idea is the dataset embedding  so that their RL policy can be trained to work across diverse datasets. Pros: 1. The approach performs well against all the baselines, and also achieves good cross-task generalisation in the tasks they evaluated on.  2. In particular, they alsoevaluated on test datasets with fairly different statistics from the training datasets, which isnt very common in most meta-learning papers today,  so it’s encouraging that the method works in that regime. Cons: 1. The embedding strategy, especially the representative and discriminative histograms, is complicated.  It is unclear if the strategy is general enough to work on harder problems / larger datasets, or with higher dimensional data like images.  More evidence in the paper for why it would work on harder problems would be great.  2. The policy network would have to output a probability for each datapoint in the dataset U,  which could be fairly large,  thus the method is computationally much more expensive than random sampling.  A section devoted to showing what practical problems could be potentially solved by this method would be useful. 3. It is unclear to me if the results in table 3 and 4 are achieved by retraining from scratch with an RBF SVM, or by freezing the policy network trained on a linear SVM and directly evaluating it with a RBF SVM base learner. Significance/Conclusion: The idea of meta-learning or learning to learn is fairly common now.  While they do show good performance,  it’s unclear if the specific embedding strategy suggested in this paper will generalise to harder tasks.  Comments: There’s lots of typos,  please proof read to improve the paper.
The authors present a derivation of previous work of [1]. In particular they propose the method of using the error signal of a dynamics model as curiosity for exploration, such as [1], but without any additionaly auxiliary methods. This the author call Curiosity by Bootstrapping Feature (CBF).\n \nIn particular they show over  a set of auxiliary learning methods (hindsight ER, inverse dynamics model[1]) there is\nnot a clear cut edge one method has over the other (or over using no auxilirary method all, that is CBF).\n\n Overall I think the novelty is too limited for acceptance. The main point of the authors (heterogeneous results\nover different auxilirary learning methods),  is not suprising at all, and to be expected. The method the authors introduce\nis just a submodule of already published results[1].\n\n For instance, section 4 discusses challenges related to these class of approaches such as the presence of stochasticity. Had the authors proposed a solution to these challenges that would have benefited the paper greatly.\n\n Minor: The light green link color make the paper hard on the eye, I suggest using [hidelinks] for hyperref.\n Figure 2 is  very small and hard to read.\n\n\n [1] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by\nself-supervised prediction. In ICML, 2017
The paper is clear and well written. It is an incremental modification of prior work (ResNeXt) that performs better on several experiments selected by the author;  comparisons are only included relative to ResNeXt. This paper is not about gating (c.f., gates in LSTMs, mixture of experts, etc) but rather about masking or perhaps a kind of block sparsity,  as the "gates" of the paper do not depend upon the input:  they are just fixed masking matrices (see eq (2)). The main contribution appears to be the optimisation procedure for the binary masking tensor g.  But this procedure is not justified:  does each step minimise the loss?  This seems unlikely due to the sampling.  Can the authors show that the procedure will always converge?  It would be good to contrast this with other attempts to learn discrete random variables  (for example, The Concrete Distribution: Continuous Relaxation of Continuous Random Variables, Maddison et al, ICLR 2017).
In this work, the objective is to analyze the robustness of a neural network to any sort of attack. This is measured by naturally linking the robustness of the network to the local Lipschitz properties of the network function.  This approach is quite standard in learning theory,  I am not aware of how original this point of view is within the deep learning community. This is estimated by obtaining values of the norm of the gradient (also naturally linked to the Lipschitz properties of the function) by backpropagation.  This is again a natural idea.
The authors proposed to supplement adversarial training with an additional regularization that forces the embeddings of clean and adversarial inputs to be similar. The authors demonstrate on MNIST and CIFAR that the added regularization leads to more robustness to various kinds of attacks. The authors further propose to enhance the network with cascaded adversarial training, that is, learning against iteratively generated adversarial inputs, and showed improved performance against harder attacks. The idea proposed is fairly straight-forward. Despite being a simple approach, the experimental results are quite promising. The analysis on the gradient correlation coefficient and label leaking phenomenon provide some interesting insights. As pointed out in section 4.2, increasing the regularization coefficient leads to degenerated embeddings. Have the authors consider distance metrics that are less sensitive to the magnitude of the embeddings, for example, normalizing the inputs before sending it to the bidirectional or pivot loss, or use cosine distance etc.? Table 4 and 5 seem to suggest that cascaded adversarial learning have more negative impact on test set with one-step attacks than clean test set, which is a bit counter-intuitive. Do the authors have any insight on this? Comments: 1. The writing of the paper could be improved. For example, "Transferability analysis" in section 1 is barely understandable; 2. Arrow in Figure 3 are not quite readable; 3. The paper is over 11 pages. The authors might want to consider shrink it down the recommended length.
The authors propose a mechanism for learning task-specific region embeddings for use in text classification.  Specifically, this comprises a standard word embedding an accompanying local context embedding.  The key idea here is the introduction of a (h x c x v) tensor K, where h is the embedding dim (same as the word embedding size), c is a fixed window size around a target word, and v is the vocabulary size.  Each word in v is then associated with an (h x c) matrix that is meant to encode how it affects nearby words,  in particular this may be viewed as parameterizing a projection to be applied to surrounding word embeddings.  The authors propose two specific variants of this approach, which combine the K matrix and constituent word embeddings (in a given region) in different ways.  Region embeddings are then composed (summed) and fed through a standard model.  Strong points--- + The proposed approach is simple and largely intuitive:  essentially the context matrix allows word-specific contextualization.  Further, the work is clearly presented. + At the very least the model does seem comparable in performance to various recent methods (as per Table 2),  however as noted below the gains are marginal  and I have some questions on the setup. + The authors perform ablation experiments,  which are always nice to see.  Weak points--- - I have a critical question for clarification in the experiments.  The authors write 'Optimal hyperparameters are tuned with 10% of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets'  -- is this true for *all* models, or only the proposed approach?  - The gains here appear to be consistent,  but they seem marginal.  The biggest gain achieved over all datasets is apparently .7,  and most of the time the model very narrowly performs better (.2-.4 range).  Moreoever, it is not clear if these results are averaged over multiple runs of SGD or not  (variation due to initialization and stochastic estimation can account for up to 1 point in variance  -- see "A sensitivity analysis of (and practitioners guide to) CNNs..." Zhang and Wallace, 2015.) - The related work section seems light.  For instance, there is no discussion at all of LSTMs and their application to text classificatio (e.g., Tang et al., EMNLP 2015)  -- although it is noted that the authors do compare against D-LSTM,  or char-level CNNs for the same (see Zhang et al., NIPs 2015).  Other relevant work not discussed includes Iyyer et al. (ACL 2015).  In their respective ways, these papers address some of the same issues the authors consider here.  - The two approaches to inducing the final region embedding (word-context and then context-word in sections 3.2 and 3.3, respectively) feel a bit ad-hoc.  I would have appreciated more intuition behind these approaches.  Small comments---There is a typo in Figure 4 -- "Howerver" should be "However"
General comment ==============  Low-rank decomposing convolutional filters has been used to speedup convolutional networks at the cost of a drop in prediction performance. The authors a) extended existing decomposition techniques by an iterative method for decomposition and fine-tuning convolutional filter weights, and b) and algorithm to determine the rank of each convolutional filter. The authors show that their method enables a higher speedup and lower accuracy drop than existing methods when applied to VGG16. The proposed method is a useful extension of existing methods but needs to evaluated more rigorously. The manuscript is hard to read due to unclear descriptions and grammatical errors. Major comments ============= 1. The authors authors showed that their method enables a higher speedup and lower drop in accuracy than existing methods when applied to VGG16. The authors should analyze if this also holds true for ResNet and Inception, which are more widely used than VGG16. 2. The authors measured the actual speedup on a single CPU (Intel Core i5). The authors should measure the actual speedup also on a single GPU. 3. It is unclear how the actual speedup was measured. Does it correspond to the seconds per update step or the overall training time? In the latter case, how long were models trained? 4. How and which hyper-parameters were optimized? The authors should use the same hyper-parameters for all methods (Jaderberg, Zhang, Rank selection). The authors should also analyze the sensitivity of speedup and accuracy drop depending on the learning rate for ‘Rank selection’. 5. Figure 4: the authors should show the same plot for more convolutional layers at varying depth from both VGG and ResNet. 6. The manuscript is hard to understand and not written clearly enough. In the abstract, what does ‘two-pass decomposition’, ‘proper ranks’, ‘the instability problem’, or ‘systematic’ mean? What are ‘edge devices’, ‘vanilla parameters’? The authors should also avoid uninformative adjectives, clutter, and vague terms throughout the manuscript such as ‘vital importance’ or ‘little room for fine-tuning’. Minor comments ============= 1. The authors should use ‘significantly’ only if a statistical hypothesis was performed. 2. The manuscript contains several typos and grammatical flaws, e.g. ‘have been widely applied to have the breakthrough’, ‘The CP decomposition factorizes the tensors into a sum of series rank-one tensors.’, ‘Our two-pass decomposition provides the better result as compared with the original CP decomposition’. 3. For clarity, the authors should express equation 5 in terms of Y_1, Y_2, Y_3, and Y_4. 4. Equation 2, bottom: C_in, W_f, H_f, and C_out are undefined at this point.
In this paper, the authors studied the problem of semi-supervised few-shot classification, by extending the prototypical networks into the setting of semi-supervised learning with examples from distractor classes.   The studied problem is interesting,  and the paper is well-written.  Extensive experiments are performed to demonstrate the effectiveness of the proposed methods.   While the proposed method is a natural extension of the existing works (i.e., soft k-means and meta-learning). On top of that, It seems the authors have over-claimed their model capability at the first place  as the proposed model cannot properly classify the distractor examples but just only consider them as a single class of outliers.  Overall, I would like to vote for a weakly acceptance regarding this paper.
This paper proposes to use neural network and gradient descent to automatically design for engineering tasks. It uses two networks, parameterization network and prediction network to model the mapping from design parameters to fitness. It uses back propagation (gradient descent) to improve the design. The method is evaluated on heat sink design and airfoil design. This paper targets at a potentially very useful application of neural networks that can have real world impacts. However, I have three main concerns: 1) Presentation. The organization of the paper could be improved. It mixes the method, the heat sink example and the airfoil example throughout the entire paper. Sometimes I am very confused about what is being described. My suggestion would be to completely separate these three parts: present a general method first, then use heat sink as the first experiment and airfoil as the second experiment. This organization would make the writing much clearer. 2) In the paragraph above Section 4.1, the paper made two arguments. I might be wrong, but I do not agree with either of them in general. First of all, "neural networks are good at generalizing to examples outside their train set". This depends entirely on whether the sample distribution of training and testing are similar and whether you have enough training examples that cover important sample space. This is especially critical if a deep neural network is used since overfitting is a real issue. Second, "it is easy to imagine a hybrid system where a network is trained on a simulation and fine tuned ...". Implementing such a hybrid system is nontrivial due to the reality gap. There is an entire research field about closing the reality gap and transfer learning. So I am not convinced by these two arguments made by this paper. They might be true for a narrow field of application. But in general, I think they are not quite correct. 3) The key of this paper is to approximate the dynamics using neural network (which is a continuous mapping) and take advantage of its gradient computation. However, many of dynamic systems are inherently discontinuous (collision/contact dynamics) or chaotic (turbulent flow). In those scenarios, the proposed method might not work well and we may have to resort to the gradient free methods. It seems that the proposed method works well for heat sink problem and the steady flow around airfoil, both of which do not fall into the more complex physics regime. It would be great that the paper could be more explicit about its limitations. In summary, I like the idea, the application and the result of this paper. The writing could be improved. But more importantly, I think that the proposed method has its limitation about what kind of physical systems it can model. These limitation should be discussed more explicitly and more thoroughly.
This paper proposes the concept of optimal representation space and suggests that a model should be evaluated in its optimal representation space to get good performance. It could be a good idea if this paper could suggest some ways to find the optimal representation space in general, instead of just showing two cases. It is disappointing, because this paper is named as "finding optimal representation spaces ...". In addition, one of the contributions claimed in this paper is about introducing the "formalism" of an optimal representation space. However, I didn't see any formal definition of this concept or theoretical justification. About FastSent or any other log-linear model, the reason that dot product (or cosine similarity) is a good metric is because the model is trained to optimize the dot product, as shown in equation 5 --- I think this simple fact is missed in this paper. The experimental results are not convincing, because I didn't find any consistent pattern that shows the performance is getting better once we evaluated the model in its optimal representation space. There are statements in this paper that I didn't agree with 1) Distributional hypothesis from Harris (1954) is about words not sentences. 2) Not sure the following line makes sense: "However, these unsupervised tasks are more interesting from a general AI point of view, as they test whether the machine truly understands the human notion of similarity, without being explicitly told what is similar"
The paper proposes a deep learning framework called DeePa that supports multiple dimensions of parallelism in computation to accelerate training of convolutional neural networks. Whereas the majority of work on parallel or distributed deep learning partitions training over bootstrap samples of training data (called image parallelism in the paper), DeePa is able to additionally partition the operations over image height, width and channel. This gives more options to parallelize different parts of the neural network. For example, the best DeePa configurations studied in the paper for AlexNet, VGG-16, and Inception-v3 typically use image parallelism for the initial layers, reduce GPU utilization for the deeper layers to reduce data transfer overhead, and use model parallelism on a smaller number of GPUs for fully connected layers. The net is that DeePa allows such configurations to be created that provide an increase in training throughput and lower data transfer in practice for training these networks. These configurations for parellism are not easily programmed in other frameworks like TensorFlow and PyTorch. The paper can potentially be improved in a few ways. One is to explore more demanding training workloads that require larger-scale distribution and parallelism. The ImageNet 22-K would be a good example and would really highlight the benefits of the DeePa in practice. Beyond that, more complex workloads like 3D CNNs for video modeling would also provide a strong motivation for having multiple dimensions of the data for partitioning operations.