launch/ampere huggingface dataset:
SUMMARY. The paper presents an extension of word2vec for structured features. The authors introduced a new compatibility function between features and, as in the skipgram approach, they propose a variation of negative sampling to deal with structured features. The learned representation of features is tested on a recommendation-like task.  ---------- OVERALL JUDGMENT The paper is not clear  and thus I am not sure what I can learn from it. From what is written on the paper I have trouble to understand the definition of the model the authors propose and also an actual NLP task where the representation induced by the model can be useful. For this reason, I would suggest the authors make clear with a more formal notation, and the use of examples, what the model is supposed to achieve. ---------- DETAILED COMMENTS When the authors refer to word2vec is not clear if they are referring to skipgram or cbow algorithm,  please make it clear. Bottom of page one: "a positive example is 'semantic'",  please, use another expression to describe observable examples,  'semantic' does not make sense in this context. Levi and Goldberg (2014)  do not say anything about factorization machines,  could the authors clarify this point? Equation (4), what do i and j stand for?  what does \beta represent?  is it the embedding vector?  How is this formula related to skipgram or cbow? The introduction of structured deep-in factorization machine should be more clear with examples that give the intuition on the rationale of the model. The experimental section is rather poor,  first, the authors only compare themselves with word2ve (cbow),  it is not clear what the reader should learn from the results the authors got. Finally, the most striking flaw of this paper is the lack of references to previous works on word embeddings and feature representation,  I would suggest the author check and compare themselves with previous work on this topic.
Twin Networks: Using the Future as a Regularizer ** PAPER SUMMARY ** The authors propose to regularize RNN for sequence prediction by forcing states of the main forward RNN to match the state of a secondary backward RNN. Both RNNs are trained jointly and only the forward model is used at test time. Experiments on conditional generation (speech recognition, image captioning), and unconditional generation (MNIST pixel RNN, language models) show the effectiveness of the regularizer. ** REVIEW SUMMARY ** The paper reads well, has sufficient reference. The idea is simple and well explained. Positive empirial results support the proposed regularizer. ** DETAILED REVIEW ** Overall, this is a good paper. I have a few suggestions along the text but nothing major. In related work, I would cite co-training approaches. In effect, you have two view of a point in time, its past and its future and you force these two views to agree, see  (Blum and Mitchell, 1998) or Xu, Chang, Dacheng Tao, and Chao Xu. "A survey on multi-view learning." arXiv preprint arXiv:1304.5634 (2013). I would also relate your work to distillation/model compression which tries to get one network to behave like another. On that point, is it important to train the forward and backward network jointly or could the backward network be pre-trained? In section 2, it is not obvious to me that the regularizer (4) would not be ignored in absence of regularization on the output matrix. I mean, the regularizer could push h^b to small norm, compensating with higher norm for the output word embeddings. Could you comment why this would not happen? In Section 4.2, you need to refer to Table 2 in the text. You also need to define the evaluation metrics used. In this section, why are you not reporting the results from the original Show&Tell paper? How does your implementation compare to the original work? On unconditional generation, your hypothesis on uncertainty is interesting and could be tested. You could inject uncertainty in the captioning task for instance, e.g. consider that multiple version of each word e.g. dogA, dogB, docC which are alternatively used instead of dog with predefined substitution rates. Would your regularizer still be helpful there? At which point would it break?
The authors purpose a method for creating mini batches for a student network by using a second learned representation space to dynamically selecting  examples by their 'easiness and true diverseness'.  The framework is detailed  and results on MNIST, cifar10 and fashion-MNIST are presented.  The work presented is novel but there are some notable omissions:   - there are no specific numbers presented to back up the improvement claims;  graphs are presented but not specific numeric results - there is limited discussion of the computational cost of the framework presented  - there is no comparison to a baseline in which the additional learning cycles used for learning the embedding are used for training the student model. - only small data sets are evaluated.  This is unfortunate  because if there are to be large gains from this approach, it seems that they are more likely to be found in the domain of large scale problems, than toy data sets like mnist.
This paper proposed a new optimization framework for semi-supervised learning based on derived inversion scheme for deep neural networks.  The numerical experiments show a significant improvement in accuracy of the approach.
The authors present an algorithm for training ensembles of policy networks that regularly mixes different policies in the ensemble together by distilling a mixture of two policies into a single policy network, adding it to the ensemble and selecting the strongest networks to remain (under certain definitions of a "strong" network).  The experiments compare favorably against PPO and A2C baselines on a variety of MuJoCo tasks,  although I would appreciate a wall-time comparison as well,  as training the "crossover" network is presumably time-consuming. It seems that for much of the paper, the authors could dispense with the genetic terminology altogether - and I mean that as a compliment.  There are few if any valuable ideas in the field of evolutionary computing  and I am glad to see the authors use sensible gradient-based learning for GPO, even if it makes it depart from what many in the field would consider "evolutionary" computing.  Another point on terminology that is important to emphasize - the method for training the crossover network by direct supervised learning from expert trajectories is technically not imitation learning but behavioral cloning.  I would perhaps even call this a distillation network rather than a crossover network.  In many robotics tasks behavioral cloning is known for overfitting to expert trajectories,  but that may not be a problem in this setting as "expert" trajectories can be generated in unlimited quantities.
The authors report a number of experiments using off-the-shelf sentence embedding methods for performing extractive summarisation, using a number of simple methods for choosing the extracted sentences.  Unfortunately the contribution is too minor, and the work too incremental, to be worthy of a place at a top-tier international conference such as ICLR.  The overall presentation is also below the required standard.  The work would be better suited for a focused summarisation workshop,  where there would be more interest from the participants. Some of the statements motivating the work are questionable.  I don't know if sentence vectors *in particular* have been especially successful in recent NLP (unless we count neural MT with attention as using "sentence vectors").  It's also not the case that the sentence reordering and text simplification problems have been solved, as is suggested on p.2.  The most effective method is a simple greedy technique.  I'm not sure I'd describe this as being "based on fundamental principles of vector semantics" (p.4). The citations often have the authors mentioned twice. The reference to "making or breaking applications" in the conclusion strikes me as premature to say the least.
This paper proposes to bring together multiple inductive biases that hope to correct for inconsistencies in sequence decoding.  Building on previous works that utilize modified objectives to generate sequences, this work proposes to optimize for the parameters of a pre-defined combination of various sub-objectives.  The human evaluation is straight-forward and meaningful to compensate for the well-known inaccuracies of automatic evaluation.  While the paper points out that they introduce multiple inductive biases that are useful to produce human-like sentences,  it is not entirely correct that the objective is being learnt as claimed in portions of the paper.  I would like this point to be clarified better in the paper.  I think showing results on grounded generation tasks like machine translation or image-captioning would make a stronger case for evaluating relevance.  I would like to see comparisons on these tasks.
This paper suggests a simple yet effective approach for learning with weak supervision.  This learning scenario involves two datasets, one with clean data (i.e., labeled by the true function) and one with noisy data, collected using a weak source of supervision.   The suggested approach assumes a teacher and student networks,  and builds the final representation incrementally, by taking into account the "fidelity" of the weak label when training the student at the final step.  The fidelity score is given by the teacher, after being trained over the clean data,  and it's used to build a cost-sensitive loss function for the students.  The suggested method seems to work well on several document classification tasks.  Overall, I liked the paper.   I would like the authors to consider the following questions - - Over the last 10 years or so, many different frameworks for learning with weak supervision were suggested (e.g., indirect supervision, distant supervision, response-based, constraint-based, to name a few).   First, I'd suggest acknowledging these works and discussing the differences to your work.  Second - Is your approach applicable to these frameworks?   It would be an interesting to compare to one of those methods  (e.g., distant supervision for relation extraction using a knowledge base), and see if by incorporating fidelity score, results improve.  - Can this approach be applied to semi-supervised learning?  Is there a reason to assume the fidelity scores computed by the teacher would not improve the student in a self-training framework? - The paper emphasizes that the teacher uses the student's initial representation, when trained over the clean data.   Is it clear that this step in needed?  Can you add an additional variant of your framework when the fidelity score are  computed by the teacher when trained from scratch?  using different architecture than the student?
Quality: The work focuses on a novel problem of generating text sample using GAN and a novel in-filling mechanism of words.  Using GAN to generate samples in adversarial setup in texts has been limited due to the mode collapse and training instability issues.  As a remedy to these problems an in-filling-task conditioning on the surrounding text has been proposed.  But, the use of the rewards at every time step (RL mechanism) to employ the actor-critic training procedure could be challenging computationally challenging. Clarity: The mechanism of generating the text samples using the proposed methodology has been described clearly.  However the description of the reinforcement learning step could have been made a bit more clear. Originality: The work indeed use a novel mechanism of in-filling via a conditioning approach to overcome the difficulties of GAN training in text settings.  There has been some work using GAN to generate adversarial examples in textual context too to check the robustness of classifiers.  How this current work compares with the existing such literature? Significance: The research problem is indeed significant  since the use of GAN in generating adversarial examples in image analysis has been more prevalent compared to text settings.  Also, the proposed actor-critic training procedure via RL methodology is indeed significant from its application in natural language processing. pros: (a) Human evaluations applications to several datasets show the usefulness of MaskGen over the maximum likelihood trained model in generating more realistic text samples. (b) Using a novel in-filling procedure to overcome the complexities in GAN training. (c) generation of high quality samples even with higher perplexity on ground truth set. cons: (a) Use of rewards at every time step to the actor-critic training procure could be computationally expensive. (b) How to overcome the situation where in-filling might introduce implausible text sequences with respect to the surrounding words? (c) Depending on the Mask quality GAN can produce low quality samples.  Any practical way of choosing the mask?
This paper presents a method for clustering based on latent representations learned from the classification of transformed data after pseudo-labellisation corresponding to applied transformation. Pipeline: -Data are augmented with domain-specific transformations. For instance, in the case of MNIST, rotations with different degrees are applied. All data are then labelled as "original" or "transformed by ...(specific transformation)". -Classification task is performed with a neural network on augmented dataset according to the pseudo-labels. -In parallel of the classification, the neural network also learns the latent representation in an unsupervised fashion. -k-means clustering is performed on the representation space observed in the hidden layer preceding the augmented softmax layer. Detailed Comments: (*) Pros -The method outperforms the state-of-art regarding unsupervised methods for handwritten digits clustering on MNIST. -Use of ACOL and GAR is interesting, also the idea to make "labeled" data from unlabelled ones by using data augmentation. (*) Cons -minor: in the title, I find the expression "unsupervised clustering" uselessly redundant since clustering is by definition unsupervised. -Choice of datasets: we already obtained very good accuracy for the classification or clustering of handwritten digits. This is not a very challenging task. And just because something works on MNIST, does not mean it works in general. What are the performances on more challenging datasets like colored images (CIFAR-10, labelMe, ImageNet, etc.)? -This is not clear what is novel here since ACOL and GAR already exist. The novelty seems to be in the adaptation to GAR from the semi-supervised to the unsupervised setting with labels indicating if data have been transformed or not.
The paper proposed a copula-based modification to an existing deep variational information bottleneck model, such that the marginals of the variables of interest (x, y) are decoupled from the DVIB latent variable model, allowing the latent space to be more compact when compared to the non-modified version.  The experiments verified the relative compactness of the latent space, and also qualitatively shows that the learned latent features are more 'disentangled'.  However, I wonder how sensitive are the learned latent features to the hyper-parameters and optimizations? Quality: Ok.  The claims appear to be sufficiently verified in the experiments.  However, it would have been great to have an experiment that actually makes use of the learned features to make predictions.  I struggle a little to see the relevance of the proposed method without a good motivating example. Clarity: Below average.  Section 3 is a little hard to understand.  Is q(t|x) in Fig 1 a typo?  How about t_j in equation (5)?  There is a reference that appeared twice in the bibliography (1st and 2nd). Originality and Significance: Average.  The paper (if I understood it correctly) appears to be mainly about borrowing the key ideas from Rey et. al. 2014 and applying it to the existing DVIB model.
In the Following, pros and cons of the paper are presented. Pros ------- 1. Many real-world applications. 2. Simple architecture and can be reproduced (if given enough details.) Cons----------------------1. Ablation study showing whether bidirectional LSTM contributing to the similarity would be helpful. 2. Baseline is not strong.  How about using just LSTM? 4. It is suprising to see that only concatenation with MLP is used for optimization of capturing regularities across languages.  5. Equation-11 looks like softplus function more than vanilla ReLU. 6. How are the similarity assessments made in the gold standard dataset.  The cost function used only suggest binary assessments.  Please refer to some SemEval tasks for cross-lingual or cross-level assessments.  As binary assessments may not be a right measure to compare articles of two different lengths or languages. Minor issues------------1. SNS is meant to be social networking sites? 2. In Section 2.2, it denote that 'as the figure demonstrates'.  No reference to the figure. 3. In Section 3, 'discussed in detail' pointed to Section 2.1 related work section.  Not clear what is discussed in detail there. 4. Reference to Google Translate API is wrong. The paper requires more experimental analysis to judge the significance of the approach presented.
This paper uses an information geometric view on hierarchical models to discuss a bias - variance decomposition in Boltzmann machines,  presenting interesting conclusions,  whereby some more care appears to be needed for making these claims.  The paper arrives at the main conclusion that it is possible to reduce both the bias and the variance in a hierarchical model.  The discussion is not specific to deep learning nor to Boltzmann machines, but actually addresses hierarchical exponential family models.  The methods pertaining hierarchical models are interesting and presented in a clear way.  My concern are the following points: The main theorem presents only a lower bound, meaning that it provides no guarantee that the variance can indeed be reduced.  The paper seems to ignore that a model with hidden variables may be singular,  in which case the Fisher metric is not positive definite and the Cramer Rao bound has no meaning.  This interferes with the claims and derivations made in the paper in the case of models with hidden variables.  The problem seems to lie in the fact that the presented derivations assume that an optimal distribution in the data manifold is given (see Theorem 1 and proof), effectively making this a discussion about a fully observed hierarchical model.  In particular, it is not further specified how to obtain θˆB(s) in page 6 before (13).  Also, in page 5 the paper states that ``it is known that the EM-algorithm can obtain the global optimum of Equation (12) (Amari, 2016, Section 8.1.3)''.  However, what is shown in that reference is only that:  (Theorem 8.2., Amari, 2016) ``The KL-divergence decreases monotonically by repeating the E-step and the M-step. Hence, the algorithm converges to an equilibrium.''  A model with hidden variables can have several global and local optimisers  (see, e.g. https://arxiv.org/abs/1709.05276).  The critical points of the EM algorithm can have a non trivial structure,  as has been observed in the case of non negative rank matrix varieties  (see, e.g., https://arxiv.org/pdf/1312.5634.pdf).  OTHER In page 3, ``S_\beta is e-flat and S_\alpha ... '', should this not be the other way around?  (See also page 5 last paragraph of Section 2.)  Please also indicate the precise location in the provided reference.   All pages up to page 5 are introduction.  Section 2.3. as presented is very vague and does not add much to the discussion.  In page 7, please explain E ψ(θˆ )^2 −ψ(θ∗ )^2=0
The paper introduces a method for learning graph representations (i.e., vector representations for graphs).  An existing node embedding method is used to learn vector representations for the nodes.  The node embeddings are then projected into a 2-dimensional space by PCA.  The 2-dimensional space is binned using an imposed grid structure.  The value for a bin is the (normalized) number of nodes falling into the corresponding region.  The idea is simple and easily explained in a few minutes.  That is an advantage.  Also, the experimental results look quite promising.  It seems that the methods outperforms existing methods for learning graph representations.  The problem with the approach is that it is very ad-hoc.  There are several (existing) ideas of how to combine node representations into a representation for the entire graph.  For instance, averaging the node embeddings is something that has shown promising results in previous work.  Since the methods is so ad-hoc (node2vec -> PCA -> discretized density map -> CNN architecure) and since a theoretical understanding of why the approach works is missing, it is especially important to compare your method more thoroughly to simpler methods.  Again, pooling operations (average, max, etc.) on the learned node2vec embeddings are examples of simpler alternatives.  The experimental results are also not explained thoroughly enough.  For instance, since two runs of node2vec will give you highly varying embeddings (depending on the initialization), you will have to run node2vec several times to reduce the variance of your resulting discretized density maps.  How many times did you run node2vec on each graph?
In this paper the authors give a nice review of clustering methods with deep learning and a systematic taxonomy for existing methods. Finally, the authors propose a new method by using one unexplored combination of taxonomy features. The paper is well-written and easy to follow. The proposed combination is straightforward, but lack of novelty. From table 1, it seems that the only differences between the proposed method and DEPICK is whether the method uses balanced assignment and pretraining. I am not convinced that these changes will lead to a significant difference. The performance of the proposed method and DEPICK are also similar in table 1. In addition, the experiments section is not comprehensive enough as well. the author only tested on two datasets. More datasets should be tested for evaluation. In addition, It seems that nearly all the experiments results from comparison methods are borrowed from the original publications. The authors should finish the experiments on comparison methods and fill the entries in Table 1. In summary, the proposed method is lack of novelty compare to existing methods. The survey part is nice, however extensive experiments should be conducted by running existing methods on different datasets and analyzing the pros and cons of the methods and their application scenarios. Therefore, I think the paper cannot be accepted at this stage.
While I acknowledge that training generative models with binary latent variables is hard,  I'm not sure this paper really makes valuable progress in this direction.  The only results that seem promising are those on binarized MNIST, for the non-convolutional architecture,  and this setting isn't particularly exciting.  All other experiments seem to suggest that the proposed model/algorithm is behind the state of the art.  Moreover, the proposed approach is fairly incremental, compared to existing work on RWS, VIMCO, etc. So while this work seem to have been seriously and thoughtfully executed,  I think it falls short of the ICLR acceptance bar.