launch/ampere huggingface dataset:
The authors propose a new network architecture for RL that contains some relevant inductive biases about planning. This fits into the recent line of work on implicit planning where forms of models are learned to be useful for a prediction/planning task. The proposed architecture performs something analogous to a full-width tree search using an abstract model (learned end-to-end). This is done by expanding all possible transitions to a fixed depth before performing a max backup on all expanded nodes. The final backup value is the Q-value prediction for a given state, or can represent a policy through a softmax. I thought the paper was clear and well-motivated. The architecture (and various associated tricks like state vector normalization) are well-described for reproducibility. Experimental results seem promising but I wasn’t fully convinced of its conclusions. In both domains, TreeQN and AtreeC are compared to a DQN architecture, but it wasn’t clear to me that this is the right baseline. Indeed TreeQN and AtreeC share the same conv stack in the encoder (I think?), but also have the extra capacity of the tree on top. Can the performance gain we see in the Push task as a function of tree depth be explained by the added network capacity? Same comment in Atari, but there it’s not really obvious that the proposed architecture is helping. Baselines could include unsharing the weights in the tree, removing the max backup, having a regular MLP with similar capacity, etc. Page 5, the auxiliary loss on reward prediction seems appropriate, but it’s not clear from the text and experiments whether it actually was necessary. Is it that makes interpretability of the model easier (like we see in Fig 5c)? Or does it actually lead to better performance? Despite some shortcomings in the result section, I believe this is good work and worth communicating as is.
This paper proposes a new theoretically-motivated method for combining reinforcement learning and imitation learning for acquiring policies that are as good as or superior to the expert.  The method assumes access to an expert value function (which could be trained using expert roll-outs) and uses the value function to shape the reward function and allow for truncated-horizon policy search.  The algorithm can gracefully handle suboptimal demonstrations/value functions,  since the demonstrations are only used for reward shaping,  and the experiments demonstrate faster convergence and better performance compared to RL and AggreVaTeD on a range of simulated control domains.  The paper is well-written and easy to understand. My main feedback is with regard to the experiments: I appreciate that the experiments used 25 random seeds!  This provides a convincing evaluation. It would be nice to see experimental results on even higher dimensional domains such as the ant, humanoid, or vision-based tasks,  since the experiments seem to suggest that the benefit of the proposed method is diminished in the swimmer and hopper domains compared to the simpler settings. Since the method uses demonstrations,  it would be nice to see three additional comparisons: (a) training with supervised learning on the expert roll-outs, (b) initializing THOR and AggreVaTeD (k=1) with a policy trained with supervised learning, and (c) initializing TRPO with a policy trained with supervised learning.  There doesn't seem to be any reason not to initialize in such a way, when expert demonstrations are available,  and such an initialization should likely provide a significant speed boost in training for all methods. How many demonstrations were used for training the value function in each domain?  I did not see this information in the paper. With regard to the method and discussion: The paper discusses the connection between the proposed method and short-horizon imitation and long-horizon RL, describing the method as a midway point.  It would also be interesting to see a discussion of the relation to inverse RL,  which considers long-term outcomes from expert demonstrations.  For example, MacGlashn & Littman propose a midway point between imitation and inverse RL [1]. Theoretically, would it make sense to anneal k from small to large? (to learn the most effectively from the smallest amount of experience) [1] https://www.ijcai.org/Proceedings/15/Papers/519.pdf Minor feedback: - The RHS of the first inequality in the proof of Thm 3.3 seems to have an error in the indexing of i and exponent, which differs from the line before and line after
The authors propose a procedure to generate an ensemble of sparse structured models.  To do this, the authors propose to (1) sample models using SG-MCMC with group sparse prior, (2) prune hidden units with small weights, (3) and retrain weights by optimizing each pruned model.  The ensemble is applied to MNIST classification and language modelling on PTB dataset.  I have two major concerns on the paper.  First, the proposed procedure is quite empirically designed.  So, it is difficult to understand why it works well in some problems.  Particularly. the justification on the retraining phase is weak.  It seems more like to use SG-MCMC to *initialize* models which will then be *optimized* to find MAP with the sparse-model constraints.  The second problem is about the baselines in the MNIST experiments.  The FNN-300-100 model without dropout, batch-norm, etc. seems unreasonably weak baseline.  So, the results on Table 1 on this small network is not much informative practically.  Lastly, I also found a significant effort is also desired to improve the writing.  The following reference also needs to be discussed in the context of using SG-MCMC in RNN. - "Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling", Zhe Gan*, Chunyuan Li*, Changyou Chen, Yunchen Pu, Qinliang Su, Lawrence Carin
The quality of the paper is good, and clarity is mostly good.  The proposed metric is interesting,  but it is hard to judge the significance without more thorough experiments demonstrating that it works in practice. Pros:- clear definitions of terms  - overall outline of paper is good  - novel metric Cons - text is a bit over-wordy, and flow/meaning sometimes get lost.  A strict editor would be helpful,  because the underlying content is good  - odd that your definition of generalization in GANs appears immediately preceding the section titled "Generalisation in GANs"  - the paragraph at the end of the "Generalisation in GANs" section is confusing.  I think this section and the previous ("The objective of unsupervised learning") could be combined, removing some repetition, adding some subtitles to improve clarity.  This would cut down the text a bit to make space for more experiments.  - why is your definition of generalization that the test set distance is strictly less than training set ?  I would think this should be less-than-or-equal  - there is a sentence that doesn't end at the top of p.3: "... the original GAN paper showed that [ends here]"  - should state in the abstract what your "notion of generalization" for gans is, instead of being vague about it  - more experiments showing a comparison of the proposed metric to others (e.g. inception score, Mturk assessments of sample quality, etc.) would be necessary to find the metric convincing  - what is a "pushforward measure"? (p.2)  - the related work section is well-written and interesting,  but it's a bit odd to have it at the end.  Earlier in the work (e.g. before experiments and discussion) would allow the comparison with MMD to inform the context of the introduction  - there are some errors in figures that I think were all mentioned by previous commentators.
This work addresses an important and outstanding problem: accurate long-term forecasting using deep recurrent networks. The technical approach seems well motivated, plausible, and potentially a good contribution, but the experimental work has numerous weaknesses which limit the significance of the work in current form. For one, the 3 datasets tested are not established as among the most suitable, well-recognized benchmarks for evaluating long-term forecasting. It would be far more convincing if the author’s used well-established benchmark data, for which existing best methods have already been well-tuned to get their best results. Otherwise, the reader is left with concerns that the author’s may not have used the best settings for the baseline method results reported, which indeed is a concern here (see below). One weakness with the experiments is that it is not clear that they were fair to RNN or LSTM, for example, in terms of giving them the same computation as the TT-RNNs. Section Hyper-parameter Analysis” on page 7 explains that they determined best TT rank and lags via grid search. But presumably larger values for rank and lag require more computation, so to be fair to RNN and LSTM they should be given more computation as well, for example allowing them more hidden units than TT-RNNs get, so that overall computation cost is the same for all 3 methods. As far as this reviewer can tell, the authors offer no experiments to show that a larger number of units for RNN or LSTM would not have helped them in improving long-term forecasting accuracies, so this seems like a very serious and plausible concern. Also, on page 6 the authors say that they tried ARMA but that it performed about 5% worse than LSTM, and thus dismissing direct comparisons of ARMA against TT-RNN. But they are unclear whether they gave ARMA as much hyper-parameter tuning (e.g. for number of lags) via grid search as their proposed TT-RNN benefited from. Again, the concern here is that the experiments are plausibly not being fair to all methods equally. So, due to the weaknesses in the experimental work, this work seems a bit premature. It should more clearly establish that their proposed TT-RNN are indeed performing well compared to existing SOTA.
This paper is well constructed and written. It consists of a number of broad ideas regarding density estimation using transformations of autoregressive networks. Specifically, the authors examine models involving linear maps from past states (LAM) and recurrence relationships (RAM). The critical insight is that the hidden states in the LAM are not coupled allowing considerable flexibility between consecutive conditional distributions. This is at the expense of an increased number of parameters and a lack of information sharing. In contrast, the RAM transfers information between conditional densities via the coupled hidden states allowing for more constrained smooth transitions. The authors then explored a variety of transformations designed to increase the expressiveness of LAM and RAM. The authors importantly note that one important restriction on the class of transformations is the ability to evaluate the Jacobian of the transformation efficiently. A composite of transformations coupled with the LAM/RAM networks provides a highly expressive model for modelling arbitrary joint densities but retaining interpretable conditional structure. There is a rich variety of synthetic and real data studies which demonstrate that LAM and RAM consistently rank amongst the top models demonstrating potential utility for this class of models. Whilst the paper provides no definitive solutions, this is not the point of the work which seeks to provide a description of a general class of potentially useful models.
CONTRIBUTION The main contribution of the paper is not clearly stated.   To the reviewer, It seems “life-long learning” is the same as “online learning”.   However, the whole paper does not define what “life-long learning” is. The limited budget scheme is well established in the literature.  1. J. Hu, H. Yang, I. King, M. R. Lyu, and A. M.-C. So. Kernelized online imbalanced learning with fixed budgets. In AAAI, Austin Texas, USA, Jan. 25-30 2015.   2. Y. Engel, S. Mannor, and R. Meir. The kernel recursive least-squares algorithm. IEEE Transactions on Signal Processing, 52(8):2275–2285, 2004. It is not clear what the new proposal in the paper. WRITING QUALITY The paper is not well written in a good shape.  Many meanings of the equations are not stated clearly, e.g., $phi$ in eq. (7).  Furthermore, the equation in algorithm 2 is not well formatted.  DETAILED COMMENTS 1. The mapping function $phi$ appears in Eq. (1) without definition. 2. The last equation in pp. 3 defines the decision function f by an inner product.  In the equation, the notation x_t and i_t is not clearly defined.   More seriously, a comma is missed in the definition of the inner product. 3. Some equations are labeled but never referenced, e.g., Eq. (4). 4. The physical meaning of Eq.(7) is unclear.   However, this equation is the key proposal of the paper.    For example, what is the output of the Eq. (7)?  What is the main objective of Eq. (7)?   Moreover, what support vectors should be removed by optimizing Eq. (7)?   One main issue is that the notation $phi$ is not clearly defined.    The computation of f-y_r\phi(s_r) makes it hard to understand.   Especially,  the dimension of $phi$ in Eq.(7) is unknown.  ABOUT EXPERIMENTS 1.	It is unclear how to tune the hyperparameters. 2.	In Table 1, the results only report the standard deviation of AUC.  No standard deviations of nSV and Time are reported.
This paper studies new off-policy policy optimization algorithm using relative entropy objective and use EM algorithm to solve it.  The general idea is not new, aka, formulating the MDP problem as a probabilistic inference problem.  There are some technical questions: 1. For parametric EM case, there is asymptotic convergence guarantee to local optima case;  However, for nonparametric EM case, there is no guarantee for that.  This is the biggest concern I have for the theoretical justification of the paper. 2. In section 4, it is said that Retrace algorithm from Munos et al. (2016) is used for policy evaluation.  This is not true.  The Retrace algorithm, is per se, a value iteration algorithm.  I think the author could say using the policy evaluation version of Retrace, or use the truncated importance weights technique as used in Retrace algorithm, which is more accurate. Besides, a minor point: Retrace algorithm is not off-policy stable with function approximation, as shown in several recent papers, such as “Convergent Tree-Backup and Retrace with Function Approximation”.  But this is a minor point if the author doesn’t emphasize too much about off-policy stability. 3. The shifting between the unconstrained multiplier formulation in Eq.9 to the constrained optimization formulation in Eq.10 should be clarified.  Usually, an in-depth analysis between the choice of \lambda in multiplier formulation and the \epsilon in the constraint should be discussed,  which is necessary for further theoretical analysis.  4. The experimental conclusions are conducted without sound evidence.  For example, the author claims the method to be 'highly data efficient' compared with existing approaches,  however, there is no strong evidence supporting this claim.  Overall, although the motivation of this paper is interesting,  I think there is still a lot of details to improve.
The manuscript proposes two objective functions based on the manifold assumption as defense mechanisms against adversarial examples.  The two objective functions are based on assigning low confidence values to points that are near or off the underlying (learned) data manifold while assigning high confidence values to points lying on the data manifold.  In particular, for an adversarial example that is distinguishable from the points on the manifold and assigned a low confidence by the model, is projected back onto the designated manifold such that the model assigns it a high confidence value.  The authors claim that the two objective functions proposed in this manuscript provide such a projection onto the desired manifold and assign high confidence for these adversarial points.  These mechanisms, together with the so-called shell wrapper around the model (a deep learning model in this case) will provide the desired defense mechanism against adversarial examples. The manuscript at the current stage seems to be a preliminary work that is not well matured yet.  The manuscript is overly verbose and the arguments seem to be weak and not fully developed yet.  More importantly, the experiments are very preliminary and there is much more room to deliver more comprehensive and compelling experiments.
The main idea of this paper is to replace the feedforward summation y = f(W*x + b) where x,y,b are vectors, W is a matrix by an integral \y = f(\int W \x + \b) where \x,\y,\b are functions, and W is a kernel.  A deep neural network with this integral feedforward is called a deep function machine.  The motivation is along the lines of functional PCA:  if the vector x was obtained by discretization of some function \x, then one encounters the curse of dimensionality as one obtains finer and finer discretization.  The idea of functional PCA is to view \x as a function is some appropriate Hilbert space, and expands it in some appropriate basis.  This way, finer discretization does not increase the dimension of \x (nor its approximation), but rather improves the resolution.  This paper takes this idea and applies it to deep neural networks.  Unfortunately, beyond rather obvious approximation results, the paper does not get major mileage out of this idea.  This approach amounts to a change of basis -  and therefore the resolution invariance is not surprising.  In the experiments, results of this method should be compared not against NNs trained on the data directly, but against NNs trained on dimension reduced version of the data (eg: first fixed number of PCA components).  Unfortunately, this was not done.  I suspect that in this case, the results would be very similar.
This paper presents a so-called cross-view training for semi-supervised deep models. Experiments were conducted on various data sets and experimental results were reported. Pros:* Studying semi-supervised learning techniques for deep models is of practical significance. Cons:* The novelty of this paper is marginal. The use of unlabeled data is in fact a self-training process. Leveraging the sub-regions of the image to improve performance is not new and has been widely-studied in image classification and retrieval. * The proposed approach suffers from a technical weakness or flaw. For the self-labeled data, the prediction of each view is enforced to be same as the assigned self-labeling. However, since each view related to a sub-region of the image (especially when the model is not so deep), it is less likely for this region to contain the representation of the concepts (e.g., some local region of an image with a horse may exhibit only grass); enforcing the prediction of this view to be the same self-labeled concepts (e.g,“horse”) may drive the prediction away from what it should be ( e..g, it will make the network to predict grass as horse). Such a flaw may affect the final performance of the proposed approach. * The word “view” in this paper is misleading. The “view” in this paper is corresponding to actually sub-regions in the images * The experimental results indicate that the proposed approach fails to perform better than the compared baselines in table 2, which reduces the practical significance of the proposed approach.
This paper presents a nearest-neighbor based continuous control policy. Two algorithms are presented: NN-1 runs open-loop trajectories from the beginning state, and NN-2 runs a state-condition policy that retrieves nearest state-action tuples for each state. The overall algorithm is very simple to implement and can do reasonably well on some simple control tasks, but quickly gets overwhelmed by higher-dimensional and stochastic environments. It is very similar to "Learning to Steer on Winding Tracks Using Semi-Parametric Control Policies" and is effectively an indirect form of tile coding (each could be seen as a fixed voronoi cell). I am sure this idea has been tried before in the 90s but I am not familiar enough with all the literature to find it (A quick google search brings this up: Reinforcement Learning of Active Recognition Behaviors, with a chapter on nearest-neighbor lookup for policies: https://people.eecs.berkeley.edu/~trevor/papers/1997-045/node3.html). Although I believe there is work to be done in the current round of RL research using nearest neighbor policies, I don't believe this paper delves very far into pushing new ideas (even a simple adaptive distance metric could have provided some interesting results, nevermind doing a learned metric in a latent space to allow for rapid retrainig of a policy on new domains....), and for that reason I don't think it has a place as a conference paper at ICLR. I would suggest its submission to a workshop where it might have more use triggering discussion of further work in this area.
There could be an interesting idea here,  but the limitations and applicability of the proposed approach are not clear yet.  More analysis should be done to clarify its potential.  Besides, the paper seriously needs to be reworked.  The text in general, but also the notation, should be improved. In my opinion, the authors should explain how to apply their algorithm to more general network architectures, and test it, in particular to convnets.  An experiment on a modern dataset beyond MNIST would also be a welcome addition. Some comments:- The method is present as a fully-connected network training procedure.  But the resulting network is not really fully-connected, but modular.  This is clear in Fig. 1 and in the explanation in Sect. 3.1.  The newly added hidden neurons at every iteration do not project to the previous pool of hidden neurons.  It should be stressed that the networks end up with this non-conventional “tiled” architecture.  Are there studies where the capacity of such networks is investigated, when all the weights are trained concurrently. - It wasn’t clear to me whether the memory reallocation could be easily implemented in hardware.  A few references or remarks on this issue would be welcome. - The work “Efficient supervised learning in networks with binary synapses” by Baldassi et al. (PNAS 2007) should be cited.  Although usually ignored by the deep learning community, it actually was a pioneering study on the use of low resolution weights during inference while allowing for auxiliary variables during learning. - Coming back my main point above, I didn’t really get the discussion on Sect. 5.3.  Why didn’t the authors test their algorithm on a convnet?  Are there any obstacles in doing so?  It seems quite important to understand this point,  as the paper appeals to technical applications and convolution seems hard to sidestep currently. - Fig. 3: xx-axis: define storage efficiency and storage requirement. - Fig. 4: What’s an RSBL?  Acronyms should be defined. - Overall, language and notation should really be refined.  I had a hard time reading Algorithm 1,  as the notation is not even defined anywhere.  And this problem extends throughout the paper. For example, just looking at Sect. 4.1, “training and testing data x is normalized…”, if x is not properly defined, it’s best to omit it;   “… 2-dimentonal…”, at least major typos should be scanned and corrected.