Task: Language Identification
Input (Spanish, English): This paper proposes a new Bayesian neural network architecture for predicting the values of learning curves during the training of machine learning models. This is an exploratory paper, in that the ultimate goal is to use this method in a Bayesian optimization system, but for now the experiments are limited to assessing the quality of the predictions. This builds on previous work in Domhan, 2015, however in this work the model incorporates information from all tested hyperparameter settings rather than just extrapolating from a single learning curve. This paper also explores two MCMC methods for inference: SGLD and SGHMC, but I couldn’t tell if either of these were tested in Domhan, 2015 as well.  The performance seems overall positive, particularly in the initial phase of each curve where there is very little information. In this case, as expected, sharing knowledge across curves helps. One regime which did not seem to be tested, but might be very informative, is when some curves in the training set have been mostly, or fully observed. This might be a case where sharing information really helps.  Something that concerns me about this approach is the timing. The authors stated that to train the network takes about 20-60 seconds. In the worst case, with 100 epochs, this results in a little over 1.5 hours spent training the Bayesian network. This is a non-trivial fraction of the several hours it takes to train the model being tuned.  The Bayesian network makes many separate predictions, as shown in Figure 2. It would be interesting to see how accurate some of these individual pieces are. For example, did you bound the asymptotic value of the learning curve, since you mostly predicted accuracy? If not, did the value tend to lie in [0,1]?  Below are some minor questions/comments.  Figure 1 axes should read “validation accuracy” Figure 6 can you describe LastSeenValue (although it seems self-explanatory, it’s good to be explicit) in the bottom left figure, and why isn’t it used anywhere else as a baseline? Figure 7 and Table 1 are you predicting just the final value of the curves? Or every value along each curve, conditioned on the previous values? Why do you only use 5 basis functions? Does this sufficiently capture all of the flexibility of these learning curves? Would more basis functions help or hurt?
Output (English): en