In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
--------
Question: The authors propose a simple idea. They penalize confident predictions by using the entropy of the predictive distribution as a regularizer. The authors consider two variations on this idea. In one, they penalize the divergence from the uniform distribution. In the other variation, they penalize distance from the base rates. They term this variation "unigram" but I find the name odd as I've never seen multi-class labels described as unigrams before. What would a bigram be?   The idea is simple,  and while it's been used in the context of reinforcement learning, it hasn't been popularized as a regularizer for improving generalization in supervised learning.   The justifications for the idea still lacks analysis. And the author responses comparing it to L2 regularization have some holes. A simple number line example with polynomial regression makes clear how L2 regularization could prevent a model from badly overfitting to accommodate every data point. In contrast, it seems trivial to fit every data point and satisfy arbitrarily high entropy. Of course, the un-regularized optimization is to maximize log likelihood, not simply to maximize accuracy.  And perhaps something interesting may be happening at the interplay between the log likelihood objective and the regularization objective. But the paper doesn't indicate precisely what.  I could imagine the following scenario: when the network outputs probabilities near 0, it can get high loss (if the label is 1). The entropy regularization could be stabilizing the gradient, preventing sharp loss on outlier examples. The regularization then might owe mainly to faster convergence. Could the authors analyze the effect empirically, on the distribution of the gradient norms?   The strength of this paper is its empirical rigor. The authors take their idea and put it through its paces on a host of popular and classic benchmarks spanning CNNs and RNNs. It appears that on some datasets, especially language modeling, the confidence penalty outperforms label smoothing.   At present, I rate this paper as a borderline contribution but I'm open to revising my review pending further modifications.   Typo: In related work: "Penalizing entropy" - you mean penalizing low entropy

Answer: en


Question: While I understand the difficulty of collecting audio data from animals, I think this type of feature engineering does not go in the right direction. I would rather see a model than learns the feature representation from data.  I would think it should be possible to collect a more substantial corpus in zoos / nature etc, and then train a generative model. The underlying learned feature representation could be then used to feed a classifier. I'm not familiar with the particularities of this task, it's hard to judge the improvements by using chirplets.

Answer: en


Question: In this work, the authors extend MS-COCO by adding an incorrect caption to each existing caption, with only one word of difference. The authors demonstrate that two state-of-the-art methods (one for VQA and one for captioning) perform extremely poorly at a) determining if a caption is fake, b) determining which word in a fake caption is wrong, and c) selecting a replacement word for a given fake word.  This work builds upon a wealth of literature regarding the underperformance of vision/language models relative to their apparent capacities. I think this work makes concrete some of the big, fundamental questions in this area: are vision/language models doing "interesting" things, or not? The authors consider a nice mix of tasks and models to shed light on the "broken-ness" of these settings, and perform some insightful analyses of factors associated with model failure (e.g., Figure 3).  My biggest concerns with the paper are similarity to Ding et al. That being said, I do think the authors make some really good points; Ding et al. generate similar captions, but the ones here differ by only one word and *still* break the models -- I think that's a justifiably fundamental difference. That observation demonstrates that Ding et al.'s engineering is not a requirement, as this simple approach still breaks things catastrophically.  Another concern is the use of NeuralTalk to select the "hardest" foils.              While a clever idea, I am worried that the use of this model creates a risk of self-reinforcement bias, i.e., NeuralTalk's biases are now fundamentally "baked-in" to FOIL-COCO.   I think the results section could be a bit longer, relative to the rest of the paper (e.g. I would've liked more than one paragraph -- I liked this part!)  Overall, I do like this paper, as it nicely builds upon some results that highlight defficiencies in vision/language integration. In the end, the Ding et al. similarity is not a "game-breaker," I think -- if anything, this work shows that vision/language models are so easy to fool, Ding et al.'s method is not even required.  Small things:  I would've liked to have seen another baseline that simply concatenates BoW + extracted CNN features and trains a softmax classifier over them. The "blind" model is a nice touch, but what about a "dumb" vision+langauge baseline? I bet that would do close to as well as the LSTM/Co-attention. That could've made the point of the paper even stronger.  330: What is a supercategory? Is this from WordNet? Is this from COCO? I understand the idea, but not the specifics.  397: has been -> were  494: that -> than  693: artefact -> undesirable artifacts (?)  701: I would have included a chance model in T1's table -- is 19.53% [Line 592] a constant-prediction baseline? Is it 50% (if so, can't we flip all of the "blind" predictions to get a better baseline?) I am not entirely clear, and I think a "chance" line here would fix a lot of this confusion.  719: ariplane  ~~ After reading the author response...  I think this author response is spot-on. Both my concerns of NeuralTalk biases and additional baselines were addressed, and I am confident that these can be addressed in the final version, so I will keep my score as-is.

Answer:
en