Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper produces a new method call TVmax and presents that the selective visual attention could improve the score in the Image Captioning task. Different from the fusedmax[2] which fuses attention in one dimension, the proposed method encourages the sparse attention over contiguous 2D regions. Compared with the softmax function, the sparsemax[1]  and the TVmax are able to sparse the visual attention very well. The paper also evaluates the score in both automated metrics and the human rating. Experiments show that the sparse visual attention achieves higher performance with a little computational cost.

One problem in this paper is that the author applies their proposed TVmax on Image Captioning  task, however it only achieves a little improvement on the automated metrics compared with the baseline(softmax). I wonder whether there is a better task for evaluating the visual attention. 

Although the proposed method (TVmax) is slightly worse than the sparsemax in the automated metrics, it is still promising in multimodal problems. 

Therefore, My decision leans to a weak accept.

Some questions:
1.From the experiments, the proposed method achieved only a little higher performance than the baseline(softmax). Could you please show some reasons about that?
2.Could you show some results of TVmax on the other task in order to show the effectiveness of the proposed method?


[1]From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification.      André F. T. Martins, Ramón Fernandez Astudillo
[2]A Regularized Framework for Sparse and Structured Neural Attention .            Vlad Niculae, Mathieu Blondel
review_title: Official Blind Review #376

final_score: rating: 3: Weak Reject
review_content: This paper proposes two sparsifying methods of computing attention weights, dubbed sparsemax and TVmax, which appear to slightly improve objective and subjective image captioning scores.    The sparsifying projections are posed as optimization problems, and algorithms for their computation, along with formula for their gradients are given.  Proof of the optimality of these algorithms relies significantly on prior work, so could not be checked deeply without bringing in additional sources.  

It is not clear that the motivation for these sparsifying objectives is sound.   The conventional softmax approach to attention weights should be capable of producing attention weights near zero, which would be effectively sparse, especially if the pre-activations, z_i, in equation (1), are allowed to have a large enough range.   It's not clear why weights should need to be zero exactly in the ignored regions, since being near zero should be sufficient to contribute almost nothing to the subsequent weighted sum.    So it is also not clear why the strict sparsity itself, as opposed to the effective sparsity of the softmax, should explain the differences in Figure 1, and in the results.  In particular it is unclear why the strict sparsity should prevent repetition; when looking at the weight distributions in the two cases, a more likely story seems to be that the weight distributions don't repeat as much from one word to the next in the second case, but there is no clear reason to attribute this to sparsity.   The pictures of the attention weights are lacking a color scale, so it is impossible to see how close to zero it comes in the unattended regions, although the gray color values chosen for these regions might be misleading. 
The TVmax approach, in addition to sparsity, also constrains the non-zero region to be contiguous.   To the extent that this improves performance, this presumably introduces an inductive bias that matches the data.   It is unclear why this fails to produce better objective scores than sparsemax, while producing better human ratings.   In any case it is not clear why this should necessarily be a good inductive bias for all images, although it is plausible that it helps in some cases.  
In many neural network problems, what makes a difference has more to do with the optimizability of the gradients, than the specific activations per se, and that might be the case here too, although the paper does not analyze this aspect of the proposed models.   

Overall the paper is flawed by the lack of clarity in the motivation for the proposed methods, and the lack of retrospective analysis and understanding of why the proposed methods should improve results. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of applying attention to the problem of image captioning. To this end the authors first apply full attention where the probabilities are computed using softmax before applying the recently proposed Sparsemax - which essentially computes probabilities from scores by performing a projection on to the probability simplex. The authors then propose a variant of Sparsemax which they call TVMax, which has the property that it encourages assigning probability weights to contiguous regions in 2D space unlike Sparsemax which has no such incentive. The main idea is to augment the Sparsemax projection loss with a Lasso like penalty which penalizes assigning different attention probabilities to contiguous regions in the image. The authors then compare their TVMax approach with softmax and Sparsemax attention for image captioning and show improvements on the MSCOCO and Flickr datasets.

The idea of applying additional structural constraints on the sparsity structure induced by Sparsemax is a cool idea, and I like the idea of incentivizing contiguous pixels to have similar attention probabilities. Sparse attention patterns seems like an important direction of research with the motivation of either 1) improving generalization over full attention or 2) scaling to inputs of length where full attention is not feasible. This seems like a good progress in the first direction. The major weakness I see in this work is that the authors only limited their experiments to image captioning. It would be interesting to see if their approach could benefit other tasks such as machine translation, image generation etc. The other issue with their approach is that it doesn't seem to scale well - if I understand correctly their algorithm takes O(n^2logn) for sequence length n. The other issue potentially could be weak baselines since the authors use an LSTM for their caption generation network instead of Transformer.

[Edit: After going through reviewer discussion, I updated my score to reject. I am not convinced of the motivation for sparse attention unless it is for long sequences, since otherwise the regular softmax should be able to assign 0's to the un-needed items. Moreover, for generalization one can use attention dropout which is simpler instead.]
review_title: Official Blind Review #2

Meta-review: This paper presents sparse attention mechanisms for image captioning. In addition to recent sparsemax based method, authors proposed to extend it by incorporating structural constraints in 2D images, which is called TVMAX. The proposed methods are shown to improve the quality of captioning, particularly in terms of fewer erroneous repetitions, and obtain better human evaluation scores. 
Through reviewer discussion, one reviewer updated the score to rejection. A major concern raised by the reviewers is that the motivation of introducing sparse attention is not clear, and the reason why it improves the quality (particularly, why it can reduce repetition) is not convincing. While we understand it is plausible for long sequences as in text domain, we are not convinced that it is really necessary for image captioning problems. Although authors seem to have some ideas, we cannot see how they will be reflected in the paper so I’d like to recommend rejection.
I recommend authors to polish the paper with a clearer description of the motivation and high-level analysis of the method as well as testing on other visual tasks to show its generality. 

Decision: Reject