Title: No title found

Abstract: Attention networks show promise for both vision and language tasks, by emphasizing relationships between constituent elements through weighting functions. Such elements could be regions in an image output by a region proposal network, or words in a sentence, represented by word embedding. Thus far the learning of attention weights has been driven solely by the minimization of task specific loss functions. We introduce a method for learning attention weights to better emphasize informative pair-wise relations between entities. The key component is a novel center-mass cross entropy loss, which can be applied in conjunction with the task specific ones. We further introduce a focused attention backbone to learn these attention weights for general tasks. We demonstrate that the focused supervision leads to improved attention distribution across meaningful entities, and that it enhances the representation by aggregating features from them. Our focused attention module leads to state-of-the-art recovery of relations in a relationship proposal task and boosts performance for various vision and language tasks.

1. INTRODUCTION:

Complex tasks involving visual perception or language interpretation are inherently contextual. In an image of an office scene, for example, a computer mouse may be too small to detect but the recognition of a computer keyboard might hint at its presence and its possible locations. The study of objects in their context is a cornerstone of much past computer vision work (Rabinovich et al., 2007) . Scene categories are often determined by the relationships between objects or environments commonly found in them (Zhou et al., 2017) while in natural language processing words must be interpreted in relation to other words or phrases in sentences. Machine learning algorithms that learn object to object or word to word relationships have thus been sought. Among them, attention networks have shown great promise for the task of learning relationship attention weights between entities (Veličković et al., 2017; Vaswani et al., 2017 ). As a recent example, the scaled dot product attention module from Vaswani et al. (2017) achieves state of the art performance in language translation tasks.

We here propose to explicitly supervise the learning of attention weights between elements of a data source using a novel center-mass cross entropy loss. The minimization of this loss increases relation weights between entity pairs which are more commonly observed in the data, but without the need for handcrafted frequency measurements. We design a focused attention network that is end-to-end trainable and which explicitly learns pairwise element affinities without the need for relationship annotations in the data. Multiple experiments demonstrate that such focused attention improves upon the baseline as well as attention without focus, for both computer vision and natural language processing tasks. In a relationship proposal task the use of this backbone achieves results comparable to the present state-of-the-art (Zhang et al., 2017) , even without the use of ground truth relationship labels. The use of ground truth labels for focused attention learning leads to a further 25% relative improvement, as measured by a relationship recall metric.

2. MOTIVATION:

Attention Networks -The Present State The modeling of relations between objects as well as objects in their common contexts has a rich history in computer vision (Rabinovich et al., 2007; Torralba, 2003; Galleguillos & Belongie, 2010) . Deep learning based object detection systems leverage attention models to this end, to achieve impressive performance in recognition tasks. The scaled dot product attention module of Vaswani et al. (2017) , for example, uses learned pairwise Figure 1 : A comparison of recovered relationships on the MIT67 dataset, with training only on the minicoco dataset. The reference object is surrounded by a blue box and regions with which it learns relationships are shown with orange boxes with the relationship weights visible in red text (zoom-in on the PDF). Left: Relation Networks (Hu et al., 2018) often learn weights between a reference object and its immediate surrounding context, while Focused Attention Networks better emphasize relationships between distinct and spatially separated objects. Right: Relation Networks can suffer from a poor selection of regions to pair, or low between object relationship weights in comparison to Focused Attention Networks. attention weights between region proposal network (RPN) generated bounding boxes in images of natural scenes (Hu et al., 2018) to boost object detection. Pixel level attention models have also been explored to aid semantic segmentation (Zhao et al., 2018) and video classification (Wang et al., 2018) .

Current approaches to learn the attention weights often do not reflect relations between entities in a typical visual scene. In fact, for a given reference object (region), relation networks (Hu et al., 2018) tend to predict high attention weights with scaled or shifted bounding boxes surrounding the same object instance. This is likely because including surrounding context, or simply restoring missing parts of the reference object, boosts object detection. The learned relationship weights between distinct objects (regions) are also often small in magnitude. Typical qualitative examples comparing Relation Networks with our Focused Attention Network are shown in Figure 1 , with a quantitative comparison reported in Section 5. Similar situations can occur in applications of attention networks to natural language processing tasks. In document classification, for example, attention weights learned using Hierarchical Attention Networks (Yang et al., 2016) tend to concentrate on a few words in a sentence, while our Focused Attention Network leads to more distributed attention, allowing for more comprehensive sentence level features, as illustrated in Figure 2 . given word to its sentence level context (see Section 4.5). The sentence is in a clockwise direction starting from 12 o'clock. "ait": weights learned using Hierarchical Attention Networks (Yang et al., 2016) , "unsup": weights from the unsupervised case of our Focused Attention Network module, and "sup": weights from the supervised case. Right: semantic word-to-word relationship labels used in the supervision of our network (see Section 3.2).

Attention Networks -Limitations A present limitation of attention networks in various applications is the use of only task specific losses as their training objectives. There is little work thus far on explicitly supervising the learning of weights, so as to be more distributed across meaningful entities. For example, Relation Networks (Hu et al., 2018) and those applied to segmentation problems, such as PSANet (Zhao et al., 2018) , learn attention weights solely by minimizing categorical cross entropy for classification, L1 loss for bounding box localization or pixel-wise cross entropy loss for semantic segmentation (Zhao et al., 2018) . In language tasks including machine translation (Vaswani et al., 2017) and document classification (Yang et al., 2016) , the attention weights are also solely learned by minimizing the categorical cross entropy loss. In this article we refer to such attention networks as being unsupervised.

Whereas attention aggregation with learned weights boosts performance for these specific tasks, our earlier examples provide evidence that relationships between distinct entities may not be adequately captured. We shall address this limitation by focusing the attention weight learning using a novel center-mass cross entropy loss, as discussed in the following section.

3. FOCUSING THE ATTENTION:

Given our goal of better reflecting learned attention weights between distinct entities, we propose to explicitly supervise the learning of attention relationship weights. We accomplish this by introducing a novel center-mass cross entropy loss.

3.1. PROBLEM STATEMENT:

Given a set of N entities that are generated by a feature embedding framework, which can be a region proposal network (RPN) (Ren et al., 2015) or a word embedding layer with a bidirectional LSTM (Yang et al., 2016) , for the i-th entity we define f i as the embedding feature. To compute the relatedness or affinity between entity m and entity n, we define an attention function F which computes the pairwise attention weight as

ω mn = F(f m , f n ).

(1)

A specific form of this attention function applied in this paper is reviewed in Section 4.1, and it originates from the scaled dot product attention module of Vaswani et al. (2017) .

We can now build an attention graph G whose vertices m represent entities in a data source with features F = {f m } and whose edge weights {ω mn } represent pairwise affinities between the vertices. We define the graph adjacency matrix for this attention graph as W. We propose to supervise the learning of W so that the matrix entries ω mn corresponding to entity pairs with high co-occurrence in the training data gain higher attention weights.

3.2. SUPERVISION TARGET:

We now discuss how to construct ground truth supervision labels in matrix form to supervise the learning of the entries of W. For visual recognition tasks we want our attention weights to focus on relationships between objects from different categories, so for each entry t mn of the ground truth relationship label matrix T , we assign t mn = 1 only when: 1) entities m and n overlap with two different ground truth objects' bounding boxes with intersection over union (IOU) > 0.5 and 2) their category labels c m and c n are different. We provide a visualization of such ground truth relationships in Figure 4 of the appendix. For language tasks we want the attention weights to reveal meaningful word pairs. For example, semantic relationships between nouns and nouns, verbs and nouns, nouns and adjectives, adverbs and verbs, and adverbs and adjectives should be encouraged. To this end, we build a simple word category pair dictionary of valid pairings (see Figure 2 ) and assign label t mn = 1 when the word category pair c m and c n is found in this semantic pair dictionary.

Center-Mass Intuitively, we would like W to have high affinity weights at those entries where t mn = 1, and low affinity weights elsewhere, i.e., the attention weights should concentrate on the 1's in the ground truth relationship label matrix T . We capture this via a notion of center-mass M of ground truth relation weights, defined as

EQUATION

where W = sof tmax(W) is a matrix-wise softmax operation.

3.3. CENTER-MASS CROSS ENTROPY LOSS:

The key to our approach is the introduction of a center-mass cross entropy loss, which aims to focus attention weight learning so that ω mn is high for pairs of commonly occurring distinct entities. The Hu et al. (2018) . Bottom middle: we add a scene recognition branch to the backbone. Bottom right: we insert the Focused Attention Module into a Hierarchical Attention Network (Yang et al., 2016) . Here "FC" stands for fully connected layer and "Conv" stands for convolutional layer.

loss is computed as

EQUATION

When minimizing this center-mass loss more frequently occurring 1-labeled pairs in the matrix will cumulatively receive stronger emphasis, for example, human-horse pairs versus horse-chair pairs in natural images. Furthermore, when supervising the attention learning in conjunction with another task specific loss, the matrix entries that reduce the task loss will also be optimized. The resultant dominant ω mn entries will not only reflect entity pairs with high co-occurrence, but will also help improve the main objective. The focal term (1 -M) r (Lin et al., 2017) helps narrow the gap between well converged center-masses and those that are far from convergence. For example, with a higher center-mass value the gradient on the log loss will be scaled down, whereas for a lower center-mass the gradient will be scaled up. The focal term prevents the network from committing only to the most dominant ω mn entries, and thus promotes diversity. We choose r = 2 in our experiments, motivated by the model ablation study conducted in Section 5.3.

4. NETWORK ARCHITECTURE:

Our focused attention module originates from the scaled dot product attention module in Vaswani et al. (2017) . We discuss our network structures for the focused attention weight learning backbone and various specific tasks, as shown in Figure 3 , with implementation details provided in the appendix.

4.1. SCALED DOT PRODUCT ATTENTION NETWORK:

We briefly review the computation of attention weights in Vaswani et al. (2017) , given a pair of nodes from the attention graph defined in Section 3.1. Let an entity node consist of its feature embedding, defined as f . Given a reference entity node m, such as one of the blue boxes in Figure 1 , the attention weight ωmn indicates its affinity to a surrounding entity node n. It is computed using a softmax activation over the scaled dot products ω mn defined as:

EQUATION

Both W K and W Q are matrices and so this linear transformation projects the embedding features f m and f n into metric spaces to measure how well they match. The feature dimension after projection is d k . With the above formulation, the attention graph affinity matrix is defined as W = {ω mn }.

4.2. FOCUSED ATTENTION NETWORK (FAN) BACKBONE:

In Figure 3 (top) we illustrate the base Focused Attention Network architecture. Here the dot product attention weights W go through a matrix-wise softmax operation to generate the attention matrix output W, that is used for the focused supervision with the center-mass cross entropy loss defined in Section 3.3. We shall refer to this loss term as relation loss. In parallel, a row-wise softmax is applied to W to output the coefficients W agg , which are then used for attention weighted aggregation: f m out = n ω mn agg f n . The aggregated output from the FAN module is sent to a task specific loss function. The entire backbone is end-to-end trainable, with both the task loss and the relation loss.

4.3. OBJECT DETECTION AND RELATIONSHIP PROPOSALS:

In Figure 3 (bottom left) we demonstrate how to generalize the FAN module for object detection and relationship proposal generation. The network is end-to-end trainable with detection loss, RPN loss and our relation loss. In addition to the ROI pooling features F ∈ R N obj ×1024 from the Faster R-CNN backbone of Ren et al. (2015) , contextual features F c from attention aggregation are applied to boost detection performance: F c = W agg F. The final feature descriptor for the detection head is Hu et al. (2018) . In parallel, the attention matrix output W ∈ R N ×N is used to generate relationship proposals by finding the top K weighted pairs in the matrix.

F d = F + F c , following

4.4. SCENE CATEGORIZATION TASK:

In Figure 3 (bottom middle) we demonstrate how to apply the FAN module to scene categorization. Since there are no bounding box annotations in most scene recognition datasets, we adopt a pre-trained FAN detection module, described in Section 4.3, in conjunction with a newly added convolution branch, to perform scene recognition. From the convolution backbone, we apply an additional convolution layer followed by a global average pooling to acquire the scene level feature descriptor F s . The FAN module takes as input the object proposals' visual features F, and outputs the aggregation result as the scene contextual feature F c . The input to the scene classification head thus becomes F meta = concat(F s , F c ), and the class scores are output.

4.5. DOCUMENT CATEGORIZATION TASK:

In Figure 3 (bottom right) we demonstrate how to apply the FAN module to a document classification task, using Hierarchical Attention Networks (Yang et al., 2016) . We insert the FAN module into the word level attention layer, but making it parallel to the original word-to-sentence attention module, and denote it as "FAN-hatt". FAN module learns word-to-word attention and outputs a sentence level descriptor. The FAN descriptor is concatenated with the output from the word-to-sentence attention to result in enhanced sentence level embedding. The rest of the computation follows the original paper (Yang et al., 2016) , where a sentence level attention model is applied and a final document level descriptor is abstracted.

5. EXPERIMENTS:



5.1. DATASETS AND NETWORK TRAINING:

Datasets. We evaluate our FAN architecture using the following datasets: VOC07, MSCOCO, Visual Genome, MIT67, 20 Newsgroups, Yahoo Answers. Details are provided in the appendix.

Vision Network Training Following Section 4.3, we first train the detection-and-relation joint framework end-to-end with a detection task loss and a relation loss on the minicoco dataset, dubbing this network "FAN-minicoco". We report detection results as well as relation learning quality. We then fine tune the scene task structure on the MIT67 dataset, using the pre-trained FAN-minicoco network (see Section 4.4), and report scene categorization performance. Details on the network architecture, the input/output dimensions and the hyper-parameters are in the appendix. Language Network Training We train the FAN models with Hatt (Yang et al., 2016) as the backbone. Details on the architecture and hyper-parameters are in the appendix.

5.2. EVALUATION METRICS:

Object Detection Task. For the VOC07 and MSCOCO datasets the evaluation metric is mAP (mean average precision), as defined in (Everingham et al., 2010; Lin et al., 2014) . Relationship Proposal Task. We evaluate the learned relationships using a recall metric which measures the percentage of ground truth relations that are covered in the predicted top K relationship list, which is consistent with (Zhang et al., 2017; Zellers et al., 2018; Xu et al., 2017) . A detailed definition is in the appendix. Classification Tasks. For MIT67, 20 Newsgroups and Yahoo Answers, where the task is to classify scenes or documents, we use classification accuracy as the evaluation metric.

5.3. MODEL ABLATION STUDY:

Prior to evaluating the FAN model we carry out ablation studies to examine ways of supervising the center mass and different choices of loss functions.

Focused Supervision Strategies:

We consider different approaches to training the focused attention, using the detection-and-relation joint framework from Section 4.3 on the VOC07 dataset for each case. First, we apply a row-wise softmax over the pre-activation matrix W and calculate the center-mass in a row-wise manner and apply the center-mass cross entropy loss accordingly. We refer to this as "row". Second, we apply the supervision explained in Section 3.3 but without the use of the focal term, and refer to this as "mat". Third, we add the focal term to the matrix supervision, referring to this as "mat-focal". The results in Table 1 show that the focused attention weights, when supervised using the center-mass cross entropy loss with a focal term (Section 3.3), are better concentrated on inter-object relationships, as reflected by the recall metric (Section 5.2) when compared with the unsupervised case. In all further experiments, unless stated otherwise, we apply the matrix supervision with the focal term, since it provides the best performance. Design of Loss Functions. We conduct a second ablation study to examine additional loss functions for optimizing the center mass as well as varying focal terms r, as introduced in Section 3.3. Defining VOC07 smooth L1 L2 r = 0 r = 1 r = 2 r = 3 r = 4 mAP@0.5 79.6 ± 0.2 79.7 ± 0.2 79.4 ± 0.1 79.5 ± 0.2 79.9 ± 0.2 79.8 ± 0.1 79.7 ± 0.2 recall@5k 60.3 ± 0.3 64.6 ± 0.5 62.1 ± 0.3 66.5 ± 0.2 69.9 ± 0.3 68.7 ± 0.4 67.1 ± 0.3 Table 3 : An evaluation of smooth L 1 and L 2 loss functions and variations of the focal loss factor r, on the VOC07 dataset. The results are reported as percentages (%) averaged over 3 runs.

x = 1 -M ∈ [0, 1], we consider loss function variants

EQUATION

)

in addition to the focal loss. The results in Table 3 show that focal loss is in general better than smooth L1 and L2 when supervising the center mass. In subsequent experiments we apply focal loss with r = 2, which empirically gives the best performance.

5.4. RELATIONSHIP PROPOSAL TASK:

Table 2 shows the evaluation of relationships learned on the Visual Genome dataset, by the unsupervised Focused Attention Network model "unsup" (similar to (Hu et al., 2018) ), the Focused Attention Network supervised with weak relation labels described in Section 3.2 "sup-cate", and supervision with human annotated ground truth relation labels "sup-gt". We also include the reported recall metric from Relationship Proposal Networks (Zhang et al., 2017) , which is a state-of-the-art level relationship learning network with strong supervision, using ground truth relationships. Our center-mass cross entropy loss does not require potentially costly human annotated relationship labels for learning, yet it achieves the same level of performance as the present state-of-the-art (Zhang et al., 2017) (the green curve in Table 2 ). When supervised with the ground truth relation labels instead of the weak labels (Section 3.2), we significantly outperform relation proposal networks (by about 25% in relative terms for all K thresholds) with this recall metric (the red curve in Table 2 ).

5.5. OBJECT DETECTION TASK:

In Table 4 we provide object detection results on the VOC07 and MSCOCO datasets. In both cases we improve upon the baseline and slightly outperform the unsupervised case (similar to Relation Networks (Hu et al., 2018) ). This suggests that relation weights learned using our focused attention network are at least as good as those from (Hu et al., 2018) , in terms of object detection performance.

VOC07 base F-RCNN FAN + L det (Hu et al., 2018) (2015) and L rel for the center mass relation loss defined in section 3.3.

We adopt the FAN-minicoco network (Section 5.1), and add an additional scene task branch to fine tune it on MIT67, as discussed in Section 4.4. Table 5 shows the results of applying this model to the MIT67 dataset. We refer to the backbone as "CNN" (first column), which is the left branch in Figure 3 (bottom middle). In the second column we apply the same network further fine-tuned on minicoco before training on MIT67. In the third column we include the detection branch, which is the right branch in Figure 3 (bottom middle) but remove its FAN module. In the fourth and fifth columns we apply the full scene architecture in Figure 3 (bottom middle), using FAN-minicoco network pretrained without (unsupervised) and with (supervised) relation loss, respectively. The FAN supervised case (fifth column) demonstrates a non-trivial improvement over the baseline (third column) and also significantly outperforms the unsupervised case (fourth column). This suggests that the relation weights learned solely by minimizing detection loss do not generalize well to a scene task, whereas those learned by our Focused Attention Network supervised by weak relations labels can. We hypothesize that recovering informative relations between distinct objects, which is what our Focused Attention Network is designed to do, is particularly beneficial for scene categorization. We present document classification results in Table 6 . We provide a comparison between the base Hierarchical Attention Networks (Hatt), and FAN-hatt, explained in Section 4.5, with and without relation loss supervision. The supervised (semantic) focused attention supervision results in an improvement over both the unsupervised case and the baseline, particularly when the training data per category is small (top two rows). This advantage diminishes, however, when the training volume is dramatically increased, suggesting that in the latter case a baseline network is able to perform equally well. In addition, the qualitative distributions of word importance factors (see appendix for an exact definition) in Figure 2 suggest that focused semantic attention encourages more diversity in attention weight learning, which in turn leads to better document classification performance.



6. CONCLUSION:

Our Focused Attention Network is versatile, and allows the user to direct the learning of attention weights in the manner they choose. The application of the FAN module allows multiple informative entities in the data source to contribute to the learned feature representation, abstracting diverse aspects of the data. In multiple experiments we have demonstrated the benefit of learning relations between distinct objects for computer vision tasks, and between lexical categories (words) for document classification tasks. It not only boosts performance in object detection, scene categorization and document classification, but also leads to state-of-the-art performance in a relationship proposal task.

In the future we envision its use as a component for deep learning architectures where supervised control of relationship weights is desired, since it is adaptable, modular, and end-to-end trainable in conjunction with a task specific loss.

between different object instances is focused. That is, as long as object proposal a and object proposal b are different object instances in an image, we consider a possible relationship between them and assign t ab = T [a, b] = 1. We refer to this as different instance focused supervision. We provide a visual example of the above mentioned supervision targets in Figure 4 . In Table 7 , we provide object detection results on VOC07 when training the FAN model using different supervision targets.

Different Category Different Instance:

VOC07 varying T Diff Instance Diff Category avg mAP (%) 47.6 ± 0.1 48.2 ± 0.2 mAP@0.5 (%) 79.5 ± 0.2 79.9 ± 0.2

Table 7 : Detection results on the VOC07 dataset when varying supervision targets, where we show maen accuracies over 3 runs.

Language Tasks. In our paper we focused attention between word categories according to English grammar. Here we consider additional ways of supervising the word-to-word relationships. In different category supervision, we consider a word pair (a, b) to have a valid ground truth relationship when a and b belongs to different lexical categories. In same category supervision, we consider a word pair (a, b) to have a valid ground truth relationship when a and b belong to the same lexical categories. In different word supervision, we consider a word pair (a, b) to have a valid ground truth relationship when a and b are different words. We provide a comparison of using the aforementioned supervision targets on the 20News dataset, in 

A.5 CONVERGENCE OF CENTER-MASS:

We provide additional results illustrating the convergence of center-mass M training in Table 9 . The center mass is a element wise multiplication between the post softmax attention weight matrix W and the ground truth label matrix T , defined as M = W T ∈ [0, 1]. More details are in Section (documents), and the word embedding dimension is set to 100. The maximum number of words in a sentence is set to be N w = 30, and the maximum number of sentences in a document is set to be N s = 15. Therefore, the word level Focused Attention Network's attention weight matrix W has a dimension of 30 × 30, for a single sentence. The output dimension for Bi-LSTMs is set to be 100, and the attention dimension in attention models is also set to be 100. The Adam optimizer (Kingma & Ba, 2015) is applied with an initial learning rate of 1e -3. The network is trained end-to-end with categorization loss and relation loss for 15 epochs. Further details regarding input/output dimensions of intermediate layers can be found in Figure 5 (right).

C SCALING OF LOSS TERMS:

We applied a balancing factor λ to properly weight the relation loss when training in conjunction with the main objective loss.

Detection and relation proposal:

We applied L = L det + λL rel , λ = 0.01 (6) where L det is defined in (Ren et al., 2015) , which is a combination of RPN losses and detection head losses.

Scene categorization:

The benefit of focused attention comes from a pretrained detection model and the scene task itself does not optimize the relation loss.

Document Classification:

For language tasks, we applied L = L cls + λL rel , λ = 0.1 (7) where L cls is a standard cross entropy loss for the classification task.

D IMPLEMENTATION DETAILS:

We ran multiple trials (3-5) of our experiments and reported error bars in our results, and observed that the numbers are relatively stable and are reproducible. Furthermore, we plan to release our code upon acceptance of this article. Given that all our datasets are publicly available this will allow other researchers to both reproduce our experiments and use our Focused Attention Network module for their own research.

D.1 VISION TASKS:

We implemented the center-mass cross entropy loss as well as the Focused Attention Module using MxNet. For the Faster R-CNN backbone, we adopted the source code from Relation Networks (Hu et al., 2018) .

Scene Categorization. In order to maintain the learned relationship weights from the pre-trained module, which helps encode object co-occurrence context in the aggregation result, we fix the parameters in the convolution backbone, RPN layer and Focused Attention Network module, but make all other layers trainable. Fixed layers are shaded in grey in Figure 3 (bottom middle).

D.2 LANGUAGE TASKS:

We implemented the Hierarchical Attention Networks according to Yang et al. (2016) in Keras with a TensorFlow backend. The word-to-word Focused Attention Network module as well as the center-mass cross entropy loss, are also implemented in the same Keras based framework.

D.3 RUNTIME AND MACHINE CONFIGURATION:

All our experiments are carried out on a linux machine with 2 Titan XP GPUs, an Intel Core i9 CPU and 64GBs of RAM. The Figures and Tables referred to in the following text are those in the main article.

• Figure 4 , Relationship Proposal. For a typical run of Visual Genome Focused Attention

Network training, it takes 55 hours for 8 epochs using the above machine configuration. We also determined that when compared with unsupervised cases of the above experiments, the use of the Focused Attention Network module does not add any noticeable run time overhead.

E ADDITIONAL VISUALIZATIONS:

Word Importance in a Sentence In Figure 6 , we provide additional visualizations of the word importance factor in a sentence (defined in Section 4.5 of the main article), using the same format as that used in Figure 2 in the main article. Visual Relationships We now provide additional qualitative visualizations showing typical relationship weights learned by our method. In Figure 7 , we visualize the predicted relationship on images from the MIT67 dataset, using a pre-trained Focused Attention Network on the minicoco dataset, referred to as FAN-minicoco, as discussed in Section 5.1 of the main article. We compare this with the corresponding unsupervised case, which is similar to Relation Networks (Hu et al., 2018) .

:



Figures and Tables:

Figure2: Left: Visualization of the word importance factor, which models the contribution of a given word to its sentence level context (see Section 4.5). The sentence is in a clockwise direction starting from 12 o'clock. "ait": weights learned using Hierarchical Attention Networks(Yang et al., 2016), "unsup": weights from the unsupervised case of our Focused Attention Network module, and "sup": weights from the supervised case. Right: semantic word-to-word relationship labels used in the supervision of our network (see Section 3.2).

Figure 4: The visualization of supervision targets for vision tasks. The blue box indicates a fixed reference object a and the orange boxes indicate the objects b that have ground truth relationship with a, for which we assign T [a, b] = 1. Left: different category supervision. Note that the sheep in the blue box is not related to the other sheep in the image. Right: different instance supervision. The sheep in the blue box now has a relationship to other sheep (in yellow boxes).

Figure 6: Additional visualization of the word importance factor in a sentence. See Section 3.2 and the caption of Figure 2 of the main article for an explanation.

Recall comparison for the Visual Genome dataset with varying top K, where the ground truth relation labels are human annotated.

Table content: <table/>

Object Detection Results. mAP@0.5: average precision over a bounding box overlap threshold as IOU = 0.5. avg mAP: averaged mAP over multiple bounding box overlap thresholds. VOC07 experiments are reported over 3 runs, demonstrating stability. L det stands for detection task loss as defined in Ren et al.

Table content: <table><tr><td>FAN + L det + L rel</td></tr></table>

MIT67 Scene Categorization Results. The important entries are averages over 3 runs. See the text in Section 5.6 for a discussion. For details regarding F s , F c and F , see Section 4.4.

Table content: <table><tr><td/><td>CNN</td><td>CNN</td><td>CNN + ROIs</td><td colspan="2">CNN + FAN-unsup CNN + FAN-sup</td></tr><tr><td>Pretraining</td><td colspan="3">Imgnet Imgnet+COCO Imgnet+COCO</td><td>Imgnet+COCO</td><td>Imgnet+COCO</td></tr><tr><td>Features</td><td>FS</td><td>FS</td><td>FS, max(F)</td><td>FS, FC</td><td>FS, FC</td></tr><tr><td>Accuracy (%)</td><td>75.1</td><td>76.8</td><td>78.0 ± 0.3</td><td>77.1 ± 0.2</td><td>80.2 ± 0.3</td></tr></table>

Document categorization results for the 20 Newsgroups and Yahoo Answers datasets, with the results averaged over 5 trials. For the Yahoo dataset, we train on sub-sampled training sets and report results on the full test set.

Table content: <table><tr><td>Datasets</td><td>Train Vol. Per Cate</td><td colspan="3">Hatt (Yang et al., 2016) FAN-hatt-unsup FAN-hatt-sup</td></tr><tr><td>20 News</td><td>550</td><td>64.0 ± 0.3</td><td>64.6 ± 0.2</td><td>65.6 ± 0.2</td></tr><tr><td>Yahoo-mini</td><td>5,000</td><td>64.9 ± 0.2</td><td>65.2 ± 0.2</td><td>66.0 ± 0.1</td></tr><tr><td>Yahoo-half</td><td>70,000</td><td>72.2 ± 0.1</td><td>72.1 ± 0.1</td><td>72.4 ± 0.1</td></tr></table>

, with the supervision constraints becoming stricter from the leftmost column to the rightmost column.

Table content: <table><tr><td>20news varying T</td><td colspan="3">Diff Word Same Category Diff Category</td><td>Semantic</td></tr><tr><td>Accuracy (%)</td><td>64.7 ± 0.2</td><td>64.9 ± 0.2</td><td>65.1 ± 0.1</td><td>65.6 ± 0.2</td></tr></table>

Document categorization results for the 20 Newsgroups dataset when varying the supervision target, where we show mean accuracies over 5 runs.

Table content: <table/>

• Table 1 Object, Detection. For a typical run of VOC07 Focused Attention Network training, it takes 4 hours when training for 8 epochs. For a typical run on minicoco , it takes 26 hours using the same setup. • Table 2, Scene Categorization. For a typical run of the MIT67 dataset, it takes 2 hours when training for 8 epochs. • Table 3, Document Classification. For a typical run of the 20 Newsgroup dataset, it takes 30 minutes for 15 epochs.

Table content: <table/>

Peer review (iclr2020):
This paper proposes to learn attention among pairs of input units (they call these entities) as a matrix of attention values. These when aggregated along a row gives a summary of related entities for each input entity, and when aggregated overall gives the *soft* entity-pairs in an input. They apply it on four image-related tasks: object detection, categorization, relation , and scene detection and one document classification task. Direct supervision for learning the entity-pair attention is only available from the relation proposal task. They also induce indirect supervision from the object detection task using co-occurrence of objects in an image. For text they hypothesize attention pair supervision based on POS tags. Their experiments show that only for the relation proposition task their method provides significant gains over existing methods, that too in the fully supervised mode. For scene categorization their method is good only with supervision on attention relation. For object detection their gains are insignificant and for text classification they show gains only with the labeled data is small. I have the following concerns about this work: 1. Computing pairwise attention is computationally expensive, and the gains they obtain seem too small to justify. The only significant gains are for the relation proposal task where such pairwise attention is expected to be the norm. 2. Their experiments do not perform a fair comparison over the number of parameters and running time. The FAN module inserts lot more parameters and quadratically blows up the attention vector length. If we normalize on the number of parameters and running time, it is unclear if the gains of FAN over baseline will continue to hold. For example, in the document classification task, they concatenate the FAN descriptor with usual word to sentence attention. What if a multi-headed attention is used on the word to sentence layer? It is unclear if FAN will score over multi-headed attention. 3. Their coverage of related work and empirical comparison with existing work is sketchy. Authors have not done justice to the multiple prior work that seek to supervise attention or model the relationship between attention of multiple entities. Several papers on supervising attention but in the section *Attention Networks – Limitations* the authors do not mention these. I list some of these: Syntax-Directed Attention for Neural Machine Translation Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Tiejun Zhao AAAI 2018. Siddhesh Khandelwal, Leonid Sigal: AttentionRNN: A Structured Spatial Attention Mechanism. CoRR abs/1905.09400 (2019) Attentive Relational Networks for Mapping Images to Scene Graphs Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, Jiebo Luo; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3957-3966 Supervised Attentions for Neural Machine Translation Haitao Mi Zhiguo Wang Abe Ittycheriah https://www.aclweb.org/anthology/D16-1249.pdf Question for authors: --------------------------------- Please clarify the exact way in which the FAN module that provides word-pair attention is converted into a sentence descriptor. From Figure~3 it appears that you do row-wise max-pooling but please specify clearly in section 4.5, preferably with a formula. The form of Equation 3 needs more justification. Why this specific form? Presentation: ------------------ The first section is hard to read and requires significant improvement in writing. I give some suggestions: Typo in sentence: *for the task of learning relationship attention weights between entities* *relation weights* is an awkward construct. A citation is required on *Relation Networks* in their first mention in this sentence: * Typical qualitative examples comparing Relation Networks with our Focused Attention Network are shown in Figure 1, with a quantitative comparison reported in Section 5.* The sentences in the paper are often hard to parse. Here is are some examples: *The scaled dot product attention module of Vaswani et al. (2017), for example, uses learned pairwise attention weights between region proposal network (RPN) generated bounding boxes in images of natural scenes (Hu et al., 2018) to boost object detection.* *In fact, for a given reference object (region), relation networks (Hu et al., 2018) tend to predict high attention weights with scaled or shifted bounding boxes surrounding the same object instance. This is likely because including surrounding context, or simply restoring missing parts of the reference object, boosts object detection* The illustration of attention in Figure 2 is not at all clear.