Experiments ::: Relation Extraction Performance	We first compare our proposed SeG with aforementioned approaches in Table TABREF19 for top-N precision (i.e., P@N). As shown in the top panel of the table, our proposed model SeG can consistently and significantly outperform baseline (i.e., PCNN+ATT) and all recently-promoted works in terms of all P@N metric. Compared to PCNN with selective attention (i.e., PCNN+ATT), our proposed SeG can significantly improve the performance by 23.6% in terms of P@N mean for all sentences; even if a soft label technique is applied (i.e., PCNN+ATT+SL) to alleviate wrongly labeled problem, our performance improvement is also very significant, i.e., 7.8%. Compared to previous state-of-the-art approaches (i.e., PCNN+HATT and PCNN+BAG-ATT), the proposed model can also outperform them by a large margin, i.e., 10.3% and 5.3% , even if they propose sophisticated techniques to handle the noisy training data.

Compared to selective attention, one major advantage of the proposed gating mechanism is that, it performs stably and promisingly even if only one sentence appears in a bag and thus keeps the consistency across all training examples. The experiments on NYT dataset demonstrate that our approach achieves a new state-of-the-art performance in terms of both AUC and top-n precision metrics.

Compared to the baseline framework (i.e., selective attention for multi-instance learning), SeG is able to produce entity-aware embeddings and rich-contextual representations to facilitate downstream aggregation modules that stably learn from noisy training data. Moreover, SeG uses gate mechanism with pooling to overcome problem occurring in selective attention, which is caused by one-sentence bags. In addition, it still keeps a light-weight structure to ensure the scalability of this model. The experiments and extensive ablation studies on New York Time dataset  show that our proposed framework achieves a new state-of-the-art performance regarding both AUC and top-n precision metrics for distantly supervised relation extraction task, and also verify the significance of each proposed module. Particularly, the proposed framework can achieve AUC of 0.51, which outperforms selective attention baseline by 0.14 and improves previous state-of-the-art approach by 0.09.

Conclusion	In this paper, we propose a brand-new framework for distantly supervised relation extraction, i.e., selective gate (SeG) framework, as a new alternative to previous ones. It incorporates an entity-aware embedding module and a self-attention enhanced selective gate mechanism to integrate task-specific entity information into word embedding and then generates a complementary context-enriched representation for PCNN. The proposed framework has certain merits over previously prevalent selective attention when handling wrongly labeled data, especially for a usual case that there are only one sentence in the most of bags. The experiments conduct on popular NYT dataset show that our model SeG can consistently deliver a new benchmark in state-of-the-art performance in terms of all P@N and precision-recall AUC. And further ablation study and case study also demonstrate the significance of the proposed modules to handle wrongly labeled data and thus set a new state-of-the-art performance for the benchmark dataset. In the future, we plan to incorporate an external knowledge base into our framework, which may further boost the prediction quality by overcoming the problems with a lack of background information as discussed in our error analysis.

Experiments ::: Metrics	Following previous works , , , , we use precision-recall (PR) curves, area under curve (AUC) and top-N precision (P@N) as metrics in our experiments on the held-out test set from the NYT dataset. To directly show the perfomance on one sentence bag, we also calculate the accuracy of classification (Acc.) on non-NA sentences.

Proposed Approach ::: Self-Attention Enhanced Neural Network	Previous works of relation extraction mainly employ a piecewise convolutional neural network (PCNN)  to obtain contextual representation of sentences due to its capability of capturing local features, less computation and light-weight structure. However, some previous works  find that CNNs cannot reach state-of-the-art performance on a majority of natural language processing benchmarks due to a lack of measuring long-term dependency, even if stacking multiple modules. This motivates us to enhance the PCNN with another neural module, which is capable of capturing long-term or global dependencies to produce complementary and more powerful sentence representation. Hence, we employ a self-attention mechanism in our model due to its parallelizable computation and state-of-the-art performance. Unlike existing approaches that sequentially stack self-attention and CNN layers in a cascade form , , we arrange these two modules in parallel so they can generate features describing both local and long-term relations for the same input sequence. Since each bag may contain many sentences (up to 20), a light-weight networks that can can efficiently process these sentences simultaneously is more preferable, such as PCNN that is the most popular module for relation extraction.

These verify the effectiveness of our approach over previous works when solving the wrongly labeled problem that frequently appears in distantly supervised relation extraction. Moreover, for proposed approach and comparative ones, we also show AUC curves and available numerical values in Figure FIGREF31 and Table TABREF32 respectively. The empirical results for AUC are coherent with those of P@N, which shows that, our proposed approach can significantly improve previous ones and reach a new state-of-the-art performance by handling wrongly labeled problem using context-aware selective gate mechanism. Specifically, our approach substantially improves both PCNN+HATT and PCNN+BAG-ATT by 21.4% in aspect of AUC for precision-recall.

Experiments ::: Error Analysis ::: Isolated Sentence in Bag	Each sentence in a bag can be regarded as independent individual and do not have any relationship with other sentences in the bag, which possibly leads to information loss among the multiple sentences in the bag when considering classification over bag level.

Experiments ::: Baselines and Competitive Approaches	We compare our proposed approach with extensive previous ones, including feature-engineering, competitive and state-of-the-art approaches, which are briefly summarized in the following. Mintz  is the original distantly supervised approach to solve relation extraction problems with distantly supervised data. MultiR  is a graphical model within a multi-instance learning framework that is able to handle problems with overlapping relations. MIML  is a multi-instance, multi-label learning framework that jointly models both multiple instances and multiple relations. PCNN+ATT  employs a selective attention over multiple instances to alleviate the wrongly labeled problem, which is the principal baseline of our work. PCNN+ATT+SL  introduces an entity-pair level denoising method, namely employing a soft label to alleviate the impact of wrongly labeled problem. PCNN+HATT  employs hierarchical attention to exploit correlations among relations. PCNN+BAG-ATT  uses an intra-bag to deal with the noise at sentence-level and an inter-bag attention to deal with noise at the bag-level.

For this reason, there is only one light-weight self-attention layer in our model. This is contrast to  yu2018qanet and  wu2019pay who stack both modules many times repeatedly. Our experiments show that two modules arranged in parallel manner consistently outperform stacking architectures that are even equipped with additional residual connections ). The comparative experiments will be elaborated in Section SECREF34 and SECREF35.

Particularly, by removing entity-aware embedding (i.e, SeG w/o Ent) and self-attention enhanced selective gate (i.e., SeG w/o Gate w/o Self-Attn), it shows 11.5% and 1.8% decreases respectively in terms of P@N mean for all sentences. Note that, when dropping both modules above (i.e., SeG w/o ALL), the framework will be degenerated as selective attention baseline , which again demonstrates that our proposed framework is superior than the baseline by 15% in terms of P@N mean for all sentences. To verify the performance of selective gate modul when handling wrongly labeled problem, we simply replace the selective gate module introduced in Eq.(DISPLAY_FORM15) with selective attention module, namely, SeG+Attn w/o Gate, and instead of mean pooling in Eq.(DISPLAY_FORM15), we couple selective gate with selective attention to fulfill aggregation instead mean-pooling, namely, SeG+Attn. Across the board, the proposed SeG still deliver the best results in terms of both metrics even if extra selective attention module is applied.

However, aside from the relative position features, we argue that the embeddings of both the head entity $e_h$ and tail entity $e_t$ are also vitally significant for relation extraction task, since the ultimate goal of this task is to predict the relationship between these two entities. This hypothesis is further verified by our quantitative and qualitative analyses in later experiments (Section SECREF35 and SECREF39). The empirical results show that our proposed embedding can outperform the widely-used way in prior works . In particular, we propose a novel entity-aware word embedding approach to enrich the traditional word embeddings with features of the head and tail entities. To this end, a position-wise gate mechanism is naturally leveraged to dynamically select features between relative position embedding and entity embeddings. Formally, the embeddings of head and tail entities are denoted as $\mathbf {v}^{(h)}$ and $\mathbf {v}^{(t)}$ respectively.

Experiments ::: Ablation Study	To further verify the effectiveness of each module in the proposed framework, we conduct an extensive ablation study in this section. In particular, SeG w/o Ent denotes removing entity-aware embedding, SeG w/o Gate denotes removing selective gate and concatenating two representations from PCNN and self-attention, SeG w/o Gate w/o Self-Attn denotes removing self-attention enhanced selective gate. In addition, we also replace the some parts of the proposed framework with baseline module for an in-depth comparison. SeG+ATT denotes replacing mean-pooing with selective attention, and SeG w/ stack denotes using stacked PCNN and self-attention rather than in parallel. The P@N results are listed in the bottom panel of Table TABREF19, and corresponding AUC results are shown in Table TABREF36 and Figure FIGREF37. According to the results, we find that our proposed modules perform substantially better than those of the baseline in terms of both metrics.

Experiments ::: Dataset	In order to accurately compare the performance of our model, we adopt New York Times (NYT) dataset , a widely-used standard benchmark for distantly supervised relation extraction in most of previous works , , , , which contains 53 distinct relations including a null class NA relation. This dataset generates by aligning Freebase with the New York Times (NYT) corpus automatically. In particular, NYT dataset contains 53 distinct relations including a null class NA relation referred to as the relation of an entity pair is unavailable. There are 570K and 172K sentences respectively in training and test set.

Two examples of one-sentence bag are shown in Table TABREF1. These results indicate that, in training phrase the selective attention module is enforced to output a single-valued scalar for $80\%$ examples, leading to an ill-trained attention module and thus hurting the performance. Motivated by aforementioned observations, in this paper, we propose a novel Selective Gate (SeG) framework for distantly supervised relation extraction. In the proposed framework, 1) we employ both the entity embeddings and relative position embeddings  for relation extraction, and an entity-aware embedding approach is proposed to dynamically integrate entity information into each word embedding, yielding more expressively-powerful representations for downstream modules; 2) to strengthen the capability of widely-used piecewise CNN (PCNN)  on capturing long-term dependency , we develop a light-weight self-attention ,  mechanism to capture rich dependency information and consequently enhance the capability of neural network via producing complementary representation for PCNN; and 3) based on preceding versatile features, we design a selective gate to aggregate sentence-level representations into bag-level one and alleviate intrinsic issues appearing in selective attention.

In plainer terms, this means any possible relation between two entities hold true in at least one distantly-labeled sentence rather than all of the them that contains those two entities. In particular, instead of generating a sentence-level label, this framework assigns a label to a bag of sentences containing a common entity pair, and the label is a relationship of the entity pair on knowledge graph. Recently, based on the labeled data at bag level, a line of works , , , ,  under selective attention framework  let model implicitly focus on the correctly labeled sentence(s) by an attention mechanism and thus learn a stable and robust model from the noisy data. However, such selective attention framework is vulnerable to situations where a bag is merely comprised of one single sentence labeled; and what is worse, the only one sentence possibly expresses inconsistent relation information with the bag-level label. This scenario is not uncommon. For a popular distantly supervised relation extraction benchmark, e.g., NYT dataset , up to $80\%$ of its training examples (i.e., bags) are one-sentence bags. From our data inspection, we randomly sample 100 one-sentence bags and find $35\%$ of them is incorrectly labeled.

Proposed Approach ::: Selective Gate	Given a sentence bag $B = [s_1, \dots , s_m]$ with common entity pair, where $m$ is the number of sentences. As elaborated in Section SECREF6, we can obtain $\mathbf {S} = [\mathbf {s}_1, \dots , \mathbf {s}_m]$ and $\mathbf {U} = [\mathbf {u}_1, \dots , \mathbf {u}_m]$ for each sentence in the bag, which are derived from PCNN and self-attention respectively. Unlike previous works under multi-instance framework that frequently use a selective attention module to aggregate sentence-level representations into bag-level one, we propose a innovative selective gate mechanism to perform this aggregation. The selective gate can mitigate problems existing in distantly supervised relation extraction and achieve a satisfactory empirical effectiveness. Specifically, when handling the noisy instance problem, selective attention tries to produce a distribution over all sentence in a bag; but if there is only one sentence in the bag, even the only sentence is wrongly labeled, the selective attention mechanism will be low-effective or even completely useless.

Lastly, to explore the influence of the way to combine PCNN with self-attention mechanism, we stack them by following the previous works , i.e., SeG w/ Stack. And we observe a notable performance drop after stacking PCNN and self-attention in Table TABREF36. This verifies that our model combining self-attention mechanism and PCNN in parallel can achieve a satisfactory result. To further empirically evaluate the performance of our method in solving one-sentence bag problem, we extract only the one-sentence bags from NYT's training and test sets, which occupy 80% of the original dataset. The evaluation and comparison results in Table TABREF33 show that compared to PCNN+ATT, the AUC improvement (+0.13) between our model and PCNN+ATT on one-sentence bags is higher than the improvement of full NYT dataset, which verifies SeG's effectiveness on one-sentence bags. In addition, PCNN+ATT shows a light decrease compared with PCNN, which can also support the claim that selective attention is vulnerable to one-sentence bags.

Self-Attention Enhanced Selective Gate with Entity-Aware Embedding for Distantly Supervised Relation Extraction	Distantly supervised relation extraction intrinsically suffers from noisy labels due to the strong assumption of distant supervision. Most prior works adopt a selective attention mechanism over sentences in a bag to denoise from wrongly labeled data, which however could be incompetent when there is only one sentence in a bag. In this paper, we propose a brand-new light-weight neural framework to address the distantly supervised relation extraction problem and alleviate the defects in previous selective attention framework. Specifically, in the proposed framework, 1) we use an entity-aware word embedding method to integrate both relative position information and head/tail entity embeddings, aiming to highlight the essence of entities for this task; 2) we develop a self-attention mechanism to capture the rich contextual dependencies as a complement for local dependencies captured by piecewise CNN; and 3) instead of using selective attention, we design a pooling-equipped gate, which is based on rich contextual representations, as an aggregator to generate bag-level representation for final relation classification.

Related Work	Recently, many works ,  employed selective attention  to alleviate wrongly labeled problem existing in distantly supervised RE. For example,  han2018hierarchical propose a hierarchical relation structure attention based on the insight of selective attention. And,  ye2019distant extend the sentence-level selective attention to bag-level, where the bags have same relation label. Differing from these works suffering from one-sentence bag problem due to the defect of selective attention, our proposed approach employ a gate mechanism as an aggregator to handle this problem. There are several works recently proposed to couple CNN with self-attention , ,  for either natural language processing or computer vision. For example,  yu2018qanet enrich CNN's representation with self-attention for machine reading comprehension. Unlike these works stacking the two modules many times, we arrange them in parallel instead of to ensure model's scalability. In addition, some previous approach explore the importance of entity embedding for relation extraction , , which usually need the support external knowledge graph and learn the entity embeddings over the graph. In contrast, this approach considers the entity embeddings within a sentence and incorporate them with relative position feature without any external support.

Proposed Approach	As illustrated in Figure FIGREF2, we propose a novel neural network, i.e., SeG, for distantly supervised relation extraction, which is composed of following neural components.

Proposed Approach ::: Entity-Aware Embedding	Given a bag of sentences $B^k = \lbrace s^k_1, \dots , s^k_{m^k}\rbrace $ where each sentence contains common entity pair (i.e., head entity $e^k_h,$ and tail entity $e^k_t$), the target of relation extraction is to predict the relation $y^k$ between the two entities. For a clear demonstration, we omit indices of example and sentence in remainder if no confusion caused. Each sentence is a sequence of tokens, i.e., $s = [w_1, \dots , w_n]$, where $n$ is the length of the sentence. In addition, each token has a low-dimensional dense-vector representation, i.e., $[\mathbf {v}_1, \cdots , \mathbf {v}_n] \in \mathbb {R}^{d_w \times n}$, where $d_w$ denotes the dimension of word embedding. In addition to the typical word embedding, relative position is a crucial feature for relation extraction, which can provide downstream neural model with rich positional information , .

Relative positions explicitly describe the relative distances between each word $w_i$ and the two targeted entities $e_h$ and $e_t$. For $i$-th word, a randomly initialized weight matrix projects the relative position features into a two dense-vector representations w.r.t the head and tail entities, i.e., $\mathbf {r}^{e_h}_i$ and $\mathbf {r}^{e_t}_i\in \mathbb {R}^{d_r}$ respectively. The final low-level representations for all tokens are a concatenation of the aforementioned embeddings, i.e., $\mathbf {X}^{(p)} = [\mathbf {x}^{(p)}_1, \cdots , \mathbf {x}^{(p)}_n] \in \mathbb {R}^{d_p \times n}$ in which $\mathbf {x}^{(p)}_i = [\mathbf {v_i}; \mathbf {r}^{e_h}_i; \mathbf {r}^{e_t}_i]$ and $d_p = d_w + 2\times d_r$.

The position-wise gating procedure is formulated as in which $\mathbf {W}^{(g1)}\in \mathbb {R}^{d_h \times 3d_w}$ and $\mathbf {W}^{(g2)}\in \mathbb {R}^{d_h \times d_p}$ are learnable parameters, $\lambda $ is a hyper-parameter to control smoothness, and $\mathbf {X} = [\mathbf {x}_1, \dots , \mathbf {x}_n] \in \mathbb {R}^{d_h \times n}$ containing the entity-aware embeddings of all tokens from the sentence.

Note that almost $80\%$ of bags from popular relation extraction benchmark consist of only one sentence, and many of them suffer from the wrong label problem. In contrast, our proposed gate mechanism is competent to tackle such case by directly and dynamically aligning low gating value to the wrongly labeled instances and thus preventing noise representation being propagated. Particularly, a two-layer feed forward network is applied to each $\mathbf {u}_j$ to sentence-wisely produce gating value, which is formally denoted as where, $\mathbf {W}^{(g1)} \in \mathbb {R}^{3d_c \times d_h}$, $\mathbf {W}^{(g2)} \in \mathbb {R}^{d_h \times d_h}$, $\sigma (\cdot )$ denotes an activation function and $g_j \in (0, 1)$. Then, given the calculated gating value, an mean aggregation performs over sentence embeddings $[\mathbf {s}_j]_{j=1}^m$ in the bag, and thus produces bag-level vector representation for further relation classification.

Proposed Approach ::: Self-Attention Enhanced Neural Network ::: Piecewise Convolutional Neural Network	This section provides a brief introduction to PCNN as a background for further integration with our model, and we refer readers to  zeng2015distant for more details. Each sentence is divided into three segments w.r.t. the head and tail entities. Compared to the typical 1D-CNN with max-pooling , piecewise pooling has the capability to capture the structure information between two entities. Therefore, instead of using word embeddings with relative position features $\mathbf {X}^{(p)}$ as the input, we here employ our entity-aware embedding $\mathbf {X}$ as described in Section SECREF3 to enrich the input features. First, 1D-CNN is invoked over the input, which can be formally represented as where, $\mathbf {W}^{(c)} \in \mathbb {R}^{d_c \times m \times d_h}$ is convolution kernel with window size of $m$ (i.e., $m$-gram).

Then, to obtain sentence-level representation, a piecewise pooling performs over the output sequence, i.e., $\mathbf {H}^{(c)} = [\mathbf {h}_1, \dots , \mathbf {h}_n]$, which is formulated as In particular, $\mathbf {H}^{(1)}$, $\mathbf {H}^{(2)}$ and $\mathbf {H}^{(3)}$ are three consecutive parts of $\mathbf {H}$, obtained by dividing $\mathbf {H}$ according to the positions of head and tail entities. Consequently, $\mathbf {s} \in \mathbb {R}^{3d_c}$ is the resulting sentence vector representation.

Proposed Approach ::: Self-Attention Enhanced Neural Network ::: Self-Attention Mechanism	To maintain efficiency of proposed approach, we adopt the recently-promoted self-attention mechanism , , , ,  for compressing a sequence of token representations into a sentence-level vector representation by exploiting global dependency, rather than computation-consuming pairwise ones . It is used to measure the contribution or importance of each token to relation extraction task w.r.t. the global dependency. Formally, given the entity-aware embedding $\mathbf {X}$, we first calculate attention probabilities by a parameterized compatibility function, i.e., where, $\mathbf {W}^{(a1)}, \mathbf {W}^{(a2)} \in \mathbb {R}^{d_h \times d_h}$ are learnable parameters, $\operatornamewithlimits{softmax}(\cdot )$ is invoked over sequence, and $\mathbf {P}^{(A)}$ is resulting attention probability matrix. Then, the result of self-attention mechanism can be calculated as in which, $\sum $ is performed along sequential dimension and $\odot $ stands for element-wise multiplication.

And, $\mathbf {u} \in \mathbb {R}^{d_h}$ is also a sentence-level vector representation which is a complement to PCNN-resulting one, i.e., $\mathbf {s}$ from Eq.(DISPLAY_FORM9).

Acknowledgements	This research was funded by the Australian Government through the Australian Research Council (ARC) under grants LP180100654 partnership with KS computer. We also acknowledge the support of NVIDIA Corporation and Google Cloud with the donation of GPUs and computation credits respectively.

This procedure is formalized as Finally, $\mathbf {c}$ is fed into a multi-layer perceptron followed with $|C|$-way $\operatornamewithlimits{softmax}$ function (i.e., an $\operatornamewithlimits{MLP}$ classifier) to judge the relation between head and tail entities, where $|C|$ is the number of distinctive relation categories. This can be regarded as a classification task . Formally,

Proposed Approach ::: Model Learning	We minimize negative log-likelihood loss plus $L_2$ regularization penalty to train the model, which is written as where $\mathbf {p}^k$ is the predicted distribution from Eq.(DISPLAY_FORM16) for the $k$-th example in dataset $\mathcal {D}$ and $y^k$ is its corresponding distant supervision label.

Experiments	To evaluate our proposed framework, and to compare the framework with baselines and competitive approaches, we conduct experiments on a popular benchmark dataset for distantly supervised relation extraction. We also conduct an ablation study to separately verify the effectiveness of each proposed component, and last, case study and error analysis are provided for an insight into our model.

Experiments ::: Training Setup	For a fair and rational comparison with baselines and competitive approaches, we set most of the hyper-parameters by following prior works , , and also use 50D word embedding and 5D position embedding released by ,  for initialization, where the dimension of $d_h$ equals to 150. The filters number of CNN $d_c$ equals to 230 and the kernel size $m$ in CNN equals to 3. In output layer, we employ dropout  for regularization, where the drop probability is set to $0.5$. To minimize the loss function defined in Eq.DISPLAY_FORM18, we use stochastic gradient descent with initial learning rate of $0.1$, and decay the learning rate to one tenth every 100K steps.

Experiments ::: Case Study	In this section, we conduct a case study to qualitatively analyze the effects of entity-aware embedding and self-attention enhanced selective gate. The case study of four examples is shown in Table TABREF38. First, comparing Bag 1 and 2, we find that, without the support of the self-attention enhanced selective gate, the model will misclassify both bags into NA, leading to a degraded performance. Further, as shown in Bag 2, even if entity-aware embedding module is absent, proposed framework merely depending on selective gate can also make a correct prediction. This finding warrants more investigation into the power of the self-attention enhanced selective gate; hence, the two error cases are shown in Bags 3 and 4. Then, to further consider the necessity of entity-aware embedding, we show two error cases for SeG w/o Ent whose labels are /location/location/contains and NA respectively in Bag 3 and 4. One possible reason for the misclassification of both cases is that, due to a lack of entity-aware embedding, the remaining position features cannot provide strong information to distinguish complex context with similar relation position pattern w.r.t the two entities.

Experiments ::: Error Analysis	To investigate the possible reasons for misclassification, we randomly sample 50 error examples from the test set and manually analyze them. After human evaluation, we find the errors can be roughly categorized into following two classes.

Experiments ::: Error Analysis ::: Lack of background	We observe that, our approach is likely to mistakenly classify relation of almost all the sentences containing two place entities to /location/location/contains. However, the correct relation is /location/country/capital or /location/country/administrative_divisions. This suggests that we can incorporate external knowledge to alleviate this problem possibly caused by a lack of background.

Introduction	Relation extraction (RE) is one of the most fundamental tasks in natural language processing, and its goal is to identify the relationship between a given pair of entities in a sentence. Typically, a large-scale training dataset with clean labels is required to train a reliable relation extraction model. However, it is time-consuming and labor-intensive to annotate such data by crowdsourcing. To overcome the lack of labeled training data,  mintz2009distant presents a distant supervision approach that automatically generates a large-scale, labeled training set by aligning entities in knowledge graph (e.g. Freebase ) to corresponding entity mentions in natural language sentences. This approach is based on a strong assumption that, any sentence containing two entities should be labeled according to the relationship of the two entities on the given knowledge graph. However, this assumption does not always hold. Sometimes the same two entities in different sentences with various contexts cannot express a consistent relationship as described in the knowledge graph, which certainly results in wrongly labeled problem. To alleviate the aformentioned problem,  riedel2010modeling proposes a multi-instance learning framework, which relaxes the strong assumption to expressed-at-least-one assumption.

Question: By how much do they outperform previous state-of-the-art in terms of top-n precision?
Answer: Outperforms PCNN+HATT by 10.3% and PCNN+BAG-ATT by 5.3%