SAMRank: Unsupervised Keyphrase Extraction using Self-Attention Map in BERT and GPT-2
Authors: Byungha Kang
Youhyun Shin
Venue: EMNLP 2023 Main (2023)
Keywords: Unsupervised Keyphrase Extraction
Pre-trained Language Model
Self-Attention Map
BERT
GPT-2
TL;DR: SAMRank is a novel unsupervised keyphrase extraction approach using only self-attention map in a PLM, outperforming most embedding-based models without the need for calculating embedding similarities between phrases and sentences.

Abstract: We propose a novel unsupervised keyphrase extraction approach, called SAMRank, which uses only a self-attention map in a pre-trained language model (PLM) to determine the importance of phrases. Most recent approaches for unsupervised keyphrase extraction mainly utilize contextualized embeddings to capture semantic relevance between words, sentences, and documents. However, due to the anisotropic nature of contextual embeddings, these approaches may not be optimal for semantic similarity measurements. SAMRank as proposed here computes the importance of phrases solely leveraging a self-attention map in a PLM, in this case BERT and GPT-2, eliminating the need to measure embedding similarities. To assess the level of importance, SAMRank combines both global and proportional attention scores through calculations using a self-attention map. We evaluate the SAMRank on three keyphrase extraction datasets: Inspec, SemEval2010, and SemEval2017. The experimental results show that SAMRank outperforms most embedding-based models on both long and short documents and demonstrating that it is possible to use only a self-attention map for keyphrase extraction without relying on embeddings. Source code is available at https://github.com/kangnlp/SAMRank.

SAMRank: Unsupervised Keyphrase Extraction using Self-Attention Map in BERT and GPT-2
Byungha Kang and Youhyun Shin'
Department of Computer Science and Engineering, Incheon National University (bhkang, yhshinj@inu.ac. kr
Abstract
to be domain-specific. In contrast, unsupervised keyphrase extraction uses only the information in- trinsic to the document. Traditional methods rely on statistical information, such as TF-IDF (Jones, 2004), and graphs based on the co-occurrence of words (Mihalcea and Tarau, 2004). However, these methods often fall short of deciphering latent mean- ings in the text, resulting in suboptimal perfor- mance. The introduction of deep learning-based language models changed this paradigm, by en- abling the extraction of phrases closely connected semantically to the document through the calcula- tion of the degree of similarity between the docu- ment and phrase embeddings.
We propose a novel unsupervised keyphrase extraction approach, called SAMRank, which uses only self-attention map in a pre- trained language model (PLM) to determine the importance of phrases. Most recent ap- proaches for unsupervised keyphrase extrac- tion mainly utilize contextualized embeddings capture semantic relevance between words, sentences, and documents. However, due to the anisotropic nature of contextual embeddings, these approaches may not be optimal for se- mantic similarity measurements. SAMRank proposed here computes the importance of phrases solely leveraging self-attention map a PLM, in this case BERT and GPT-2, elimi- nating the need to measure embedding similari- ties. To assess the level of importance, SAM- Rank combines both global and proportional attention scores through calculations using self-attention map. We evaluate the SAMRank three keyphrase extraction datasets: Inspec, SemEva12010, and SemEva12017. The exper- imental results show that SAMRank outper- forms most embedding- based models both long and short documents and demonstrating that possible to use only self-attention map for keyphrase extraction without relying on embeddings. Source code is available at https://gi thub. cwm/ wowm//nnnll//////ooommccm SAMMRakk.
However, this paper does not utilize embed- dings. The rationale behind this is that contex- tualized representations are not well-suited to tasks based similarity. A study that has geometri- cally analyzed the representations of PLMs such as ELMo (Peters al., 2018), BERT (Devlin et al., 2019), and GPT-2 (Radford et al., 2019). have postulated that the contextualized representa- tions of these language models exhibit anisotropy (Ethayarajh, 2019). These representations tend to concentrate on specific directions of the vec- tor space, becoming increasingly anisotropic in higher layers. Indeed, the average cosine similar- ity of two word representations randomly chosen from BERT and GPT-2 markedly increases with the layer depth. Given that embeddings typically come from the last layer's 's representation a PLM, anisotropic embeddings can compromise tasks that require similarity-based rankings. To counteract this, BWRank (Wang and Li, 2022) improved the performance of keyphrase extraction by isotropi- cally reconstructing BERT's embeddings by reduc- ing dimensions through whitening operation.
1 Introduction
Keyphrase extraction refers to process of identify- ing the words or phrases that signify the primary themes of document. It has wide range of appli- cations, including document summarization, infor- mation retrieval, and topic modeling. The method- ologies used for keyphrase extraction are typically categorized into supervised and unsupervised ap- proaches. Although supervised keyphrase extrac- tion yields excellent performance, it relies heav- ily on large quantities of labeled data and tends
In an earlier work entitled "What Does BERT Look At? An Analysis of BERT's Attention" (Clark et al., 2019), the authors sought to un- derstand how BERT's attention mechanism oper-
"Corresponding author.

---

Ciabal Attertia (Solne Sing
Properional Attchoo Celesaaion the dae)
[CE the Ingortienn
que
Mgorthmms employlal iechriques such semaaisc networss do produce wowoolieiiine menea agoothis aie small IB sizn and are able 16 replace lodiiion Ituestion dasabased Iney
constralned
SAM Chlls-tomation Map!
Agorthin can [BEM]
quistion genoration 100 Intewowo
service oxari stoll
this paper. prosen Saggnniim Ierialtonnss Dereration roltaaiin succtlens alpniiiinss
solvabic
impooyil lestin suae much asmmrrtic natoon ea Aisttsttss vettaal quesilara, Theas aew serell siow and able replace Inadiiiiaa queetiinn ilelstinens: They are particllelly suttatle wmerrattonnn on the conelmined mobile platforms. The algorthme applledi quection generatin job moniow.
exam, etc)
Figure Visualization of final token-leved scores. The top figure uses the self-attention map from BERT (2nd head of the 5th layer), and the bottom one from GPT- (Ist head of the 1tth layer). Deeper colors indicate higher scores.
Candsase Phoses [uteaa
Candicaie
Figure I: Overview of SAMRank.
attention and are highly attentive to phrases receiv- ing global attention. An overview of SAMRank is presented in Figure I and the final token scores calculated through SAMRank can be visualized, depicted in Figure 2.
ates by determining how much attention a specific word gives to other words in each head of BERT. Their findings indicate that certain BERT heads were skilled in identifying grammatical structures and coreferences such as direct objects of verbs and objects of prepositions. This suggests that in transformer-based PLMs, such as BERT that em- ploy a multi-head attention mechanism to handle linguistic complexity, certain heads may specialize in particular tasks. Building on this research, we can posit that some heads of transformer-ben based PLM may specialize in capturing keyphrases given their importance in providing understanding of the thematic essence of the text and considering the likelihood that some heads may have been trained to detect keyphrases during the pre- pre-training process the PLM.
In this paper, we independently analyze the self- attention map of each head each layer to identify heads that excel in keyphrasc extraction. We extract the self-attention map from all 144 heads in BERT and GPT-2, using three representative keyphrase extraction datasets: Inspec, SemEval2010, and Se- mEval2017. The performance of each head is eval- uated, with experimental results showing that SAM- Rank outperforms most embedding-baid based methods in specific heads of both BERT and GPT-2. This paper's 's main contributions are three-fold:
We propose SAMRank, a novel and efficient approach that extracts keyphrases solely with self- self-attention map, eliminating the need for embeddings and similarity measurements.
Therefore, this paper shifts the focus from PLM embeddings to the self-attention map afforded by each head in the PLM. We propose SAMRank, a novel approach that leverages only the self- attention map of a transformer-based PLM to ex- tract keyphrases. SAMRank calculates a global attention score by summing the attention weights received by the token from all tokens within the document. also redistributes the global atten- tion score proportionally to other tokens that are focusing on globally significant tokens based on the attention weights they have allocated, yielding a proportional attention score. The combination of these two scores determines the final score for to- kens. By aggregating the final scores of each token within candidate phrase, SAMRank calculates the total score for the phrase and determines its overall significance. This results in ranking of candidate phrases based on their total scores, thus allowing the extraction of phrases that both receive high
We demonstrate not only the capability of encoder- based language models like BERT, but also decoder-based language models such GPT-2, for keyphrase extraction. Further- more, we provide evidence that GPT-2 can out- perform BERT in keyphrase extraction tasks.
We discover the existence of heads transformer-basee PLMs that are more pro- ficient in keyphrase extraction compared to baselines.
2 Background
The transformer model proposed in "Attention Is All You Need" (Vaswani al., 2017) introduced the attention mechanism, bringing about revo- lution in the field of natural language processing.

---

In this section, we introduce a self-attention map. which is the computational output of self-attention, the core mechanism of the transformer. Addi- tionally, we discuss two transformer- based PLMs, BERT and GPT-2, which have demonstrated supe- rior performance in various natural language pro- cessing tasks.
concatenated to form the final layer output.
2.2 BERT
BERT (Devlin et al., 2019) is model based on the encoder of the transformer, which utilizes self- attention, resulting in square matrix for each head's self-attention map. In this paper, the BERT-base model', which comprises 12 layers, each equipped with 12 attention heads, yielding a total of 144 heads. The maximum input token length for BERT-base is 512.
2.1 Self-Attention Map
The transformer's pivotal component is the self- attention mechanism, which models intricate inter- dependencies among tokens within a text by quan- tifying the attention each token pays to the others. The self-attention mechanism of the transformer calculates the scaled dot product between each to- ken's query vector (Q) and the key vectors (K s) of all tokens, then passes this through a softmax function to compute the attention weights. The attention weights are multiplied by each token's value vector (V), and these vectors are summed to generate the final output vector for the query token.
2.3 GPT-2
GPT-2 (Radford al.. 2019) is a model based on the decoder of the transformer, which uses masked self-attention, creating a lower triangular matrix for each head's self-attention map. In this paper, we use the 1 117M size model of GPT-2?. The GPT-2 model, consists of 12 layers, each with 12 attention heads, totaling 144 heads. The maximum input token length for GPT-2 is 1024.
Since each token in a text is served as a query Q once in self-attention, the aggregation of the attention weights of all tokens yields a square ma- trix, known as the self-attention map (SAM). SAM is a crucial piece of information that provides in- sights into how tokens relate to each other. The i-th row of SAM represents the attention weights of the i-th token of the text when it is served query Q --mmnning, how much it 'attends to' the other tokens-with with the sum of each row equaling I, Conversely, the i-th column of SAM represents the attention weights received by the i-th token from the other tokens when it is served as value K.
Methodology
Our proposed methodology involves ranking phrases based on the self-attention map (SAM) of transformer-bant
language models, in this case BERT and GPT-2. Each row of SAM, which a self-attention weights vector, is perceived as each token allotting its attention according to its rele- vance with other tokens. The tokens that are seman- tically connected to many other tokens more likely to receive this 'attention'. Hence, the global attention score of each token is computed by sum- ming the attention weight it received from all the tokens in the text. Then, similar to the attention op- eration (multiplying the V vector by the attention weight), the global attention score is distributed in proportion to the attention weight assigned by the token. This results in the calculation of the propor- tional attention score, where tokens strongly asso- ciated with globally significant tokens are assigned high scores. The final score for token is calculated by combining the global attention score and the pro- portional attention score. For phrases, their scores are computed as the accumulated sum of the scores of their constituent tokens across all occurrences in the document. Based on these phrase-level scores, the phrases are then ranked to determine their over- all importance. Table provides an instance of
While self-attention considers the relationships with all tokens in the input text, masked self- attention 'masks' or disregards tokens that appear after the current token. This means that each token pays attention only to the tokens that appeared be- fore it. Consequently, a masked self-attention map is lower triangular matrix where all values above the diagonal are 0.
Transformer is an encoder-decoder model, with the encoder employing the self-attention mech- anism and the decoder using the masked self- attention mechanism. The transformer is multi- layered, with the attention mechanism operating in parallel across multiple heads within each layer. Each head has independent Q, K, and weights, meaning they process the same input text from var- ious perspectives. The outputs from each head are
hht https:://hugiingaace chttps://huggingfaac co/gpt2

---

these extraction results for specific text.
the middle layer (Clark al., 2019). Also, the first token of GPT-2, which performs masked self- attention, receives very high attention weights from the beginning of the document. We set the global attention score for BERT's [SEP] tokens and GPT- 2's first input tokens to 0 because their global at- tention scores can be excessively high, which can hinder the understanding of correlation between each token.
3.1 Candidate Generation
We use the module implemented in EmbedRank (Bennani-Smires et al., 2018) to extract keyphrase candidates. This module employs the Stanford- CoreNLP tool to tokenize and POS tag the doc- uments and then uses NLTK*Ss RegexpParser to extract noun phrases, which are defined as words or series of words tagged with 'NN', 'NNS', 'NNP', 'NNPS', and 'JJ'. These extracted noun phrases become candidate phrases.
3.4 Proportional Attention Score Calculation
If token not globally important but has very high correlation with an important token, the token should also be considered an important token. If token assigns very high attention weight to an important token, it can be interpreted as having a high correlation with the important token. We allow such tokens to be ranked high by redistributing the global attention score in proportion the attention weight they assigned the important token.
3.2 Self-Attention Map Extraction
In this paper, extract self-attention maps (SAMs) from each layer and head of BERT and GPT-2 using Hugging Face's transformers library". We input the entire document into the PLM at once, as opposed to inputting each sentence separately. This enables us to capture the correlations between tokens that reflect the overall context of the doc- ument. If the number of tokens in the document exceeds the model's maximum number of input tokens, we divide the entire document into sev- eral segments with the same number of tokens and extract SAM for each. For BERT, we input text without [PAD] tokens the structure of the [CLS] text [SEP]. GPT-2 is input directly without any special tokens.
B = A diag(G)
(3)
This can be calculated by multiplying each token's 's global attention score by each element in the corre- sponding column of SAM.
B
BYd
(4)
R 2
Byg
Then, we normalize the values of each column by dividing them by the sum of the column to make each column have values between and 1, thus obtaining matrix B'.
3.3 Global Attention Score Calculation
SAMRank initially calculates the score the token- level and later aggregates the phrase-level. We calculate each token's global attention score as the total sum of attention weights it received from all the tokens the document. The self- attention map is a matrix A. its size determined follows: input token number (n) x input token number (n). Therefore, we can calculate each to- ken's 's global attention score (G²,) by determining the column sum for each column of SAM.
P = B
(5)
The proportional attention score, which is calcu- lated proportional to each token's global impor- tance and its attention weight, is calculated by per- forming a row sum operation for each row of the updated SAM (B').
A = [A]ux nxn
(1)
3.5 Phrase-level Score Aggregation
G
Aji
(2)
The final score the token-level is determined by the sum of the global attention score (GG,) and the proportional attention score (Pt,) of each token:
However, BERT's [SEP] tokens receive very high attention weights from other tokens from
St. =t, Pt; (6)
hhttps://stanfordnnpp www.giiiti//oooooooo io/CoreNLP http://///tttubccm///// http://ggtt...moggggggggg///aaaa
The score at the lerase-Level is calculated as the sum of the final importance scores of the tokens

---

that make up the phrase:
Tenching teasagened scance all-peealllecs Froet dasare sadics Jasaice ruppece Lac 1905 were a decals of cuseroou change insragercca science (MS: odacioes. White the cullek the legitning of decade was sure bleck. the reealisance MS edacatio boceghn about by the use of spoit as the pervay deliney vebile for ququttttiee modelng nechsigee eigpes rncibed mach brightr new The pope rakes inveatey
Sp. - Σ Ste (7)
cueen con Mi odacation suggnen onne precisin now dinatives are detiare supper spoon for cxteasiss to concskor
iEP
Fimend Tran
nov evageren sciece speatideers, quamization maddeling edacation daci-
agrpon syoress
The final score at the phrase-level is calculated as the sum of the phrase scores calculated at each location where the phrase appears in the document:
ressageree eicnes; decisioe modele commmess change allcaaion serpon
NAMRRrk (UERT)
sproditiin:: gonnivtte modeling schoopus: maaagroan steres, decidine
apport: decbler appan moteets
NAMIREE (LP7T-3)
qpesoiiasts maaaepete sckonnn promisy now deatione Soason suppert
00kana: desbiie appoot
Sp Σ Sp kEP
(8)
Table An example of top keyphrase extraction re- sults by JointGI and SAMRank from the Inspec dataset.
If a phrase consists of single word, and if that word is used as subword of another candidate phrase, the score at that location is not included in the calculation because the score of the subword is calculated in the context of the phrase which the subword belongs, and this score should be added to the score of the phrase. Therefore, a single word is only scored at positions where used inde- pendently and not as a subword of other candidate phrases. Also, single words are often most likely to receive higher scores simply because they are more frequent in the document than other candidate phrases. To correct this, the final score of single word calculated as the average score by dividing the total score of the word by the frequency of the word.
Datasets
Inspec
SemEval2010
SemEval2017
Docs
500
100
493
Avg. Words
135
1589
194
Avg. Sents
6
68
7
Avg. Keys
9
12
17
Unigram
13.47%
19.52%
24.59%
Bigram
52.66%
54.57%
33.61% %
Trigram
24.86%
19.02%
17.40%
Table Statistics datasets.
To evaluate the performance of keyphrase ex- traction, we calculate the Fl measure for the top 5, top 10, and top 15 keyphrases predicted by the model. This is done after ranking the predicted keyphrases, applying NLTK's s PorterStemmer for stemming, and eliminating duplicates.
For long documents that exceed the maximum number of input tokens, first we divide them into multiple segments and apply SAMRank indepen- dently to each segment, after which we integrate each result. That is, the final score of the phrase is ranked by calculating it as the sum of the scores that the phrase obtained in each segment.
4.2 Baselines and Implementation Details
We compare SAMRank with statistics-based meth- ods: TF-IDF (Jones, 2004), YAKE (Campos al., 2018); the graph-based methods: TextRank (Mi- halcea and Tarau, 2004), SingleRank (Wan and Xiao, 2008), TopicRank (Bougouin et al., 2013). PositionRank (Florescu and Caragea, 2017); static embedding methods: EmbedRank (Bennani- Smires et al., 2018), and contextual embedding- based methods: SIFRank (Sun et al., 2020), At- tentionRank (Ding and Luo, 2021), MDERank (Zhang et al., 2022), JointGL (Liang et al., 2021). Most of these methods evaluate performance out- comes by stemming the predicted phrases and the ground truth phrases and then removing duplicates. However, JointGL, which shows the highest per- formance among the existing methods, does not remove duplicates after stemming. This may have inflated the reported performance. To ensure fair comparison, we reproduce JointGL's performance by removing duplicates after stemming. We use the optimal hyperparameters reported in the JointGI. paper. For SemEval2017, we use the same hyper- parameters used with Inspec.
4 Experiments
4.1 Datasets and Evaluation Metrics
We evaluate the proposed method using three com- monly used datasets: Inspec (Hulth, 2003), Se- mEval2010 (Kim al., 2010), and SemEval2017 (Augenstein et al., 2017). Both Inspec and Se- mEval2017 datasets consist of scientific paper ab- stracts, and no document in these datasets exceeds BERT's maximum input length of 512 tokens. In contrast, the SemEva12010 dataset comprises full- length ACM papers, most of which exceed GPT- 2's maximum input token count of 1024. In line with previous studies (Saxena et al., 2020; Liang al., 2021), we use the test composed of 100 documents and their annotated set of keyphrases. Table 2 presents the statistical information of each dataset.

---

Inspec FI @ 10
SemEval2010
SemEval2017
Model
FI@5
FI@IE
F1@5
FI@10 10
@1@15 15
F1@5
FI@ 10
F1@15 15
Statistics based Models
TF-IDF
11.28
13.88
13.83
2.81
3,48
3.91
12.70
16.26
16.73
YAKE
18.08
19.62
20.11
11.76
14.4
15.19
11.84
18.
20.55
Graph traph-based Models
TextRank
27.04
25.08
36.65
3.80
5.38
7.65
16.43
25.83
30.50
SingleRank
27.79
34.46
36.05
5.90
9.02
10.58
18.23
27.73
31.73
TopicRnak
25.38
28,46
29.49
12.12
12.90
13.54
17.10
22.62
24.87
PositionRank
28.12
32.87
33.32
9.84
13.34
14.33
18.23
26.30
30.55
Static Embedding- based Models
EmbedRank d2v
31.51
37.94
37.96
3.02
5.08
7.23
20.21
29.59
33.94
EmbedRank s2y
29.88
37.09
38.40
5.40
8.91
10.06
Contextual Embedding based Models
SIFRank
29.11
38,80
39.59
22.59
32.85
38.10
AttentionRank
24.45
32.15
34.49
11.39
15.12
16.66
23.59
34.37
38.21
MDERank
27.85
34.36
36.40
13.05
18.27
20.35
20.37
31.21
36.63
JointGL
30.82
36.28
36.67
10.78
13.67
14.64
20.49
29.63
34.05
Self-Attention Map base-baed Models (Ours)
SAMRank (BERT)
33.96
39.35
39.73
15.28
18.36
18.03
24.08
33.40
37.53
SAMRank (GPT-2)
33.92
39.44
39.72
15.88
19.49
19.03
24.80
34.75
38.78
Table 3: Performance comparison of baselines and SAMRank on F1 @55, F 10, and F1@ 15. The bold indicates cases where SAMRank showed improved performance compared baseline models. The underline indicates the best performance model.
4.3 Results
GPT-2 pointed out that the average intra-sentence similarity, the cosine similarity between each word representation within sentence in GPT-2, is lower than is in BERT (Ethayarajh, 2019). We also conjecture that while BERT encodes the context by making each word in sentence share mean- ing, GPT-2 maintains the unique meaning of each word while encoding the context. Therefore, it can be hypothesized that the characteristic of GPT-2, which preserves the unique meaning of each word during contextualization without dilution, worked favorably, resulting in GPT-2 showing higher per- formance than BERT.
In Table 3, the experimental results for SAMRank are presented. We select the head that achieved the highest F1 @ 15 score from total of 144 heads. In Inspec, the eleventh head of the third layer of BERT and the first head of the eleventh layer of GPT- 2 performed best. Both SAMRank using BERT and GPT-2 surpass the Fl @5 @ 10, and F1 15 scores of existing baselines. In SemEval2010, the sixth head of the twelfth layer of BERT and the eleventh head of the eleventh layer of GPT-2 obtain the highest performance. Our model achieves state- of-the-art (SOTA) performance FI @5, F1 @ 10, and outperforms all other baselines F1 @ 15, cept for MDERank. These results demonstrate that SAMRank also performs well when extracting keyphrases from long documents. In ScmEval2017, the second head of the fifth layer of BERT and the first head of the eleventh layer of GPT-2 show the highest performance, achieving SOTA results on FI @55, Fl @ 10, and FI @ 15.
4.4
Ablation Study
4.4.1
Contribution of Global and Proportional Attention Score
Table presents the performance evaluation when the global attention score and proportional attention score, components of SAMRank, are used indepen- dently for the head showing the highest Fl @ 15 performance. Even when using only the global tention score, both BERT and GPT-2 showed fairly high performance outcomes across all datasets. However, the FI @1 FI@ 10, and Fl 15 scores are approximately 1% lower on average across all
Our proposed SAMRank outperforms almost all existing baselines across all datasets. We note that the performance of GPT-2 is slightly higher than BERT, except for the Inspec dataset. An earlier work that analyzed the embeddings of BERT and

---

Inspec
SemEval2010
SemEval2017
Model
FI@5
Fi @ 10
@ 15
F1@5
FI@
F1 @ 15
F1@5
@I@ 10
F1@1
SAMRank (BERT)
33,96
39.35
39.73
15.28
18.36
18.03
24.08
33.40
37.53
only Global
33.43
38.04
38.36
14.95
16.26
17.41
23.39
33.29
36.89
only Proportional
32.63
38.54
39.04
14.77
17.65
17.39
23,46
32.98
36.89
SAMRank (GPT-2)
33,92
39.44
39.72
15.88
19.49
19.03
24.80
34.75
38.78
only Global
33.66
39.24
38.33
14.96
18.01
18.76
24.07
33.82
37.13
only Proportional
25.02
32.74
34.23
11.27
14.29
14.87
19.76
28.72
32.98
Table 4: Experimental results on the impact of cach attention score: global and proportional attention score. Comparison of performances using only the global attention score (only Global) and only proportional attention score (only Proportional). The bold indicates the best performance within each group (BERT and GPT-2)..
Inspec
SemEval2010
SemEval2017
Model
FI @
FI @
FI@I5 15
FI@5
FI@ 10
Fl@15 15
FI@5
Fl @
@
Backward Redistribution
SAMRank (BERT)
33.96
39.35
39.73
15.28
18.36
18.03
24.08
33.40
37.53
SAMRank (GPT-2)
33.92
39.44
39.72
15.88
19.49
19.03
24.80
34.75
38.78
Forward Redistribution
SAMRank (BERT)
33.96
39.67
39.78
16.08
18.77
18.27
23.04
32.49
37.18
SAMRank (GPT-2)
33.50
38.80
39.04
17.17
18.40
18.90
25.18
34.60
38.22
Table 5: Experimental results on the comparison of global attention score redistribution directions for computing the proportional attention score. Evaluation with different distribution directions, both backward and forward.
datasets compared to when both the global and pro- portional attention scores are used. Additionally, performance degradation is more evident when only the proportional attention score is used, Es- pecially for SAMRank (GPT-2), when only the proportional attention score was used, the Fl @5, FI@ 10, and Fl @ 15 scores decreased by an aver- age of approximately 8% across all datasets. These results indicate that the global attention score is factor that more crucially affects the performance of SAMRank. However, the best performance found when both the proportional attention score and global attention score were used in all cases, both scores are essential in SAMRank, and a more accurate keyphrase extraction possible by combining them.
periments with the forward approach as well. The results show that different heads exhibit the highest performance in the forward approach and the back- ward approach. SAMRank based BERT heads performs better with the forward approach on the Inspec and SemEval2010 datasets. However, on the SemEval2017 dataset, the original backward ap- proach achieves better performance. On the other hand, SAMRank based on GPT-2 heads mostly shows better performance with the original back- ward approach, with the forward approach only showing better performance F-1 @5 of the Se- mEval2010 and SemEval2017 datasets. Therefore, the performance based on the distribution direction of the global attention score varied depending on the dataset and the model.
We conclude that the backward and forward ap- proaches evaluate the importance of tokens from different perspectives. However, the average per- formance difference hased on the distribution direc- tion is not significant. This shows that the overall approach of SAMRank, which calculates the global attention score and redistributes it based on the rel- evance of each token individually, effective.
4.4.2 Direction of the Global Attention Score Redistribution
Table 5 presents the results of the experiments for both the backward and forward approaches. In Section 3.4, we propose a method of redistributing the global attention score in proportion to the at- tention weight each token receives (the backward approach). Conversely, each token could also redis- tribute the global attention score in proportion to the attention weights it allocates to other tokens as a query Q (the forward approach). We conduct ex-
5 Analysis of Heads
To identify heads that specialized in keyphrase ex- traction, we evaluate the performance each

---

Inspec
SemEva12010
SemEval2017
PLM
@1@5
Fl 10
F1@15 15
F1 @5
Fl @ 10
Fl @ 15
F1@5
FI@ 10
Fl @1@15
BERT
33.96
39.35
39.73
15.28
18.36
18.03
24.08
33.40
37.53
GPT-2
33.92
39.44
39.72
15.88
19.49
19.03
24.80
34.75
38.78
Llama 7B
33.28
39.57
39.27
16.69
18.66
18.51
23.22
33.38
37.63
Llama 2 13B
34.07
39.75
39.51
16.11
18.47
19.12
23.94
33.97
38.04
Llama 2 70B
35.06
40.01
39.76
17.03
19.02
18.90
24.88
35.32
38.30
Table 6: Performance of SAMRank on LLMss: Llama 2 7B, Llama 13B, and Llama 2 70B.
the 144 heads in BERT and GPT-2. The F1@11 15 performance of each head across all layers for the three datasets detailed in Appendix A. We find that the performance of specific heads shows con- sistent patterns across all three datasets. In BERT, heads mainly located in the third and fifth layers demonstrate high performance, whereas in GPT-2, heads in the eleventh layer are particularly per- formant. However, the head yielding the highest performance varies across the datasets. This in- dicates that specific heads in transformer-based PLMs do pay substantial attention to phrases per- ceived as keyphrases by humans, but their perfor- mance varies depending on the type and length of the text, suggesting that each head might adopt different perspectives or attention patterns when identifying keyphrases.
form experiments on: Llama 2 7B 6 -lama 2 13B 7, and Llama 2 70B 8, Llama 7B consists of 32 layers, each with 32 heads. Llama 2 13B consists of 40 layers, each with 40 heads. Llama 2 70B con- sists of 80 layers, each with 64 heads. The results are presented in Table 6.
In our experiments, Llama 2 models mostly show comparable performance to BERT and GPT- 2. However, the Llama 2 70B model surpasses both BERT and GPT-2 in the Inspec and SemEva12017 datasets, achieving the highest scores in Fl @5 across all datasets. These superior performances in Fll @5 are also observed in our experiments with BERT and GPT-2, indicating a consistent trend across different PLM-based models. Importantly, among the Llama 2 models, there is a clear trend that an increase in the number of parameters leads to better performance. This observation supports the prevalent notion in LLM research that increas- ing model size can lead to improved performance.
Moreover, we observe that the intra-layer perfor- mance variance in BERT is larger than in GPT-2, This might suggest that some heads in BERT spe- cialize in tasks unrelated to semantic keyphrase extraction, possibly focusing on syntactic or other non-semantic tasks. Despite the relatively small performance differences among the pepperforriigg heads across different layers in BERT, identify some particular heads (first and eleventh head) in the eleventh layer of GPT-2 that significantly out- performs other heads, including those in BERT, This suggests the possibility of certain heads in GPT-2 being highly specialized for keyphrase iden- tification.
Related Work
Traditional unsupervised keyphrase extraction (UKE) approaches broadly divided into two cat- egories: statistical-based and graph-based methods. Statistical upproaches, like TF-IDF (Jones, 2004) and YAKE (Campos et al.. 2018), use information such as word frequency and distribution. Graph- based methods, like TextRank (Mihalcea and Tarau, 2004), SingleRank (Wan and Xiao, 2008), Topi- cRank (Bougouin et al., 2013), and PositionRank (Florescu and Caragea, 2017), represent a docu- ment as graph and rank phrases based on the graph's characteristics.
6 SAMRank on LLMs
SAMRank is based on the self-attention mecha- nism, so it can be used in transformer models other than BERT or GPT-2. We conduct keyphrase traction experiments applying SAMRank the large language models (LLMs) that have recently attracted significant attention in the field of natural language processing. We utilize Llama 2, recently released by Meta (Touvron et al., 2023). We per-
Traditional unsupervised keyphrase extraction approaches that rely solely on surface-level fea- tures often fail extract semantically significant
"https://huggingee Lomeee/eeeeeeeeeaam/mma//aaaaa- 1aaaa/aaaaa/aa77 2 hhttss:/hugginggfaceeu cw/neeaaoooaaeeeeeeeeeecco
Ll ammb-2-13
"https: hetiss//huggingg www.iinffae.cceoeeeee -
Llama-

---

keyphrases. To address these limitations, meth- ods utilizing embeddings emerged. EmbedRank (Bennani-Smires et al., 2018) an example that employs static embeddings such as Sent2Vec and Vec2 Subsequently, a new trend in keyphrase extraction emerged with the advent of PLMs. SIFRank (Sun et al., 2020) utilizes ELMo, a PLM capable of generating contextual embeddings.
based PLMs, beyond the scope of keyphrase ex- traction.
Limitations
SAMRank utilizes just single self-attention map (SAM) from one out of 144 heads, As evidenced by the results in the Appendix, certain heads consis- tently exhibit high performance across all datasets, but the best performing head does not always re- main the same. Moreover, tradc-off in perfor- mance is observed among the heads. Some heads demonstrate high performance on Fl @5 but low on Fl @ 15, while others show the opposite pattern. This variability indicates that the optimal head lection may vary depending on the type and length of the text and the specific evaluation metric, neces- sitating additional human exploration for optimal results. Therefore, rather than solely relying on a single self-attention map, combining outputs from multiple high -errforrrrminng heads could be expected to yield not only more stable, but also more gen- eralized performance applicable various types of text. The method of combining multiple SAMs would make a good topic for future research.
More recently, transformer-based models like BERT have been widely adopted. AttentionRank (Ding and Luo, 2021) applies BERT's self-attention and cross-attention mechanisms to keyphrase iden- tification, examining the relevance of a candi- date keyphrase within its sentence and the tire document. Masked Document Embedding Rank (MDERank) (Zhang et al., 2022) introduces a masking strategy, ranking keyphrases by com- paring the similarity of the source and masked document embeddings. This approach benefits from a customized BERT model trained via self- supervised contrastive learning method. JointGL (Liang et al., 2021) merges local and global con- texts for keyphrase extraction, creating graph structure capture local information and calculat- ing semantic similarity for global perspective.
8 Conclusion
Additionally, when processing long documents, we divide the text into segments with equal to- ken counts, extract keyphrases from each segment, and then integrate the results to draw out the final keyphrases. The motivation behind this approach of equal segmentation is reveal the importance of tokens under identical conditions across all seg- ments. However, this method can lead a loss of context from the original long text, potentially reducing accuracy. Thus, applying the SAMRank approach to models that can handle more extended token inputs, such as BigBird (Zaheer et al., 2020) or Longformer (Beltagy et al., 2020), could be an interesting topic for future research.
In this paper, we present a new perspective on keyphrase extraction that moves away from the conventional embedding- based approach and solely leverages the information from the self-attention map of PLM. The proposed SAMRank extracts keyphrases based on a combination of the global at- tention score, which identifies tokens that attract at- tention within a document, and the proportional at- tention score, which finds tokens deeply associated with important tokens. Experimental results on three representative keyphrase extraction datascts demonstrate that SAMRank consistently outper- forms embedding- based models, improving Fl @5 and F1 @ 10 scores by approximately 11.5% and 3% on average, respectively.
Acknowledgements
These results suggest that some heads in PLMs, particularly BERT and GPT-2, could be specialized in capturing keyphrases. We also demonstrate that the self-attention map of these heads can provide more useful information for keyphrase extraction than embeddings. Future work could potentially utilize the SAMRank approach to enhance the in- terpretability by thoroughly analyzing the specific roles and functionality of each head in transformer-
This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ICAN(ICT Challenge and Advanced Network of HRD) sup- port
-TT./I//---0---2222--0000 pervised by the IITP(Institute for Information & Communications Technology Planning & Evalua- tion).
su-

---

References
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, EL, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNN.P-IJCNI.)) pages 55-65, Hong Kong. China. Association for Computational Linguisties.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10: SciencelE extracting keyphrases and relations from scientific publications. In Proceedings of the 1ith International Workshop on Semantic Evaluation (SemEval-2017) pages 546- 555. Vancouver, Canada. Association for Computa- tional Linguistics.
Corina Florescu and Comelia Caragea. 2017. Posi- tionRank: An unsupervised approach to keyphrase extraction from scholarly documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 105--1115, Vancouver, Canada. Association for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The mongddocument transformer.
Kamil Bennani- Smires, Claudiu Musat, Andreea Hoss- mann, Michael Baeriswyl, and Martin Jaggi. 2018. Simple unsupervised keyphrase extraction using sen- tence embeddings. In Proceedings of the 22nd Con- ference on Computational Natural Language Learn- ing, pages 221-229, Brussels, Belgium. Association for Computational Linguistics.
Anette Hulth. 2003. Improved automatic keyword traction given more linguistic knowledge. Pro- ceedings of the 2003 Conference on Empirical Meth- ods in Natural Language Processing, pages 216 223.
Adrien Bougouin, Florian Boudin, and Béatrice Daille. 2013. TopicRank: Graph-based topic ranking for keyphrase extraction. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 543-551, Nagoya, Japan. Asian Federation of Natural Language Processing.
Karen Spärek Jones. 2004. A statistical interpretation of term specificity and its application in retrieval. J. Documentation, 6055)-93-502.
Su Nam Kim, Olena Medelyan, Min-Yeu Kan, and Tim- othy Baldwin. 2010. SemEval-2010 task Auto- matic keyphrase extraction from scientific articles. In Proceedings of the Sth International Workshop Semantic Evaluation, pages 21 21-26, Uppsala. Sweden. Association for Computational Linguistics.
Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge. Célia Nunes, and Adam Jatowt. 2018. Yake! collection- intrrindenndeee automatic key- word extractor. In Advances in Information Retrieval 40tth European Conference on IR Research, ECIR 2018, Grenoble, France, March 26-29, 2018, Pro- ceedings, volume 10772 of Lecture Notes in Com- puter Science, pages 806-810 Springer.
Mikalai Krapivin and Maurizio Marchese. 2009. Large dataset for keyphrase extraction.
Xinnian Liang, Shuangzhi Wu, Mu Li, and Zhoujun Li. 2021. Unsupervised keyphrase extraction by jointly modeling local and global context. In P Proceedings of the 202 Conference on Empirical Methods in Nat- ural Language Processing, pages 155-164, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276 226. Florence, Italy, Association for Com- putational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidircctional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark. Kenton Lee, and Luke Zeulemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Haoran Ding and Xiao Luo. 2021. AttentionRank: Un- supervised keyphrase extraction using self and cross attentions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 1919-1928, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

---

Arnav Saxena, Mudit Mangal, and Goonjan Jain. 2020. KeyGames: game theoretic approach to automatic keyphrase extraction. In Proceedings of the 28th International Conference on Computational Linguis- lics, pages 2037-2048, Barcelona, Spain (Online). International Committee Computational Linguis- tics.
A Performance of All Heads in BERT and GPT-2
We evaluate the keyphrase extraction performance of all the heads in BERT and GPT-2 for three datasets (Inspec, SemEval2010, SemBvvl20017) us- ing the SAMRank. Figures 3 and 4 represent the F1 @ 15 score results of the heads of each layer of BERT and GPT-2 respectively a graph form. The horizontal axis of the graph signifies the layer or- der, and the heads from to 12 in each layer are denoted by different symbols. The graph at the top shows the performance for the Inspec dataset, the one in the middle for SemEval2010, and the one at the bottom for the SemEval2017 dataset.
Yi Sun, Hangping Qiu, Yu Zheng. Zhongwei Wang, and Chaoran Zhang. 2020. Sifrank: A new base- line for unsupervised keyphrase extraction based on pre-trained language model. IEEE Access, 8:10896- 10906.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière. Naman Goyal. Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Cite arxiv2230.11997.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cexsing Systems, volume 30. Curran Associates, Inc.
Xiaojun Wan and Jianguo Xiao. 2008 Single document keyphrase extraction using neighborhood knowledge. In Proceedings of the Twenty-Thinl AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illi- nois, USA, July 13-17, 2008, pages 855 860. AAAI Press.
011-
Haoyu Wang and Jian Li. 2022. Unsupervised keyphrase extraction from single document based on bert. In 2022 International Seminar on Computer Science and Engineering Technology (SCSET), pages 267-270.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tamon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pages 17283 17297. Curran Associates, Inc.
Linhan Zhang, Qian Chen, Wen Wang, Chong Deng, ShiLiang Zhang, Bing Li, Wei Wang, and Xin Cao. 2022. MDERank: A masked document embedding rank approach for unsupervised keyphrase extraction. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 396-409, Dublin, Ireland, Association for Computational Linguistics.
Figure 3: Performance (F1 @ 15) of BERT's heads in each layer for three datasets.

---

040
Krapivin
read:
Model
F1@5
F1@
F1@15
TextRank
6.04
9,43
9.95
SingleRank
8.12
10.53
10.42
TopicRnak
8.94
9.01
8.30
MultipartitRann
9.29
9.35
9.16
YAKE
8,09
9.35
11.05
EnbbedBankkSennt2oo)M-MMin
8.44
10.47
10.71
SIFRank(ELMM))
1.62
2.52
3.00
EmbodRank(BERTT
4.05
6.60
7.84
MDERaak(BERT)
11.78
12.58
Leper
MDDERIKKKPEBEBRT>>
12.35
14.31
13.31
SR (BERT)
16.35
15.91
14.52
SAMRank (GPT-22
17.49
16.46
14.92
410
Table 7: Performance of SAMRank the Krapivin dataset for very long documents.
C SAMRank on Long Documents
Transformer-based Pre-trained Language Models (PLMs), which are inherently based on attention mechanisms, have demonstrated outstanding per- formance. However, the inherent characteristic of attention mechanisms is that their computational cost increases with the length of the input. Mod- els like BERT and GPT-2 have token input limit, typically set 512 1024 tokens. To address this limitation, SAMRank divides longer documents into smaller segments. Each segment is processed separately, and the scores from each segment are ag- gregated to derive the final keyphrases. To validate the effectiveness of this approach on very long doc- uments, conduct experiments with this method the Krapivin dataset (Krapivin and Marchese, 2009), comprised of scientific full papers that aver- age about 8,500 words. The experimental results are presented in Table The performance the baselines was referenced from the MDERank paper (Zhang et al., 2022).
Figure 4: Performance (F1 @ 15) of GPT-2's heads in each layer for three datasets.
B Visualization of Actual Maps and Token-level Scores
Figures 5 and 6 visualize the self-attention map (SAM) obtained by inputting example documents from SemEval2017, and the token-level scores cal- culated through the SAM. The SAM in Figure is extracted from the 5-2 (layer-head) of BERT, which shows the highest performance in SemEval2011 The SAM in Figure 6 is extracted from the 11-1 head of GPT-2, which shows the highest perfor- mance SemEval2017. The figures below the SAM visualize the final token-level scores obtained by combining the proposed global attention score and proportional attention score in the paper.
SAMRank, when utilizing BERT and GPT-2, achieves the highest Fl scores across all three met- rics, demonstrating its effeetiveness in handling very long documents. This suggests that even when only part the entire document inputted and the context compromised, attention weights still strongly focuses on the keyphrases. Moreover, the FI @ shows exceptionally high performance compared to other metrics, indicating a tendency for strong attention concentration a few tokens within single attention map.

---

[CLS] in this paper we present algorithms for automatic generation of logic reasoning questions the algorithms are able to construct questions that are sol ##vable with unique solutions. the algorithms employ ai techniques such as semantic networks to produce verbal questions these algorithms are small in size and are able to replace traditional question databases they are particularly suitable for implementation on the memory constrained mobile platforms the algorithms can be applied to question generation for job interview civil service exam etc
[SEP]
Figure 5: Visualization of example self-attention map extracted from BERT's head (5-2) and token-level scores calculated by combining global attention and proportional attention score.

---

In this paper, we present algorithms for generattin of logic reasoning questions, The algorithms are able to construct questions that are solvable with unique solutions, The algorithms employ techniques such as semantic networks to produce verbal questions.
These algorithms are small in size and are able to replace traditional
question databases. They are particularly suitable for implementation on the memory constrained mobile platforms. The algorithms can be applied to question generation for job interview, civil service exam, etc.
Figure 6: Visualization of example self-attention map extracted from GPT-2's head (11-1) and token-level scores calculated by combining global attention score and proportional attention score.


Reviews:
[{"review_id": "yR8VzqV1Y4", "reviewer": "Reviewer_G3r8", "summary": "", "questions": "Could you please elaborate further on the results presented in Table 3?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the opportunity to respond to your insightful questions. Your expertise in this area is invaluable to us, and we take your comments very seriously as we strive to improve the quality of our work.\n\\\n&nbsp;\n\n###  **Detailed description of Table 3**\n\nAs observed in Table 3, the **F1@5 performance of SAMRank is significantly higher across all datasets compared to F1@10 and F1@15.** This higher score at F1@5 can be attributed to an interesting phenomenon observed in specific heads of Transformer-based Pretrained Language Models (PLMs), which is further detailed in the Self-Attention Maps presented in Appendix B.\n\nIn Figures 5 (BERT’s Self-Attention Map) of Appendix B, exhibit vertical lines of darker colors corresponding to the tokens that form the phrase 'logic reasoning questions’. This implies that the majority of tokens in the text are assigning relatively higher attention weights to each token in the phrase 'logic reasoning questions.' Particularly, Figure 6, which represents the Self-Attention Map from the 11th layer's 1st head of GPT-2, reveals that a substantial number of attention weights are allocated to tokens such as 'algorithms,' 'automatic generation', 'logic reasoning questions' and 'AI’.\n\nTherefore, it can be inferred that SAMRank's notably **high performance in F1@5 is due to the intriguing phenomenon where attention is heavily focused on the tokens that make up keyphrases in specific heads of the Transformer.**\n\\\n&nbsp;\n### **Additional experimental results on Krapivin**\n\nMoreover, we have conducted additional experiments on the Krapivin dataset and observed similar trends. Despite the Krapivin documents being quite long, averaging around 9,000 words, they contain relatively few ground truth keyphrases, averaging about 6. **SAMRank achieves state-of-the-art performance in F1@5, F1@10, and F1@15 metrics on this dataset (Krapivin) as well, significantly outperforming baseline models, especially in F1@5.**\n\n| Dataset                 | Krapivin                |\n|-------------------------|-------------------------|\n| #Doc                    | 460                     |\n| Avg. #words             | 9057                    |\n| Avg. #Sents             | 345                     |\n| Avg. #Keys              | 6                       |\n| #unigram                | 17.79%                  |\n| #bigram                 | 62.21%                  |\n| #trigram                | 16.35%                  |\n\n\n\nResults on Krapivin dataset:\n| Model                     | F1@5  | F1@10 | F1@15  |\n|---------------------------|-------|-------|--------|\n| TextRank                  | 6.04  | 9.43  | 9.95   |\n| SingleRank                | 8.12  | 10.53 | 10.42  |\n| TopicRnak                 | 8.94  | 9.01  | 8.30   |\n| MultipartiteRank          | 9.29  | 9.35  | 9.16   |\n| YAKE                      | 8.09  | 9.35  | 11.05  |\n| EmbedRank(Sent2Vec)+MMR   | 8.44  | 10.47 | 10.71  |\n| SIFRank(ELMo)             | 1.62  | 2.52  | 3.00   |\n| EmbedRank(BERT)           | 4.05  | 6.60  | 7.84   |\n| JointGL                   | 5.47  | 6.12  | 6.60   |\n| MDERank(BERT)             | 11.78 | 12.93 | 12.58  |\n| MDERank(KPEBERTab)        | 12.91 | 14.36 | 13.58  |\n| MDERank(KPEBERTre)        | 12.35 | 14.31 | 13.31  |\n| **SAMRank (BERT)**         | **16.31** | **15.20** | **13.87**  |\n| **SAMRank (GPT-2)**        | **16.70** | **15.97** | **14.63**  |\n\nWe plan to enhance the manuscript by adding the additional experiment result for the Krapivin dataset to Table 3, and we also intend to provide a detailed explanation of Table 3 in the 4.3 Results section as you suggested.\n\\\n&nbsp;\n\nWe are grateful for your time and thorough review, which are instrumental in enhancing the quality of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces SAMRank, a novel unsupervised keyphrase extraction method that leverages the self-attention map of a pre-trained language model (PLM) to determine the importance of phrases. The proposed approach is technical sound.\n\n2. The motivation behind this work is both clear and convincing. Also, the paper is well-organized and easy to follow.\n\n3. The experimental results show that SAMRank outperforms most embedding-based models on both long and short documents and demonstrating that it is possible to use only a self-attention map for keyphrase extraction without relying on embeddings.", "reasons_to_reject": "On some datasets, the results of SAMBank are not significantly better than the baseline method.", "questions_to_authors": "Could you please elaborate further on the results presented in Table 3?", "paper_topic_and_main_contributions": "This paper presents SAMRank, a novel unsupervised keyphrase extraction method that utilizes a self-attention map from a pre-trained language model (PLM) to determine phrase importance. The motivation for this approach is twofold. Firstly, contextualized representations, as observed in PLMs like ELMo, BERT, and GPT-2, often possess anisotropy, making them less suitable for similarity-based tasks. Secondly, previous studies have indicated that certain heads in transformer-based PLMs, such as BERT, specialize in specific tasks, thereby highlighting the potential of using self-attention maps for keyphrase extraction.\n\nThe authors' description of the motivation for their model design is detailed and convincing, making the paper easy to comprehend and follow. The experimental section includes a thorough comparison of SAMRank with baseline approaches on three public datasets. The results obtained from these experiments demonstrate the effectiveness of SAMRank in keyphrase extraction.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UuWvkTeFZ5", "reviewer": "Reviewer_aUyH", "summary": "", "questions": "(1)In ，In Section 3.5, Bert is in token, what is the boundary of the phrase given? Please give a detailed explanation. \n\n\n(2) This proposed method only uses the Self-Attention Map Extraction and  Attention Score, and its performance can achieve better results. Please give a detailed explanation.\n(3) Add some data sets to evaluate the proposed model,  such as Krapivin (Krapivin M, Autaeu A, Marchese M. Large dataset for keyphrases extraction. Technical Report, University of Trento, 2008.).\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful suggestions on our paper, which contribute to enhancing the quality of our work.\n\\\n&nbsp;\n### **Question 1 : What is the boundary of the phrase given?**\n\nAs you rightly pointed out, both BERT and GPT-2 provide self-attention maps at the token level. We calculate the scores for each token in the text and subsequently sum the scores of tokens that constitute the candidate phrases. For better clarity, we refer to Figure 5 in Appendix B, where a self-attention map from one of BERT's heads is visualized. The score of a phrase, like \"logic reasoning questions,\" is determined by summing the scores of its constituent tokens, specifically the 13th ('logic'), 14th ('reasoning'), and 15th ('questions') tokens in the text. While the word 'questions' may appear multiple times in the text, only the 15th token's score will contribute to the score of the phrase 'logic reasoning questions'.\n\nTo implement this, we first remove tokenizer-specific special characters like '##' (BERT) and 'Ġ’ (GPT-2). Next, **we store the start and end token indices of each candidate phrase within the text.** If a certain phrase appears multiple times in the text, we store the indices for each occurrence. Only the tokens that exactly match the candidate phrase, without stemming, will contribute to its final score. **We plan to add more detailed information in Section 3 of the revised manuscript.**\n\\\n&nbsp;\n### **Question 2 : Detailed explanation on why SAMRank achieves better results**\n\nThe performance of SAMRank is not solely attributed to our proposed methodology but also to inherent characteristics commonly observed in Transformer-based pre-trained language models like BERT and GPT-2. **Our model builds upon the observation that certain heads in these Transformer models are more inclined to focus on semantically important keyphrases.** Through qualitative analysis conducted on the SemEval2017 dataset, as visualized in Appendix B (Figure 5), we observed this attention behavior. Furthermore, we extended this qualitative evaluation to GPT-2 and found similar tendencies (Figure 6). This behavior is not confined to specific models; In additional experiments, we confirmed its universality by applying SAMRank to a recently released large language model, Llama2, achieving comparable results.\n\\\n&nbsp;\n\nSE2010: SemEval2010\n\nSE2017: SemEval2017\n|                | Inspec |      |      | SE2010 |      |      | SE2017 |      |      |\n|----------------|--------|------|------|-------------|------|------|-------------|------|------|\n| Model          | F1@5   | F1@10| F1@15| F1@5        | F1@10| F1@15| F1@5        | F1@10| F1@15|\n| SAMRank(BERT)  | 33.96  | 39.35| 39.73| 15.28       | 18.36| 18.03| 24.08       | 33.40| 37.53|\n| SAMRank(GPT-2) | 33.92  | 39.44| 39.72| 15.88       | 19.49 | 19.03| 24.80   | 34.75| 38.78|\n| **SAMRank(Llama7B)** | 34.26 | 39.04| 39.20| 16.69      | 18.65| 18.50| 23.21       | 33.37| 37.62|\n| **SAMRank(Llama13B)** | 34.06| 39.74 | 39.50| 16.11       | 18.46| 19.03| 23.93       | 33.96| 38.04|\n\nWe also intended to conduct experiments on Llama2 70B, but due to time constraints, we were unable to include them. We will complete the experiments and add the performance results for Llama2 70B in the revised version of the paper.\n\nIn summary, the strength of SAMRank lies not just in its algorithmic design **but also in leveraging the intrinsic characteristics of Transformer models that inherently focus on semantically important keyphrases.** This is a behavior confirmed across different models and datasets. Therefore, the effectiveness of SAMRank is, in part, a reflection of the pre-trained models' built-in capability to give attention to keyphrases.\n\nWe are grateful for the request for additional clarification on this matter. The current manuscript may not fully explain why SAMRank excels with just the use of self-attention maps; however, we hope to address this gap in the next manuscript submission.\n\\\n&nbsp;\n### **Question 3 : Additional experimental results on Krapivin**\n\nWe are grateful for the suggestion to evaluate our model on the Krapivin dataset. Upon conducting additional experiments, **our model achieved state-of-the-art results. These statistics and results will be included in the revised paper.**\n\n\n| Dataset                 | Krapivin                |\n|-------------------------|-------------------------|\n| #Doc                    | 460                     |\n| Avg. #words             | 9057                    |\n| Avg. #Sents             | 345                     |\n| Avg. #Keys              | 6                       |\n| #unigram                | 17.79%                  |\n| #bigram                 | 62.21%                  |\n| #trigram                | 16.35%                  |\n\nResults on Krapivin dataset:\n| Model                     | F1@5  | F1@10 | F1@15  |\n|---------------------------|-------|-------|--------|\n| TextRank                  | 6.04  | 9.43  | 9.95   |\n| SingleRank                | 8.12  | 10.53 | 10.42  |\n| TopicRnak                 | 8.94  | 9.01  | 8.30   |\n| MultipartiteRank          | 9.29  | 9.35  | 9.16   |\n| YAKE                      | 8.09  | 9.35  | 11.05  |\n| EmbedRank(Sent2Vec)+MMR   | 8.44  | 10.47 | 10.71  |\n| SIFRank(ELMo)             | 1.62  | 2.52  | 3.00   |\n| EmbedRank(BERT)           | 4.05  | 6.60  | 7.84   |\n| JointGL                   | 5.47  | 6.12  | 6.60   |\n| MDERank(BERT)             | 11.78 | 12.93 | 12.58  |\n| MDERank(KPEBERTab)        | 12.91 | 14.36 | 13.58  |\n| MDERank(KPEBERTre)        | 12.35 | 14.31 | 13.31  |\n| **SAMRank(BERT)**         | **16.31** | **15.20** | **13.87**  |\n| **SAMRank(GPT-2)**        | **16.70** | **15.97** | **14.63**  |\n\nWe appreciate your insights and the time you have invested in reviewing our paper. We hope to submit a revised paper that reflects your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "（1）The proposed method seems to show fairly good results on three representative keyphrase extraction datasets.\n\n (2) The  source code has been released.", "reasons_to_reject": "(1)The innovation of the proposed method is limited using self-attention map.  \n\n(2) This proposed method only uses the Self-Attention Map Extraction and  Attention Score, and its performance can achieve better results. Please give a detailed explanation\n\n(3) Add some data sets to evaluate the proposed model,  such as Krapivin (Krapivin M, Autaeu A, Marchese M. Large dataset for keyphrases extraction. Technical Report, University of Trento, 2008.).\n", "questions_to_authors": "(1)In ，In Section 3.5, Bert is in token, what is the boundary of the phrase given? Please give a detailed explanation. \n\n\n(2) This proposed method only uses the Self-Attention Map Extraction and  Attention Score, and its performance can achieve better results. Please give a detailed explanation.\n(3) Add some data sets to evaluate the proposed model,  such as Krapivin (Krapivin M, Autaeu A, Marchese M. Large dataset for keyphrases extraction. Technical Report, University of Trento, 2008.).\n", "paper_topic_and_main_contributions": "The topic of this paper is about approaches for data- and compute efficiency.\n\nThe main contributions of this paper are as follows: \n\n(1)This paper presents a SAMRank model for keyphrase extraction that solely leverages the information from the self-attention map of PLM. \n\n (2)The proposed extracts keyphrases based on a combination of the global attention score, which identifies tokens that attract attention within a document, and the proportional attention score, which finds tokens deeply associated with important tokens.\n\n (3)Experimental results demonstrate that SAMRank is effective on three representative keyphrase extraction datasets.\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Krapivin M, Autaeu A, Marchese M. Large dataset for keyphrases extraction. Technical Report, University of Trento, 2008"}, "extra_text": {}}, {"review_id": "iU6YCvFCB2", "reviewer": "Reviewer_4f1S", "summary": "", "questions": "\n\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the time and effort you've invested in reviewing our paper. Your detailed feedback provides invaluable insights for improving our work.\n\\\n&nbsp;\n### **Technological Novelty**\n\nWe appreciate your observation that existing methods already combine transformer-based models with traditional unsupervised techniques. Our baseline models, such as AttentionRank, MDERank, and JointGL, indeed leverage transformer-based models. However, it's important to underline that SAMRank takes a fundamentally different approach. **While existing models rely on the embeddings generated by transformers**, our work exploits solely the Self-Attention Maps of specific transformer heads.\n\nWe've discussed in the Introduction section the potential limitations of embedding-based methods, citing research that shows the cosine similarity between arbitrary word embeddings from upper transformer layers approaches 1. **Embedding-based methods may not be suitable for tasks requiring ranking by similarity, hence SAMRank bypasses this limitation by focusing solely on Self-Attention Map.**\n\nSAMRank is rooted in an intriguing behavior observed in specific transformer heads, which closely aligns with how humans often underline or highlight keyphrases while reading. This behavior is observed in specific heads of Transformer models like BERT and GPT-2, which focus attention on semantically important keyphrases. In Appendix B's Figure 5 (BERT’s Self-Attention Map), for instance, the majority of tokens in the text are shown to assign higher attention weights to the phrase 'logic reasoning question'. This pattern is further substantiated in Figure 6, which visualizes attention allocation in the 11th layer's 1st head of GPT-2, highlighting attention weights towards tokens such as 'algorithms,' 'automatic generation,' 'logic reasoning questions,' and 'AI.' \n\n**As the first study to leverage this intriguing attribute inherent in Transformers, our method is significantly different** from existing transformer-based methods that aim to improve performance through embedding similarities.\n\\\n&nbsp;\n### **Comparing with Supervised Keyphrase Generation Models**\n\nYour suggestion to include keyphrase generation models like CopyRNN and SEG-Net as baselines is constructive. However, these models are **supervised**, meaning they are trained on labeled keyphrase data. In contrast, **SAMRank is unsupervised and does not rely on gold keyphrases or training.** \n\nMoreover, the very nature of generative models allows them to predict both present and absent keyphrases, making a direct comparison challenging. Extractive models like SAMRank are designed to predict 'present keyphrases,' which are phrases directly appearing in the source text. In contrast, generative models are trained not only on present keyphrases label but also on absent keyphrases label, enabling them to predict both. As a result, the performance of generative models is often separately evaluated for 'present' and 'absent' keyphrases. This is different from extractive models, which are evaluated based on a ground truth that includes both present and absent keyphrases. Even if we were to compare only the present keyphrase prediction performance of generative models, **the difference in the ground truth (gold keyphrases) used for evaluation makes a direct comparison between the two types of models less straightforward.**\n\nIt's worth noting that recent EMNLP or ACL papers on unsupervised keyphrase extraction, such as AttentionRank (Ding & Luo, 2021), JointGL (Liang et al., 2021) and MDERank (Zhang et al., 2022),  have also chosen to compare only with other unsupervised extraction models. We were following this research trend for a methodologically consistent and fair comparison with existing approaches. But we **will include a discussion on generative models like CopyRNN and SEG-Net in the Related Work section to provide a comprehensive view.**\n\\\n&nbsp;\n### **Additional Dataset**\n| Dataset                 | Krapivin                |\n|-------------------------|-------------------------|\n| #Doc                    | 460                     |\n| Avg. #words             | 9057                    |\n| Avg. #Sents             | 345                     |\n| Avg. #Keys              | 6                       |\n| #unigram                | 17.79%                  |\n| #bigram                 | 62.21%                  |\n| #trigram                | 16.35%                  |\n\nResults on Krapivin dataset:\n| Model                     | F1@5  | F1@10 | F1@15  |\n|---------------------------|-------|-------|--------|\n| TextRank                  | 6.04  | 9.43  | 9.95   |\n| SingleRank                | 8.12  | 10.53 | 10.42  |\n| TopicRnak                 | 8.94  | 9.01  | 8.30   |\n| MultipartiteRank          | 9.29  | 9.35  | 9.16   |\n| YAKE                      | 8.09  | 9.35  | 11.05  |\n| EmbedRank(Sent2Vec)+MMR   | 8.44  | 10.47 | 10.71  |\n| SIFRank(ELMo)             | 1.62  | 2.52  | 3.00   |\n| EmbedRank(BERT)           | 4.05  | 6.60  | 7.84   |\n| JointGL                   | 5.47  | 6.12  | 6.60   |\n| MDERank(BERT)             | 11.78 | 12.93 | 12.58  |\n| MDERank(KPEBERTab)        | 12.91 | 14.36 | 13.58  |\n| MDERank(KPEBERTre)        | 12.35 | 14.31 | 13.31  |\n| **SAMRank(BERT)**         | **16.31** | **15.20** | **13.87**  |\n| **SAMRank(GPT-2)**        | **16.70** | **15.97** | **14.63**  |\n\nWe've evaluated SAMRank on the Krapivin dataset as you suggested and achieved state-of-the-art performance, demonstrating its effectiveness even on very long documents. **These results will be added to Table 3.** \n\\\n&nbsp;\n### **SAMRank with LLM (Llama2)**\nSE2010: SemEval2010\n\nSE2017: SemEval2017\n|                | Inspec |      |      | SE2010 |      |      | SE2017 |      |      |\n|----------------|--------|------|------|-------------|------|------|-------------|------|------|\n| Model          | F1@5   | F1@10| F1@15| F1@5        | F1@10| F1@15| F1@5        | F1@10| F1@15|\n| SAMRank(BERT)  | 33.96  | 39.35| 39.73| 15.28       | 18.36| 18.03| 24.08       | 33.40| 37.53|\n| SAMRank(GPT-2) | 33.92  | 39.44| 39.72| 15.88       | 19.49 | 19.03| 24.80   | 34.75| 38.78|\n| **SAMRank(Llama7B)** | 34.26 | 39.04| 39.20| 16.69      | 18.65| 18.50| 23.21       | 33.37| 37.62|\n| **SAMRank(Llama13B)** | 34.06| 39.74 | 39.50| 16.11       | 18.46| 19.03| 23.93       | 33.96| 38.04|\n\nAdditionally, we conducted further experiments using the recently-released Llama2 model to assess SAMRank's performance across different transformer architectures. Preliminary results indicate that the model outperforms the proposed baselines, further **supporting the generalizability of our approach**.\n\nWe also intended to conduct experiments on Llama2 70B, but due to time constraints, we were unable to include them. We will complete the experiments and add the performance results for Llama2 70B in the revised version of the paper.\n\\\n&nbsp;\n\nWe deeply appreciate your invaluable insights and thoughtful suggestions. We hope that our responses and additional experiments address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper contributes a new idea, i.e.,  in an unsupervised model, a self-attention map in BERT and GPT-2 is used to extract keyphrases. \n\n2. The paper is detailed in each step and the source code has been released.\n", "reasons_to_reject": "1. The technological novelty of this paper is a bit limited as some keyphrase prediction methods have combined the transformer-based models with the traditional unsupervised method.\n\n2. The comparable methods only include some unsupervised keyphrase extraction models. The latest neural keyphrase generation models, such as CopyRNN (Meng et al., 2017) and SEG-Net (Ahmad et al., 2021), should be included to evaluate the proposed model.\n\n3. More datasets, such as KP20k (Meng et al., 2017) and Krapivin (Krapivin et al., 2009), should be used to evaluate the proposed model.", "questions_to_authors": "\n\n", "paper_topic_and_main_contributions": "Keyphrase prediction is an interesting task in NLP. The paper proposes an unsupervised keyphrase extraction model, which uses only a self-attention map in BERT and GPT-2l to compute the importance of the candidate phrases.\n ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tNtRPxNx4n", "reviewer": "Reviewer_Tn1j", "summary": "", "questions": "Suggestion: \n1. Have you contemplated investigating the disparities between attention mechanisms in large-scale language models (e.g., 7B, 13B, ~100B) and their smaller-scale counterparts?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our sincerest thanks for your comprehensive and thoughtful review of our paper. Your insightful concerns have been invaluable in helping us further refine and improve our work.\n\\\n&nbsp;\n\n### **The disparities between attention mechanisms in LLM and their smaller-scale counterparts**\n\nIn our study, we observed promising results for keyphrase extraction using both encoder-based BERT and decoder-based GPT-2. This led us to wonder about the efficacy of newer models like GPT-3.5 and GPT-4, which were not open-sourced at the time of our research. Fortunately, the recent release of the open-source Llama2 models allowed us to extend our experiments. We used the Llama2 7B and 13B models, which are significantly larger than BERT and GPT-2, containing up to 1,600 attention heads (about 11 times BERT and GPT-2).\n\nSE2010: SemEval2010\n\nSE2017: SemEVal2017\n|                | Inspec |      |      | SE2010 |      |      | SE2017 |      |      |\n|----------------|--------|------|------|-------------|------|------|-------------|------|------|\n| Model          | F1@5   | F1@10| F1@15| F1@5        | F1@10| F1@15| F1@5        | F1@10| F1@15|\n| SAMRank(BERT)  | 33.96  | 39.35| 39.73| 15.28       | 18.36| 18.03| 24.08       | 33.40| 37.53|\n| SAMRank(GPT-2) | 33.92  | 39.44| 39.72| 15.88       | 19.49 | 19.03| 24.80   | 34.75| 38.78|\n| **SAMRank(Llama7B)** | 34.26 | 39.04| 39.20| 16.69      | 18.65| 18.50| 23.21       | 33.37| 37.62|\n| **SAMRank(Llama13B)** | 34.06| 39.74 | 39.50| 16.11       | 18.46| 19.03| 23.93       | 33.96| 38.04|\n\nOur experiments with Llama2 generally mirrored the performance of BERT and GPT-2, with some instances of improved performance. This similarity in performance allows us to hypothesize that there may not be significant disparities in attention mechanisms between smaller models and LLMs when it comes to specific heads focusing on keyphrases.\n\n- We observed an improvement in the F1@5 metric in the SemEval2010 dataset when using Llama2. We hypothesize that **this is because LLMs like Llama2 are capable of processing a larger number of tokens, thus allowing for a more contextually complete Self-Attention Map** to be generated for keyphrase extraction. \n\n- Our additional experiments show that the size of the language model doesn't significantly affect keyphrase extraction performance.  **We cautiously speculate that while larger models can process language from more diverse perspectives, it doesn't necessarily make individual heads more effective.**\n\n- We also intended to conduct experiments on Llama2 70B, but due to time constraints, we were unable to include them. **We will complete the experiments and add the performance results for Llama2 70B in the revised version of the paper.**\n\nTo offer deeper insights into Large Language Models (LLMs), we plan to include a comprehensive analysis of performance across various heads in Appendix A and visualizations of Self-Attention Maps in Appendix B of our paper.\n\\\n&nbsp;\n\nIn closing, we appreciate your time, effort, and the expertise that went into your constructive review. We hope that additional experiments have addressed your insightful concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 This paper proposes an intriguing approach that distinguishes itself from conventional representation-based unsupervised keyphrase extraction algorithms. Instead, this study utilizes the values from the attention matrices of BERT and GPT to quantify the importance of candidate keyword phrases, resulting in a good performance.\n\n2 The experimental section of this paper is remarkably comprehensive, and the experimental results are highly impressive. These experiments thoroughly validate the effectiveness of the proposed method.", "reasons_to_reject": "1. While the method proposed in this paper is the first to utilize attention matrices for keyphrase extraction, there is a lack of sufficient research on how to select the appropriate layers and heads of attention matrices.\n\n2. I have a potentially valid concern regarding the lack of exploration of the method proposed in this paper in relation to large models at the current stage.", "questions_to_authors": "Suggestion: \n1. Have you contemplated investigating the disparities between attention mechanisms in large-scale language models (e.g., 7B, 13B, ~100B) and their smaller-scale counterparts?", "paper_topic_and_main_contributions": "This paper presents a study on unsupervised keyphrase extraction, with the central motivation being how to leverage the weight information from the attention matrices in pre-trained models during the text representation process to ascertain the importance of different tokens. The experimental section demonstrates that the proposed approach based on attention matrices outperforms methods based on embedding representation and similarity measures, thereby providing evidence for the efficacy of the SAMRank algorithm proposed in this paper. The experiments conducted in this study are comprehensive and effectively establish that the observed improvements stem from the proposed SAMRank algorithm.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel unsupervised keyphrase extraction method called SAMRank, which uses the self-attention map of PLMs to determine the importance of phrases. The idea of computing scores based on self-attention map is simple but still novel in this task. More datasets are encouraged.
Decision: Accept-Main