# Improved Graph-Based Arabic Hotel Review Summarization Using Polarity Classification
## 4 Methodology
### Evaluation  
To evaluate the quality of the generated summary, we used two metrics, Bilingual Evaluation Understudy (\(BLEU\)) and Recall-Oriented Understudy for Gisting Evaluation (\(ROUGE\)). \(BLEU\) is a precision-based metric for evaluating a generated sentence with respect to a reference sentence [42]. The range of the \(BLEU\) score is between 0 and 1. The higher the value, the higher the similarity of the candidate text to the original text. In this study, we used human summarization provided by the two native Arabic speakers in place of the original text. \(BLEU\) uses a modified precision formula that assesses the similarity between the human summary and the generated summary. Equation (4) defines the modified precision \(P_{n}\) as follows:  
\[P_{n}=\frac{\sum_{C\in\{Candidates\}}\sum_{n-gram\ c\ C\ countClip(n-gram)}}{ \sum_{C^{c}\in\{Candidates\}}\sum_{n-gram\ c\ C^{c}\ count(n-gram)}} \tag{6}\]  
\(Count\ (n-gram)\) represents the \(n-gram\) number for the candidate in the test set, and \(CountClip\ (n-gram)\) is the clipped \(n-gram\) number for the candidate sentences. To maximize precision, \(BLEU\) avoids too short summaries by using a brevity penalty \(BP\) factor, which is computed by Equation (5) below:  
\[BP= \begin{cases}1&if\ |c|>|r|\\ e^{(1-\frac{|r|}{|c|})}&if\ |c|<|r|\end{cases}  \tag{7}\]  
where \(|c|\) is the length of the summary, and \(|r|\) is the length of the human summary. Then, the \(BLEU\) score is computed by Equation (6) as follows:  
\[BLEU=BP.\exp\left(\sum_{n=1}^{N}w_{n}\ \log P_{n}\right) \tag{8}\]where \(N\) is the number of \(n-gram\), \(w_{n}\) is the weight for each modified precision, the baseline is set to \(N\) = 4, \(w_{n}\) is 1/4 = 0.25, and \(P_{n}\) is the modified precision [42]. _ROUGE_ is a recall-based evaluation metric developed by [PERSON] [44] to compare the automatically generated summary with the gold standard or the human-written summary. The algorithm used by _ROUGE_ count the number of units such as \(n-gram\), word sequences, and word pairs that appeared in both summaries under comparison [44]. _ROUGE_ has many variations; the most used ones are _ROUGE_ - \(N\) and _ROUGE_-L. _ROUGE_ - \(N\) is an \(n-gram\) recall between a generated summary and the gold standard. _ROUGE_ - \(N\) is computed as follows:  
\[ROUGE-N=\frac{\sum_{\mathcal{S}\in(Ref)}\sum_{n-gram\ \mathcal{S}}countMatch(n-gram)}{ \sum_{\mathcal{S}\in(Ref)}\sum_{n-gram\ \mathcal{S}}count(n-gram)} \tag{9}\]  
where _Ref_ represents the reference summaries and \(n\) is the \(n-gram\). \(countMatch\) (\(n-gram\)) is the maximum number of \(n-gram\) in both the reference summaries and the corresponding candidate summary. For example, _ROUGE_-1 refers to the overlap of unigrams between the generated summary and the human reference summary; similarly, _ROUGE_-2 denotes the overlap of bigrams between the generated summary and the human summary. _ROUGE_-L, longest common subsequence (\(LCS\)), is the common maximum length sequence between two given sequences, \(X\) and \(Y\)[44]. The larger the value of _ROUGE_-L, the more similar the two summaries. To evaluate summaries using _ROUGE_-L, the F-measure is computed to estimate the similarity between two summaries, \(X\) of length \(m\) and \(Y\) of length \(n\), where \(X\) is a human summary sentence, and \(Y\) is the generated summary sentence. Thus, _ROUGE_-L is computed by finding F-measure as follows:  
\[R =\frac{LCS(X,\ Y)}{m} \tag{10}\] \[P =\frac{LCS(X,\ Y)}{n}\] (11) \[F =\frac{\left(1+\beta^{2}\right)\ R\ P}{R+\beta^{2}\ P} \tag{12}\]  
Such that \(LCS\) (\(X\), \(Y\)) is the length of the longest common subsequence of \(X\) and \(Y\), while \(\beta\) = \(P\)/\(R\) when \(\partial F\)/\(\partial R\) = \(\partial F\)/\(\partial P\). To summarize, _BLEU_ focuses on precision, or how much the words or \(n-gram\) in the system-generated summary appear in the human summary. In contrast, _ROUGE_ focuses on recall, i.e., how much the words or \(n-gram\) in the human summary appear in the system summary.