SlowGuess/NAACL_2022_OCR huggingface dataset:
BLONDE: An Automatic Evaluation Metric for Document-level Machine Translation
Yuchen Eleanor Jiang $^{1*}$  Tianyu Liu $^{1}$  Shuming Ma $^{2}$  Dongdong Zhang $^{2}$  Jian Yang $^{2}$  Haoyang Huang $^{2}$  Rico Sennrich $^{3}$  Mrinmaya Sachan $^{1}$  Ryan Cotterell $^{1}$  Ming Zhou $^{4}$
$^{1}$ ETH Zürich  $^{2}$ Microsoft Research Asia  $^{3}$ Universität Zürich  $^{4}$ Langboat.com {yucjiang, tianyu.liu, ryan.cotterell, mrinmaya.sachan}@inf.ethz.ch {shuming.ma, dongdong.zhang, t-jianya, haohua}@microsoft.com sennrich@cl.uzh.ch ming.zhou@chuangxin.com
Standard automatic metrics, e.g., BLEU, are not reliable for document-level MT evaluation. They can neither distinguish document-level improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations. This paper introduces a novel automatic metric  $\mathrm{BLONDE}^1$  to widen the scope of automatic MT evaluation from the sentence to the document level. BLONDE takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. We conduct extensive comparisons on a newly constructed document-level translation dataset BwB. The experimental results show that BLONDE possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances. In a large-scale human study, BLONDE also achieves significantly higher Pearson's  $r$  correlation with human judgments compared to previous metrics.
https://github.com/EleanorJiang/BlonDe
Over the past years neural machine translation (NMT) models have become the models of choice in Machine Translation (MT; Luong et al., 2015; Vaswani et al., 2017; Zhang et al., 2018, inter alia). Although some recent work (Hassan et al., 2018; Popel, 2018; Bojar et al., 2018) suggests that NMT has achieved human parity at the sentence level, the reliability of these human-parity claims was quickly contested by Läubli et al. (2018, 2020),
Figure 1: BLONDE is more selective than BLEU for document-level MT, and also shows a larger quality difference between human and machine translations.
showing that there is a larger difference between human and machine translation quality when intersentential context is taken into account.
Therefore, document-level machine translation has received increased attention in the MT community. However, despite various modeling advances, the MT community still lacks an efficient and effective evaluation metric for document-level translation. Standard evaluation metrics for MT, e.g., BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and METEOR (Banerjee and Lavie, 2005), focus on the quality of translations at the sentence level and do not consider discourse-level features.
Thus, test suites that perform context-aware evaluation by targeting characteristic discourse-level phenomena have been proposed (Hardmeier et al., 2015; Guillou and Hardmeier, 2016; Burchardt et al., 2017; Isabelle et al., 2017; Rios Gonzales et al., 2017; Muller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Guillou and Hardmeier, 2018, inter alia) for document-level MT. However, such test suites need to be re-created for new domains or even language pairs, and their construction can be very labor-intensive. We still lack an
*Most of the work was done while the first author was an intern at Microsoft Research Asia.
1Bilingual Evaluation of Document Translation.
Human Language Technologies, pages 1550 - 1565
easy-to-use automatic metric that can reliably discriminate the quality of document-level translation.
In this paper, we curate a large-scale document-level parallel corpus (BWB) from heterogeneous data sources, and quantify document-level translation mistakes by performing a large human study. On this dataset, we found that inconsistency, $^{2}$  ellipsis, and ambiguity were the most noticeable phenomena critical for document-level MT, together amounting to  $86.73\%$  of MT mistakes. Based on this analysis, we propose BLONDE, an automatic metric that evaluates translation quality at the document level. At the core of the metric is the similarity-based bijection between subsets of reference and system categories, e.g., pronouns, inflected forms, discourse relations and lexicons, and phrases, e.g., named entities. It computes recall, precision and F1, along with the corresponding measure of  $n$ -grams. Furthermore, BLONDE can incorporate human annotation easily by computing scores of human-annotated categories in the same way.
We compare BLONDE with 11 other metrics and demonstrate that BLONDE is better at distinguishing between context-aware and context-agnostic MT systems. We also observe that the degree to which BLONDE correlates with sentence-level metrics (e.g., BLEU) is lower than the degree to which the sentence-level metrics correlate with each other. This signals that BLONDE indeed captures additional aspects of translation quality beyond the sentence-level. Finally, our human evaluation also reveals significantly higher Pearson's  $r$  correlation coefficients between BLONDE and human assessments compared to other metrics.
2 BWB: Bilingual Web Book Dataset
To design a metric that is sensitive to document-level phenomena, we first curate a document-level Chinese-English parallel corpus, called BwB (Bilingual Web Books). BwB consists of Chinese web novels across multiple genres (sci-fi, romance, action, fantasy, comedy, inter alia) and their corresponding English translations crawled from the Internet.
Dataset Creation. The novels are translated by professional native English speakers, and are corrected by editors. The sentence alignment of
Statistic	Train	Test	Dev	Total
#Docs	196,304	80	79	196,463
#Sents	9,576,566	2,632	2,618	9,581,816
#Words	325.4M	68.0K	67.4K	460.8M
Table 1: Statistics of the BwB dataset.
the training set is done by Bluealign $^3$  (Sennrich and Volk, 2011). We hired four bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting BwB parallel corpus and observe an alignment accuracy rate of  $93.1\%$ . We further asked the same batch of annotators to correct such misalignments in both the development and the test set. The details of the corpus creation and quality control are described in Appendix A.
Statistics. Table 1 summarizes the statistics of the BwB dataset. It is a much larger dataset, and contains longer documents and richer discourse phenomena compared to all previous document-level datasets (Lison and Tiedemann, 2016; Koehn and Knowles, 2017; Barrault et al., 2019; Koehn, 2005; Liu and Zhang, 2020). To the best of our knowledge, this is the largest Chinese-English document-level translation dataset to date.
Dataset Split. We treat chapters in our books as documents. The maximum, median, and minimum number of sentences per document are 46, 30 and 18, respectively. To prevent any train-test leakage, we split the dataset into a training, development and a test set such that chapters from the same book are part of the same split. We use 377 books for training, and randomly select 80 and 79 documents from the 3,018 documents in the remaining 6 books as the development and test sets, respectively. $^{4}$
3 Analyzing Discourse Errors
Next, we conduct a human study on the test set of BwB, in which we identify and categorize the discourse errors made by MT systems that are not captured in sentence-level evaluation. This human study is conducted by eight professional translators. The annotators are asked to classify translation errors into DOCUMENT-level and SENTENCE-level errors (some cases can be both). SENTENCE-level
<sup>3</sup>https://github.com/rsennrich/Blealign
One document in the development set was dropped due to its poor annotation quality.
2By inconsistency we mean the mistakes related to coreference and lexical cohesion (Carpuat, 2009; Guillou, 2013).
Error Type	#	%
NO ERROR	451	17.1%
SENTENCE	1351	51.3%
DOCUMENT	1893	71.9%
INCONSISTENCY	1695	64.4%
NAMED ENTITY	1139	43.3%
TENSE	1018	38.7%
ELLIPSIS	534	20.3%
PRONOUN	456	17.3%
OTHER	103	4.0%
AMBIGUITY	193	7.3%
Table 2: The statistics of translation errors in human analysis.
errors refer to those errors that render the translations to be inadequate or not fluent as stand-alone sentences, while DOCUMENT-level errors reflect a coherence violation across multiple sentences in the document. DOCUMENT-level errors are further categorized according to the linguistic phenomena leading to the lower performance in the context-dependent translation.[5]
Table 2 shows the result of our error analysis. A substantial proportion of translations have document-level errors (71.9%). This supports the claim that BwB contains rich discourse phenomena that common MT systems cannot address. We observe that three categories, i.e., inconsistency (64.4%), ellipsis (20.3%) and ambiguity (7.3%), account for the vast majority of document-level errors. Below we discuss these three categories of DOCUMENT-level errors and the design intuitions behind BLONDE.
Inconsistency. We consider two kinds of consistency in translation: lexical and grammatical. Lexical consistency is defined as a repetitive term that keeps the same translation throughout the whole document (Carpuat and Simard, 2012). Inconsistent translation of named entities can significantly impact translation output, although BLEU may not be adversely affected (Agrawal and Singla, 2012; Hermjakob et al., 2008). Therefore, in the design of BLONDE, we also focus on the reiteration of named entities (e.g., Qiao in Figure 3). On the other hand, typical examples of grammatical consistency are tense and gender consistency. Tense consistency refers to the tense being compatible with the context, rather than being exactly the same across the whole document. Tense inconsistency can arise when the source language is an isolating language and does not mark tense explicitly, e.g., in Chinese,
SRC 你在看(kan)什么？《复仇者联盟》。
REF What are you watching? The Avengers.
MT What are you looking at? The Avengers.
Figure 2: An example of ambiguity. 看(kan) corresponds to look, see, watch and view. The correct translation can only be inferred from the next sentence (The Avengers).
and the target language is a synthetic language, e.g., English (teal in Figure 3), where tense is marked explicitly. In the same spirit, the same entity should maintain a consistent grammatical gender.
Ellipsis. Ellipsis denotes the omission from a clause of one or more words that are nevertheless inferred in the context of the remaining elements (Yamamoto and Sumita, 1998; Voita et al., 2019). Translation errors arise when there are elliptical constructions in the source language while the target language does not allow for the same types of ellipsis. For example, the ellipsis of subjects or objects is very common in Chinese while it is ungrammatical in English—especially for pronouns. In Figure 3, she (Qiao) is omitted in Chinese. However, it is hard to guess the gender of Qiao from this stand-alone sentence: the correct pronoun choice can only be inferred from context (there is a her in the previous sentence). Another common form of ellipsis is the omission of discourse markers, especially when the source language has more zero connective structures (Po-Ching and Rimmington, 2004) than the target language. In the example, However and So are ignored in SRC, which misleads the sentence-level system MTA to ignore the discourse relations between sentences.
Ambiguity. Translation ambiguity occurs when a word in one language can be translated in more than one way into another language (Tokowicz and Degani, 2010). The cross-language ambiguity comes from several sources of within-language ambiguity including lexical ambiguity, polysemy, and near-synonymy. A unified feature of these is that ambiguous terms satisfy the form of one-to-many mappings. For the example in Figure 2, the word  $\text{kan}$  can be translated to look, see, watch or view. Without access to the context, all the lexical choices are sensible.
It is worth noting that the metric proposed in this study can be applied to a wider range of language pairs by extending the definition of grammatical consistency.
5The annotation guidelines are described in Appendix B.
ENTITY E	TENSE V	PRONOUN P	DM M
SRC	a) 小乔(Qiao)看着(look)相片回忆(recall)起了二十年前。b) 那个满脸胡须的男人(man)正是(be)她(she)的新婚丈夫。c) 那却是(be)他们之间初次见面(meet)。d) 小乔(Qiao)一见到他(he)心里就咯噔(jolt)了一下, 嗷的站(stand) 起来。	[[Qiao]]	[VBD, VBZ]	[masculine, feminine, epicene, neuter]	[contingency, temporal, expansion, comparison]
REF	a) Qiao looked at the photo and recalled twenty years ago.	[1]	[2,0]	[0,0,0,0]	-
b) This bearded man was her newlywed husband.	[0]	[1,0]	[0,1,0,0]	[0,0,0,0]
c) [yet] this was the first time they were meeting with each other.	[0]	[2,0]	[0,0,1,0]	[0,0,0,1]
d) [So] Qiao's heart jolted as soon as [[she]] saw him, and [[she]] quickly stood up.	[1]	[2,0]	[1,2,0,0]	[1,0,0,0]
MTA	a) Qiao looked at the photo and recalled twenty years ago.	[1]	[2,0]	[0,0,0,0]	-
b) This bearded man is her newlywed husband.	[0]	[0,1]	[0,1,0,0]	[0,0,0,0]
c) This is the first time they meet with each other.	[0]	[0,2]	[0,0,1,0]	[0,0,0,0]
d) Joe's heart is squeaky as soon as [[he]] saw him, and [[he]] quickly stands up.	[0]	[0,2]	[3,1,0,0]	[0,0,0,0]
MTB	a) Qiao looked at the photo and recalled the past twenty years ago.	[1]	[2,0]	[0,0,0,0]	-
b) This man with the beard was her newlywed husband.	[0]	[1,0]	[0,1,1,0]	[0,0,0,0]
c) [However], that was the first time they met.	[0]	[2,0]	[0,0,1,0]	[0,0,0,1]
d) [So] as soon as Qiao saw him, [[her]] heart became squeaky, and [[she]] swiftly stood up.	[1]	[2,0]	[1,2,0,0]	[1,0,0,0]
Figure 3: An example containing inconsistency and ellipsis in BwB. For inconsistency, the same entities are marked in the same color (Qiao and Husband), and verbs are marked in teal. For ellipsis, omissions are marked with  $\mathbb{I}$ . DM stands for discourse markers  $(\mathbb{I})$ . The translation mistakes are underlined. MTB is intuitively a better system than MTA to human readers.
4 BLONDE
The aforementioned document-level phenomena have little impact on the  $n$ -gram statistics of translations. However, as is shown in Section 3, they can be key considerations for human readers when evaluating translations at the document level. Standard automatic metrics ignore the importance of contextual coherence of translations, which implies that the document-level nuances are not being properly modeled (Zhou et al., 2008; Xiong and Zhang, 2014). In this section, we describe BLONDE, an automatic metric that explicitly tracks discourse phenomena.
4.1 Document-Level Evaluation
We first give the formulation of measuring discourse phenomena. We define a document  $\mathfrak{D} = [S_1, \ldots, S_N]$  as a sequence of  $N$  sentences. We take a sentence  $S$  of length  $T$  to be a string of tokens  $t_1 \cdots t_T$  where each token  $t_i$  is taken from the vocabulary  $\mathcal{V}$ . Let spans  $(S) = \{m_1, m_2, \ldots\}$  be the set of spans in the sentence  $S$ . Here, a span is a subsequence of the tokens in  $S = t_1 \cdots t_T$ .
Let us assume that we are interested in  $K$  discourse categories. Each of these categories captures a discourse phenomenon of interest. As shown in Section 3, named entity inconsistency, tense inconsistency and pronoun ellipsis make up the majority of discourse errors (67.8%) on the data analyzed. We therefore introduce three types of categories: ENTITY, TENSE and PRONOUN. In addition,
we introduce discourse markers DM as a category, which are the essential contextual links between the various discourse segments (See Figure 3).
For a certain discourse category of interest,  $k$ , we assume that there are  $L_{k}$  features. In our case, the features of ENTITY  $\mathcal{E}$  are a list of named entities in  $\mathfrak{D}$ ; the features of TENSE are  $\mathcal{V} = [\text{MD},\text{VBD},\text{VBN},\text{VBP},\text{VBZ},\text{VBG},\text{VB}]$ , the features of PRONOUN are  $\mathcal{P} = [\text{masculine},\text{feminine},\text{neuter},\text{epicene}]$ ; the features of DM are  $\mathcal{M} = [\text{contingency},\text{temporal},\text{expansion},\text{comparison}]$ . Note that different categories can have different numbers of features and the number of features can be dynamic:  $\mathcal{E}$  depends on  $\mathfrak{D}$  while  $\mathcal{V}$  and  $\mathcal{P}$  are fixed. The intuition behind this is that we want to encourage the system output to keep consistent tense and pronouns as well as the consistent translation for a specific named entity.
Let us now define  $\mathbf{C}_{k,l}(S) \subseteq \operatorname{spans}(S)$  as the set of spans in  $S$  that share the  $l^{\mathrm{th}}$  feature in the  $k^{\mathrm{th}}$  discourse category. To give a concrete example, let us assume that TENSE is the  $k^{\mathrm{th}}$  category and VBD is the  $l^{\mathrm{th}}$  feature in this category. The corresponding  $\mathbf{C}_{k,l}(S)$  is the set of the spans (in this case, unigrams) tagged with VBD in the sen
$^{7}$ MD: Modal; VBD: Verb (past tense verb); VBN: Verb (past participle); VBP: Verb (non-3rd person singular present); VBZ: Verb (3rd person singular present); VBG: Verb (gerund or present participle); VB: Verb (base form).
8masculine: he, him, his, himself; feminine: she, her, hers, herself, neuter: it, its, itself; epicene: they, them, their, theirs, themselves.
${}^{9}$  A detailed explanation is provided in Table 6.
tence  $S$ . In Figure 3, all the spans are colored. We then let  $\mathbf{C}_k(S)$  refer to a vector of size  $L_{k}$  where each element of that vector is the set  $\mathbf{C}_{k,l}(S)$ . The sets of spans for ENTITY, TENSE, PRONOUN and DM can be produced by an NER model, a POS tagger, a rule-based string match and a discourse marker, respectively. We also define a weight vector  $\pmb{w}_{k} = [w_{k,l}:l\in \{1,\dots ,L_{k}\} ]$  for each discourse category  $k$ , where each entry  $w_{k,l}$  corresponds to the weight given to a feature.
We then define the discourse representation of sentence  $S$  as the concatenation of all categories:
$$
\mathbf {C} (S) = \left[ \mathbf {C} _ {k} (S): k \in \{1, \dots , K \} \right] \quad (1)
$$
Similarity. Let  $\mathbf{sim}:\mathbf{C}(S^s)\times \mathbf{C}(S^r)\to$ $\mathbb{R}^K$  represent a similarity vector which measures category-wise similarity between the discourse representations of two sentences  $S^r$  and  $S^s$ . Each entry of the vector sim takes non-negative values: The entry being zero if  $S^s$  and  $S^r$  have no shared spans with the same discourse category.
The similarity vector sim defined here can be implemented in several ways. A possible implementation of sim can be achieved by counting the number of functionally similar spans for each feature and then taking a weighted sum over all features:
$$
\begin{array}{l} \mathbf {s i m} \left(S ^ {s}, S ^ {r}\right) = \tag {2} \\ [ \mathbf {s i m} _ {k} (S ^ {s}, S ^ {r}): k \in \{1, \dots , K \} ] \\ \end{array}
$$
where each entry  $\mathbf{sim}_k$  is defined as follows:
$$
\mathbf {s i m} _ {k} \left(S ^ {s}, S ^ {r}\right) = \tag {3}
$$
$$
\boldsymbol {w} _ {k} \odot \min  \left(\operatorname {c o u n t} \left(\mathbf {C} _ {k} \left(S ^ {s}\right)\right), \operatorname {c o u n t} \left(\mathbf {C} _ {k} \left(S ^ {r}\right)\right)\right)
$$
$$
\begin{array}{l} \operatorname {c o u n t} \left(\mathbf {C} _ {k} (\cdot)\right) \tag {4} \\ = \left[ | \mathbf {C} _ {k, l} (\cdot) |: l \in \{1, \dots , L _ {k} \} \right] \\ \end{array}
$$
denotes the cardinality of  $\mathbf{C}_{k,l}$  applied entry-wise and min denotes the minimum function applied element-wise. Intuitively,  $\mathbf{sim}_k$ , measures the number of functionally similar spans shared by  $S^s$  and  $S^r$ . Assume that TENSE is the  $k^{\mathrm{th}}$  category. $^{10}$
It is worth noting that there are many other reasonable ways to operationalize sim. For ENTITY,
partial credit could be assigned to two named entities if they have overlapping tokens; for TENSE and PRONOUN, partial credit could be assigned to two similar categories, e.g., VBP and VB; for DM, partial credit could be assigned according to the sense hierarchy and the confidences in the detected discourse markers. We leave the expansion of the sim definition to future work.
A Document-level Similarity Measure. Now we turn from measuring the similarity at the sentence level to the document level. We first lift  $\mathbf{sim}(\cdot, \cdot)$  to measure the similarity between two documents:
$$
\operatorname {s i m} \left(\mathfrak {D} ^ {s}, \mathfrak {D} ^ {r}\right) = \sum_ {\substack {S ^ {s} \in \mathfrak {D} ^ {s}, \\ S ^ {r} \in \mathfrak {D} ^ {r}}} \operatorname {s i m} \left(S ^ {s}, S ^ {r}\right) \tag{5}
$$
where the sum is applied element-wise.
We then define  $\mathbf{sim}(\cdot, \cdot)$  for a system document  $\mathfrak{D}^s$  and a set of reference documents  $\mathbb{D}^r = \{\mathfrak{D}^{r_1}, \mathfrak{D}^{r_2}, \ldots\}$  by aggregating the  $\mathbf{sim}$  of all sentences in  $\mathfrak{D}^s$  and  $\mathbb{D}^r$ :
$$
\operatorname {s i m} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) = \sum_ {S ^ {s} \in \mathfrak {D} ^ {s}} \bigoplus_ {S ^ {r} \in \mathbb {D} ^ {r}} \operatorname {s i m} \left(S ^ {s}, S ^ {r}\right) \tag {6}
$$
Here,  $\oplus$  is a generic aggregator over multiple references, e.g.,  $\oplus = \max$ , if we take the reference which has the maximum similarity with the system output; or  $\oplus = \sum$  if we sum up the similarity scores of all references. Again,  $\oplus$  is applied element-wise.[11] We also reuse the notation  $\mathbf{sim}(\cdot, \cdot)$  for two sets of documents  $\mathbb{D}^s$  and  $\mathbb{D}^r$ :
$$
\operatorname {s i m} \left(\mathbb {D} ^ {s}, \mathbb {D} ^ {r}\right) = \bigoplus_ {\mathfrak {D} ^ {s} \in \mathbb {D} ^ {s}} \operatorname {s i m} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) \tag {7}
$$
Note that the similarity vector can also be computed for the same (set of) documents. For example, if sim is implemented as counting the number of functionally similar spans for each feature, then, sim  $(\mathfrak{D}^s,\mathfrak{D}^s)$  and sim  $(\mathbb{D}^r,\mathbb{D}^r)$  denote the total number of spans of each category in the system output and the reference, respectively. $^{12}$
Scoring. We are now ready to define the "goodness" of a system output with respect to our discourse phenomenon of interest. We compute the
<sup>10</sup>In Figure 3,  $\mathbf{sim}_k(U_b^{\mathrm{MTA}}, U_b^{\mathrm{REF}}) = 0$  since MTA mistranslated the verbs as present tense due to the exclusion of context. The total similarity  $\mathbf{sim}(\mathfrak{D}^{\mathrm{MTA}}, \mathfrak{D}^{\mathrm{REF}})$  is the number of functionally similar spans across all features:  $(1, 2, 4, 0)$  for  $(\mathcal{E}, \mathcal{V}, \mathcal{P}, \mathcal{M})$ . Here, we assume all category weights are 1.
11In Figure 3, since we only have one reference,  $\mathbf{sim}(\mathfrak{D}^{\mathrm{MTA}},\mathbb{D}^r) = \mathbf{sim}(\mathfrak{D}^{\mathrm{MTA}},\mathfrak{D}^{\mathrm{REF}}) = (1,2,4,0)$
12In Figure 3,  $\mathbf{sim}(\mathfrak{D}^{\mathrm{MTA}},\mathfrak{D}^{\mathrm{MTA}}) = (1,7,6,0)$  and  $\mathbf{sim}(\mathbb{D}^r,\mathbb{D}^r) = \mathbf{sim}(\mathfrak{Q}^{\mathrm{REF}},\mathfrak{Q}^{\mathrm{REF}}) = (2,7,5,2)$ .
precision, recall and F1 for all  $K$  discourse categories defined as follows:
$$
\boldsymbol {p} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) = \frac {\operatorname {s i m} \left(\mathfrak {D} ^ {s} , \mathbb {D} ^ {r}\right)}{\operatorname {s i m} \left(\mathfrak {D} ^ {s} , \mathfrak {D} ^ {s}\right)}, \tag {8}
$$
$$
\boldsymbol {r} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) = \frac {\operatorname {s i m} \left(\mathfrak {D} ^ {s} , \mathbb {D} ^ {r}\right)}{\operatorname {s i m} \left(\mathbb {D} ^ {r} , \mathbb {D} ^ {r}\right)}, \tag {9}
$$
$$
\boldsymbol {F} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) = 2 \cdot \frac {\boldsymbol {p} \odot \boldsymbol {r}}{\boldsymbol {p} + \boldsymbol {r}}. \tag {10}
$$