# A Data-Driven Approach to Discovering Process Choreography
## 4 Materials and Methods
### Evaluation Metrics  
The discovered process choreography is evaluated in a supervised manner, for which a reference inter-organizational process model (\(RIOBP\)) is required. This model contains the process logic, behavior, and real interaction of the participants of the IOBP. Then, process choreography data are extracted from the \(RIOBP\) model, including the message-type tasks, the subtype of each message-type task, and the message exchange sequence (noted as _relevant relationships_ previously). Furthermore, in the set \(PME\) (see Method 6), the relationships selected by our proposal are searched and counted because they met the condition of exceeding the threshold \(U_{a}\) and with the highest value in the measure of cosine similarity, which we call \(found\ relationships\). Moreover, relations with a cosine similarity value greater than or equal to the threshold \(U_{a}\) are recovered. These relationships are identified as _recovered relationships_. Then, \(found\ relationships\) are the subset of _relevant relationships_ found by our proposal in the set _recovered relationships_; that is,  
\[found\ relationships\subset recovered\ relationships|found\ relationships\in relevant\ relationships \tag{2}\]  
Considering the above, we propose the following metrics to evaluate the discovered process choreography:  
* **Relationship Precision (RP)** is the proportion of _relevant relationships_ encountered out of all _recovered relationships_. This metric evaluates whether the complete process choreography has been discovered (\(found\ relationships=|relevant\ relationships\cap recovered\ relationships|\)) without adding relations not found in the process choreography of the reference model (_relevant relationships_). \[RP:\frac{|relevant\ relationships\cap recovered\ relationships|}{ recovered\ relationships}\] (3)
* **Relationship Recall (RR)** is the proportion of \(found\ relationships\) matching _relevant relationships_. This metric indicates the percentage of the process choreography discovered versus the process choreography of the reference IOBP model _relevant relationships_. \[RR:\frac{|relevant\ relationships\cap recovered\ relationships|}{relevant\ relationships}\] (4)
* **F-score of the Relationship (fsR)** is the harmonic mean between the RP and RR, which allows us to determine the performance of the proposed approach. This metric indicates the ability of the method to discriminate between relevant and non-relevant relationships. \[FsR=2\times\frac{RP\ \times RR}{RP+RR}\] (5)  
Furthermore, the quality of intra-organizational business process models are evaluated through the metrics _precision_, _recall_, and _generalization_, as defined in [44; 45]. These metrics have as input an event log and an intra-organizational business process model, comparing the information available in the event log and the discovered business process model. In our experimentation, it is also required to evaluate the inter-organizational business process model (\(IOBP\)) built from the merged event log (\(Lf\)), for which it is required to implement the following modifications in the process logic of the \(IOBP\) model, to represent it as an intra-organizational model, making it possible to determine the quality of the model discovered through the application of the _precision_, _recall_, and _generalization_ metrics. Let us consider the \(IOBP\) model and the collaborative \(Lf\) model previously generated. In the \(IOBP\) model, the following elements contained in the _pools_BPMNI = \((i,o,T,G,E_{m})\) and \(\text{BPMNR}=(i^{\prime},o^{\prime},T^{\prime},G^{\prime},E^{\prime}_{m})\) are updated:  
1. the initial activity \(i\) and \(i^{\prime}\) of the BPMNI and BPMNR models, respectively, are eliminated, and
2. a unique initial activity \(I\) is added, from which the intra-organizational models start, formed as \(\text{BPMNI}=(I,o,T,G,E_{m})\) and \(\text{BPMNR}=(I,o^{\prime},T^{\prime},G^{\prime},E^{\prime}_{m})\).