## 7 Evaluation
### Experiment Setup  
As a thorough evaluation of the proposed NLP extension requires research and development of multiple tools, our practical experimentation is limited to certain aspects of this complex solution. The main goal of this evaluation is to verify whether the performance of NLP-enhanced partial M2M transformations is superior to partial transformations with no NLP capabilities (i.e., the original solution [14]). To test our hypothesis, transformations were executed in two modes:  
1. Applying a simple LEFT(0, 1, ') operator (described in Section 5.1), which implements the \"first word is a verb\" pattern (i.e., \"<VERB><NOUN>|<NOUN PHRASE>\"); this was done to process all activity-like concept names in the original solution.
2. Applying our proposed NLP extension, which uses NLP operators to recognize and extract the noun and verb phrases composed of any number of words.  
In other words, in this experiment, we limited our evaluation to the extraction of noun and verb phrases; these capabilities are utilized in the use cases presented in Section 6.1, Section 6.3, and Section 6.4. Other NLP-related features of the developed NLP extension are out of the scope of this particular experiment simply because they have no analogs in the original solution to be compared with. Other than that, evaluating the other features on their own was also left out of consideration for the following reasons: (1) While abbreviation resolution is discussed in several papers, it is very context-dependent and, in the context of our work, can often be simplified to dictionary-based expansion; and (2) detection of semantic relationships (synonyms, hypernyms, homonyms, and meronyms) can be simplified to performing queries in lexical databases (e.g., WordNet [62]); even though alternative approaches exist (as stated in Section 5.2), so far we have not been successful at finding implementations suitable for practical testing.  
For our experimental input dataset, we selected a predefined number of BPMN process models and UML use case models, which were acquired from various available outside sources. The final set of input models consisted of the following:  
* 20 UML use case models collected freely from the Internet;20 BPMN process models selected from the large set of Signavio BPMN models provided by the BPM Academic Initiative [86].  
The UML class model was selected as the output model of our experiment. Note that it could be safely replaced with the SBVR fact model or virtually any other model being implemented as a UML profile (e.g., our research group has developed such implementation for SBVR [15]). Yet, we chose the UML class model for simplicity, because it is more established and well known in the modeling community.  
Next, the acquired set of input models was processed by extracting the names of _Task_ elements (for BPMN process models) and _UseCase_ elements (for UML use case models) for the actual experimentation. It was considered that the _Task_ and _UseCase_ elements would contain at least one verb phrase and one noun phrase. The extracted elements were then cleaned of grammatical errors and other inconsistencies. We also excluded entries that would lead to ternary associations in the output results, because this would require more advanced processing to match verb phrases with the corresponding noun phrases, which was not yet considered at this stage of the research. However, entries that had a single verb phrase with multiple noun phrases in conjunctive or disjunctive form were considered for processing, as such structures would be transformed to multiple valid tuples in the output results.  
Finally, we applied manual extraction of output results was applied to acquire the expected outputs, which would be considered as the \"gold standard\" for benchmarking and evaluation. For a more fair comparison of the original solution with the NLP-enhanced development, we also disabled the implemented verb tense normalization feature, which was not implemented in the original approach.