## 3 Methodology
### Salient Terms Extraction  
The objective of this component is to identify the key terms in CPG contents, using both supervised and unsupervised machine-learning techniques. This objective is achieved in three steps; guideline preprocessing, interpretable model training, and salient terms identification. The _guideline preprocessing_ transform CPGs to machine-processable format by tokenization, stemming, case transformation, stop word removal, and synonym identification. We trained a set of supervised machine-learning models comprising of decision tree and rule induction, and unsupervised algorithms LDA, and word2 vec to find the key contributing terms in a CPG for taking sentence classification decision. These techniques were selected due to their results transparency and effectiveness in the classification task. We applied various parameter settings for each model to check its classification accuracy and extract the final terms, which are then used for making the classification decision. As an example in the decision tree model, we apply gain ratio, information gain, accuracy, and Gini index splitting criteria. We also evaluate the models' behaviors with and without feature selection. In feature selection, filter-based and wrapper-based techniques were applied to limit the number of features and nodes of the final model by eliminating irrelevant features. However, identifying the correct number of features is still an open research issue, in this study, we used the grid search technique [32] to dynamically set the number of features for a model. The algorithm used for dynamic features selection is given in Algorithm 1. We check the terms considered by the model generated after feature selection to get a valuable list of salient terms considered by the model. The example of the decision tree model is shown in Figure 2. The decision tree model have considered total 8 unique salient terms including \"cosmopolitan\", \"reach\", \"aged\", \"adult\", \"channel\", \"condition\", \"person\", and \"bespeak\" for distinguishing recommendation sentences from non-recommendation sentences in a CPG. We considered all terms as salient terms, which are extracted by given models with all possible settings. A list of partial salient terms considered by various machine-learning models is given in Table 1. We shared a list of unique salient terms with human experts, hereafter knowledge engineers (KEs), for reconsideration of their extracted patterns which leads to changes in the KEs extracted patterns and causes increase in the final classification accuracy. Figure 1: Process flow of the proposed recommendation identification technique. ```
Input : Guidelines \(G=\{g_{1},g_{2},g_{3},...,g_{n}\}\), Models \(M=\{m_{1},m_{2},m_{3},...,m_{n}\}\) Result: SelectedFeatures \(F^{\prime}=\{f^{\prime}_{1},f^{\prime}_{2},f^{\prime}_{3},...,f^{\prime}_{m}\} \mid m\leq n\) foreach\(g\) in \(G\)do  Features \(F\gets getFeatures(g)\)  ModelAccuries \(MA=[]\) foreach\(m\) in \(M\)do  FeaturesAccuracies \(FA=[]\) forRange \(r\) in \([5-length(F)]\)do  FeatureSubSet \(FS=F.slice(0,r)\) \(Model=TrainModel(m,FS)\) \(accuracy=Model.evaluate(g)\)  FeaturesAccuracies.\(push(accuracy)\)  end for  ModelAccuries.\(push(FeaturesAccuracies)\)  end for  AverageAccuries \(AV=getAverage(MA)\)  MaxAccuray \(\gets max(AV)\) \(F^{\prime}=F.slice(0,MaxAccuray.index)\)  end for
```  
**Algorithm 1**Grid search-based dynamic feature selection algorithm.