## 2 The Proposed Method for Integration of Uncertain Expert Knowledge to the System Model
### Validation of Expert Knowledge  
Based on the results, expert knowledge can be evaluated by comparing the estimated empirical distributions (\(q(x_{i})\)) of the parameters and the outputs with the original ones of the experts (\(p_{e}(x_{i})\)). Moreover, incorrect expert judgments can be identified and eliminated by this analysis. One of the most simple techniques to compare two probability distributions is the measure of Jaccard index, which will be used as goodness values of expert judgments. As the original method is able to represent the similarity of two datasets, a generalization of the technique is needed here to apply. Firstly, the two density functions to compare should be discretized by a \(\Delta x_{i}\) value, and the density functions should be evaluated at those points. Then, the Jaccard index of the two resulting datasets can be calculated [28]. Thereby, the goodness values of the expert judgments:  
Figure 5: Schematic figure of resampling. The generated \(s_{n}\in(0,1]\) random numbers are represented by orange arrows and the weights of the sample elements are denoted by \(w_{i}^{j}\) (\(j=1,\ldots,N\)). If an arrow points to the interval of \(w_{i}^{m}\), \(x_{i}^{m}\) is chosen as new sample element. \[g_{i,e}=J(\mathbf{p_{e}},\mathbf{q})=\frac{\sum\limits_{k=1}^{K}\min(p_{e,k},q_{k})} {\sum\limits_{k=1}^{K}\max(p_{e,k},q_{k})} \tag{12}\]  
where \(g_{i,e}\) marks the goodness value belonging to the judgment of the \(e\)th expert about \(x_{i}\), \(\mathbf{p_{e}}\) and \(\mathbf{q}\) represent the two datasets gained from \(p_{e}(x_{i})\) and \(q(x_{i})\), \(K\) refers to the size and \(p_{e,k}\) and \(q_{k}\) denote their elements, respectively. The scheme of the method can be seen in Figure 6.  
Applying the technique to compare the resulting empirical probability densities with the original ones based on experts, expert assessments can be characterized by the goodness values calculated (\(g_{i,e}\)), thus being evaluated. The resulted goodness values are also applicable to give weights to the initial uniform distributions of the experts (\(p_{e}(x_{i})\)) and to generate an outer iteration circle to further refine the estimation. Weights can be created by normalizing the goodness values for each \(x_{i}\):  
\[\tilde{g}_{i,e}=\frac{g_{i,e}}{\sum\limits_{e=1}^{E}g_{i,e}} \tag{13}\]  
where \(g_{i,e}\) refers to the normalized goodness value of the judgment of the \(e\)th expert about \(x_{i}\). Using these weights, new aggregated distributions can be created from the initial ones (similarly to Equation (6) but non-equal weights here) as follows:  
\[p(x_{i})=\sum\limits_{e=1}^{E}\tilde{g}_{i,e}p_{e}(x_{i}) \tag{14}\]  
Thus, feedback is obtained from the estimated distributions, as seen in Figure 7.  
Figure 6: Comparison of two probability distribution by calculating the Jaccard index. The \(p_{e}(x_{i})\) and \(q(x_{i})\) distributions are discretized by \(\Delta x_{i}\) and probability density functions are evaluated at that points. The elements two datasets are compared pairwise: the maximum and minimum values are denoted with red and green color, respectively. Then, Equation (12) is used to calculated Jaccard index, which is the ratio of the areas under the red and green curves, actually. Figure 7: Outline of the outer iteration circle. The initial distributions from experts are merged again in every iteration according to the normalized goodness values (\(\tilde{g}_{i,e}\)) gained by comparing these distributions to the estimated ones resulted in the inner circle. Weighting the initial distributions according to their correctness gives the possibility of eliminating the incorrect expert judgments, thus refining the estimation. In this study, the convergence in the outer circle is also investigated. The proposed method is summarized in Algorithm 1. \(N_{\Theta}\) and \(N_{y}\) refer to the number of uncertain model parameters and outputs.