## 2 Operator Knowledge Reliability Assessment Method
### Exam as a Knowledge Level Diagnosis Form
Let us adopt the following, simplified assumptions:  
* stack states are independent of each other;
* all stacks are equally important;
* stacks were developed using the same \"technology\" and under the same conditions. With these assumptions, it is easy to observe a certain analogy between generating a batch of selections and developing a trainee knowledge system. It is justified to refer to the methods employed in statistical quality control. This requires adopting a certain grading rule. It is customary to arbitrarily determine certain quantitative knowledge mastery levels (i.e., threshold values): \(\Omega_{0}\), \(\Omega_{1}\),..., \(\Omega_{J}\), while (1):  
\[\Omega_{0}<\Omega_{1}<\ldots<\Omega_{J}, \tag{1}\]  
where: \(J\)--knowledge class set population (usually also indicates the grade set population). These levels divide the knowledge area into quantitative grade classes. Knowledge mastery level implies belonging to a class determined by the exam grade:  
\[\big{(}\Omega_{j-1}\ \ \leq\ \ \Omega\ \ <\ \ \Omega_{j}\big{)}\qquad \Rightarrow\qquad j\ \text{class}\quad(j\ \text{grade});\qquad j\ \ =\ \ 1,2,\,\ldots,J, \tag{2}\]  
where: \(j\)--assessment number, \(\Omega\)--examined person's knowledge level. Different systems (levels and populations) of knowledge threshold values and grade names are encountered in teaching. Figure 8 shows selected examples of threshold values for 2-, 4-, and 7-value knowledge level classifications. Let us assume that each response will be considered a fundamental sample containing \(k\) elements. The state of each element within this sample is assessed bivalently (0 or 1). Therefore, each response is graded as \(0\div k\) points. Total \(Q\) points for all graded responses is the combined point score. Of course, the \(Q\) number determines the knowledge level in terms of the tested stacks. If we award the examined person an \(S_{j}\) grade on this basis, we can ask what is the probability that the entire knowledge mastery level falls within the range: \([\Omega_{j-1},\,\Omega_{j}]\), or what is the probability that a trainee deserves an \(S_{j}\) grade upon receiving \(Q\) points and a maximum (i.e., possible) number of points \(Q_{max}\). Specifically, we ask about the probability of the assessment accuracy, which is the diagnosis reliability (3):  
\[P\big{(}\big{(}\Omega_{j-1}\ \ \leq\ \ \Omega\ \ <\ \ \Omega_{j}\big{)}\ \mid Q,\ Q_{\text{max}}\ \big{)}\ \ =\ \ \ P\big{(}S_{j}\,\mid Q,\ Q_{\text{max}}\,\big{)}, \tag{3}\]  
The probability of an examined person deserving an \(S_{j}\) grade can be determined using the expressions (4) and (5):  
* for a continuous knowledge distribution:  
Figure 8: Knowledge level classification examples, where: 2—two-value grade set, 4—four-value grade set, 7—seven-value grade set. \[P\big{(}S_{j}\,|\,Q,\,Q_{\max}\big{)} = \frac{\int\limits_{\Omega_{j-1}}^{\Omega_{j}}f(\Omega)\ (1-\Omega)^{(Q_{\max}-Q)}\ \Omega^{Q}\ d\Omega}{\int\limits_{0}^{1}f( \Omega)\ (1-\Omega)^{(Q_{\max}-Q)}\ \Omega^{Q}\ d\Omega}, \tag{4}\]
* for a discrete knowledge distribution: Figure 4:0 (5)  
It should be emphasized that the expressions (4) and (5) are analogues of the expressions employed within statistical quality control for testing the defectiveness of the total population, based on determined sample defectiveness. Based on the expression (4) or (5), the examined person is awarded the \(S_{j}\) grade, which is the most probable, i.e.,:  
Figure 4:1  
As can be easily observed, grade reliability or the examination result credibility depends on:  
* maximum number of grade points;
* grading principle, i.e., above all, on the system of trainee knowledge grading interval boundaries. Graphs in Figure 9 are an illustration of sample results of applying the aforementioned relationships. These graphs show grade credibility functions for a case of 7-value knowledge level classification and 5 examination questions that enable obtaining a maximum number of grade points of \(Q_{\max}=25\), upon an adopted equal knowledge distribution \(f(\Omega)\).