## 2 Operator Knowledge Reliability Assessment Method  
### Exam as a Knowledge Level Diagnosis Form  
An approximate model of a system implementing an examination process is illustrated by Figure 4. It is a case where a user is tested in a traditional way. The examiner is responsible for most testing process elements. It should be noted that both selecting question sets and an answer template are the responsibility of the examiner. The user's final grade is impacted by the subjectivity of question selection, grade template, and unintentional errors (mistakes) in grading. The position of certain decision blocks has been modified in order to integrate the model from Figure 4 with the SCADA system user testing system. Please note that elements previously burdening the examiner have been moved to the \"testing application\" block in the presented model (Figure 5) that takes into account the use of a testing application within the examination process. This enables:  
* obtaining a universal general of questions (diagnostic stimulations) that allows to generate a certain number of questions from a pool (selected or drawn--depending on the settings);
* automating and objectifying labor-intensive stages of the examiner's work (comparing answers and threshold grade generation). The issue related to the reliability of an examination process as a process for diagnosing the knowledge and skill level of an ATS operator has been discussed in [50]. Let us consider trainee (here: SCADA operator) knowledge and examination process elements in technical terms. It can be assumed (in simplification) that the knowledge exhibited by the trainee is a system consisting of a set of information (S1) and its processing system (S2) (Figure 6). This system provides responses (R) to questions asked (P). Stimulations (P) are--usually--in the form of problems to be solved, tests to complete, etc. Responses (R)--which are the answers of the examined person--are usually theoretical and mathematical descriptions, calculation results, explanations, etc. Figure 4: Illustration of the examination-testing system and process. Figure 5: Illustration of an examination and testing system and process modified using the application. Please note that the knowledge system functions under the presence of interference (Z) that include, e.g., mental stress of the examined person, unfavorable impact of other examination system, and surrounding elements. Various knowledge structure models can be considered. These include, e.g., a serial or totalizing structure. Let us assume the issue associated with diagnosing knowledge of totalizing and threshold nature as the basis for a further discussion (Figure 7). The totalizing structure consists of a set of thematic stacks (autonomous knowledge fragments), made up of elements whose responses add up. An answer is correct only when the total responses from elements of a stimulated stack exceed an adopted threshold. It is possible to determine numerous thresholds, which enables multi-value classification of responses (answers). Real knowledge structures are considerably more complex and similar to the combined structure model [15]. Stacks in this structure can be of fuzzy nature. Each stimulation may induce several different responses. To obtain complete information on the knowledge level exhibited by a diagnosed object, it is necessary to study all stacks, i.e., apply as many stimulations and receive as many responses as there are stacks. If the area of diagnosed knowledge is large (thus, the number of stacks), it may turn out that the number of required stimulations and graded responses is quite large that studying all stacks is out of the question. In most cases, knowledge diagnosis is based on testing a certain number of stacks--which constitutes, in fact, examining a knowledge sample. Figure 6: Trainee knowledge system model. Figure 7: Knowledge structure models.