## CHAPTER 5. EVALUATION OF RESULTS

#### Contents

  ----------- --------------------------------------------------- -------
  Paragraph   Title                                               Page

  5.01        Quality and Timeliness Goals                        5-1

  5.02        Measuring Quality and Timeliness                    5-1

  5.03        Standard Defect Rates                               5-2

  5.04        Random Sampling Error                               5-3

  5.05        Control Charts                                      5-3

  5.06        RLC and Central Office Case Reviews                 5-4
  ----------- --------------------------------------------------- -------

+------------+---------------------------------------------------------+
| ##### 5.0  | a\. From a program perspective, it is important that    |
| 1 Quality  | Loan Guaranty operations be uniform and consistent from |
| and Timeli | office to office over time. Although stations vary      |
| ness Goals | greatly in workload, resources, etc., all stations are  |
|            | administering a single program and, thus, each          |
|            | station\'s performance should be measured against the   |
|            | same set of goals.                                      |
|            |                                                         |
|            | > b\. The national goals for accuracy and compliance    |
|            | > are listed in appendix B, table 26A. The national     |
|            | > goals for timeliness are listed in appendix B, table  |
|            | > 26B.                                                  |
+------------+---------------------------------------------------------+

+------------+---------------------------------------------------------+
| ###        | a\. Accuracy and compliance will be measured under each |
| ## 5.02 Me | quality criteria schedule in terms of the number of     |
| asuring Qu | defects or incorrect actions (NO answers) as a          |
| ality and  | percentage of all reviewed actions. NA answers indicate |
| Timeliness | that, with respect to a particular case, there was no   |
|            | opportunity to take a correct action, reflecting good   |
|            | quality, or to make an error, reflecting poor quality.  |
|            | Thus, NA answers are subtracted from the total number   |
|            | of answers to determine the number of actions on which  |
|            | the quality measurement will be based. The total number |
|            | of actions will be the number of questions on the       |
|            | schedule times the number of cases reviewed, minus the  |
|            | total number of NA answers. Quality for each schedule   |
|            | will be stated as the number of YES answers as a        |
|            | percent of the number of actions. The defect rate for   |
|            | each schedule will be stated as the number of NO        |
|            | answers as a percent of the numbers of actions.         |
+------------+---------------------------------------------------------+

#####  

+------------+---------------------------------------------------------+
| #####      | > b\. Timeliness for each time count will be stated as  |
|            | > the number of actions which were taken within the     |
|            | > number of days considered timely (see appendix B,     |
|            | > table 26B) as a percentage of the total number of     |
|            | > cases reviewed in which timeliness could be measured. |
|            | > Thus, cases in which a given time count did not       |
|            | > apply, that were excluded or otherwise could not be   |
|            | > measured, will not be used to determine if the        |
|            | > standards are being met.                              |
+------------+---------------------------------------------------------+

+------------+---------------------------------------------------------+
| ##         | a\. Although stations must work towards meeting         |
| ### 5.03 S | national goals for accuracy and compliance, the         |
| tandard De | standard defect rate for accuracy and compliance for    |
| fect Rates | each QCS (Quality Criteria Schedule) will be the        |
|            | station\'s average defect rate for that schedule for    |
|            | the preceding fiscal year. This standard defect rate    |
|            | becomes the baseline of the control chart for that      |
|            | schedule. The accuracy and compliance standard defect   |
|            | rate is not a goal to be sought after. It is a baseline |
|            | from which changes in the quality of operations can be  |
|            | observed. Improvements in the overall level of quality  |
|            | in each work process will result in changes in the      |
|            | baseline from year to year. Aberrant patterns on the    |
|            | control charts will alert management to changes in the  |
|            | operation which must be analyzed so that the underlying |
|            | problems may be addressed. Stations should also compare |
|            | their standard defect rates with national defect goals  |
|            | as listed in appendix B, table 26A. Such goals          |
|            | represent a mark towards which station operational      |
|            | efforts should be directed. These are long term goals   |
|            | which will be considered as met when the local standard |
|            | defect rate baseline for a QCS equals or is below the   |
|            | national goal line for that QCS.                        |
+------------+---------------------------------------------------------+
| #####      | b\. Timeliness goals; i.e., the number of days a        |
|            | particular work process should take, are established on |
|            | a nationwide basis as targets to be met. Timeliness     |
|            | goals appear in table 26B in appendix B. The table      |
|            | includes the goals for all time counts in quality       |
|            | criteria schedules as well as the timeliness            |
|            | requirements for certain other work processes. Average  |
|            | timeliness rates (or timeliness standards) expressed as |
|            | a percent of cases that meet the timeliness goals, will |
|            | vary from station to station like the standard defect   |
|            | rates for accuracy and compliance. [NOTE]{.underline}:  |
|            | Timeliness averages vary, timeliness goals do not.      |
|            | Timeliness control charts for each time count should be |
|            | prepared using the average percent of cases completed   |
|            | timely during the preceding fiscal year as the          |
|            | baseline. Stations should also graph the average number |
|            | of days required to complete each work process.         |
+------------+---------------------------------------------------------+

+------------+---------------------------------------------------------+
| ##         | > a\. The probability of random sampling error must be  |
| ### 5.04 R | > considered when applying the standards. Some          |
| andom Samp | > probability of error must be attached to any sample   |
| ling Error | > result, which is in fact only one of a number of      |
|            | > different possible sample results. Even a 100 percent |
|            | > review for any given period, if used to determine the |
|            | > underlying quality level of output from a continuing  |
|            | > work process, is merely a variable sample. The degree |
|            | > of this variability is inversely related to the       |
|            | > sample size and directly related to the percentage of |
|            | > defects in the end product.                           |
|            |                                                         |
|            | b\. The limits within which the random sampling error   |
|            | can be expected to range for any given sample size and  |
|            | underlying standard defect rate will be found in the    |
|            | \"Three Standard Deviation Control Limits for Percent   |
|            | in Error\" tables in Manual M20-2*, Quality Control*,   |
|            | appendix E. A more detailed explanation of the          |
|            | principles and techniques involved in determining       |
|            | control limits may be found in M20-2, chapter 4.        |
+------------+---------------------------------------------------------+

+------------+---------------------------------------------------------+
| #####      | a.  Stations will establish and maintain control charts |
|  5.05 Cont |     > in                                                |
| rol Charts |                                                         |
|            | > > accordance with M20-2, chapter 5, for each QCS.     |
|            | > > Separate control charts will be maintained for      |
|            | > > timeliness counts in appropriate QCSs. Control      |
|            | > > charts will be maintained on an annual basis and be |
|            | > > updated monthly. The baseline will be the average   |
|            | > > defect rate for the previous fiscal year. The upper |
|            | > > and lower control limits will be based on three     |
|            | > > standard deviations of the average monthly volume   |
|            | > > of actions reviewed during the previous fiscal      |
|            | > > year. Control charts should indicate not only the   |
|            | > > current year\'s baseline but also should indicate   |
|            | > > the national goal for each QCS as listed in         |
|            | > > appendix B, table 26A. Stations may wish to overlay |
|            | > > the previous year\'s control chart with the current |
|            | > > year\'s for comparative purposes. In addition to    |
|            | > > maintaining copies of the control charts in the     |
|            | > > office of the chief, they should be posted in their |
|            | > > respective sections for the benefit of employees    |
|            | > > who perform the work processes being tracked.       |
|            | >                                                       |
|            | > > b\. Proper use of the control charts as outlined in |
|            | > > M20-2, paragraph                                    |
|            | >                                                       |
|            | > > 5.04, will reveal whether a particular work process |
|            | > > is \"in control\" or "out of control.\" Although    |
|            | > > the most obvious out of control situation is        |
|            | > > identified on the chart by a plot outside the       |
|            | > > control limits, there are many other plot patterns  |
|            | > > that indicate out of control situations. Every      |
|            | > > effort must be made to isolate the cause(s) of out  |
|            | > > of control situations and to initiate corrective    |
|            | > > actions as appropriate.                             |
+------------+---------------------------------------------------------+
| #####      |                                                         |
+------------+---------------------------------------------------------+

+------------+---------------------------------------------------------+
| #####      | a\. Periodically cases listed on the sampling control   |
| 5.06 RLC a | registers will be called to RLCs and/or Central Office  |
| nd Central | for review and evaluation. These cases will be the same |
|  Office Ca | ones reviewed by the field station. The number of cases |
| se Reviews | to be submitted, the method of their selection, and the |
|            | time of their submission are described in chapter 2 of  |
|            | this manual. Each submitted case which has been         |
|            | reviewed by the field station will be accompanied by a  |
|            | copy of VA Form 26-8448a*, Quality Review Record*,      |
|            | showing the field station findings. The first two pages |
|            | of the quality review record will be submitted in all   |
|            | cases with the third page for schedules with more than  |
|            | nine questions.                                         |
|            |                                                         |
|            | b\. Answers to some questions may require verification  |
|            | of information and/or material which may not be in the  |
|            | case folder. Stations must make reasonable efforts to   |
|            | include this material or otherwise document the answer  |
|            | to each question when evidence is not in the folder on  |
|            | cases submitted for RLC or Central Office review.       |
|            | However, extensive additional documentation of folders  |
|            | for the sole purpose of enabling the RLCs or Central    |
|            | Office to perform quality reviews should not be         |
|            | attempted. Specific instructions will be provided at    |
|            | the time cases are requested. Reviews of eligibility    |
|            | determinations will become part of the stations'        |
|            | Systematic Analysis of Operations, but may also be done |
|            | during field survey visits.                             |
|            |                                                         |
|            | \[ \]                                                   |
+------------+---------------------------------------------------------+