# Presentation: 062467

## EQAInformation ManagementScoring, Analysis, and Report

**Proficiency Testing - **
**From Concept to Reality**
**Prof Harold Richardson M.D., F.R.C.P.C**

- February, 2002

## Data Management and Programming

**Database system**
- Information to be captured
- Database design
- # records
- # and size of the fields
- Linkages
- Coding

**Notes:**
Critical to your information management process is the design of the surveys – what EQA challenges are you going to issue – how do you wish to identify each participant – what results do you wish to receive from them – do you need information on their methods – their reference intervals and so on.  You will need to decide up-front the information you wish to capture and then design a database to accommodate this information as well as facilitate the analysis of the data.
Depending on the size and scope of your program, your database can be paper-based, an electronic flat-field based system or a commercial relational one.  If you decide that a paper-based system is too limiting, cumbersome or simply too much work then look at whether an application such as “Excel” can meet your needs. To go beyond this level likely you will have to use a commercial RDMS and that means you will need a lot of support from Information Services.
In any event, you will need to know the number of participants for each survey, whether one record or more per participant per survey, the number of fields in each record and the size of each field to accommodate present and future data, the linkages you wish to have for analysis purposes, and how you wish to code non-numerical data. 

## Collecting Test Results

**Paper versus Electronic**
- Mail
- Fax
- E-mail
- Browser enabled internet transfer
**Worksheets – paper or electronic**
- Lab ID
- Survey ID
- Analyte or test result
- Method codes

**Notes:**
Traditionally, EQA schemes have sent participants a paper copy of a worksheet and have received survey results in the form of a completed worksheet.  Using the mail for delivery and return is SLOW.  Using courier services is faster but more expensive.  Faster turn-around-times can be achieved using fax transmission and this facility is readily available in most countries.
 Whether mail, courier, fax, or even e-mail, is used data received from each participant must be captured to your database.  This data transfer is prone to error and relatively expensive in terms of people time.  We use two data officers to key in the information independently and then verify as necessary.
Participant direct data entry using browser-enabled web-based transfer is a goal we are working toward.  Presently this sophisticated electronic data transfer is beyond many of our participants and is resource limited.
Let’s turn back to the reality of the worksheet.  At the very least you will need - a unique identifer for each participating laboratory that carries over from survey to survey – a survey identifier that confirms the type e.g. BACT for a bacteriology survey, and the sequence of that survey in a similar series e.g. BACT02/02 – the results of the participant examination – the relevant method codes

## Developing Appropriate Scoring Criteria

**Acceptable/unacceptable performance**
**Significant error/ minor error/ discordant result**
**Assess against reference value, consensus value or peer group mean**
**Assign score or partial score for correct or acceptable result**
**Assign negative score for errors or for inappropriate result or interpretation**

**Notes:**
When the time comes to evaluate the survey results you will need criteria that should reflect published performance standards or benchmarks for the examination or test analyte.  If published performance standards cannot be identified or you judge that they are not applicable to your EQA scheme then you will have to identify the level of performance that you expect.  The criteria you choose should be able to classify the performance of any participant as acceptable or unacceptable.
Should performance fall below the expected you will have to characterize it in some way – fail – discordant – minor error – significant error.  Ideally it provides you with an opportunity to dialogue with the participant with a view to improving the quality of the performance.
You must establish a performance target for each examination or test within a survey.  In some cases you can use an external reference value, in others you may establish an internal reference value, or a consensus or peer group mean as the target value.  
Although it is not recommended that you create a rank order list of all participants, it is useful to have a scoring system that assigns a mark to each component of the survey.

## How to analyse data

**Set due date for results and close survey**
**Capture and verify the data**
**Analyse data by**
- Specific examination, test, or analyte
- All methods
- Method specific groupings [10]

**Notes:**
As part of your survey process you will need to determine and publicize the due date for results and the closure of the survey.  The due date is that day on which results must be received by your office e.g. midnight on February 28.  No allowance beyond that time – it is not fair to other participants if you allow late entry of results and it will delay your total turn-around time.
As I said earlier, all the data received by the due date must be captured and verified before you begin the analysis.  If you receive nonsense data, you will need a process of handling it – are you going to correct it or not?  The process must be standardized so that it is applied fairly and equitably to all participants.  How are you going to handle non-participation or partial participation?
As part of the survey design you will have decided how the data is to be analysed. If the survey is one of a series conducted over time then you will wish to apply the same analysis to each of the surveys since likely you will wish to compare the findings in one survey with each of the others.  Depending on the laboratory discipline or type of challenge in the survey you may wish to look only at the results of the total laboratory examination, or that the results for a test or the values determined for a particular analyte.
If your participants are using different methods e.g. in chemistry then you may have to stratify your analysis based on specific method groupings.  We do this as long as there are more than 10 users in the method group otherwise we compare against an all methods mean.  Rarely do we use an externally determined reference value as the target.

## How to analyse data

**Apply scoring scheme**
**Flag performance for review**
**Review and clarify anomalous or discrepant results**
**Identify unacceptable results**
**Communicate with the laboratory**
**Assign errors**

**Notes:**
Whether your survey is quantitative or qualitative, the basis of your scoring scheme is the intended result and the range of acceptable results.  You should not rank order participants on the basis of your scoring scheme.  Rather the score allows you to scan performance and quickly identifies those participant results that require more detailed review e.g. those in the lowest tenth percentile.  Some countries have different requirements and assign a pass or fail category based on a percentage of the maximum score on the survey.
In the quantitative disciplines, rules that govern performance are generally based on statistical analysis of the results such as ±10%cv or ±2sd of the mean.  Acceptable statistical analysis is required and you must avoid undue influence resulting from rank outliers.
Anomalous or discrepant results should be clarified as far as possible on the basis of evidence you have available.  If you cannot explain them then call them unacceptable and communicate with the participant.  I will deal with this aspect more fully in my later talk.
Be cautious in how you assign errors – it tends to create a very negative response from the participant.

## The Survey Report

**Protect confidentiality at all times**
**Timely, informative, comprehensive**
**Date of report**
**Include identifiers**
- Laboratory name and address
- Responsible person or authority
- EQA/PT scheme
- Survey number
- Examination, test or analytes included

**Notes:**
The survey report is critical to improving performance in the participant laboratory.  The EQA scheme organizer must require all staff and technical/scientific advisors/assessors to consider all information on participant performance as absolutely confidential.
The report should be generated as quickly as possible following the closing date for the survey.  The laboratory should have the possibility of examining its performance while it is still fresh in their minds.
All necessary information and identifiers must be included in the report.  ISO Guide 43 part 1 provides information on what should be included but frequently isn’t, such as the means used to ensure homogeneity of the challenge material, the uncertainty of measurement etc
On this slide I have included those items that are in the “must include” category

## The Survey Report

**Show **
- all results submitted by the laboratory
- All reference/target values
- Summary of all participant findings
- Subgroup by method as appropriate
**Display results graphically**
**Indicate acceptable performance**
**Provide advice on interpretation**
**Include comments**

**Notes:**
The survey report has two primary purposes
Allows the participant to compare their results to the target performance and to their peer group
Triggers a root cause analysis of unexpected results and allows the participant to carry out corrective actions to limit a recurrence
As far as possible the report should be user friendly and should state the range of acceptable performance
When statistical analysis has been carried out the report should state the type of analysis applied and provide advice on the interpretation
Written comment should be provided that places the survey findings in the context of valid scientific/medical literature and indicates a best practice guideline.

## Sending the Report

**Send preliminary reference value report immediately after close of survey**
**Provide definitive report ASAP by:**
- Mail or courier service
- Fax
- Secure access to your Web site
- E-mail notification

**Notes:**
Whenever possible it is helpful to send out a preliminary report immediately after the close of the survey.  At the least this report should provide participants with the expected results so that they can compare these with their own results in real time.  We post our preliminary report to our web page but we used to send it by fax.
You should set targets for the time when the definitive report on the survey is to be released.  Ideally this should happen with the least delay compatible with data capture, analysis and review.  In most cases a 2 month turn-around-time is the least that is possible.
If you have a web-site and post your reports then try to arrange for e-mail notification of your clients

## Archiving Results Data

**Are there legal or regulatory requirements?**
**Do you have a statute of limitations?**
**Is the program voluntary or mandatory?**
**Primary purpose is to track performance with time**

**Notes:**
How do you deal with the long term storage of your scheme’s data?  You do need to do it for look back purposes and to meet future legal challenges that you or your clients may face or to meet local regulatory or legal requirements.
In many countries there may be a statute of limitations.  Where I work it is 7 years and I must keep all information for that length of time.
In a voluntary program you need keep the information to satisfy your client requirements
As a minimum you do have to track performance over time if you are going to provide your clients with support of ongoing quality improvement

## Archiving Results Data

**Beware of heat and humidity**
**Paper or electronic format**
**Accessibility**
- How?
- How frequently?
- Why?

**Notes:**
Paper storage of information is likely to be unsatisfactory.  Heat and humidity affect the integrity of the records.  It is bulky, takes up valuable storage space, and becomes virtually impossible to search effectively.
Electronic data storage has made a huge difference to our ability to retain information and to search the database for a specific record or for a linked set of fields.
As you design your data archiving system there are questions to be considered that all relate to accessibility.  Do you need the data on-line?  If stored off-line then how often will you need to look at the archived data and what are you likely to want to do with it.

## Archiving Results Data

**If hard drive used to store data you must have back-up**
- Floppy disk
- CD or DVD
- Optical disk
**Store as ASCII files or maintain software to read archived files**
**Consider off-site on-line data storage ****—****encrypted/secure**

**Notes:**
Hard drives in desktop computers are vulnerable.  They crash.  They can be fried by an unexpected power surge.  You will need a continually updated  virus protection system.  You can inadvertently delete a file.  You must plan for a backup system.
Your backup can be on floppy disks but make them read only when the disk is full.  CD or DVD format gives you much greater storage capacity on many fewer disks but your computer must be capable of writing to this format.  As a mid-way point you could use ZIP disks that give you much greater storage than a floppy although you will need a ZIP drive at about $60 and each disk at $10. For more extensive data storage other media are available such as tape archiving or optical disc storage.
One of the problems with retrieval of archived data some time in the future is that the software application is no longer active.  We store all of our data as ASCII files
A number of commercial organizations now offer secure and encrypted off-site on-line data storage.  It is a relatively cheap alternative to setting up your own system but only if your are running a large EQA scheme.