Abstract:
A test results screening tool allows capture of test item and test results data and provides a reviewer with a robust set of tools for entering screening criteria and display of test results by both test subject and test center. The tool includes parametrically sets flags for unwanted testing behaviors including test item harvesting, test subjects with pre-knowledge of the test subject matter, and proxy test takers for some a portion of a test.

Description:
RELATED APPLICATIONS 
       [0001]    This patent claims priority to U.S. Provisional Application 61/602,882, filed Feb. 24, 2012, which is hereby incorporated by reference for all purposes. This application is also related to U.S. patent application titled TEST PALLET ASSEMBLY AND FAMILY ASSIGNMENT, Attorney Docket No.: 31844/46708 owned by a common assignee of this application and filed on the same day as this application, which is also incorporated by reference herein for all purposes. 
     
    
     TECHNICAL FIELD 
       [0002]    This patent is directed to a tool for identifying credentialing test score outliers and anomalous candidate behavior. More particularly, the tool is directed to identification of test subjects that appear to be cheating or taking the test for the purpose of promoting cheating. 
       BACKGROUND 
       [0003]    Testing to determine capability in a particular field has long been of interest. Testing is used in virtually all areas of academics, enterprise and government to evaluate candidates seeking certification in certain professional capacities, including real estate, law, medicine, etc. As in any testing environment, attempts to achieve an unfair advantage, or to profit from capturing information about such a test, are a reality. 
         [0004]    Examples of attempts to achieve an unfair advantage include pre-knowledge, that is, gaining access to test items (e.g., questions) before the beginning of a test, proxy testing, a situation where a person represents himself or herself as being another person during the test, and item harvesting, the intent to capture test content for the purpose of distribution to later test takers. 
         [0005]    Test developers or psychometricians routinely analyze both operational items (questions that contribute towards a candidate&#39;s score) and pretest items (field questions that do not contribute towards a candidate&#39;s score). A primary purpose of analyzing operational items is to verify the key, or correct answer, to questions prior to computing and reporting scores. Pretest items are similarly analyzed in order to assess item quality for possible inclusion on future exams. In addition to these routine analyses, operational and pretest response data can also be analyzed to assess possible threats to the security and therefore, integrity, of the exam. This line of analysis is commonly referred to as “outlier detection,” and is a subarea of a larger topic, “data forensics.” 
       SUMMARY 
       [0006]    A unified outlier detection tool allows entry of test scores from a number of test instances, for example, scores from tests administered at different facilities, and further allows a reviewer to examine the test data for a number of suspicious pre-defined indications and flag corresponding instances for further follow up. The tool also allows identification of unexpected test item responses with respect to both operational and pretest items. 
         [0007]    The tool allows automatic and systematic identification of anomalous records for closer examination, and to subsequently identify instances of potential test item harvesting, pre-knowledge, and proxy test taking. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]      FIG. 1  illustrates a representative testing cycle; 
           [0009]      FIG. 2  is a block diagram of an exemplary test delivery system; 
           [0010]      FIG. 3  is a block diagram of a second exemplary test delivery system; 
           [0011]      FIG. 4  is an exemplary test data input area; 
           [0012]      FIG. 5  is a graph showing exemplary test result distribution; 
           [0013]      FIG. 6  is a graph showing an exemplary scatter chart of two test factors; 
           [0014]      FIG. 7  is a graph showing exemplary distribution of anomalous item response times; 
           [0015]      FIG. 8  illustrates an exemplary initial results screening area; 
           [0016]      FIG. 9  illustrates an exemplary test record summary area; 
           [0017]      FIG. 10  illustrates an exemplary test record detail area showing results for harvest and pre-knowledge screening; 
           [0018]      FIG. 11  illustrates an exemplary test record detail area showing results for proxy screening; 
           [0019]      FIG. 12  illustrates an exemplary summary results area showing a compilation of screening results; 
           [0020]      FIG. 13  illustrates an exemplary form showing test item descriptive data; 
           [0021]      FIG. 14  illustrates an exemplary summary results form by test center; 
           [0022]      FIG. 15  is an exemplary method of using an outlier detection tool; and 
           [0023]      FIG. 16  is an exemplary computer system suitable for hosting an outlier detection tool. 
       
    
    
     DETAILED DESCRIPTION 
       [0024]    Although the following text sets forth a detailed description of numerous different embodiments, it should be understood that the legal scope of the description is defined by the words of the claims set forth at the end of this disclosure. The detailed description is to be construed as exemplary only and does not describe every possible embodiment since describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims. 
         [0025]    It should also be understood that, unless a term is expressly defined in this patent using the sentence “As used herein, the term ‘______’ is hereby defined to mean . . . ” or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope based on any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this patent is referred to in this patent in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term by limited, by implication or otherwise, to that single meaning. Finally, unless a claim element is defined by reciting the word “means” and a function without the recital of any structure, it is not intended that the scope of any claim element be interpreted based on the application of 35 U.S.C. §112, sixth paragraph. 
         [0026]      FIG. 1  illustrates a representative testing cycle  10 . The testing cycle  10  involves development of test items  12 , that is, individual questions or other items requiring a response from a test subject. Using the test items  12 , a test may be designed  14  to meet certain requirements of the test client. As discussed above, some tests may be targeted at ascertaining a test subject&#39;s level against a certain metric, such as grade level in reading. Other tests may be targeted at determining a test subject&#39;s latent ability at a given pass/fail test cut point. Tests may be generated  14  according to the goals of the test and the presentation of test items. Some tests may be set in a predetermined order, such as a paper test, or test items may be presented to a test subject at an order determined during the administration of the test, as in an adaptive or dynamic test. After the test is administered  18 , a scoring and reporting process  20  is performed to notify the test client of the scores, to rate scored and pre-test test items, to screen for abnormal test results, etc. The exemplary embodiments described here are, for the sake of simplicity but without limitation, directed to pass/fail testing, such as a board test for professional licensing. 
         [0027]    The remainder of the discussion in this patent generally corresponds to activities associated with block  20  of  FIG. 1 .  FIG. 2  is a block diagram of an exemplary test delivery system  200 . In an exemplary embodiment, a collection of individual test forms, called a pallet  204 , may be delivered via a network  206 , such as the Internet, to a testing center  202 . Each test form is composed of a number of test items. A test item is an individual item requiring a response, such as, but not limited to, a question. Test items may include text, graphics, images, videos, etc. At the testing center  202 , individual test forms may be used by test subjects at testing workstations  210 ,  212 . Alternatively, as is known in the art, a set of test items larger than the number of test items delivered to an individual test subject (not depicted) may be delivered to the testing center  202  and an individual test may be administered in an adaptive manner by presenting succeeding items based on previous responses by the test subject. The form of test delivery is not relevant to the function of the outlier identification tool described herein. 
         [0028]      FIG. 3  is a block diagram of a second exemplary test delivery system  220  illustrating that a pallet of tests  222 , made up of individual test forms or a set of test items may be delivered via a network  224  to a number of test centers  226   a - 226   n.  As discussed further below, the outlier detection tool supports analysis of test results for the same or similar tests delivered at multiple facilities. 
         [0029]    The outlier test tool may be embodied as a program or program modules stored on a computer memory and executed on a computer. In an exemplary embodiment, the outlier test tool may be implemented using a computer-based spreadsheet program, such as, but without limitation to, Microsoft® Excel®. 
         [0030]    For the sake of illustration of an embodiment of the outlier detection tool, representative forms are used to show data input and results tabulations. 
         [0031]      FIG. 4  illustrates an exemplary test results input area and shows a portion of a test data input area  230 . In the exemplary embodiment the input data fields include statistical results information by test subject (entry). Additional information including response by test item may be captured in this or a similar form (not depicted). In an embodiment, raw test results data may be pre-processed to develop Rasch measures using, for example, a tool such as Winsteps®. In the Rasch model, the probability of a specified response (e.g. right/wrong answer) is modeled as a function of person and item parameters. Specifically, in the simple Rasch model, the probability of a correct response is modeled as a logistic function of the difference between the person and item parameter. In most contexts, the parameters of the model pertain to the level of a quantitative trait possessed by a person or item. For example, test item parameters pertain to the difficulty of test items while person parameters pertain to the ability or attainment level of people who are assessed. The higher a person&#39;s ability relative to the difficulty of an item, the higher the probability of a correct response on that item. 
         [0032]    In preparation for selection of screening criteria, discussed below, a representative embodiment may present charts of various data so that the outlier detection tool operator can make an informed decision about where to set cut off points for various available criteria. These representative charts may be generated using standard techniques, such as programming macros, available in the base spreadsheet or similar program.  FIG. 5  illustrates combined Operational (OP) test item theta  240 . Theta in this context represents a test subject&#39;s latent ability corresponding to a center value of test item ratings. OP Theta is the test subject&#39;s performance on test items designated as operational, or scored test items. Other test items may be designated as Pre-test (PT). Pre-test items are those test items presented for the first time to test subjects and are not used in the computation of subjects&#39; operational scores. The pre-test items are evaluated based on responses received as to their value in use as operational items in future tests. The graph of  FIG. 5  may be used to set an OP Theta value for flagging as being atypical and worthy of further inspection. 
         [0033]      FIG. 6  is a graph showing an exemplary scatter chart  242  of OP Theta vs. test subject average response time (so called Rapid Response). The scatter chart  242  highlights candidate scores (latent ability) compared to the number of extremely rapid item responses produced by each. In an embodiment, hovering over a particular item may produce a pop up  244  showing the values for the individual graph point. The x-y coordinates may be used in a look-up function to identify the test subject identifier corresponding to that data point. 
         [0034]      FIG. 7  is an exemplary graph showing an exemplary distribution of extremely rapid response times. The information may be used as described below with respect to Table 1 to select screening criteria values to set up automatic outlier flagging. 
         [0035]    Table 1 below illustrates representative screening criteria, as found for example, on an exemplary screening criteria entry area of the outlier detection tool. The individual criteria may vary according to test type and preference. The values shown are for example only. An element of the outlier detection tool is the ability to vary the values based on an initial analysis of the results in summary graphs illustrated in  FIGS. 5-7 . Each column of values, that is “Value,” “Percent Rank,” and “Percentile” may each be designated independently, allowing complex selection of screening criteria based not only on an individual test result, but also the test result in comparison with the other test results in the result set. 
         [0000]    
       
         
               
               
               
               
             
               
               
               
               
             
           
               
                 TABLE 1 
               
               
                   
               
               
                 Outlier Flagging Criteria 
                 Value 
                 % Rnk 
                 % ile 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 Total Raw Scores 
                   
                   
                   
               
               
                 A. The top X % of TOTAL SCORES will be considered HIGH. X = 
                 1.00% 
                 99.00% 
                 211.44 
               
               
                 B. The bottom X % of TOTAL SCORES will be considered LOW. 
                 1.00% 
                 1.00% 
                 66 
               
               
                 X = 
               
               
                 Person Rasch Measures 
               
               
                 A. The top X % of OPERATIONAL RASCH MEASURES will be 
                 55.00% 
                 45.00% 
                 −0.3766 
               
               
                 considered HIGH. X = 
               
               
                 B. The bottom X % of OPERATIONAL RASCH MEASURES will be 
                 45.00% 
                 45.00% 
                 −0.3766 
               
               
                 considered LOW. X = 
               
               
                 A. The top X % of PRETEST RASCH MEASURES will be 
                 45.00% 
                 55.00% 
                 −0.1203 
               
               
                 considered HIGH. X = 
               
               
                 B. The bottom X % of PRETEST RASCH MEASURES will be 
                 55.00% 
                 55.00% 
                 −0.1203 
               
               
                 considered LOW. X = 
               
               
                 Misfitting Responses 
               
               
                 A. The top X % of INFIT VALUES will be considered outliers. X = 
                 0.50% 
                 99.50% 
                 1.200 
               
               
                 A. The top X % of OUTFIT VALUES will be considered outliers. X = 
                 1.00% 
                 99.00% 
                 1.374 
               
               
                 Unexpected Responses 
               
               
                 A. Item RESPONSES that have ≦ X % probability of occurring given 
                 15.00% 
               
               
                 the item difficulty and the person&#39;s ability will be considered 
               
               
                 HIGHLY UNEXPECTED. X = 
               
               
                 A. STANDARDIZED RESIDUALS between the observed and 
                 1.95 
               
               
                 expected item scores given item difficulty and person ability will be 
               
               
                 considered MISFITTING if their absolute values are greater than X. 
               
               
                 X = 
               
               
                 B. The top X % of HIGHLY UNEXPECTED RESPONSE COUNTS 
                 1.00% 
                 99.00% 
                 33.000 
               
               
                 will be considered outliers. X = 
               
               
                 Randomness in Responding 
               
               
                 A. A difference of at least X % between the % of MODEL 
                 10.00% 
               
               
                 MATCHING RESPONSES EXPECTED and the % of MODEL 
               
               
                 MATCHING RESPONSES OBSERVED will be considered 
               
               
                 HIGHLY RANDOM. X = 
               
               
                 Total Test Time 
               
               
                 A. The top X % of TOTAL TESTING TIMES (min) will be 
                 0.75% 
                 99.25% 
                 342.5 
               
               
                 considered LONG. X = 
               
               
                 B. The bottom X % of TOTAL TESTING TIMES (min) will be 
                 0.65% 
                 0.65% 
                 115.4 
               
               
                 considered SHORT. X = 
               
               
                 Item Response Times 
               
               
                 A. An ITEM RESPONSE TIME of X seconds or more will be 
                 500 
               
               
                 considered SLOW. X = 
               
               
                 A. An ITEM RESPONSE TIME of X seconds or less will be 
                 10 
               
               
                 considered RAPID. X = 
               
               
                 B. The top X % of SLOW RESPONSE TIME COUNTS will be 
                 0.10% 
                 99.90% 
                 4.0 
               
               
                 considered outliers. X = 
               
               
                 B. The top X % of RAPID RESPONSE TIME COUNTS will be 
                 1.10% 
                 98.90% 
                 39.6 
               
               
                 considered outliers. X = 
               
               
                 Omitted Responses 
               
               
                 A. The top X % of OMITTED RESPONSE COUNTS will be 
                 0.80% 
                 99.20% 
                 7.7 
               
               
                 considered outliers. X = 
               
               
                 Pre-Knowledge Flagging Criteria 
               
               
                 A. The top X % of PERCENTAGEs OF OP ITEMS WITH 
                 0.00% 
                 100.00% 
                 12.5% 
               
               
                 MISFITTING CORRECT ANSWERS will be considered outliers. 
               
               
                 X = 
               
               
                 A. The top X % of DELTAS between % of OP MISFITTING 
               
               
                 CORRECT ANSWERS and % of PT MISFITTING CORRECT 
               
               
                 ANSWERS will be considered outliers. X = 
               
               
                 A. The bottom X % of PERCENTAGEs OF OP ITEMS WITH 
               
               
                 MISFITTING INCORRECT ANSWERS will be considered outliers. 
               
               
                 X = 
               
               
                 A. The bottom X % of DELTAS between % of OP MISFITTING 
               
               
                 INCORRECT ANSWERS and % of PT MISFITTING INCORRECT 
               
               
                 ANSWERS will be considered outliers. X = 
               
               
                 23. The top X % of PERCENTAGEs OF OP ITEMS with RAPID 
                 0.90% 
                 99.10% 
                 9.0% 
               
               
                 CORRECT ANSWERS will be considered outliers. X = 
               
               
                 24. The top X % of DELTAs between % of OP ITEMS with RAPID 
                 0.20% 
                 99.80% 
                 26.9% 
               
               
                 CORRECT ANSWERS and the % of PT ITEMS with RAPID 
               
               
                 CORRRECT ANSWERS will be considered outliers. X = 
               
               
                 25. The top X % of PERCENTAGEs of OP ITEMS witih 
               
               
                 MISFITTING, RAPID, CORRECT ANSWERS will be considered 
               
               
                 outliers. X = 
               
               
                 26. The top X % of DELTAs between % of OP ITEMS with 
               
               
                 MISFITTING, RAPID, CORRECT ANSWERS and % of PT ITEMS 
               
               
                 with MISFITTING, RAPID, CORRECT ANSWERS will be 
               
               
                 considered outliers. X = 
               
               
                 27. The bottom X % of DIFFERENTIAL PERSON FUNCTIONING 
                 3.50% 
                 3.50% 
                 −2.58 
               
               
                 T-TEST VALUES with the focus on PT items (persons measure 
               
               
                 lower on pretest items than on operational items) will be considered 
               
               
                 outliers. X = 
               
               
                 28. The top X % of ADJUSTED T-TESTS BETWEEN OP Theta and 
                 3.20% 
                 96.80% 
                 2.59 
               
               
                 PT Theta will be considered outliers. X = 
               
               
                 Proxy Flagging Criteria 
               
               
                 29. The top X % of AM OPERATIONAL THETA VALUES will be 
                 46.00% 
                 54.00% 
                 −0.27908 
               
               
                 considered HIGH. X = 
               
               
                 30. The bottom X % of AM OPERATIONAL THETA VALUES will 
                 54.00% 
                 54.00% 
                 −0.27908 
               
               
                 be considered LOW. X = 
               
               
                 31. The top X % of PM OPERATIONAL THETA VALUES will be 
                 55.00% 
                 45.00% 
                 −0.2825 
               
               
                 considered HIGH. X = 
               
               
                 32. The bottom X % of PM OPERATIONAL THETA VALUES will 
                 45.00% 
                 45.00% 
                 −0.2825 
               
               
                 be considered LOW. X = 
               
               
                 33. The top X % of DIFFERENTIAL PERSON FUNCTIONING T- 
                 0.30% 
                 99.70% 
                 3.456096 
               
               
                 TEST VALUES with the focus on PM items (persons measure higher 
               
               
                 on PM items than on AM items) will be considered outliers. X = 
               
               
                 34. The top X % of DIFFERENTIAL PERSON FUNCTIONING T- 
                 0.10% 
                 99.90% 
                 2.905968 
               
               
                 TEST VALUES with the focus on AM items (persons measure higher 
               
               
                 on AM items than on PM items) will be considered outliers. X = 
               
               
                 35. The top X % of ADJUSTED T-TESTS BETWEEN AM OP Theta 
                 0.30% 
                 99.70% 
                 3.04615 
               
               
                 and PM OP Theta will be considered outliers. X = 
               
               
                 35. The bottom X % of ADJUSTED T-TESTS BETWEEN AM OP 
                 3.00% 
                 3.00% 
                 −3 
               
               
                 Theta and PM OP Theta will be considered outliers. X = 
               
               
                   
               
             
          
         
       
     
         [0036]      FIG. 8  illustrates an exemplary initial results screening area  260 . Statistical data corresponding to each test subject may be displayed in rows as illustrated. Items relating to flagged results corresponding to entries such as those designated in Table 1, above, may be automatically displayed using conditional results from a comparison of screening criteria and actual results. For example, randomness in results  262  is flagged, short overall test length  264 , and others as depicted. A reviewer can then note these test subjects, for example, by ID as successive results are considered. 
         [0037]      FIG. 9  illustrates an exemplary test record summary area  270 , showing test items  272  vertically and by test subject  274  horizontally. In this exemplary embodiment, test items are ordered by difficulty (Rasch score) and test subjects are ranked by OP Theta. Records will also be flagged  276  if there are a greater proportion of rapid correct operational test responses than rapid correct pretest responses, etc. In an embodiment, a statistical analysis measure, such as a Fisher Exact Test utility, may be used to allow the user to more closely examine patterns in the relationships between two variables, such as rapid responses and correct responses. The use of such a tool will give a probability value to any observed unbalanced relationships between variables (e. g., the probability that the observed relationship between rapid correct and incorrect answers or other relationship, would happen by chance). 
         [0038]      FIG. 10  illustrates an exemplary test record detail area  280  showing results for harvest and pre-knowledge screening. Tags (Tag  1  and Tag  2 ) columns are driven by comparisons of screening criteria and test results and outliers are automatically flagged  284 . Harvest flags are driven by test times, particularly on individual questions that are not commensurate with the rest of the group. The presumption is that a person attending only to appropriate the test items and response alternatives will not be concerned with deliberating over a correct response but will simply move through the material without regard for the response. Referring back to  FIG. 9 , Case Number 798, Person ID 133, selected response C a disproportionate 110 times. Returning to  FIG. 10 , Person ID 133 is also flagged as having statistically significant better responses on OP test items that have appeared on other tests than PT test items making their first appearance, leading to suspicion of having pre-knowledge. Reviewer comments  282  may be entered and transferred to a summary page, discussed below. 
         [0039]      FIG. 11  illustrates an exemplary test record detail area  290  showing results for proxy screening. Proxy screening characterizes test results between morning (AM) and afternoon (PM) sessions of a test, when two different people may attend the separate sessions. Proxy screening compares AM and PM Theta scores and flags those test subjects whose score differences exceed the proxy flagging criteria illustrated in Table 1. 
         [0040]      FIG. 12  illustrates an exemplary summary results area  320  showing a compilation of screening results. The summary results area  320  shows a compilation of test subjects for whom flags  322  have been raised, previously entered comments, and a reviewer&#39;s final status determination  324 . 
         [0041]      FIG. 13  illustrates an exemplary form showing test item descriptive data  340 . This information may be input along with test results data prior to screening with the outlier detection tool. In an embodiment, the test item descriptive data  340  may be generated using the Winstep® tool. 
         [0042]      FIG. 14  illustrates an exemplary summary results form by test center  350 . Here each test center participating in a particular test run is shown with a summary of suspected anomalous behavior as well as an overall pass rate. This data allows a reviewer to observer patterns by test center to see if a particular site may be the target of an attack or may have another issue associated with overall testing integrity. 
         [0043]      FIG. 15  is a flow chart of an exemplary method  400  of using of an outlier detection tool. Initially, use of the outlier detection tool may begin by capturing pre-processed test results. As shown with respect to  FIGS. 4 and 17  and as discussed above, various items related to a particular test may be entered, including test results by test item (block  402 ) and test subject as well as test item information (block  404 ). Other identifying information, such as test dates and locations may also be included. 
         [0044]    A display of various results in graphical form (block  406 ) may be presented for use by a reviewer in selecting screening criteria values. Screening criteria values may be entered (block  408 ) and used in calculation of various item flags as discussed below. 
         [0045]    A first results area may be displayed (block  410 ) showing initial tabular results with items meeting screening criteria values flagged for review. A second results area may be displayed (block  412 ) showing test results with items flagged meeting pre-knowledge screening criteria and test harvest screening criteria. A third results area may be displayed (block  414 ) showing test results with items flagged meeting test proxy screening criteria. 
         [0046]    A summary results area may be displayed (block  416 ) showing a summary of test results including previously flagged items meeting screening criteria and corresponding reviewer comments made in the first and second results areas. 
         [0047]    Corresponding to specific test item metrics, scatter charts for unexpected correct and unexpected incorrect responses to test items may be displayed (block  418 ) for use in analyzing test item suitability. 
         [0048]    With reference to  FIG. 16 , an exemplary system for implementing the claimed method and apparatus includes a general purpose computing device in the form of a computer  110 . Components shown in dashed outline are not technically part of the computer  110 , but are used to illustrate the exemplary embodiment of  FIG. 16 . Components of computer  110  may include, but are not limited to, a processor  120 , a system memory  130 , a memory/graphics interface  121  and an I/O interface  122 . The system memory  130  and a graphics processor  190  may be coupled to the memory/graphics interface  121 . A monitor  191  or other graphic output device may be coupled to the graphics processor  190 . 
         [0049]    A series of system busses may couple various system components including a high speed system bus  123  between the processor  120 , the memory/graphics interface  121  and the I/O interface  122 , a front-side bus  124  between the memory/graphics interface  121  and the system memory  130 , and an advanced graphics processing (AGP) bus  125  between the memory/graphics interface  121  and the graphics processor  190 . The system bus  123  may be any of several types of bus structures including, by way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus and Enhanced ISA (EISA) bus. As system architectures evolve, other bus architectures and chip sets may be used but often generally follow this pattern. For example, companies such as Intel and AMD support the Intel Hub Architecture (IHA) and the Hypertransport™ architecture, respectively. 
         [0050]    The computer  110  typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer  110  and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer executable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, data stores, or other physical storage elements that physically embody electronic data and excludes any propagated media such as radio waves or modulated carrier signals. 
         [0051]    The system memory  130  includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)  131  and random access memory (RAM)  132 . The system ROM  131  may contain permanent system data  143 , such as computer-specific data that may be used as a seed for generating random numbers or nonces, for example, for use in item selection and statistical calculations. RAM  132  typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processor  120 . By way of example, and not limitation,  FIG. 16  illustrates operating system  134 , application programs  135 , other program modules  136 , and program data  137 . 
         [0052]    The I/O interface  122  may couple the system bus  123  with a number of other busses  126 ,  127  and  128  that couple a variety of internal and external devices to the computer  110 . A serial peripheral interface (SPI) bus  126  may connect to a basic input/output system (BIOS) memory  133  containing the basic routines that help to transfer information between elements within computer  110 , such as during start-up. 
         [0053]    A super input/output chip  160  may be used to connect to a number of ‘legacy’ peripherals, such as floppy disk  152 , keyboard/mouse  162 , and printer  196 , as examples. The super I/O chip  160  may be connected to the I/O interface  122  with a bus  127 , such as a low pin count (LPC) bus, in some embodiments. Various embodiments of the super I/O chip  160  are widely available in the commercial marketplace. In one embodiment, bus  128  may be a Peripheral Component Interconnect (PCI) bus. 
         [0054]    The computer  110  may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,  FIG. 16  illustrates a hard disk drive  140  that reads from or writes to non-removable, nonvolatile magnetic media. The hard disk drive  140  may be a conventional hard disk drive. 
         [0055]    Removable media, such as a universal serial bus (USB) memory  153 , firewire (IEEE 1394), or CD/DVD drive  156  may be connected to the PCI bus  128  directly or through an interface  150 . Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. 
         [0056]    The drives and their associated computer storage media discussed above and illustrated in  FIG. 16 , provide storage of computer readable instructions, data structures, program modules and other data for the computer  110 . In  FIG. 16 , for example, hard disk drive  140  is illustrated as storing operating system  144 , application programs  145 , other program modules  146 , and program data  147 . Note that these components can either be the same as or different from operating system  134 , application programs  135 , other program modules  136 , and program data  137 . Operating system  144 , application programs  145 , other program modules  146 , and program data  147  are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer  110  through input devices such as a mouse/keyboard  162  or other input device combination. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processor  120  through one of the I/O interface busses, such as the SPI  126 , the LPC  127 , or the PCI  128 , but other busses may be used. In some embodiments, other devices may be coupled to parallel ports, infrared interfaces, game ports, and the like (not depicted), via the super I/O chip  160 . 
         [0057]    The computer  110  may operate in a networked environment using logical communication ports to one or more remote computers, such as a remote computer  180  via a network interface controller (NIC)  170 . The remote computer  180  may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer  110 . The logical connection between the NIC  170  and the remote computer  180  depicted in  FIG. 16  may include a local area network (LAN), a wide area network (WAN), or both, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. 
         [0058]    In summary, the outlier detection tool benefits test reviewers with a convenient and comprehensive tool for analysis and flagging test result outliers indicating potential test conditions such as test subject pre-knowledge, proxy test takers, and test item harvesting. 
         [0059]    1. In a first aspect, a computer  110  having computer executable instructions stored in a computer storage media ( 130 ,  140 ) that when executed on a processor ( 120 ) of the computer implement a tool for use in analysis of test results comprising: 
         [0060]    a screening module implementing a screening criteria entry area for entering a plurality of screening criteria indicating results to be flagged as being anomalous as meeting the screening criteria; 
         [0061]    an input area module implementing an input area for entry of test results; a calculation engine module implementing a calculation engine that develops statistical analyses of the test results; 
         [0062]    an initial results screening module implementing an initial results screening area showing the statistical analysis by test subject, including display of test results meeting the screening criteria for behavior inconsistent with legitimate test taking; 
         [0063]    a test record summary module implementing a test record summary area displaying comprehensive test subject results by test item, including flagged test results data meeting the screening criteria; 
         [0064]    a test record detail module implementing a test record detail area comprising an indication of test subjects meeting a test pre-knowledge screening criteria; and 
         [0065]    a proxy screening module implementing a proxy screening area comprising an indication of test subjects meeting a test proxy screening criteria. 
         [0066]    2. The tool of aspect 1, wherein behavior inconsistent with legitimate test taking comprises one of an interrupted test session and evidence of item harvesting. 
         [0067]    3. The tool of aspects 1-2, further comprising a summary area module implementing a summary area compiling all test subjects having any statistical result meeting any screening criteria. 
         [0068]    4. The tool of aspects 1-3, further comprising a test center results module implementing a test center results area displaying a comparison of composite test subject results by test center. 
         [0069]    5. The tool of aspects 1-4, wherein the input area module accepts test result values comprising a test identifier, a test subject identifier, a test facility identifier, a raw response, a correct/incorrect response value, and a response time. 
         [0070]    6. The tool of aspects 1-5, wherein the screening criteria entry module plots a composite result for at least one screening criteria as a visual indicator of possible screening criteria set points. 
         [0071]    7. The tool of aspects 1-6, wherein the screening criteria entry module includes a test subject cross reference module that accepts plot points and returns a test subject identifier. 
         [0072]    8. The tool of aspects 1-7, wherein the screening criteria entry module includes a flagging criteria entry module that designates alert levels for test metrics comprising: raw score values, total test time, and item response time. 
         [0073]    9. The tool of aspects 1-8, wherein the screening criteria entry module includes a value designation area that accepts screening criteria values for test metrics comprising: operational test item Rasch measures, pre-test item Rasch measures, misfitting responses, unexpected responses, response randomness, pre-knowledge statistical analysis and proxy statistical analysis. 
         [0074]    10. The tool of aspects 1-9, wherein the test record summary module orders test items by most difficult to least difficult on a first axis and orders test subjects by test score on a second axis. 
         [0075]    11. The tool of aspects 1-10, wherein the initial results screening module includes a first tool for calculating statistical tests of outlier relationships between two variables. 
         [0076]    12. The tool of aspects 1-11, wherein the first tool is a Fisher Exact Test. 
         [0077]    13. The tool of aspects 1-12, wherein the screening criteria entry module includes a consecutive kurtosis test for calculating a statistical significance of one or more specific outlier values of a given variable. 
         [0078]    14. In a second aspect, a method of analyzing test results using a test analysis tool implemented on a computer ( 110 ), the method comprising: 
         [0079]    providing the computer having a processor ( 120 ), a user interface comprising an input capability ( 162 ) and a display ( 191 ) both coupled to the processor, a memory ( 130 ,  140 ) coupled to the processor that stores computer executable instructions that when executed on the processor ( 120 ) cause the processor ( 120 ) to execute instructions; 
         [0080]    receiving, via the input capability using instructions implemented on the processor, a plurality of screening criteria indicating results to be flagged as being anomalous as meeting the screening criteria; 
         [0081]    storing test results in the memory ( 130 ,  140 ); 
         [0082]    calculating, via instructions implemented on the processor ( 120 ), a statistical analyses of the test results via a calculation engine of the computer; 
         [0083]    applying the one or more of the plurality of screening criteria to the statistical analyses of the test results; 
         [0084]    displaying, on the display ( 191 ) of the computer ( 110 ), an initial results screening area showing the statistical analysis by test subject, including displaying test results meeting the screening criteria for behavior inconsistent with legitimate test taking; 
         [0085]    displaying, on the display ( 191 ) of the computer ( 110 ), comprehensive test subject results by test item, including displaying flagged test results data meeting the screening criteria via a test summary area; 
         [0086]    displaying, on the display ( 191 ) of the computer ( 110 ), an indication of test subjects meeting a test pre-knowledge screening criteria via a test record detail area; 
         [0087]    displaying, on the display ( 191 ) of the computer ( 110 ), an indication of test subjects meeting a test proxy screening criteria via a proxy screening area; 
         [0088]    displaying, on the display ( 191 ) of the computer ( 110 ), a summary of all test subjects having any statistical result meeting any screening criteria; and 
         [0089]    receiving, via the user interface ( 162 ) of the computer ( 110 ), supplemental data or commands responsive to display of data meeting one or more screening criteria. 
         [0090]    15. The method of aspect 14, wherein displaying of test results meeting the screening criteria for behavior inconsistent with legitimate test taking comprises displaying data meeting a criteria for an interrupted test session. 
         [0091]    16. The method of aspect 14-15, wherein displaying of test results meeting the screening criteria for behavior inconsistent with legitimate test taking comprises displaying data meeting a criteria for item harvesting. 
         [0092]    17. The method of aspect 14-16, wherein entering a plurality of screening criteria comprises entering one or more of operational test item Rasch measures, pre-test item Rasch measures, misfitting responses, unexpected responses, response randomness, pre-knowledge statistical analysis and proxy statistical analysis. 
         [0093]    18. The method of aspect 14-17, wherein developing a statistical analysis of the test results comprises using a Fisher Exact Test to calculate statistical tests of outlier relationships between two variables for a particular case. 
         [0094]    Although the foregoing text sets forth a detailed description of numerous different embodiments of the invention, it should be understood that the scope of the invention is defined by the words of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only and does not describe every possibly embodiment of the invention because describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims defining the invention. 
         [0095]    Thus, many modifications and variations may be made in the techniques and structures described and illustrated herein without departing from the spirit and scope of the present invention. Accordingly, it should be understood that the methods and apparatus described herein are illustrative only and are not limiting upon the scope of the invention.