Source: https://patents.google.com/patent/US6704741B1/en
Timestamp: 2019-07-23 10:39:05
Document Index: 567127181

Matched Legal Cases: ['art 117', 'art 262', 'art 264', 'art 322', 'art 323', 'art 343']

US6704741B1 - Test item creation and manipulation system and method - Google Patents
Test item creation and manipulation system and method Download PDF
US6704741B1
US6704741B1 US09/705,151 US70515100A US6704741B1 US 6704741 B1 US6704741 B1 US 6704741B1 US 70515100 A US70515100 A US 70515100A US 6704741 B1 US6704741 B1 US 6704741B1
US09/705,151
Dean Edward Lively, Jr.
Steven Anthony Ethofer
2000-11-02 Application filed by PSYCHOLOGICAL CORP filed Critical PSYCHOLOGICAL CORP
2000-11-02 Priority to US09/705,151 priority Critical patent/US6704741B1/en
2001-01-22 Assigned to SILVERCOIN PUBLISHING ONE, INC. reassignment SILVERCOIN PUBLISHING ONE, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ETHOFER, STEVEN ANTHONY
2001-01-22 Assigned to PSYCHOLOGICAL CORPORATION, THE reassignment PSYCHOLOGICAL CORPORATION, THE ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LIVELY, DEAN EDWARD, JR.
2001-01-22 Assigned to PSYCHOLOGICAL CORPORATION, THE reassignment PSYCHOLOGICAL CORPORATION, THE ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SILVERCOIN PUBLISHING ONE, INC
2004-03-09 Publication of US6704741B1 publication Critical patent/US6704741B1/en
A method for creating a test for administration to a student includes creating a plurality of test items using a unitary software application. Each test item has a plurality of elements, including an item stem (a question or problem), at least one option (such as a choice of answers, including one correct answer and at least one “distractor” or an open-ended question), and metadata related to the test item. Next, if desired, an image is added to the test item using the software. Each test item is then stored in an electronic database. Access to each test item in the database is provided, wherein the test items are searchable by at least one of the test item elements. The test creator then electronically selects at least one test item from the database and views at least a portion of the selected test item. If desired, the at least one selected test item is assembled into a test.
Creating items for assessment testing must be accompanied by a method for synthesizing information for the construction of an examination from some or all of the items. In the past, and continuing to the present time, test item authors have used “item cards” containing the “item” (here intended to mean the “item stem”), which refers to the question or problem, the options (correct answer and “distractors”), associated artwork and/or graphics, and statistical data. This information was pasted onto card stock to form the item card, and a collection of item cards referred to as an “item bank.” The physical card medium remains in place owing to familiarity and to a lack of electronic alternatives with desired features.
Commercial computer-based systems known in the art include those of TestBuilder (William K. Bradford Publishing Co.), MicroCAT™ (Assessment Systems Corp., St. Paul, Minn.), CATBuilder™ (CAT, Evanston, Ill.), Curriculum Director (Bookette), Examiner's Plus (Pilot Software Ltd., Ankara, Turkey), and TestMate products (Master Computer Systems, Inc., and CTB/McGraw-Hill, Monterey, Calif.).
It is therefore an object of the present invention to provide an automated system and method for creating test items.
These objects and others are attained by the present invention, a system and method for creating a test for administration to a student. The method comprises the steps of creating a plurality of test items using unitary software means. Each test item has a plurality of elements, including an item stem (a question or problem), one or more options (a choice of answers, including one correct answer and at least one of a “distractor” or an open-ended answer), and metadata-related to the test item.
It is to be understood herein that the term “student” is not intended to be limited to a school attendee; rather, a target of the test items of the present system and method may comprise, for example, a person desiring certification in a particular field or profession, such as a technical, legal, medical, or other discipline. The creation and manipulation of test items for any venue are thus intended to be encompassed under the present invention, as well as tests delivered by any means known in the art, including via hard copy, computer, and Internet access.
It is also to be understood herein that the term “item” is not intended to be limited to word-based modes; rather, alternate modes of administration such as oral/auditory, video, or graphic interactions.
FIG. 1 is a schematic diagram of the main functional components of the electronic test item creation and storage system.
A description of the preferred embodiments of the present invention will now be presented with reference to FIGS. 1-38.
Item Creation Application
The item creation application 20 permits a user to develop an item 12 via interfaces with a plurality of linked software means for entering textual and graphical data and metadata associated therewith. The form of the created item 12 is preferably adapted for electronic database storage, porting over a network to a central site and to a reviewer, and viewing in a plurality of formats.
The item creation application 20 comprises a plurality of modules, here three (FIG. 2). A central module 21, an individual module 24, and a review module 28. The individual module 24 is shown as being accessed by three item writers. Item Writer 1—Item Writer 3.
The Central Module. The central module 21 comprises a data-entry package 22 and an electronic database 23 located, for example, on a server 200 housed at a main administrative location. A user accessing the server 200 is assigned a unique code for identification purposes. In a preferred embodiment, the central module 21 comprises a multiuser database application, comprising a “front end” installed on a user's workstation 201 and a “back end” installed on the server 200. The item repository 23 stores item text, item graphics, and item information (e.g., metadata). FIG. 2. for example, indicates the item repository 23 as containing three test items 12 a-12 c. The graphics may be stored in any form known in the art, such as, but not intended to be limited to, GIF, TIFF, EPS files, and bitmapped images. The repository data may be viewed and modified as desired.
The process of data entry (FIG. 4; see also data flow diagram, FIG. 7) includes selecting the data entry form 211 and proceeding to the data collection pathway (block 300). A unique item identifier, Item ID 215 is computed (block 301) by the system 21 upon the user's entry of information via drop-down menus such as, but not intended to be limited to, Grade 216, Domain 217 (i.e., subject area), Benchmark 218, which is a term of art for “content standards” or “objective,” and Item Type 219. The remaining four numbers of Item ID 215 are computed sequentially. A second number, CID 220, is also assigned (block 302, FIG. 7) to each item as it is input; this CID 220 is unique and immutable.
If desired for presentation with the item 12, an artwork/graphics identifier is assigned (block 303) to each piece of art. The main piece of art to be viewed (see, for example, 330 in FIG. 27) is coded and entered into the appropriate block 221, with subsequent pieces of art coded and entered into the subsidiary Art ID blocks 222 (block 306). It can be seen in this exemplary case that the art identifiers comprise the Item ID 215 plus a sequential code following a period “.ARx.”
The textual portion of the item stem and responses is created (block 304) using a word processing application, here Microsoft Word, in the text entry block 223. Associated artwork is saved and linked (block 305) to the Art IDs. In the present embodiment, linked artwork may be viewed via a link to a viewing application, such as Adobe® Acrobat Reader, although this is not intended as a limitation. Herein only viewing is permitted; no editing is permitted. In an alternate embodiment editing may be permitted.
If the “Administrator” 213 application is selected from the control form 210, a password is required to display another series of selections (FIG. 5). As a first selection, the administrator may choose to import remotely data-entered items from a medium (block 330), such as a disk 31 (FIG. 2), although this is not intended as a limitation. The source for such items will be discussed in the following, under the description of the individual module 24.
Items can also be rejected (block 368, FIG. 7), which causes them to be removed from the database and placed in an “open pool” database (block 369). The linked text and graphics are moved to an “open pool” folder (block 370).
The Individual Module. The individual module 24 is installable and accessible by a user at any location, for example, a remote site. This module 24 is intended for use, for example, by an independent item writer or a vendor, although this is not intended as a limitation. Typically the item writer will save a completed item on a medium such as a disk 31, CD, or Zip™ drive; alternatively, data may be transferred over an Internet or intranet. Each user will have a password to enter upon sign-in and will have a unique code 27 with which each submitted item will be tagged.
The user creates a textual document 233, such as under a word processor such as Microsoft Word® (block 386), containing the item stem 235 and possible responses 236. The associated artwork IDs shown in FIG. 11 as Lead Art ID 234 a and Option Art ID 234 b, are also put into the record (block 387), with each graphic being linked to an artwork ID. The artwork is stored in a designated folder.
It may be appreciated by one of skill in the art that important features are provided by the present invention: the system 20 is user-friendly in incorporating interactive methodologies that have become familiar to most computer users, with Windows®-type screens having drop-down menus and labeled data-entry boxes. The inventive linking aspects of the system 20 obviate any need to re-enter data; rather, all data are automatically copied to all appropriate records. Data may be entered with formatting preferences in a way familiar to the average user, which aspect is provided by the incorporation of a word processing program.
It will also be obvious to one of skill in the art that alternate forms of data delivery and interaction therewith may be contemplated as being within the scope of the present invention, such as over the Internet in any available content form. Further, interaction with any of the contemplated classes of users may occur in alternate formats. For example, instead of Windows®-type screens appearing, the user may be led through the test item creation application with a series of queries, answers to the queries provided by the user then inserted into the appropriate place(s) in the database or directing a desired display to the user.
Item Database Application
The item database application 40 is adapted to bring item metadata, administration statistics, and artwork into a central location. In a preferred embodiment, the item bank interfaces are created on top of a commercially available package, here FileMaker® Pro v. 5.0, although this is not intended as a limitation. The database may be divided into projects, with each project in turn subdivided into sectors containing statistics, test administration, pull lists, and artwork and text. Alternatively, each project may have a separate database if its own. These divisions are not intended as limitations, however, and one of skill in the art will recognize other potential sectors and methods of categorization.
Item database 482: the item ID, item stem, correct response, distractor(s), benchmark, “item not appropriate indicator,” content area, content focus, text art
Exemplary screens for the first embodiment 47 are shown in FIGS. 18-28, with the control screen 471 having buttons for selecting subsequent screens as schematically illustrated in FIG. 16. A selection of the “item database” button 101 brings up the first of three accessible screens. Screen 111 (FIG. 19) contains a subscreen that includes the item ID number 112, passage title 113, topic 114; item stem 115, response options 116, and linked passages/art 117.
A selection of the “statistics database” button 102 brings up the first of three accessible screens. Screen 114 (FIG. 22) contains a subscreen that includes the item ID number 112 and book number 210; calibration scale 211 a and date 211 b; total number 212; item analysis run date 213; AS anchor 214; number of difficulty values 215, fit index and nonfit indicator 216; chi-squared values 217, sample size 218; and observed P value 219 on the left-hand side. On the right-hand side are located overall differential item functioning rating 220; demographic statistics 221; and passage code 222.
A selection of the “test administration database” button 103 brings up a screen 17 (FIG. 25), which includes commodity code 240; name of book 241; publication identification information 242, year 243, month 244, and name 245; grade level 246; field test form 247; and description 248.
A selection of the “pull list database” button 104 brings up a screen 18 (FIG. 26), which includes sequence number 250; page number 251; linking benchmark 252; field test or core 253; publication identification information 254, including year, month, ID, and name; vertical and horizontal linking 255; anchor 256; item stem 257; and option(s) 258.
A selection of the “art database” button 105 brings up the first of two accessible screens. Screen 119 (FIG. 27) contains a subscreen that includes new and old art names 260, book identification 261, passage or art 262, lead art form and code 263, description of art 264, passage code 265, copyright expiration date 266, comments 267, and associated art number 268.
Exemplary screens for the second embodiment 48 are shown in FIGS. 29-36, with the control screen 481 having buttons for selecting subsequent screens as schematically illustrated in FIG. 17. A selection of the “item database” button 281 brings up the first of two accessible screens. Screen 134 (FIG. 30) contains item information, including the item code 290, item stem 291, option(s) 292, comments 293, content area description 294, reporting category 295, item writer's code 296, SOL description 297, an “item not appropriate” list 298, and item art list 299.
Selecting the “statistics database” button 282 on screen 481 brings up screen 36 (FIG. 32), which displays item code 301, year 304, season 305, and form number 306. The subscreen displays grade 307, item sequence number 308, page number in the test book 309, a core item/link flag 310, N count 311, P value 312, index of reliability 313, mean square fit 314, point biserial correlation 315, point biserial correlation <0.30316, equated item difficulty 317, distractor analysis 318, bias indices 319, and prior versus compare data 320.
Selecting the “art database” button 283 on screen 481 brings up a first screen 37 (FIG. 33), which displays data including art code 321, a description of the art 322, comments on the art 323, and the copyright expiration 324.
Selecting the “passage database” button 284 on screen 481 brings up a first screen 39 (FIG. 35), which displays data including the passage code 340, the passage in text form 341, comments on the passage 342, and a list of the associated passage art 343.
These two embodiments are intended to be exemplary, and one of skill in the art will recognize the inherent flexibility of the system and its adaptability to a wide range of applications. Further, as with the test item creation application above, it may be contemplated that this application is easily amenable to content delivery in any electronic format and having interaction capabilities over, for example, the Internet. Also as above, interaction with this application may be envisioned as occurring via queries to the user rather than via Windows®-type screens.
Item Selection Application
The item selection application 50 of the present invention (FIG. 37) assists the user to assemble a test 90 from created items. It is known in the art that selecting items for inclusion in a test is both analytical and subjective. In prior art paper-based systems, item cards were arranged by some criterion, such as grade or content area. Further sorting resulted in piles based upon, for example, a range of P values or Rasch scores. The subjective phase of the selection then included reading and reviewing item stems and responses along with artwork, choosing appropriate items, sequencing those chosen items, and, if desired, readjusting with refinement on such considerations as target statistics.
In the present invention, the database 23 may be queried on one or more criteria simultaneously in a matter of seconds by viewing screen 440. The user can examine each resulting item in text or graphical form and select items believed desirable for inclusion in a test. Included on the screen 440 are statistical data for that item, content area description 442, reporting category 443, and an objective description of the item 444. The item, if desired, is then placed in a “shopping cart” by selecting button 441, not unlike those used in online shopping, or into an “item working set.”
Test Item Assembly Application
The test item assembly application 60 of the present invention (FIG. 38) comprises moving the selected test items to a medium preparatory to creating an actual test form 90. This process includes typesetting or “tagging” for publication software, such as Xyvision® Production Publisher or Quark Xpress™ and embedding required graphics, tables, and artwork; Text formatting can applied as desired, such as including boldface, italic, or underlined type and one- or two-column format, for example. A form may then be output to a device such as a printer; alternatively, a form or portion thereof may be printed directly. In addition, the output may be contemplated as occurring electronically for printing elsewhere; alternatively, the output may remain in electronic form for enabling online testing, with hard-copy output not necessary.
The database system may or may not be linked to this application. If the database system is used to store the “tagging” information, it also serves as the front end to the publishing software. An advantage of this option is the avoidance of rekeying. In addition, items that are not part of the database cannot appear in the test 90. Also, since the database drives the production process, that which is in the database and that which goes to press must be identical.
1. A method for creating a test for administration to a student comprising the steps of:
creating a plurality of test items using unitary software means, each test item having a plurality of elements comprising an item stem, a response option, and metadata related to the test item;
if desired, adding to at least one test item an image using the software means;
storing each test item in an electronic database;
providing access to each test item in the database, the test items searchable by at least one of the test item elements; and
electronically selecting at least one test item from the database;
viewing at least a portion of the selected test item;
if desired, assembling the at least one selected test item into a test;
plotting a test characteristic curve representative of an appearance of a desirable item pool:
calculating a statistical function from metadata corresponding to all test items in the assembled test;
plotting the statistical function;
if the statistical function is sufficiently close to the test characteristic curve, accepting the assembled test; and
if the statistical function is not sufficiently close to the test characteristic curve, removing a test item from the assembled test and replacing the removed test item with another test item tending to bring the statistical function sufficiently close to the test characteristic curve.
2. The method recited in claim 1, wherein the metadata further comprise at least one of statistics and occurrence data on prior administration of each test item, subject area, item type, reporting category, item identifier, and an objective.
selecting a type of view desired from an unformatted textual representation without associated graphics and a formatted representation including associated graphics and textual formatting elements; and
presenting the desired type of view to a user.
8. The method recited in claim 1, wherein the selecting step comprises:
electronically searching the database using at least one of the metadata elements as a search criterion;
if the searching step yields at least one test item, viewing at least a portion of at least one of the test items; and
if desired, electronically adding a link to a selected test item to an electronic record comprising data for forming the test.
performing a calculation on a statistical datum of all selected test items forming the test to yield a statistical value;
if the statistical value falls within the predetermined range, accepting the formed test;
if the statistical value falls outside the predetermined range, removing at least one test item from the test and replacing the removed test item with another test item tending to bring the statistical value within the predetermined range; and
if necessary, repeating the performing and removing steps until the statistical value falls within the predetermined range.
electronically searching the database using at least one of the metadata elements as a search criterion; and
outputting results of the searching step.
11. A system for creating a test for administration to a student comprising:
a processor having database software means resident thereon;
an electronic database accessible by the processor and adapted to receive for storage therein a plurality of graphical images and a plurality of test items, each test item having a plurality of elements comprising an item stem, a response option, metadata related to the test item, and, if desired, a link to at least one of the graphical images;
input means in electronic communication with the processor, adapted to provide access to each test item in the database, the test items searchable by at least one of the test item elements with the use of the database software means, the input means further adapted to permit an electronic selection of at least one test item from the database; and
output means in electronic communication with the processor adapted to display at least a portion of the selected test item;
the input means further adapted to receive an instruction to link the selected test item with a test for assembly therewith using the database software means, and further comprising a link to a software package that is adapted to:
plot a test characteristic curve representative of an appearance of a desirable item pool;
calculate a statistical function from metadata corresponding to all test items in the assembled test;
plot the statistical function, and wherein the input means is further adapted to;
if the statistical function is sufficiently close to the test characteristic curve, permit a user to accept the assembled test; and
if the statistical function is not sufficiently close to the test characteristic curve, permit a user to remove a test item from the assembled test and replace the removed test item with another test item tending to bring the statistical function sufficiently close to the test characteristic curve.
US09/705,151 2000-11-02 2000-11-02 Test item creation and manipulation system and method Active 2021-08-22 US6704741B1 (en)
US09/705,151 US6704741B1 (en) 2000-11-02 2000-11-02 Test item creation and manipulation system and method
US6704741B1 true US6704741B1 (en) 2004-03-09
ID=31888590
US09/705,151 Active 2021-08-22 US6704741B1 (en) 2000-11-02 2000-11-02 Test item creation and manipulation system and method
US (1) US6704741B1 (en)
US20060155528A1 (en) * 2005-01-10 2006-07-13 Educational Testing Service Method and system for text retrieval for computer-assisted item creation
US20120196261A1 (en) * 2011-01-31 2012-08-02 FastTrack Technologies Inc. System and method for a computerized learning system
US20130224702A1 (en) * 2012-02-24 2013-08-29 National Assoc. Of Boards Of Pharmacy Test Pallet Assembly
US9965479B2 (en) 2008-09-18 2018-05-08 Adobe Systems Incorporated Systems and methods for relinking data items
2000-11-02 US US09/705,151 patent/US6704741B1/en active Active
Bookette's Curriculum Director 3.0 Versus Harcourt's Custom Item Bank Developed in File Maker Pro V.5.0, one page. No publication date.
Cat Builder, brochure, two pages. Dec. 1999.
Cat Software System & Trade, http://www.catinc.com/softwarespecs.htm , three pages Jan. 14, 2000.
CTB Online, http://www.ctb.com/contact/index.html , one page. Jan. 14, 2000.
Master Computer Systems, http://www.mastercomputersys.com/solpro.htm , two pages. Jan. 14, 2000.
Microcat Computerized Testing System, six pages. No publication date.
Microcat Computerized Testing Systems, http://assess.com/microcat.html , four pages. Jan. 14, 2000.
Pilot Software, http://www.pilotltd.com/contact.htm , one page. Jan. 14, 2000.
Pilot Software, http://www.pilotltd.com/products.htm , three pages. Jan. 14, 2000.
Riverside Publishing, http://www.riverpub.com/feedback.html , one page. Jan. 14, 2000.
Riverside Publishing, http://www.riverpub.com/products.htm , one page. Jan. 14, 2000.
Test Bulder, http://www.kiva.net/~fulton/TB/home.htm , three pages. Jan. 14, 2000.
Test Bulder, http://www.kiva.net/˜fulton/TB/home.htm , three pages. Jan. 14, 2000.
US20110166853A1 (en) * 2005-01-10 2011-07-07 Educational Testing Service Method and System for Text Retrieval for Computer-Assisted Item Creation
US8131554B2 (en) 2005-01-10 2012-03-06 Educational Testing Service Method and system for text retrieval for computer-assisted item creation
Hatcher et al. 2013 A step-by-step approach to using SAS for factor analysis and structural equation modeling
Knodel 1993 The design and analysis of focus group studies: A practical approach
Bourdon 2002 The integration of qualitative data analysis software in research strategies: Resistances and possibilities
Cargill et al. 2013 Writing scientific research articles: Strategy and steps
Merriam 2002 Introduction to qualitative research
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LIVELY, DEAN EDWARD, JR.;REEL/FRAME:011460/0624
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SILVERCOIN PUBLISHING ONE, INC;REEL/FRAME:011460/0630
Owner name: SILVERCOIN PUBLISHING ONE, INC., IOWA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ETHOFER, STEVEN ANTHONY;REEL/FRAME:011462/0689