System and methodology for computer-facilitated development of reading comprehension test items through passage mapping

The present disclosure provides systems and methods for providing interface rendering of passages for test item generation. Methods and systems can utilize natural language processing (NLP) and machine learning (ML) models to analyze passages that were provided in historical testing environments (e.g., first passage), determine one or more similar search strings that are also found in a current passage (e.g., second passage), and provide the first and second passages to a user interface of a user device. On the display, the search string in the historical and current passages may be highlighted at the user interface to identify the similarities.

TECHNICAL FIELD

The disclosed technology relates generally to electronic testing item development, and more particularly to systems and methods for an interface rendering of passages for reading assessment test item development based on the detection and recognition of semantic and linguistic properties found in the passage.

BACKGROUND

Original text-based content for testing and educational purposes is traditionally identified and generated in a manual and laborious manner. Test developers may review pre-existing text-based content (usually referred to as “passages”) and select portions ripe for generating test items (e.g., questions and answers, or other test/assessment items). The vast number and volume of pre-existing text-based content, with varying difficulty and complexity, can make selecting portions ripe for generating test items difficult with traditional processes. Better systems and methods are needed.

BRIEF SUMMARY OF EMBODIMENTS

Traditionally, a content specialist develops items based on these passages. For example, the content specialist reads the passage to identify particular words, phrases, or groups of sentences (e.g., search strings) that could be included in future test items. This is referred to as “passage-mapping.” It tends to be time-consuming, subjective, imperfect, and inconsistent, with different content specialists finding or overlooking different testable item opportunities. Computer systems are historically incapable of automatic passage mapping and assisting in generating passage-based question and answer items, including question and answer items for reading comprehension testing, mostly due to the vast plethora of content of broad topics, unidentified search strings in previous passages, and of varying degree of difficulty.

The present disclosure provides systems and methods for providing interface rendering of passages for test item generation. The present disclosure can utilize natural language processing (NLP) and machine learning (ML) models to analyze passages that were provided in historical testing environments (e.g., first passage), determine one or more similar search strings that are also found in a current passage (e.g., second passage), and provide the first and second passages to a user interface of a user device. On the display, the search string in the historical and current passages may be highlighted at the user interface to identify the similarities. The search string may also be used to help generate new test items for the current passage. The historical and current passages may correspond with search strings that share similarities between these two passages. The analysis can help generate similar test questions in a future test, where the future test would include the current passage and the new test questions.

As a sample illustration, an input passage may include one or more search strings, including, for example, a search string corresponding with a causal relation (“as a result of”), a comparative relation (“better than”), or other information. The system disclosed herein may search for one or more of these input passages provided in historical testing environments to find search strings that are similar to search strings in a current passage. Suggested input passages may be provided and displayed. The input passages can include one or more annotations that highlight a search string that may be modeled to generate a new passage. The annotation can be associated with a content taxonomy code. The annotation can comprise an overlay or highlighting of at least a portion of the passage.

Method and systems described herein can include receiving, by an item evaluation server, an input passage. The input passage may be a corpus of text and/or comprise one or more search strings that correspond with a plurality of textual patterns. One or more of these textual patterns can include skill-specific language cues.

Method and systems described herein can include providing for display the input passage. The input passage may include one or more search strings and/or corpus of text. At least a portion of the input passage can be displayed at a Graphical User Interface (GUI).

Method and systems described herein can include tokenizing and/or searching the input passage to determine one or more skill-specific textual patterns. The input passage may include the skill-specific textual pattern, and thus be a relevant and/or “testable” to the input passage. In some examples, one or more of these textual patterns may be associated with a test skill. Text tokenizing and search can be performed upon activation of a first trigger. Passage tokenizing can include tokenizing the passage into words, sentences, and parts of speech.

Method and systems described herein can include providing a search query comprising one or more of the skill-specific textual patterns. The textual patterns can be searched in the input passage.

Method and systems described herein can include matching the skill-specific textual patterns with corresponding taxonomy codes and determining a stored item which includes the skill-specific textual pattern associated with the taxonomy code. The stored item may correspond with a generated test question corresponding with the passage. The taxonomy code can be related to or indicate, for example, a comparative relationship, causal relationship, argument, fact, opinion, reason, judgement, evidence, intent, or theme.

Method and systems described herein can include providing for display the input passage and corresponding one or more stored items that were provided with the input passage in the historical test environment. The stored item can be displayed in a second portion of the GUI. The stored item provided for display can include an overlay of at least a portion of the stored item to identify, for example, a search string or other relevant information for the test administrator. The overlay can be based on the taxonomy code and/or the skill-specific textual pattern.

Methods and systems described herein can include providing for display the input passage and/or current passage. The input passage can further include annotation associated with the taxonomy code. The annotation can include an overlay of at least a portion of the input passage based on the taxonomy code and/or the skill-specific textual pattern. Methods and systems can further include providing for display at least a portion of the input passage without the annotation of the input passage. Providing the input passage can commence upon activation of a second trigger. Activation of the second trigger can indicate a request to revert to the display of input passage.

Methods and systems described herein can include determining the stored item from the data store, based on a stored item attribute. The stored item attribute can be, for example, either of: a last test date, a question difficulty, a question discriminator, a theme, a scholastic area, or distractor.

Methods and systems described herein can implement a machine learning (ML) item evaluation model. The ML model may be trained using input passages and corresponding stored items for those input passages. The stored item may be determined by the trained machine learning item evaluation model based on the skill-specific textual pattern associated with the taxonomy code of the input passage, detection and recognition of semantic and linguistic properties, and/or weighted contributions of stored item attributes.

Methods and systems described herein can include, upon activation of the first trigger or another trigger, tokenizing, via the item evaluation server, the current passage to determine a theme of the current passage using, at least in part, a natural language processing engine of the item evaluation server. Moreover, methods and systems can determine the input passage and/or stored item from the data store based on the stored item having the same theme as the current passage.

Methods and systems described herein can include matching a skill-specific textual pattern of the current passage to a respective taxonomy code. The determined stored item can include one or more of the skill-specific textual patterns. Upon activation of a third trigger indicating a request to select a particular taxonomy code from the at least one taxonomy code, methods and systems described herein can provide for display at the first object of the GUI, the annotated current passage corresponding with the new test environment or the annotated input passage corresponding with the historical test environment. Moreover, systems and methods can provide for display at the second object the annotated stored item. The annotated stored item can be based on a user selected taxonomy code and/or the skill specific textual pattern. The user selected taxonomy code can be used to train the trained machine learning item evaluation model.

Methods and systems described herein can further include generating a test item for the current passage. The generated test item can be provided for display, including alongside the stored item. The generated test item can be based on a portion of the current passage which has a higher density of assigned taxonomy codes and a multiple choice item, each part of which is based on the corresponding portion of the current passage, a theme of the current passage, and the stored item.

Systems described herein can include a graphical user interface, a data store, and an item evaluation server. The item evaluation serve can further include a processor and a non-transitory computer readable medium with computer executable instructions embedded thereon. The computer executable instructions can be configured to cause the processor to execute the methods described herein.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The disclosure described in this document increases the efficiency of passage mapping by providing the content specialist with a systematic, computer-assisted approach. The disclosure described herein provide greater speed of identification of opportunities for generating test items based on a passage or large corpus, in any genre, style, and for any topic. The disclosure described herein can provide for identification of item opportunities that the content specialist would otherwise have missed.

Systems and methods described herein can provide improved displays for rendering passage highlighting, which can automatically and efficiently highlight language ripe for generating test items based on a taxonomy code. Improved displays described herein can scan/overlay displayed test items based on taxonomy code to increase computing efficiencies.

Systems and methods described herein can improve item display by highlighting passages and determining and rendering relevant stored test questions, which can allow for improved item performance, greater consistency between test cycles and quality standardization. Systems and methods described herein allow for processing of vast amount of historical item data, and computationally efficient, automatic determination and rendering of relevant past test items. Similarities between the tokenized passage section for basing generated test items, and the historical test items can be efficiently determined using data such as the taxonomy code. For example, search patterns for relevant new item opportunities can be based on items that have been historically provided in testing environment, and those stored items which are relevant to the passage for generating new test items can be determined and displayed. Systems and methods described herein can allow for consistency in testing because new test items can assess similar skills in similar ways as before, resulting in similarly successful item performance. Moreover, past unsuccessful items can be avoided and not suggested.

FIG. 1Aillustrates an example system for item development. System100can include item evaluation server102. Item evaluation server102can include processing module105. Processing module105may represent, for example, computing or processing capabilities found within desktop, laptop and notebook computers; hand-held computing devices (PDA's, smart phones, cell phones, palmtops, etc.); mainframes, supercomputers, workstations or servers; or any other type of computing devices as may be desirable or appropriate for this application or environment. Processing module105can also represent computing capabilities embedded within or otherwise available to a given device. For example, a computing engine might be found in other electronic devices such as, for example, digital cameras, navigation systems, cellular telephones, portable computing devices, modems, routers, WAPs, terminals and other electronic devices that might include some form of processing capability. Processing module105can be implemented using, for example, a microprocessor, controller, or other control logic.

Item evaluation server102can further include natural language and processing (NLP) engine107and/or prediction and learning engine109. NLP engine107can perform detection and recognition of semantic and linguistic properties of language. Processing module105and/or engines107,109can operate a machine learning based item evaluation model.

System100can further include data bus110, although any communication medium can be used to facilitate interaction with other components of system100or to communicate externally.

System100for interface rendering of tokenized item development can include data store111. Data associated with one or more operations of the present disclosure may be stored in data store111. Data store111may be a local or a remote data store (or a combination thereof) and may additionally be configured to receive and/or forward some or all of the stored data relating to an operation of the present disclosure. Data in data store111may be organized as database items.

Data store111may take one or more of various forms, which might include, for example, a media drive and a storage unit interface. The media drive might include a drive or other mechanism to support fixed or removable storage media. For example, a hard disk drive, a floppy disk drive, a magnetic tape drive, an optical disk drive, a CD or DVD drive (R or RW), or other removable or fixed media drive might be provided. Accordingly, storage media might include, for example, a hard disk, a floppy disk, magnetic tape, cartridge, optical disk, a CD or DVD, or other fixed or removable medium that is read by, written to or accessed by media drive912. As these examples illustrate, the storage media can include a computer usable storage medium having stored therein computer software (including for executing instructions for the item evaluation server102) or data.

In alternative embodiments, data store111might include other similar instrumentalities for allowing computer programs or other instructions or data to be loaded into the item evaluation server102and/or the processing module105. Such instrumentalities might include, for example, a fixed or removable storage unit and an interface (e.g. the reviewer interface138). Examples of such storage units and interfaces can include a program cartridge and cartridge interface, a removable memory (for example, a flash memory or other removable memory engine) and memory slot, a PCMCIA slot and card, and other fixed or removable storage units and interfaces that allow software and data to be transferred from the data store111to the item evaluation server102.

Data store111can include a plurality of data stores, including textual pattern data store114, question (item “stem”) data store116, answer data store118, multiple choice item data store (not shown, but can include stored items of question data store116and/or answer data store118), attributes data store120, and item data store122.

Corpus data store112may include entire or partial corpora from varying topics. Corpus data store112can also include general or topic specific dictionaries.

Textual patterns in textual pattern data store114can be skill-specific language cues (for example as “search strings”). For example, textual patterns that allow for understanding comparative relationships can be search strings “is greater than” or “is like.” Textual pattern data store114may also include taxonomy codes associated with the textual patterns. For example, a taxonomy code can be generated and assigned to the one or more textual patterns that are language cues for demonstrating comparative relationships. Each such taxonomy code for textual patterns indicating comprehension of a same or similar skill, can have the same portion of the taxonomy code.

Question data store116may include questions, sometimes referred to as item “stems.” In some examples, the questions may be associated with answer options from answer data store118and/or attributes in attribute data store120. Question items may be full questions, or be sentence fragments (question stems) wherein the remaining portion of the sentence may be found in the corresponding answer item.

Item data store122may include stored items provided in historical test environments, including for example question (stem) and answers, together with associated passages. Although called stored items, the data in item data store122is not limited to items or other content which have been previously tested.

Item evaluation server102might also include one or more memory engines, simply referred to herein as main memory106. For example, random access memory (RAM) or other dynamic memory might be used for storing information and instructions to be executed by processing module105. Main memory106might also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processing module105. Item evaluation server102might likewise include a read only memory (“ROM”) or other static storage device coupled to bus110for storing static information and instructions for processing module105.

In some embodiments of the disclosure, item evaluation server102may be configured to execute any one or more of the protocols, processes, models, or operations of this disclosure, provide outputs for display on electronic device160and/or reviewer interface138, and receive inputs from electronic device160and/or data store111. In various other embodiments, server102may be configured to execute any one or more operations associated with developing test items electronically, or receiving material for developing test items, such as multiple choice items, electronically from paper material (e.g. via a scanner for scanning textual content, not shown), or cooperatively with one or more other electronic devices160, computers, or workstations.

Network interface(s)130can facilitate communication with electronic device160on network134and the other components of system100. Network interface(s)130can be used to allow software and data to be transferred between item evaluation server102and or data store111, and one or more device(s)160. Examples of network interfaces130might include a modem or softmodem, a standardized network interface (such as an Ethernet, network interface card, WiMedia, IEEE 802.XX or other interface), a communications port (such as for example, a USB port, IR port, RS232 port Bluetooth® interface, or other port), or other communications interface. Software and data transferred via network interface130might typically be carried on signals, which can be electronic, electromagnetic (which includes optical) or other signals capable of being exchanged by a given network interface130. These signals might be provided to network interface130via bus110.

Bus110might carry signals and might be implemented using a wired or wireless communication medium. Some examples of bus110might include a phone line, a cellular link, an RF link, an optical link, a network interface, a local or wide area network, and other wired or wireless communications channels. The network134may be the internet, a public switchable telephone network, a local area network, private wide area network, wireless network, or any of the like for communicating with system100and or device160of the present disclosure. Although network134is shown, it is understood that the disclosure is not limited to operation with a network, but rather, the disclosure may be practiced in any suitable electronic device (or multiple devices). Accordingly, network134is not meant to limit the disclosure in any respect.

Reviewer interface138can be a graphical user interface (GUI). Reviewer interface138can facilitate receiving stored items and other user feedback, as well as displaying information to the user.

Passage module140can facilitate receiving and/or displaying passages containing text. Passage module140may transmit passages to item evaluation server102via network134. Item evaluation server102can create new stored items or update existing stored items in item data store122using information from one or more passages received from passage module140.

Trigger module143can facilitate receiving and/or producing triggers. The triggers may be generated by the reviewer/user of device160, item evaluation server102, or between other systems.

System100may further include power supply162. Additionally, a printer165may be configured to output the content from one or more operations of the disclosure, including test items, passages, annotated passages, portions of passages, or items.

FIG. 1Bshows example database entries for the system100. For example, textual pattern data store114can have database structure169and database entries corresponding to textual patterns. Stored item data store122can have database structure170with respective database entries corresponding to stored items. Database items of the textual pattern data store114database structure169can include the textual pattern text, the skill tested, the taxonomy code, and other information. The database can also reference items from other data stores, such as the date the textual pattern was last tested, and the item the textual pattern was last tested on. As an illustration, there can be an textual pattern text data entry171corresponding with the search string “suggests that,” which tests an argumentative skill173(ARG). The database169can refer to item data store122for item177where the textual pattern was last tested (i.e. by reference to an item reference number, e.g. May-17-2017-Passage-B-Question-1) and the date of the last test180(e.g. May-17-2017).

Database items of the textual pattern data store114database structure169can include the item reference number, the passage for that item, the question (stem) of that item, the answer options of the item, various attributes (e.g. which can also reference to items in the attributes data store120), such as the theme, and the dates last test, first tests, or the number of times tests. Moreover, database structure can also include the textual pattern(s) for each item, the skill tested, and the taxonomy code of that textual pattern. For example, the database structure170for the item data store122may include the item reference number177of May-17-2017-Passage-B-Question-1, with an entry for a last test date180(e.g. May-17-2017) and first test date (e.g. May-17-2014). For that item, there can be entries with the entire passage, question, answer options, and theme (e.g. natural science). These entries can be cross-referenced as data entries of other data stores of the data store111, for example the theme in the attributes120data store or the item stems in the question data store116. Further, database entries which can also be database entries of the textual pattern data store114database structure169can also be included, such as the textual pattern171“suggests that”, the skill tested172(e.g. argumentative skill), and taxonomy code173(e.g. ARG-OPI-001). New entries can be created in item data store122as system100learns and/or develops new content. As textual patterns, attributes, themes, etc. of those new passages and items are identified, those passage and item entries can be updated.

Thus, methods and systems described herein can use data to map passages and items to each other, for example via textual patterns. The identification of a taxonomy code in cross-referenced databases can increase computing efficiencies when system100determines relevant items and when system100provides an overlay to the display based on textual patterns as described herein (see at leastFIGS. 3-4below). Moreover, system100can use data to identify textual patterns in passages, and to identify previously tested material having the same textual patterns, themes, attributes, etc. Although only two database structures which refer to each other are shown, other databases in data store111can refer to each other to at least facilitate improved passage mapping and item suggestion by providing an improved display for test item generation. In systems100, database structures and references to entries thereof can automatically update, thus improving the computational efficiency and rendering of passage section highlighting and relevant items. In system100, entries in data store111may be automatically updated, for example, via network134, with real time data from testing (such as test results, question difficulty, question discrimination, or new items).

FIG. 2shows a graphical user interface (GUI) with an illustrative passage. Passage(s) can be input into a first portion205of GUI201and correspond with a new passage that has not been provided in a historical testing environment. System100can receive the passage in first portion205of GUI201via passage module140. The passage can be a user-generated passage, or a passage from some other source that may be ripe for generating test items.

Test items may include a reading comprehension item. Reading comprehension test items can be based on a taxonomy of distinct types of reading skills. Examples of such skills can include for example: understanding stated and implied comparative relationship, understanding stated and implied causal relationships, understanding argument, understanding stated and implied claims and counterclaims, distinguishing among fact, opinion, and reasoned judgment, and determining the intended meaning of words with multiple possible meanings, or the meaning of figurative language.

Such skills can involve distinctive and identifiable types of language in the passage. For example, textual patterns on understanding comparative relationships can involve comparative words or expressions in the passage (e.g. “ . . . to a greater extent than . . . ”). Textual patterns on understanding causal relationships can involve any of a large possible number of causal expressions in the passage (e.g. “ . . . resulting from . . . ”). One or more textual patterns may be used as a search string to identify comparable language between a first passage and a second passage.

GUI201can include interactive portions for receiving indications. Portions for receiving indications can include portions for receiving: an indication of a choice of a file for upload207to system100, an indication of a request to scan208, an indication for reverting to an unannotated state210. Interactive portions can be triggers for system100and can be facilitated by trigger module143. As an example, a reviewer may select choose file207to select a file from device160, which may contain a passage for display in first portion205of GUI201. System100may cause the passage to be displayed in first portion205of GUI201. Alternatively, only the first paragraph (or any other portion) may be displayed initially in first portion205of GUI201. The reviewer may select scan208which can indicate a request for the passage to be scanned. System100may scan (e.g., after tokenizing the passage) to determine the presence of one or more skill-specific textual pattern(s). System100can display a modification on the passage, for example a portion of the passage.

System100, upon receiving an indication of request to scan208, can scan the passage. Scanning can by first determining a string of characters corresponding with the passage and then scanning the string of characters. System100can automatically scan the passage as text is entered into first portion205of GUI201, or upon receiving an indication of a choice of a file for upload207.

Upon scanning, system100may tokenize the passage. Tokenizing the passage can include separating it into tokens, which can be, for example, words, sentences, and parts of speech. Tokenizing can include automatically turning portions of the passage to tokens. Tokenizing can further include removing prefixes or suffixes of words (e.g. “ing” or “ed”) to determine root words. System100can tokenize by removing special characters or whitespace. Tokenization techniques may take the form of a regular expression using one or more identifiers to divide the text input into individual words and sentences. In embodiments, identifiers may consist of punctuation or spaces. For example, the tokenization technique may transform a portion of a stored item in item data store122into one or more individual words, resulting in one or more individual terms: “Fossils”, “of”, “Ankylosaurus”, “are”, “rare”, “in”, “the”, “sediments”, “,”. System100can also pre-process by spell checking or getting rid of non-relevant segments, such as stop words. Each token can be stored into an array in data store111.

Tokens can be stored in array data structures (e.g. stores in memory), where each token can be addressed and read. For example, data storage may include of a hard disk drive, a floppy disk drive, a magnetic tape drive, an optical disk drive, a CD or DVD drive (R or RW), flash memory, cache, or other removable or fixed media drive may be provided.

After tokenization, system100may query textual pattern data store114to determine the presence of textual patterns in the passage. The textual patterns can correspond with one or more search strings and relate to skill-specific language cues. Thus, the textual patterns can be indicative of reading assessment skills. Instances of such skill-specific language cues can be identified (e.g. based on content expert suggestions, past successful test items, past manual passage mapping by content specialists, internet searches, library content). The textual patterns can gathered into large libraries of search string patterns, organized by taxonomy code. These large libraries can be used for automated passage scans. The large libraries can be implemented as textual pattern data store114.

Each passage can be scanned. Each token of the previously tokenized passage may be queried against textual pattern data store114. System100can determine the textual patterns in the passage and/or test items. For example, textual pattern data store114may contain the textual pattern “suggests that”, and “suggests that” may be identified as a textual pattern in the passage by querying textual pattern data store114. System100may indicate this textual pattern as an opportunity for developing a test item for testing understanding of argumentative reading comprehension. As another example, when the passage scan finds a “causal” textual pattern in that passage (e.g. “as a consequence”), the word or phrase can be indicated as an opportunity to develop a test item testing understanding of causal relationships.

FIG. 3shows an interface rendering of the main textual components.FIG. 3shows the graphical user interface201with annotated passage and stored item for interface rendering of textual components to allow for test item development. Upon the user selecting Scan208and finding a textual patterns in the passage, as described with reference toFIG. 2, the system100can provide for display at least a portion of the passage. For example, at least a paragraph of the passage can be displayed in the first portion205of the GUI201. The selected portion of the passage provided for display may have a higher density of found textual patterns310, or have textual patterns310of a specific taxonomy code305, or otherwise indicating an opportunity for test item development of a specific language skill. The density of found textual patterns310may be determined by the words that are part of found textual patterns, divided by the sum of those words and the words which are not part of a textual pattern. The density of found textual patterns may be determined for segments of words, for sentences, or for paragraphs.

Upon the system receiving a trigger indicating a request to Scan208and the textual patterns310of the passage determined, as described with reference toFIG. 2, system100can annotate the passage to point out the found textual patterns310. Each instance of the same textual patterns310can be annotated, for example, in the same manner.

Annotations can correspond with a string marker. String markers can be a superimposed shape, such as circle, square, underline, arrow, or pointer. String markers can be a manipulation of text, such as a highlight, text color, bolding, capitalization, or underline. In some examples, each portion of the passage (e.g. a sentence or a paragraph) can be depicted in its own line. Taxonomy code(s)305associated with the specific textual pattern(s)310can be displayed proximate to and/or as part of the overlay of the passage.

Proximate to the displayed passage, the taxonomy codes of textual patterns can be displayed in taxonomy code portion217of the GUI201. Alternatively or additionally, a listing of each such textual pattern can be displayed. The found taxonomy codes (e.g. first taxonomy code215represented by ARG, which can indicate an argumentative skill, and/or second taxonomy code220represented by OPI-001 which can indicate a specific textual pattern suggesting an argumentative skill, e.g. “suggests that”) can be arranged so as that the taxonomy codes can be displayed according to specific skill, such as ARG for argument, REL for relationship, and CAU for causal skills. In the shown example, taxonomy codes containing ARG, REL, CAU may have been found in the passage (i.e. textual patterns corresponding to indications of testing understanding/comprehension of argumentative language, relationships, or causation). The skills and/or codes in taxonomy code portion217of GUI201, can be interactive portions and can operate as input triggers to system100.

As an example, if the reviewer selects first taxonomy code215(e.g. indicating ARG for testing argument skills), taxonomy codes corresponding to the found textual patterns can be displayed. Other taxonomy codes may include second taxonomy code220ARG-OPI-001, and/or ARG-OPI-002, and/or ARG-OPI-003. A portion of the taxonomy codes or the full taxonomy codes may be displayed (e.g. OPI-001 instead inf ARG-OPI-001). Alternatively, the found textual patterns corresponding to selected taxonomy code skill can be displayed (e.g. “suggests that” instead of second taxonomy code220).

System100may retrieve taxonomy codes and/or textual patterns from textual pattern data store114. Alternatively, the textual patterns corresponding to the taxonomy codes may be displayed in this taxonomy code portion217of the GUI. Upon receiving a trigger of that selection, for example via trigger module143, system100can provide for display of an annotation of the passage displayed in first portion205of GUI201so as to distinctly point out those textual patterns310found in the passage corresponding to the selected taxonomy codes (e.g. first taxonomy code215, or second taxonomy code220). A reviewer may select one or more taxonomy code (e.g. second taxonomy code220). The taxonomy code can be related to or indicate textual patterns which convey, for example, a comparative relationship, causal relationship, argument, fact, opinion, reason, judgement, evidence, intent, or theme. The annotation can include one or more taxonomy markers or manipulations of the passage. The system100can be updated so that the textual patterns310corresponding to the one or more selected taxonomy code(s) in the taxonomy code portion217of the GUI201can be proximately displayed, (or not) depending on if the corresponding taxonomy code is selected or not in the taxonomy code portion217of the GUI201. Moreover, a selection of Revert210, can indicate a request to revert to the display of the original passage, such as the passage displayed inFIG. 2in the first portion204of the GUI201.

Upon the system receiving a trigger indicating a request to Scan208and the textual patterns of the passage are determined as described with reference toFIG. 2, system100can retrieve a stored item (e.g. from item data store122) and display it in second GUI portion330and/or third GUI portion335of GUI201.

The stored item can be a relevant item provided in a historical test environment. System100can retrieve and display one or more (e.g. two, three, four, five) stored items and automatically alter the display (e.g. by decreasing order or relevance based on the determined relevance). The stored item can be relevant to the passage if it contains one or more textual patterns340which are the same as textual pattern310in the passage, and or if the stored item has desirable attributes associated with it (e.g. as queried in attributes data store120or otherwise determined). System100can assign a relevance score to stored items, and provide for the display any relevant stored items (e.g., highest relevance score, top two or three stored items, etc.).

A taxonomy marker can be used to indicate textual pattern340of the stored item. The same taxonomy marker used to identify textual pattern310of the passage can be used for textual pattern340of the stored item, if for example the textual patterns are the same. A visually similar taxonomy marker can be used (e.g. similar color) when the textual patterns are languages cues for the same assessed skill but correspond to different language cues (or search strings in textual pattern data store114). Other visual, acoustic, or other technique to assist in passage-mapping for assessment item development may be employed.

System100can provide for display prompt325for the stored item which is provided for display. Prompt325may include a recitation of relevant textual pattern(s)340and/or taxonomy code thereof. For example, prompt325may recite: “Provided passage contains textual pattern ‘suggests that’. The following is a previously developed ARG-OPI item based on ‘suggests that’”.

The stored item provided for display can be a portion of the stored item in item data store122. The portion can be a section, paragraph, and/or sentence portion336that contains textual pattern340. The stored item provided for display can include a question and answer prompt350. The stored item provided for display can include a question and answer355as retrieved from the question and answer data store(s)116,118. The correct answer can be noted by a correct answer indication360. Moreover, distractor answers can be noted by other visual indications. Attributes which are related to the displayed stored item (not shown) can be displayed proximate to the stored item, e.g. year, theme, topic, expected time to complete, percentage of test takers which selected a particular answer, question discriminator, a first and/or last test date, a question difficulty, a scholastic area, and/or distractor answers.

The system100can allow for the content specialist to generate a test item with the assistance of the annotated passage and the annotated stored item mapping, for example, as shown inFIG. 3.

FIG. 4shows a graphical user interface with an illustrative input passage, stored item, and one or more search strings. For example, interface201includes suggested generated test item for assessment item development. The passage and/or stored item can be displayed as shown inFIG. 2-3. In addition, system100can generate, determine and/or display a test item, including a test item passage (e.g. in first portion205of the GUI), prompt415for the new test item, a question and answer, testable portion420of the test item passage, and test item question425and answer options430. The generated and displayed test item can update depending on which of the found taxonomy codes215,220are selected, or otherwise indicated as referring to a testable skill or concept. More information regarding development and generation of test items is shown with reference toFIG. 6.

FIG. 5illustrates a flow chart showing passage-mapping steps for item development. For example, flow chart500can include passage mapping steps and prior item suggestion steps for assessment item development.

At step505, the system can receive an input passage. The input passage can be received as described inFIG. 2, with relation to providing for display the passage in the first portion205of the GUI201.

At step510, the system can receive a trigger activation indicating a request to scan the passage. System100can activate trigger module143, and/or initiate an interactive portion of the GUI201for receiving the indication request to scan208. System100can automatically scan the input passage as text is entered into the first portion205of the GUI201, thus receiving text (or new text) in the first portion205of the GUI can be the trigger activation for a request to scan. In system100, the input passage can be automatically scanned when the action of choosing a file.

At step515, the system can provide a passage for display. The display may include at least a portion of the passage. For example, system100is configured to provide the passage, or portion thereof, for display as discussed in relation to the displayed passage in first portion205of GUI201inFIGS. 2-4. As with each of the steps discussed inFIG. 5, steps510and515may be interchanged.

At step520, the system can tokenize the passage and search for one or more skill-specific textual patterns in the passage. Tokenizing and searching the passage to determine the textual pattern can be performed as described with reference toFIGS. 2-4above.

At step525, the system can query textual pattern data store114to determine the skill-specific textual pattern in the passage. For example, system100can query textual pattern data store114to determine the textual patterns in the passage. The textual patterns can correspond with search strings. Each token of the passage can be queried in the textual pattern data store114. Alternatively or in addition to step525, system100can determine the textual patterns in the passage by applying a trained machine learning item evaluation model to determine the textual patterns. Training the model to identify textual patterns is elaborated on further in relation to at least step805through step825of method800inFIG. 8. Applying the model to identify textual patterns is also shown in relation toFIG. 8.

At step530, the system100can, where appropriate (i.e. where a skill-specific textual pattern was identified in step525), match the skill-specific textual pattern with a taxonomy code305. Taxonomy code305may be stored in textual pattern data store114.

In some embodiments, steps520,525, and530can be performed recursively. For example, system100can tokenize the first pretest item into words, sentences, and parts of speech and query textual pattern data store114to determine if the passage contains a skill-specific textual pattern, as shown in step525. Step525(and optionally step530) can be successively performed until all textual patterns have been searched and, if found, matched with a taxonomy code at step530.

At step535, the system can determine a stored item from item data store122. System100can determine a stored item which is relevant to the passage (or relevant testable portion thereof) at step505and determine the relevant stored item by assigning relevance score(s) to the stored item(s) in item data store122. System100can apply a trained machine learning item evaluation model to determine a relevant stored item (which is elaborated on further in relation to step850, of method800inFIG. 8). The determined stored item can have the skill-specific textual pattern which was previously identified in the passage. The determined stored item can have the skill-specific textual pattern in a question item (e.g. as queried in question data store116) and/or in a correct answer of the stored item (e.g. as queried in the answer data store118), which may indicate the specific skill-specific textual pattern related to the skill tested in that stored item. The stored item can have the most instances, or the highest density of the skill-specific textual patterns which were previously identified in the passage. The identified stored item can have the skill-specific textual pattern having the same taxonomy code found in as the passage. The identified stored item can have a skill-specific textual pattern testing the same skill as the passage.

Determining and selection of the stored item can further be based on one or more attributes of the stored item. The system100can apply a trained machine learning item evaluation model to determine a relevant stored item based on the attributes. The attribute of the stored item can be in the attributes data store120. The attributes can be assigned weights based on how relevant the attributes are to selecting a stored item. The attribute can be a last test date, a question difficulty, a question discriminator, a theme, an enemy theme, scholastic area, or distractor. Themes can be, prose, humor, literary narrative, social science (psychology, sociology, education, civics, government), natural science (biology, chemistry, physics, technology, astronomy, medicine). For example, past unsuccessful items where no or all of the test takers got the question correct (thus were poor discriminators) can be avoided and not suggested or provided for display by system100.

As an example, system100can receive the theme or topic of the passage. The theme may be received using a machine learning based model as described in relation toFIG. 7, by receiving the theme from the data store111, or by the reviewer interface138, for example. System100can then select the relevant stored item (e.g. from item data store122) that has the same theme as the passage. In some examples, system100can select a theme other than an enemy theme. For example, an enemy theme may correspond with a different theme than the theme of the passage. Enemy themes can be humor and natural science, for example.

As another example, system100may query the attributes data store120and select the stored item which has previously been tested more recently in time or least recently in time, etc. System100can apply weights to these attributes. As an illustrative sample, system100may weigh more heavily the presence of the previously determine skill-specific textual pattern at step530, less heavily the theme or topic, less heavily the percentage of students who got the answer correct, and least heaviest, for example, the last date the item had occurred in a test. Any combination of attributes, theme, and/or presence of skill-specific textual patterns, with any weight attributed thereof, can be used to determine a relevant stored item.

Although step535refers to determining a single stored item, one or more stored items can be determined (for example, the top three most, or top five most relevant items). Moreover, system100can assign relevance scores to one or more stored items. The relevance score can define the degree to which the stored item relates to the passage.

At step540, the system can provide for display the annotated stored item. The stored item can include a portion of passage336, together with an associated previously-developed question and answer prompt350, and/or the question(s) and answers355. The annotation can be based on the identified taxonomy codes and/or the textual patterns of the stored item. The annotation can be an overlay prominently depicting the skill-specific textual patterns associated with the identified taxonomy codes. The annotated stored item can be provided for display as shown inFIG. 3(see annotation, at least in the form of square taxonomy marker at textual pattern340). The annotation can include relevant taxonomy markers, display of attributes (e.g. a last test date), display of relevance score, and/or a correct answer indication360. Alternatively, the stored item can be displayed without annotation.

Although the method500refers to displaying a single stored item at step540, one or more stored items can be displayed (for example, the top three or top five most relevant items). System100can retrieve and display one or more (e.g. two, three, four, five) relevant prior assessment items (or portions thereof, or links to the file location thereof), and automatically provide for display in an organized manner (e.g. by decreasing order or relevance based on the determined relevance, with most relevant stored item closer to the passage in the display). Moreover, system100can allow for cycling through of the display of stored items, or can provide a rotating display of stored items, for example, from the most relevant to least relevant. In system100where entering new text is the trigger activation for scanning at step515, system100can automatically annotate the passage as text is entered into first portion205of the GUI, for example by highlighting textual patterns as they are entered.

At step550, the system can receive activation of a second trigger. The second trigger can indicate a request to revert the display. A selection of revert210as shown inFIGS. 2-4can indicate the request to revert the passage and/or stored item into unannotated states (i.e. without any overlay).

At step555, the system can revert the display to the passage upon the activation of the second trigger in step550. For example, the passage can be reverted to the passage as received by system100in step505, and as provided for display inFIG. 2in the first portion204of the GUI201.

FIG. 6illustrates a flow chart showing passage-mapping steps, item suggestion steps, and new item development steps in a method for item development. In method600, steps605,610,615,620,625,630,635, and640may be similar to steps505,510,515,520,525,530,535, and/or540, respectively, as described in relation to method500ofFIG. 5.

At step645, the system may generate a test item. The test item can be determined by tokenizing the passage and/or determining textual patterns and themes of the passage as described herein. The test item can further be determined by determining a relevant stored item. System100may implement detection and recognition of semantic and linguistic properties of the passage and stored item.

The generated test item can have a passage portion based on the passage. The generated test item passage portion can be based on a portion of the passage which has a higher density of assigned taxonomy codes, most frequent taxonomy code, or certain type of taxonomy code. System100can otherwise use the NLP engine107to alter the difficultly of the passage (for example to take a challenging passage and make it comprehensible to a student of a lower reading level).

The passage portion can be displayed in the first portion205of the GUI201. The test item can also have a multiple choice item. A multiple choice item can also be generated, each portion of which (i.e. a question (stem) portion and an answer portion) can be based on the corresponding portion of the passage, a theme of the passage, and/or the stored item. For example, the question item of the generated item can have semantic and linguistic properties that mirror the question of the identified stored item (e.g. in Question data store116), but with the subjects, nouns, adverbs of the stored item, specifically, around the (e.g. in the same paragraph, sentence, etc.) textual pattern of the passage as identified by the NLP engine107. The selected textual pattern can be in the generated test and/or answer item. The attributes in the attributes data store120can also be used to general the test item. For example, answers can include similar distractor ideas as used in previous developed content.

Other methods of automated reading passage generation are provided in U.S. patent application Ser. No. 16/671,120 filed on Oct. 31, 2019, and titled “Automated Reading Passage Generation,” which is herein incorporated by reference in its entirety for all purposes.

Generated test items can also be added to item data store122, and as entries in the relevant data structure170for use in systems and methods described herein. For example, at step505, the system100may receive a passage containing the sentence “The narrow muzzle of the nodosaur suggests that it had a more selective diet than Ankylosaurus, further indicating ecological separation.” “Suggests that” may be identified as a skill specific textual pattern item at step520, step525, and/or step530. If a stored item which is determined at step535(and optionally displayed at step540) can include a sentence in the passage: “Although paleontologists cannot say for sure that birds helped with the spread and evolution of flowering plants during the last half of the Cretaceous, or that flowering plants aided bird evolution, or that a combination of the two happened, the ecologically tight relationship we see today between these two suggests that they did indeed co-evolve.” The stored item can include a question and answer: “Question: In Passage B, the statement that birds and flowering plants coevolved is presented as . . . . Answer: reasoned judgment based on the ecological relationship between birds and flowering plants.” The system100can generate a prompt415for test item question and answer, and a test item question425and answer430that includes: “In Passage . . . , the statement indicating ecological separation of nodosaur and Ankylosaurus is presented as: . . . a reasoned judgment based on the ecological relationship between the nodosaur and Ankylosaurus.”

System100can generate the test item by applying a trained machine learning item evaluation model, which operates at least on the NLP engine107and/or prediction and learning engine109. Moreover, system100, upon performing steps645and650, system100can receive feedback from reviewer interface138that system100correctly or incorrectly generated the test item. Other useful feedback that can be collected by system100for the generating improved test items include additions, deletions, or comments to the suggested generated items. System100can update the trained machine learning model accordingly to the feedback.

At step650, the system can display the test item. The test item can be displayed as shown inFIG. 4. For example, the display may include a passage (e.g. in first portion205of the GUI), a prompt415for test item a question and answer options, the testable portion420of the test item passage, and test item question425and answer430.

FIG. 7illustrates a flow chart for applying a trained machine learning item evaluation model of a NLP engine to determine theme of a new passage. In method700, a trained machine learning item evaluation model may be implemented to determine a theme of passage.

At step705, the system can receive a training corpus. The training corpus can be received over network134, or can be stored in corpus data store112of system100.

At step710, the system can receive a topic-specific supplemental corpus. The topic-specific supplemental corpus can be received over the network134, or can be in the corpus data store112. The topic-specific supplemental corpus and/or the training corpus, can provide the system100with an indication of the linguistic features of a corpus of a specific topic. For example, a topic-specific supplemental corpus can be a medical dictionary or other field-specific dictionary. The method can alternatively be performed without the step710.

At step715, the system can tokenize the training corpus and/or the supplemental corpus into training words. Tokenizing the training corpus can be performed as previously described with reference to tokenization.

At step720, the system can determine, for example with the machine learning item evaluation model operating at least on the NLP engine107and/or the prediction and learning engine109, at least one training theme corresponding to the training corpus. The NLP engine107can detect and recognize semantic and linguistic properties to assist in determining the training theme.

At step725, the system can receive indications of training themes corresponding to the training corpus.

At step730, the system can receive an indication that NLP engine107and/or the prediction and learning engine109correctly determined the theme at step720. System100can produce this indication by comparing the determined training themes with the received indications in step725. Alternatively, or in addition, method700can include receiving an indication (or otherwise determining) that the machine learning based item evaluation model did not correctly determine the theme at the previous step720. Step705through step730may be repeated until the system is trained to recognize a variety of themes from a variety of training corpuses. The method700can further include updating corpus training sets accordingly.

At step735, the system can receive an input passage (or other passage which is modified from the input passage). This step can be performed as described in relation to steps505of method500ofFIG. 5. and method600ofFIG. 6.

At step740, the system can apply the machine learning based item evaluation model to determine a theme of the input passage. The machine learning based item evaluation model can be implemented by NLP engine107and/or prediction and learning engine109. Thus, systems and methods described herein can be used to train a trained machine learning item evaluation model for evaluating and determining a theme of a passage used to develop assessment items and systems and methods described herein can use the trained machine learning item evaluation model to determine the theme(s) of passages. In addition, if a theme of the passage is provided by the user, the passage can be used to further train the model, for example by using the passage as a training corpus in step705. Alternatively, the steps described herein can be used to determine theme of previously tested items in item data store122. The theme of such stored items can be included as an attribute in attributes data store120. Corpus training sets can accordingly be updated.

FIG. 8illustrates a flow chart showing determining a stored item using the NLP engine. The system may use, for example, a trained machine learning item evaluation model, operating on the NLP engine107and/or prediction and learning engine109. Method800can be used to perform step535and/or step540of method500ofFIG. 5and/or portions ofFIG. 6.

At step805, the system can obtain a training passage or a training set of passages. Training passages can be historical assessment content used in historical testing environments. Training passages can be passages from corpus data store112, stored items in item data store122, or otherwise obtained via the network134, for example by the internet or at field test locations.

System100can train the machine learning item evaluation model to determine the textual patterns in passages stored item and/or query textual pattern data store114. Training the model to identify textual patterns can be performed as described in step810through step825of the method800.

At step810, the system can tokenize the training items (or set thereof) into training words. Tokenizing can be performed as previously described herein.

At step815, the system can determine, for example with the machine learning based item evaluation model operating on the NLP engine107and/or the prediction and learning engine109, textual patterns from the training words.

At step820, the system can receive indications of textual patterns.

At step825, the system can receive an indication that the NLP engine correctly (or incorrectly) identified the textual patterns. System100can produce this indication by comparing the determined textual patterns at step815with the received indications in step820. Method800can further include updating the training set of passages and/or textual pattern data store114as needed. Step805through step825can be repeated until the system is trained to recognize textual patterns reflecting a variety of skills. Method800can be used to train systems100to learn new textual patterns, and new ways of using the textual patterns in passages. System100can identify new textual patterns relevant to developing test items for a variety of training passages.

Further, the relevant stored item (e.g. from item data store122) can be determined by system100based on a weighted contribution of attributes determined by a trained learning model. Steps830through840can refer to training the machine learning based model to determine relevant stored items based on an input passage.

At step830, the system can determine the relevant stored item. The relevant stored item can be a previously developed item in item data store122, and can be relevant to the training passage obtained at step805(or any other training passage, such as a corpus, received by the system100). The relevant previously developed item can be determined by applying prediction and learning engine109and/or NLP Engine107and based on a weighted contribution of textual patterns and/or taxonomy code, theme, and/or attributes. The theme of the training passage can be determined in the various ways previously described (for example in relation to method700ofFIG. 7, or by receiving an indication of a theme from the reviewer). The textual patterns of the training item can be determined in the various ways previously described (for example by tokenizing the training pretest item, querying textual pattern database114, and/or matching the skill specific textual pattern with a taxonomy code as described in relation to step520through step530shown inFIG. 5, or as described in relation to step805through step825of method800). Attributes can be in the attributes data store120, and can be associated with stored items. The weights in the weighted contribution of textual patterns and/or taxonomy code, theme, and/or other attributes, can be in data store111, main memory106, or a program in such memory (for example in the program for the machine learning based model). The relevant stored item can be determined based on the weighted contribution algorithm. Any combination of attributes, theme, and/or presence of skill specific textual patterns, with any weight attributed thereof, can be used to determine the relevant stored item. For example, the presence or absence of a textual pattern can be afforded greater weight than any of the attributes. System100can also determine the second most relevant, or third most relevant (etc.) stored items. The system100can also assign a relevance score to the stored items based on the determined relevance.

At step835, the system can receive an indication of a relevant stored item. System100can also receive an indication of other stored items. System100can provide a prompt or request for receiving this indication.

At step840, the system can receive an indication that the NLP engine and/or the prediction and learning engine correctly (or incorrectly) identified a relevant stored item. This indication can be determined by the system100by comparing the determined relevant stored item at step830, with the indication received at step835. System100can also determine that other relevant stored items were correctly identified (or not). System100and method800can further update the training data, and the various model weights accordingly. The weighted contribution algorithm can be updated based on step830through step840(or other training steps). Steps830through840can be repeated accordingly until the system is trained to determine relevant stored items. Thus, system100can train a machine learning based model to identify a relevant stored item based on weighted contribution algorithm, and subsequently apply that trained machine learning based model to determine one or more relevant stored item(s) for example, at step535and/or step540of method500ofFIG. 5, and/or of method600as shown inFIG. 6. Thus, a robust system that can identify relevant stored items for a variety of input passages can be created.

At step845, the system can receive an input passage. The input passage can be received as shown in steps505of method500ofFIG. 5and of method600ofFIG. 6, and step735of method700ofFIG. 7, including by the reviewer interface138. The theme of the input passage can be determined as indicated in method700as shown inFIG. 7, or otherwise received by system100. Textual patterns of the input passage can be determined as previously described.

At step850, the system can apply a machine learning based item evaluation model, for example operating on NLP engine107and/or prediction and learning engine109to the passage to determine a relevant stored item. Step850can be can be used for performing steps535and/or steps540of method500ofFIG. 5, and/or of method600as shown inFIG. 6. A relevant stored item can be determined as described herein, for example based on a weighted contribution of textual pattern(s), theme, and/or other attribute. A second most relevant, or third most relevant stored item can also be determined.

System100can then further display and annotate the stored item as previously described (e.g. step555) or otherwise perform steps described herein. System100can also receive indications (for example a feedback from the reviewer interface138) that system100correctly identified a stored item and further update the weighted contribution algorithm and/or weights accordingly. Other useful feedback that can be collected by system100for the determining relevant items include additions, deletions, or comments to the suggested relevant items, and knowledge of how the user interacts with system100after the display of the determined relevant items. For example, after display of the stored item, if a user selects another taxonomy code (or skill) from taxonomy code portion217of GUI201, system100can give less weight to the taxonomy code (or skill specific test item) in the future, if the taxonomy code in new passage310is that specific found taxonomy code tested in the determined and displayed item. The system100can update the trained machine learning model, weights, and/or database entries in the data stores111according to the feedback.

Systems and methods disclosed herein can be used to assist in developing and/or automatically generating new test passages, and multiple-choice items.

Systems and methods disclosed herein can be used to identify quantities and/or types of item opportunities across a large corpus, from which an item-rich portion could then be extracted for use as a passage on an assessment or for test prep materials.

Systems and methods disclosed herein can be used in identifying reading item assessment opportunities for passages or corpora.

As used herein, the term engine might describe a given unit of functionality that can be performed in accordance with one or more embodiments of the technology disclosed herein. As used herein, an engine might be implemented utilizing any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routines or other mechanisms might be implemented to make up a engine. In implementation, the various engines described herein might be implemented as discrete engines or the functions and features described can be shared in part or in total among one or more engines. In other words, as would be apparent to one of ordinary skill in the art after reading this description, the various features and functionality described herein may be implemented in any given application and can be implemented in one or more separate or shared engines in various combinations and permutations. Even though various features or elements of functionality may be individually described or claimed as separate engines, one of ordinary skill in the art will understand that these features and functionality can be shared among one or more common software and hardware elements, and such description shall not require or imply that separate hardware or software components are used to implement such features or functionality.

Where components or engines of the technology are implemented in whole or in part using software, in one embodiment, these software elements can be implemented to operate with a computing or processing engine capable of carrying out the functionality described with respect thereto. One such example computing engine is shown, for example, at least in item evaluation server102. Various embodiments are described in terms of this example computing engine. After reading this description, it will become apparent to a person skilled in the relevant art how to implement the technology using other computing engines or architectures.

The terms “computer program medium” and “computer usable medium” are used to generally refer to media such as, for example, memory106, data store111, and bus110. These and other various forms of computer program media or computer usable media may be involved in carrying one or more sequences of one or more instructions to a processing device for execution. Such instructions embodied on the medium, are generally referred to as “computer program code” or a “computer program product” (which may be grouped in the form of computer programs or other groupings). When executed, such instructions might enable the item evaluation server102to perform features or functions of the disclosed technology as discussed herein.

Additionally, with regard to flow diagrams, operational descriptions and method claims, the order in which the steps are presented herein shall not mandate that various embodiments be implemented to perform the recited functionality in the same order unless the context dictates otherwise.