Affective, behavioral, and cognitive processes data collection

Systems and methods are provided for the design and implementation of experiments that facilitate the investigation of process data. The experiments involve recording the completion of a task by participants and then playing back the video of task completion to automatically probe participants about their affective, behavioral, and cognitive experiences. As a result of this system, information about affective, behavioral, and cognitive processes can be more easily investigated by researchers without computer programming knowledge. Corresponding apparatuses, systems, and methods are also discussed.

TECHNICAL FIELD

The subject matter described herein relates to education, learning, and testing, and more specifically to effective generation of experimental tools for data collection of affective, behavioral, and cognitive processes in education and educational testing.

BACKGROUND

Educational researchers use a variety of tools to better understand the processes involved in effective human learning, acquisition of new knowledge, retention of knowledge, and performance during knowledge-based or skills-based testing. Some of these tools involve observing a person during performance of a task. A researcher may be interested in assessing and evaluating affective, behavioral, and cognitive processes during the performance of the task. As part of such evaluation, the person performing the task may be asked to self-evaluate their thoughts, mood, and attitudes during the performance of the task. However, such self-evaluation may distract them from performing the task, and the resulting evaluation may be skewed because of such interruptions. One of the ways to remove such artifacts from the research results is to avoid interruptions during the performance of the task, and instead playback the video taken during the performance of the task after the completion of the task. The person who performed the task can then be asked about their recollection of their affective, behavioral, and cognitive processes during the performance of the task.

A full experiment may therefore involve a human subject performing a task, video recording that performance, playing back the recording with added questions about human subject's recollection of their state of mind during performance of the task. Setting up such an experiment requires careful balancing of multiple objectives under various limitations. Currently known processes for experimental setup are labor intensive and require strong expertise both in cognition and in computer engineering. For these reasons, the existing processes are expensive, long, and difficult to replicate.

Therefore, there is a need in the technical field for systems and methods of fast, inexpensive, and repeatable experimental setup for data collection of affective, behavioral, and cognitive processes used in educational and cognition research.

SUMMARY

Processor implemented methods for evaluating a cognition-measuring effectiveness as a task are provided. Methods may include designing an experiment for a participant. The experiment may be configured to administer a task to the participant, administering the task to the participant, recording a video of the participant performing the task, playing back the video to the participant, collecting affective, behavioral, and cognitive data from the participant during a playback, and rendering data collected during the experiment for review.

Administering the task to the participant may include directing the participant to perform a pre-task activity, providing pre-task directions to the participant, directing the participant to perform a post-task activity, providing pre-playback directions to the participant, and providing end directions to the participant. Administering may also include recording in-task behaviors and verbalizations of the participant.

Processor implemented methods may include prompting the participant to make a judgment of difficulty of the task. Processor implemented methods may also include performing a think aloud protocol that asks the participant to verbalize their thought process while completing the task. Recording a video may include recording the participant's audio and task video.

Designing the experiment may include reviewing introductory information, making experiment design decisions, previewing the experiment, and publishing the experiment. Introductory information may include an overview of a type of experiment, a description of a task in the experiment, and available experiment design options.

Making experiment design decisions may include selecting channels to record during a performance of a task by the participant, selecting the task to record, selecting characteristics of a video playback, selecting stop points, creating stop point questions, linking questions to stop points, selecting video controls to make available during a playback, selecting playback data capture options, and selecting pre- and post-task activities.

Selecting channels to record during a performance of a task by a user may include selecting audio and video channels from a computer and audio and video channels from the participant. Selecting the task to record may include selecting either a desktop application or a web-based application. Selecting a video playback may include selecting either to save the video and not do a playback, or to playback the video.

Selecting stop points may include selecting manual stop points which provide information when the participant chooses to stop the playback, or selecting automatic stop points which provide information at pre-specified points and requiring the participant to provide information in order to resume the playback, or selecting both manual and automatic stop points.

The automatic stop points may include stop points occurring at a pre-defined time interval, stop points occurring at pre-defined time points, or stop points occurring at event-based locations. Creating stop point questions may include selection of a question type, the question type being a single-selection multiple-choice, multiple-selection multiple-choice, or open-ended.

Linking questions to stop points may include assigning the question to the stop point. Selecting video controls to make available during a playback may include selecting whether an observer is watching the playback with the participant. Selecting playback data capture options may include selecting to capture an identification of the participant, a stop point label, a stop point time, a stop point question, or a response of the participant.

Processor implemented methods may include comparing data collected during the playback with data collected from other participants, or calculating a scoring metric indicating an effectiveness of the experiment in assessing cognitive information from the participant.

The subject matter described herein provides many technical advantages. For example, data from affective, behavioral, and cognitive processes can be collected quickly, effectively, and with repeatability. Further, the experimental setup may be done with a high degree of automation. This, in turn, may allow a broader range of researchers to design and perform cognitive experiments that utilize information about the process of task completion.

DETAILED DESCRIPTION

Systems and methods described herein may facilitate collection of affective, behavioral, and cognitive processes data when users interact with a task. Specifically, systems and methods described herein may provide researchers with the ability to set up an automated system for experiment delivery, capture audio and/or video data of users (e.g., participants) interacting with a task (e.g., computer-based assessment or learning activity), and collect process data from users as they watch a video playback of their interaction with the task. Systems and methods described herein may also provide a user-friendly interface for researchers to set up their experiment. The interface of some embodiments may allow researchers without programming knowledge or skills to design and create a data collection that would otherwise require computer-programming expertise. In addition, some embodiments may provide researchers with information pertaining to advantages and disadvantages for different selection options in the data collection process. Thus, researchers may be scaffolded through the process of designing their experiment. Next, we first discuss the type of experiment that can be created, followed by a discussion of how systems and methods described herein enable researchers to design such an experiment.

FIG.1is a flow diagram illustrating steps of activities according to one exemplary embodiment. For experiment delivery, system100may link together a variety of activities for participants to complete. This linking of activities may involve automatically presenting each activity and/or coordinating the transition between activities. The automatic presentation and transition between activities may allow researchers to collect data from more than one participant simultaneously and/or without monitoring by researchers.FIG.1shows the flow of activities that can occur in an experiment according to an embodiment. It is important to note that the activities depicted inFIG.1are all optional, with the exception of the “Task”104. This means that researchers may have flexibility with respect to what occurs in their experiment.FIG.1is the flow of activities from the perspective of the person participating in the experiment. We will discuss the flow of events for designing an experiment by the researcher after further description of the activities inFIG.1.

After an experiment start101, certain pre-task activity102may be conducted. Pre-task activity102may include any one or more of the following: survey, presentation of information, task that will not be recorded, and another activity that the researcher would like to have participants complete.

After pre-task activity102, directions before task103may be set up. Directions before task103may include an opportunity to provide instructions to participants before they begin the task.

After directions before task103, task104may be set up. Task104may include target activity that participants may complete during the experiment. Task104may be web-based or a desktop application that can range from an interactive virtual world to a slide show, in terms of complexity. Task104is the only activity in the experiment that may be recorded and the only required activity for all experiments created with system100.

After task104, post-task activity105may be set up. Post-task activity105may include any one or more of the following: survey, presentation of information, task that will not be recorded, and another activity that the researcher would like to have participants complete. Post-task activity105may occur at the same time that the task video is being processed.

After post-task activity105, directions before video playback106may be set up. Directions before video playback106may include an opportunity to provide instructions to participants before they begin the video playback.

After directions before video playback106, video playback107may be set up. Video playback107may include participants watching a playback of the video of themselves completing the task. The video can include up to four channels that include audio and video from the participant and from the computer (i.e., task). The video may also include more than four channels. The playback107may stop at pre-specified points in the video and may pose pre-specified questions to participants. The pre-specified stop points and questions may be defined by the researcher during the experiment design.

After video playback107, directions at end108may be set up. Directions at end108may include an opportunity to provide instructions to participants at the completion of the experiment. After directions at end108, there may be an end of experiment109.

The general data collection design provided by systems and methods of some embodiments may allow for the further investigation of participants' experience with the task without interrupting the typical way in which participants would interact with the task. Prompting participants to make judgments of difficulty, for example, during the task may alter the way in which they experience the task. The video playback in systems and methods of some embodiments may allow for this kind of prompting with the full context of the interaction provided by the video. Retrospective judgments of this kind can allow for more frequent prompting than would be possible for prompts during the task. However, the retrospective judgments of systems and methods of some embodiments can also be combined with in-task judgments or verbalizations. A think aloud protocol that asks participants to verbalize their thought process while completing a task, for example, can be combined with a video playback by recording participants' audio and task video (as well as other channels, if needed). The video playback can then be utilized for further reflection or for researchers to pose follow-up questions based on participants' verbalizations during the task. Next, we describe how researchers design and create their experiment using systems and methods of some embodiments.

FIG.2is a flow diagram illustrating steps of experimental design200according to another exemplary embodiment. Systems and methods of some embodiments may have a user-friendly interface that may walk researchers through the process of designing and creating their experiment, without the need of computer-programming skills.FIG.2shows the general flow of events that may occur when researchers design their experiments. Experiment design in some embodiments may begin, after the experiment design start201, with introductory information202that may provide an overview of the type of experiment that can be designed with systems and methods of some embodiments as well as descriptions of each major activity in the experiment and/or the options available to researchers. Next, researchers may begin designing their experiment by making selections203.

Exemplary design decisions will be described further below, along with screenshots of the interface according to some embodiments. In some embodiments, selections may include one or more of the following: channels to record204, a task to be recorded205, use of video playback206(yes path2098or no path207), stop points209, playback data210, and additional options211.

Lastly, researchers may be able to preview the experiment they have designed at212. At this point in the process researchers can either publish their project, which would create the software for experiment delivery and end the experiment design213, or they can return to the experiment design process and make any revisions needed. Next, we describe some experiment design decisions available to researchers as shown inFIG.2.

FIG.3is a diagram illustrating selection of channels to record according to another exemplary embodiment. Selection of channels may be facilitated by an interactive display300, which may include a top menu301, a side bar menu302, an instructions field303, a navigation menu304, and a general menu305. The top menu301may include a name of a system and a name of a current project. The side bar menu302may include quick access blocks that may enable navigation to one of the other steps of the experiment design. The instructions field303may display instructions to a researcher for setting up this portion of the experiment. The navigation menu304may provide buttons for moving to a previous step of the experiment setup and/or to a subsequent step of the experiment setup. The general menu305may be a drop down menu of commands and/or navigational options, which may assist the researcher in developing the experiment setup.

The first design decision that researchers may make is what channel(s) to record when participants complete the task. In an embodiment, there are four available channels that can be selected. These channels may include a selection panel306for selecting audio and video channels from the computer and a selection panel307for selecting audio and video channels from the participant. Researchers can select any combination of channels that are pertinent for their experiment.

In some embodiments, additional information may be recorded and may be available for playback. Such additional information may include one or more of the following: additional audio data; additional video data; eye tracking; physiological sensors; pressure sensor from chairs.

Interactive display300may also provide tips button308that may facilitate the decision making process for researchers. The selected channels may be recorded during task completion and then may be combined into one video file. Tips button308may provide user with one or more of the following reminders.

Researcher may select the recording options that are most important for their study. The recording options that are most appropriate may depend on one or more of the following: the task being recording; the type of data to collect; the total time for data collection session. For example, if the task to be recorded does not include any sound, then the audio option for screen capture may not need to be selected.

When considering the type of data to collect, it may not be necessary to include all available recording options. For example, for a think aloud protocol, only the audio option for study participant may be needed if the facial expressions, movements, and gestures of the participant are not relevant to the goals of the study. However, if those behaviors are important for the goals of the study and a think aloud protocol is included, then both the audio and video options for study participant may be included.

The total session time may be considered, because the number of selected options, particularly the number of video options, may increase the video processing time. On the view time estimate and refine choices if needed step, it may be possible to see how the selected options impact the video processing time and contribute to the overall study session time. It may be important to carefully consider the nature of the task and the goals of the study when selecting the recording options.

FIG.4is a diagram illustrating combining participant video and computer video channels according to another exemplary embodiment. Screen400shows a combination of video channels from the participant and the computer screen during an experiment that was designed with the system according to an embodiment. Screen400may include a top menu401, a task display402, an experiment layout403, an experiment transcript404, and a participant video405. Elements402,403, and404may all be part of the task that was completed by the participant in one web browser window.

FIG.5is a diagram illustrating selection of a task to record according to another exemplary embodiment. Selection of task to record may be facilitated by an interactive display500, which may include a top menu501, a side bar menu502, an instructions field503, a navigation menu504, and a general menu505. The next design decision that researchers must make may be what task to record when completed by participants. Researchers may have the option to select either a desktop application506or a web-based application507. A desktop application506can vary in complexity from the presentation of a virtual world to a slide show of static images, and anything in between. For web-based applications507, researchers can select the type of browser508to be used for the task. For both types of tasks, researchers may also be asked to provide an estimate509of the average time to complete the task and to provide a short name510for the task. The task time estimate509may be used later for estimating the total experiment time.

FIG.6is a diagram illustrating selection of a video playback according to another exemplary embodiment. Selection of a video playback may be facilitated by an interactive display600, which may include a top menu601, a side bar menu602, an instructions field603, a navigation menu604, and a general menu605. The next design decision that researchers may make is whether they will have participants complete the video playback during the experiment,606. If researchers choose to save the video and not do a playback during the experiment,607, they may jump to the selection of additional options, otherwise they may continue to make design decisions for the video playback as shown inFIG.2.

FIG.7is a diagram illustrating selection of stop points according to another exemplary embodiment. Selection of stop points may be facilitated by an interactive display700, which may include a top menu701, a side bar menu702, an instructions field703, a navigation menu704, and a general menu705. If researchers have selected to include a video playback in their experiment (choice606inFIG.6), then the next design decisions may involve selecting how that video playback will occur. Specifically, researchers may decide when the stop points (i.e., collect process information from participants) will occur, what question(s) will be posed, how those questions will be linked to stop points, and what video controls will be available to participants.

Researchers may have three options for when stop points will occur during the video playback. Stop points can occur only manually706, only automatically707, or both manual and automatically709. Manual stop points may involve participants only providing information when they choose to stop the video. Automatic stop points, on the other hand, may be pre-specified by the researchers and may cause the video to automatically stop and require participants to provide information in order to proceed.FIG.7shows a screen700of one type of automatic stop point selected as well as enabling the use of manual stop points. When manual stop points are present, researchers can select the label for the button (e.g., Unmotivated in section709ofFIG.7) that participants may click to provide information.

There are three types of automatic stop points that researchers can choose from to include in the video playback. The first stop point type is shown inFIG.7and these may occur at a regular time interval. These time interval stop points may occur at any interval pre-defined by researchers (e.g., every 30 seconds, every 1 minute, etc.). The second stop point type may also be time-based, but these may occur at pre-defined time points specified by researchers. A particular task may have important events that occur at 2 minutes, 5 minutes, and 10 minutes, for example. In this case the researcher can set these time-based locations to be stop points. The third stop point type may be event-based locations. Event-based locations may be derived from task log files, which may enable researchers to have a stop point at the same event for each participant, even if the event occurs at different times for each participant. For example, researchers may want to gather additional process information for every participant when they complete question two in the task. In some embodiments, only one of these automatic stop point types can be selected for any video playback.

The screen700may provide tips708about when and why certain automatic stop points may be more or less advantageous. Tips button708may provide user with one or more of the following reminders.

Three types of automatic stop points may be described. Each of these automatic stop points can be combined with manual stop points. Repeated stop at a selected time interval may mean that during playback the video will automatically stop and present one question to the participant at the designated time interval. The same question may be presented at each automatic stop point. The time interval could be every 30 seconds, every minute, every 5 minutes, or any time of researcher's choosing.

Stop at specific locations in the video may mean that during playback the video may automatically stop and present one question to the participant at the designated time-based locations in the video. For example, researcher can have the video stop at 1:00 (one minute) and 2:45 into the video for all participants and present a question. At each time-based location researcher can present a different question to participants. Researcher may be limited to a maximum number of time-based locations (e.g., 5, 10, 20, or other number of locations); however, there may be no restrictions on the locations of time-based locations.

Stop at specific events in the video may mean that during playback the video may automatically stop and present one question to the participant at the designated event in the video. The events may come from the log file generated by the task. For example, researcher can have the video automatically stop when every participant is asked Question2, regardless of when Question2occurs for each participant. At each event researcher can present a different question to participants. Each event can occur once, multiple times, or not at all depending on the log file for a particular participant. Researcher may be limited to a maximum number of events (e.g., 5, 10, 20, or other number of events).

When determining what type of automatic stop point to use, it may be important to consider (a) the total task time, (b) how many questions researcher wants to pose to participants, and (c) the total study session time. Consideration of these three elements of the study may allow the researcher to determine how often it may be appropriate to present a question, what type(s) of questions may be appropriate to present, and if the researcher will be able to collect all of the necessary information during the total study session time.

FIG.8is a diagram illustrating creation of a stop point question according to another exemplary embodiment. Creation of a stop point question may be facilitated by an interactive display800, which may include a top menu801, a side bar menu802, an instructions field803, a navigation menu804, and a general menu805. A question may be posed to participants at each stop point during the video playback in order to gather process information. Researchers can choose between three types of questions, selected at a choice entry area806. One question type may be single-selection multiple-choice in which the participant can only select one option when responding (seeFIG.8). Researchers enter the text for their question and can include directions for the question and as many answer choice options as needed, in question entry area807. The second question type may be multiple-selection multiple-choice, which may be the same as the single-selection multiple-choice except that participants can select more than one option when responding. The third question type may be open-ended in which participants may provide a text-based response to the question. Researchers may also be asked to estimate the amount of time needed to answer the question in order to facilitate estimating the total experiment time.

FIG.9is a diagram illustrating linking questions to stop points according to another exemplary embodiment. Linking questions to stop points may be facilitated by an interactive display900, which may include a top menu901, a side bar menu902, an instructions field903, a navigation menu904, and a general menu905. After researchers have decided when stop points will occur and the question(s) that will be posed during the playback, they may decide how the question(s) may be linked to the stop points. In other words, researchers may decide what question may be shown when a particular stop point occurs. The linking of questions and stop points may vary based on the previous options selected by researchers for when stop points will occur.FIG.9shows the linking process when time interval automatic stop points and manual stop points have been selected (seeFIG.8). In this embodiment, researchers can only pose two different questions to participants during the playback, one question when the automatic stop point occurs and another question when the manual stop point occurs (see stop options area906inFIG.9). In the case of time-based and event-based locations, researchers can link a different question to each event, although a number of questions researchers may present at any given stop point may be restricted to one, two, four, or another number of questions. Text of questions may be edited by selecting the question from the question entry area907.

FIG.10is a diagram illustrating selection of video controls to make available according to another exemplary embodiment. Selection of video controls may be facilitated by an interactive display1000, which may include a top menu1001, a side bar menu1002, an instructions field1003, a navigation menu1004, and a general menu1005. The next video playback decision that researchers may make is what types of video controls to make available during the playback. There may be two general options shown inFIG.10that may differ on who is present during the video playback. If an observer or researcher is with the participant while watching the video playback (option1006), more video control options can be enabled. However, if only the participant is watching the video playback (option1007), then there may be a more restricted set of video control options. Specifically, researchers can choose to allow participants to fast forward and rewind based on the locations of stop points (i.e., participants cannot fast forward past an automatic stop point) and pause the video while watching, in video control selector1008. Researchers can also select to display the stop points on the video timeline, which may allow participants to know when a stop point will occur before the video stops.

FIG.11is a diagram illustrating selection of playback data capture options according to another exemplary embodiment. Selection of playback data capture options may be facilitated by an interactive display1100, which may include a top menu1101, a side bar menu1102, an instructions field1103, a navigation menu1104, and a general menu1105. If researchers have selected to do a video playback during their experiment, the next decision they may make is what data may be captured during the video playback.FIG.11shows the different data capture options available to researchers. There may be five pieces of data that can be automatically captured at each stop point: participant identification (e.g., Participant4), stop point label (e.g., manual, automatic), stop point time (i.e., time point in the video), stop point question, and participant response. There may be three additional pieces of data that can be captured as well if selected in playback data capture selector1106. The first option may be the participant response time, which may capture how long the participant takes to respond to the stop point question. The second option may be participant actions, which may capture when participants use any of the video control options that are available during the video playback. The third option may be participant response actions that occur when the participant is responding to the stop point question (e.g., select option A and then switch to option C). Researchers can also view sample data outputs of the different pieces of data in order to better understand how the data output will be formatted.

FIG.12is a diagram illustrating selection of pre and post-task activities according to another exemplary embodiment. Selection of pre and post-task activities may be facilitated by an interactive display1200, which may include a top menu1201, a side bar menu1202, an instructions field1203, and a navigation menu1204. The last design decisions that researchers may make relate to optional activities to occur during the experiment. Specifically, these may be pre- and post-task activities and direction screens (seeFIG.2). Researchers can utilize some, all, or none of these optional activities.FIG.12shows how researchers may setup pre-task activities (entry field1206) and post-task activities (entry field1206) that may be web-based, which may include providing the URL and a time estimate for activity completion. Templates for direction screens may be provided to suggest important information to relay to participants. However, researchers can replace the template information with their own as well.

After researchers have made all of the experiment design decisions in the previous steps, they may get two previews of their experiment. This information can then be used to make any necessary changes or proceed to finishing the experiment for experiment delivery. One preview may be an estimated experiment timing as shown inFIG.13. Another preview may be a visual presentation of the video playback as shown inFIG.14.

FIG.13is a diagram illustrating estimation of experiment timing according to another exemplary embodiment. Estimation of experiment timing may be facilitated by an interactive display1300, which may include a top menu1301, a side bar menu1302, an instructions field1303, and a navigation menu1304. At multiple points throughout the experiment design, researchers may be asked to provide time estimates for the different activities that participants may complete. Estimates may be provided for high-level activities1310, or for sub-activities1311. In an embodiment, the following time estimates1312may be provided: pre-task and task time, including activity1313, directions1314, and record task1315; post-task time, including activity1316and directions1317; video playback time, including video playback1318and directions1319. The information provided by researchers may then be used to calculate an estimate for the total time1309needed for one participant to complete the experiment.FIG.13shows how the total time estimate is computed as well as the design decisions that have been made by the researcher. Researchers can also edit any previous design decisions and those design changes may be used to automatically update the total time estimate.

Researchers may be provided with tips1308about how they can reduce the total time estimate if it goes beyond their time constraints for an experiment session. Tips1308may include one or more of the following. The time estimate for the processing section may only be modified by changes to the time estimated to complete the task that is recorded. The estimated processing time may not be updated based on changes to the recording options (e.g., removing video option for study participant). Time estimates for processing may vary based on the computer being used to complete the data collection. The current time estimate provided for processing may likely be at the higher end for the potential total processing time. It is important to test out researcher's project on the computer that will be used for the data collection in order to get a more accurate estimate of the processing time.

If the time estimate for the total time is too long, researcher may consider modifying one or more of the following: the number of stops during playback; the type of question(s) researcher is asking; the length of the pre-task or post-task activity. Researcher may allow for a range of completion times to insure that all participants can complete the study.

FIG.14is a diagram illustrating a video playback preview according to another exemplary embodiment. In an embodiment, the preview of the video playback may involve a visual presentation of the experiment design decisions that have been made by the researcher.FIG.14shows the preview screen1421for the video playback. In an experimental design of this embodiment, the researcher has selected to record video of both the task (1422) and the participant (1423), include time interval automatic and manual stop points, allow participants to pause the video, and display the automatic stop point locations on the video timeline. In addition, the question that is linked to the automatic stop point, with directions, may be shown (1424), as it would be to participants. Researchers can then close the preview and make any necessary changes to the design decisions.

FIG.15is a diagram illustrating experiment design end screen according to another exemplary embodiment. Experiment design end screen1500may include a top menu1501, a side bar menu1502, an instructions field1503, a navigation menu1504, and a general menu1505. After researchers have viewed the two experiment previews and have finalized their design decisions, they can either save the experiment (1506) or save and publish the experiment (1507).FIG.15shows a screenshot of these two options. If researchers select to save and publish the experiment (1507), they may be provided with information about the next steps for conducting their experiment. The next steps may include information about the structure of the software for experiment delivery as well as tips about testing the software prior to the actual data collection. After publishing their experiment, researchers may only need to return to the experimental setup if they need to make experiment design changes or to create a new experiment.

Once the experiment is designed, researcher may proceed with conducting the experiment. This may include administering a task to a participant. It may include video playback to the participant. It may include collecting affective, behavioral, and cognitive data either during the performance of the task, or during the video playback, or both. In some embodiments, data collected during the performance of the task or the playback may be compared with data collected from other participants.

In some embodiments, a scoring metric is calculated. Scoring metric may be calculated after the performance of the task or after the video playback. In some embodiments, the scoring metric may indicate an effectiveness of the experiment in assessing cognitive information from the participant.

FIGS.16A,16B, and16Cdepict example systems for implementing the approaches described herein for affective, behavioral, and cognitive processes data collection. For example,FIG.16Adepicts an exemplary system1600that includes a standalone computer architecture where a processing system1602(e.g., one or more computer processors located in a given computer or in multiple computers that may be separate and distinct from one another) includes a computer-implemented digital recording1604being executed on the processing system1602. The processing system1602has access to a computer-readable memory1607in addition to one or more data stores1608. The one or more data stores1608may include existing experiments1610as well as previously generated tasks1612. The processing system1602may be a distributed parallel computing environment, which may be used to handle very large-scale data sets.

FIG.16Bdepicts a system1620that includes a client-server architecture. One or more user PCs1622access one or more servers1624running an appropriate-level digital recording1637on a processing system1627via one or more networks1628. The one or more servers1624may access a computer-readable memory1630as well as one or more data stores1632. The one or more data stores1632may include existing experiments1634as well as previously generated tasks1638.

FIG.16Cshows a block diagram of exemplary hardware for a standalone computer architecture1650, such as the architecture depicted inFIG.16Athat may be used to include and/or implement the program instructions of system embodiments of the present disclosure. A bus1652may serve as the information highway interconnecting the other illustrated components of the hardware. A processing system1654labeled CPU (central processing unit) (e.g., one or more computer processors at a given computer or at multiple computers) may perform calculations and logic operations required to execute a program. A non-transitory processor-readable storage medium, such as read only memory (ROM)1658and random access memory (RAM)1659, may be in communication with the processing system1654and may include one or more programming instructions for performing the method of generating experiments for an individual. Optionally, program instructions may be stored on a non-transitory computer-readable storage medium such as a magnetic disk, optical disk, recordable memory device, flash memory, or other physical storage medium.

InFIGS.16A,16B, and16C, computer readable memories1607,1630,1658,1659or data stores1608,1632,1683,1684,1688may include one or more data structures for storing and associating various data used in the example systems for generating experiments for an individual. For example, a data structure stored in any of the aforementioned locations may be used to store data from XML files, initial parameters, and/or data for other variables described herein. A disk controller1690interfaces one or more optional disk drives to the system bus1652. These disk drives may be external or internal floppy disk drives such as1683, external or internal CD-ROM, CD-R, CD-RW or DVD drives such as1684, or external or internal hard drives1685. As indicated previously, these various disk drives and disk controllers are optional devices.

Each of the element managers, real-time data buffer, conveyors, file input processor, database index shared access memory loader, reference data buffer and data managers may include a software application stored in one or more of the disk drives connected to the disk controller1690, the ROM1658and/or the RAM1659. The processor1654may access one or more components as required.

A display interface1687may permit information from the bus1652to be displayed on a display1680in audio, graphic, or alphanumeric format. Communication with external devices may optionally occur using various communication ports1682.

In addition to these computer-type components, the hardware may also include data input devices, such as a keyboard1679, or other input device1681, such as a microphone, remote control, pointer, mouse and/or joystick.

Additionally, the methods and systems described herein may be implemented on many different types of processing devices by program code comprising program instructions that are executable by the device processing subsystem. The software program instructions may include source code, object code, machine code, or any other stored data that is operable to cause a processing system to perform the methods and operations described herein and may be provided in any suitable language such as C, C++, JAVA, for example, or any other suitable programming language. Other implementations may also be used, however, such as firmware or even appropriately designed hardware configured to carry out the methods and systems described herein.