Abstract:
The Learner Interaction Monitoring Systems (LiMS) is a web-based application that can interface with any web-based course delivery platform to transform the online learning environment into an active observer of learner engagement. The LiMS ‘event capture model’ collects detailed real-time data on learner behavior in self-directed online learning environments, and interprets these data by drawing on behavioral research. The LiMS offers education and training managers in corporate contexts a valuable tool for the evaluation of learner performance and course design. By allowing more detailed demonstration of ROI in education and training, LiMS allows managers to make the case for web based courseware that reflects appropriate and evidence-based instructional design, rather than budgetary constraints.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This patent application claims the benefit of, under Title 35, United States Code, Section 119(e), U.S. Provisional Patent Application No. 61/503,810, filed Jul. 1, 2011, the content of which is incorporated herein by reference. 
     
    
     FIELD OF THE INVENTION 
       [0002]    The present invention is directed to a learner interaction monitoring system, which captures the behaviors of online learners and evaluates online training courses. 
       BACKGROUND OF THE INVENTION 
       [0003]    Businesses around the world are increasingly recognizing the need to support continuous professional development and skills acquisition and foster lifelong learning in their workforce, in order to maintain competitiveness in the global marketplace. Many have recognized the potential benefits of online training and education: the flexibility offered by the web-based learning environment allows working adult learners to engage with course materials “any time, any place, any path, any pace”. Well-designed online training courses promise to make education and training available in ways that fit the busy work and life schedules of employees, and almost two decades of research into online learning has demonstrated that there is no measurable significant difference in learning outcomes between face to face and online learning modalities. In the corporate context, where exam results and course grades rarely exist as measures of learner achievement or effective online course design, it falls to training and education managers to identify reliable and valid approaches to evaluating both course design and learner performance, in order to demonstrate a significant return on the sizable investments needed to implement high quality online learning. Unfortunately, few easily implementable approaches exist. As has been noted, “in spite of the best efforts of organizations and the professional trainers&#39; associations, there are significant problems in evaluating the true impact of [online] training”. 
         [0004]    Similarly, while the value of feedback from and to learners in educational settings is well-established, most corporate training departments lack the tools to gather accurate learner feedback (direct or indirect) about their online learning experience or activities. Such information is critical in evaluating whether training courses are meeting corporate educational needs and goals. In the absence of evaluative tools that return meaningful and easily interpretable data, corporate training departments are more likely to deliver web based courseware that simply reflects budgetary restrictions, rather than appropriate and evidence-based instructional design. 
         [0005]    U.S. Pat. No. 7,131,842 to Hollingsworth, et al. disclose a method for measuring the effectiveness of a school&#39;s educational program. The method involves observing classroom teaching of students at the institution, and analyzing the observed teaching to determine effectiveness based on teacher time on task, teacher instructional effectiveness, alignment of student work to standards, and breadth of coverage of student work. However, Hollingsworth, et al. is directed to analyzing teaching and student work a traditional classroom environment, and does not teach capturing user interactions with a course delivery interface during online learning. The method does not incorporate real-time and/or ongoing analysis, but rather, only analyzes collected data after the observation is complete. 
         [0006]    U.S. Patent Application Publication No. 2011/0167013 to Pogue, et al. discloses a method of online classroom quality control involving monitoring the interaction between an online instructor and online students to determine quality of teaching. However, Pogue, et al. is primarily directed toward evaluating the performance of an instructor, and the criteria used for establishing performance include only pass rate, grades, survey results, student and teacher log-in amounts and participation. Pogue et al. is not directed to utilizing event capture to derive behavioral observations relating to course design, to granular response latency relating to dilemma and behaviors that serve an automatic function, or user interactivity with secondary resources. 
         [0007]    U.S. Patent Application Publication No. 2010/0291528 to Huerta discloses a monitoring device configured to capture real-time participant events and stimuli in a learning environment. However, the data capture is primarily audio-visual, and the device is directed primarily to natural language interpretation of speech and analyzing biometric information in a live classroom environment to evaluate the performance of a teacher by identifying educational strategies. Huerta is not directed to an online learning environment, and does not teach capturing user interactions with a course delivery interface during online learning. 
         [0008]    U.S. Patent Application Publication No. 2010/0190143 to Gal, et al. discloses a system which monitors, logs, and reports the performance of students based on their operation of molecular digital learning objects at a workstation. The system may determine whether objectives are completed within defined time periods, whether certain percentages of questions are completed correctly, or whether a student has not operated a keyboard or mouse for a particular time period. However, Gal, et al. is not directed to utilizing event capture to derive behavioral observations relating to course design, to granular response latency relating to dilemma and behaviors that serve an automatic function, or to user interactivity with secondary resources. 
         [0009]    U.S. Patent Application Publication No. 2009/0035733 to Meitar, et al. discloses a computer program for monitoring the progress of tutors and students in a classroom environment. Meitar et al. teach allocation of learning activities to students stored data relating to scoring and timing of student testing. However Meitar et al. is not directed to utilizing event capture to derive behavioral observations relating to course design, to granular response latency relating to dilemma and behaviors that serve an automatic function, or to user interactivity with secondary resources. 
         [0010]    A U.S. Patent Application Publication No. 2005/0053908 to Satheesh, et al. discloses a computer program for monitoring the progress of tutors and students in a classroom environment. Satheesh et al. teach the analysis of stored data relating to scoring and timing of student testing. However, Satheesh et al. is not directed to utilizing event capture to derive behavioral observations relating to course design, to granular response latency relating to dilemma and behaviors that serve an automatic function, or to user interactivity with secondary resources. 
       SUMMARY OF THE INVENTION 
       [0011]    Accordingly, it is an object of the present invention to provide a system and method for monitoring user interactions with an interactive learning system which overcome these deficiencies. 
         [0012]    These and other objectives are achieved by providing a system for monitoring a course of study including a server computer; a network; a user computer in communication with the server computer over the network; software executing on the server computer to transmit a course of study to said user computer over the network; software executing on said user computer to receive the course of study from said server computer and to present the course of study to a user of said user computer; software executing on said user computer to capture an interaction of the user with the course of study as interaction data; and an analysis module receiving the interaction data and generating a behavioral observation using the interaction data. 
         [0013]    In some embodiments, a second interaction of the user with the course of study is captured as second interaction data; and, the behavioral observation is generated using both the interaction data and the second interaction data. 
         [0014]    In some embodiments, a second interaction of the user with the course of study is captured as second interaction data; and, a second behavioral observation is generated using both the interaction data and the second interaction data. 
         [0015]    In some embodiments, the analysis module compares the interaction data with baseline data to generate the behavioral observation. 
         [0016]    In some embodiments, the course of study is presented to a user of the user computer using a web browser. 
         [0017]    In some embodiments, the user interactions with the course of study are captured using a web browser plug-in. 
         [0018]    In some embodiments, the interaction data includes event data and timestamp data. 
         [0019]    In some embodiments, the analysis module determines a reading time based upon the interaction data. 
         [0020]    In some embodiments, the analysis module compares the reading time with baseline data to generate the behavioral observation. 
         [0021]    In some embodiments, the behavioral observation reflects that the user is having trouble understanding a reading passage if the reading time is greater than a baseline reading time. 
         [0022]    In some embodiments, the behavioral observation reflects that a reading passage is problematic if a reading time for more than one user is greater than a baseline reading time. 
         [0023]    In some embodiments, behavioral observation reflects that the user is devoting insufficient time to a task if the reading time is less than a baseline reading time. 
         [0024]    In some embodiments, the analysis module determines a response latency based upon the interaction data. 
         [0025]    In some embodiments, analysis module compares the response latency with baseline data to generate the behavioral observation. 
         [0026]    In some embodiments, the behavioral observation reflects that the user is experiencing a dilemma if the response latency is greater than a baseline response latency. 
         [0027]    In some embodiments, the analysis module determines if a user has accessed a resource. 
         [0028]    In some embodiments, the analysis module generates a behavioral observation based upon whether the user has accessed the resource. 
         [0029]    In some embodiments, the analysis module generates a behavioral observation based upon whether the user has performed an additional interaction subsequent to accessing the resource. 
         [0030]    In some embodiments, the behavioral observation reflects that the user is disinterested in the resource if the user has not accessed the resource. 
         [0031]    In some embodiments, the analysis module updates the baseline data based upon the behavioral observation. 
         [0032]    In some embodiments, the analysis module updates the baseline data to create updated baseline data based upon a second behavioral observation of a second user. 
         [0033]    In some embodiments, the system includes a reporting module receiving the behavioral observation from the analysis module and generating a report using the behavioral observation. 
         [0034]    In some embodiments, the report comprises a behavioral grade based on the behavioral observation. 
         [0035]    In some embodiments, the report comprises a descriptive profile of the user. 
         [0036]    In some embodiments, the report comprises a comparison among more than one user. 
         [0037]    In some embodiments, the course of study comprises text, a test, a quiz, or an examination. 
         [0038]    In some embodiments, the analysis module comprises software executing on an analysis computer that is in communication with the user computer over the network. 
         [0039]    In some embodiments, the analysis module comprises software executing on the server computer. 
         [0040]    In some embodiments, the analysis module comprises software executing on the user computer. 
         [0041]    Other objects of the present invention are achieved by providing a system for monitoring a course of study including a first computer; presentation software executing on the computer to present a course of study to a user; interactions between the user and the first computer which relate to the course of study; analysis software executing on the first computer and analyzing the interactions to generate behavioral observations; and, reporting software executing on the first computer to generate a report of the behavioral observations. 
         [0042]    In some embodiments, the presentation software, the analysis software, or the reporting software executes on a second computer in communication with the first computer. 
         [0043]    Still other objects of the present invention are achieved by providing method of monitoring a course of study by providing a user computer; presenting a course of study to a user of the user computer using presentation software executing on the computer; monitoring user interactions with the course of study using monitoring software; analyzing the user interactions with the course of study to generate behavioral observations using analysis software. 
         [0044]    In some embodiments, the user computer is connected to a network and the course is transmitted to the computer from a server computer. 
         [0045]    In some embodiments, the system presents the behavioral observations as a report using reporting software. 
         [0046]    In some embodiments, the behavioral observations are used to modify the course of study. 
         [0047]    In some embodiments, the behavioral observations are used to modify the course of study while the course of study is in progress. 
         [0048]    In some embodiments, the behavioral observations are used to modify the course of study with respect to other users of the course of study. 
         [0049]    To meet the need for a robust web-based training evaluation tool, we have developed the Learner Interaction Monitoring System (LiMS) application. LiMS is a two-part web-based plug-in application that can interface with any web-based course delivery platform to transform the online learning environment into an active observer of learner engagement with course materials. Unlike the minimalist tracking tools packaged with standard Learning Management Systems (LMSs), LiMS purposely captures fine-grained data on learner activity and behaviors within the learning environment, turning the course itself into an active receiver of indirect learner feedback. Given that most web-based corporate training is self-paced, self-directed and is not instructor-led, this capacity already makes LiMS unique. 
         [0050]    Typically, LMSs capture only very limited learner performance data such as student scores and course completion status. At best, such data merely demonstrate base-level learner participation. Until recently, this has been sufficient in the business environment, where the primary role of an online course has been to satisfy regulatory training requirements. 
         [0051]    While some LMSs now capture and archive more extensive learner tracking data, they offer only very limited data reporting options, and access to archived data has only been possible via slow and cumbersome manual processes. Even more problematic is the reality that existing commercial LMSs provide little or no guidance for educators and managers to indicate which (if any) of the captured tracking variables may be pedagogically meaningful—that is to say, which of the available data points are indicative of student participation in educationally purposeful activity that may contribute to their learning or success. Today, training administrators are nevertheless expected to evaluate learner competence and course effectiveness by examining primitive or largely inaccessible data sets, in the absence of any interpretive guidance. 
         [0052]    The Learner Interaction Monitoring System (LiMS) now offers administrators a means of capturing rich and detailed real-time information about individual learner behaviors in an online learning environment, and makes use of evidence-based interpretive approaches to permit meaningful evaluation of learners and courses. As has been recognized, “ . . . you can always choose not to analyze data you have collected, but you can never analyze what you do not collect”. Most LMSs fail to collect data that can reveal information about both learners and course design. By collecting user course engagement events, such as mouse clicks or movements acting upon individual page elements such as buttons, checkboxes and lists, LiMS ensures that the detailed course interaction data of the user experience is captured for reference. Going far beyond the capture of simple event sequencing, LiMS also captures data reflecting the variable behavioral characteristics of those actions such as duration, timing and response latency. Much like a teacher in a live classroom, LiMS virtually “watches” what the student is doing in an online class and commits those actions to memory by recording the behavioral data to an SQL database. Importantly, LiMS implementation and continuing development builds on existing research to permit pedagogically meaningful interpretation of captured data. 
         [0053]    LiMS leverages a growing body of research confirming the relationships between online behaviors and learner performance, and builds on the resultant understanding that learner behavior in online learning environments offers a rich source of indirect feedback on both their learning strategies and experience, and effectiveness of course design. 
         [0054]    In the past decade, the field of ‘academic analytics’ has emerged, as a small number of investigators have begun to employ business intelligence tools and processes in the analysis of data sets captured by institutional LMSs. Studies of interactive, instructor-facilitated online courses in the higher education sector have begun to demonstrate significant positive correlations between selected simple measures of student engagement with online course content, tools and peers, and a student&#39;s ultimate achievement in a course, as measured by final course grade. While this work has highlighted the real relationships between online behaviors and learner performance, the level of individual learner analysis remains crude. Moreover, the cited studies reveal that in the group-study contexts of higher education, learner-learner interaction measures are the best predictors of success. Continuing work in this line has less to offer the world of self-directed online education and training. 
         [0055]    LiMS builds upon the basic findings of academic analytics, and extends these to the world of online corporate training by developing a detailed map of an individual learners engagement with online course materials, coupled with evidence-based interpretation. 
         [0056]    In face-to-face training environments, educators intuitively evaluate learners or lessons by observing learner activity and reactions. Learners may demonstrate interest in a subject, for example, by contributing verbal comments or by identifying themselves as eager to participate by raising their hand. Disinterested or disengaged learners are often quiet, and reluctant to engage in dialogue. Proponents of alternate assessment approaches propose that multiple behavioral indicators can and should be employed to evaluate learner performance and the effectiveness of teaching methods. They point to psychological research that supports pedagogically meaningful interpretation of a range of learner behaviors. Such research suggests that in the classroom, as well as online, the timeliness and sequence of a learner&#39;s actions contribute to an individual&#39;s learning profile. They argue that motivated educators can make use of such observations to adjust their teaching techniques and approach in order to better engage the quiet or struggling student, as well as to acknowledge and further challenge the interested. To date, visual observation of learner engagement has been largely unavailable in asynchronous learner-directed online training environments. 
         [0057]    We contend that new methods of capturing learner behavior data, such as the online behaviors recorded by LiMS, can provide some of the same valuable information as live teacher observation and can permit richer more fine-grained evaluation of learner performance online. The separation in space and time between the learner and teacher (or instructional designer) in distance learning contexts, first characterized as ‘transactional distance’, is routinely positioned as problematic. However, in online learning environments this exact condition of relative isolation may permit revelation of the learner&#39;s “true self” and allow individuals to express “phenomenally real aspects of self not often or easily expressed to others.” In the un-monitored isolation of a self-directed online course, the supposedly unobserved actions of a learner may reveal behavioral information that is much more reliable than data collected via self-reporting and self-evaluation mechanisms. 
         [0058]    “Data by itself is not information. For [an] evaluation to be credible, it has to inform and have an impact.” 
         [0059]    In a LiMS-enabled learning environment, start time is captured as soon as a learner opens a web-based course page. All elements (radio buttons, text boxes, hyperlinked text, etc) are identified on the page and reported to the database. For comprehensive elements (select boxes, list boxes, etc) each possible individual selection is recorded as an attribute of that parent element. All text (initially visible) is word counted, to allow calculation of a predicted reading completion time. Each ‘event’—mouse click, mouse-over, response change—is captured and the time noted, to establish whether the event was ‘early’ or not. Hyperlinked URLs are captured when the text is selected—‘time opened’ is stamped (as well as ‘time closed’, to establish length of time of event). When elements are interacted with by the mouse, the duration of the event is captured and a ‘behavior’ is assigned to that duration (i.e. pause, hover, etc). 
         [0060]    At the completion of each online training experience, LiMS assigns a ‘behavioral grade’ to the learner reflecting their approach to the training material when compared to a standard established by LiMS itself. At course completion, each learner, by default, is assigned a beginning learning behavioral grade of A+. As the learner&#39;s course behavior is reviewed by the LiMS engine, the grade is impacted by interpretation of captured criteria: 1.) Did the learner consistently read all of the text on each page? 2.) Did the learner access optional content when available? 3.) Did the learner answer all questions correctly? 4.) Did the learner access optional content that offered guidance to answering questions correctly? 5.) Did the learner adjust their behavioral approach during the course thereby positively impacting their performance? 
         [0061]    LiMS adjusts the learner&#39;s grade using an algorithm that computes a final assigned ‘grade’ reflecting their behavioral approach to online training materials. 
         [0062]    A descriptive profile of the learner is generated based on the course grade and the behavioral data, and is posted on the student&#39;s report page. For example, text-based reported may include statements such as: “Student X rarely accesses optional data even when it might result in performance improvement”. 
         [0063]    LiMS implementation can then be customized to allow educational designers to ask targeted questions about learner choices within a course, or to track learner behavior in relation to key course material items or events of interest. In relation to learner behaviors, for example, educators may wish to ask questions such as: Are learners spending sufficient time reading particular core course text materials? Do my learners display differential response latency to key questions, and can this provide insight into comprehension or decision making style? Additional comparison measures permits benchmarking against peers or course norms. 
         [0064]    Example: Reading Time 
         [0065]    The time interval between reading completion and initiation of a sequential event (for example, completion of a quiz), may be a useful discrepancy indicator in learner or course evaluation. Native speakers of English in North America typically read at a rate of 250-300 words per minute, when reading for comprehension. LiMS assumes a baseline reading rate of 300 words per minute, or 5 words per second, allowing the calculation of predicted reading time for all text-based course materials. As a learner moves through web-based course text, LiMS not only captures reading times for text blocks, but offers interpretive analysis describing whether a learner appears to be proceeding normally, too slowly or too quickly through reading materials. Such information can then be considered in relation to learner performance (if reading is apparently completed ‘too quickly’, is a learner giving insufficient time to the task?) and to course design (if learners regularly take too much time on selected text blocks, do these need rewriting for clarity?). 
         [0066]    Example: Response latency 
         [0067]    Similarly, response latency may also function as a discrepancy indicator. Research suggests, for example, that response latency “may be a useful measure for behaviors that serve an automatic function”. That is to say, where response to an item (for example, a quiz question), is slow (shows increased response latency), it can be hypothesized that the responder (the learner) is experiencing a dilemma. Consider, for example, a web based Ethics course in which a learner is asked to make a decision about whether or not to accept a small gift from a vendor despite company policy strictly forbidding gifts. Reading the question and selecting an answer is expected to take no more than 12 seconds, whether the learner agrees with the statement or not. If the time to complete the task falls within the expected time period, then the learner can be assumed to be committed to their answer, regardless of their selection. If the task completion time was longer than expected, we might infer a struggle in choice selection, indicating a cognitive or behavioral dilemma. 
         [0068]    In this scenario, there may be value in investigating why the learner took extra time answering the question. Research has demonstrated, for example, that response time for wrong answers is longer than for correct answers and that “respondents holding unstable attitudes need more time to respond to an opinion question than respondents who hold stable attitudes.” Further investigation may be needed to discover whether course materials explaining company policy are insufficiently clear, or whether a particular learner is in genuine disagreement with or confusion about the policy. 
         [0069]    Example: Using Additional Resources 
         [0070]    Continuing with the above scenario, suppose the question was reconstructed as: ‘Would you consider accepting a gift under $100 from a vendor?’ This time the organization&#39;s policy information about receiving gifts is hidden in a popup window. A button labeled “additional information” is included on the page that, once selected, reveals the organization&#39;s gift receiving policy. This additional interactive option introduces the possibility of considering additional behavioral indicators: Did the learner display interest in “additional information”? If selected, was the additional content open long enough to be read or closed instantly? If the button was not selected, what was the sequential action? In other words, adding additional learning resources offers greater potential for behavior tracking and interpretation. 
         [0071]    Integrating course materials and learning activities into online courses using more sophisticated media and tools (linked resources, multi-media, interactive activities) not only engages learners more effectively, but it also creates richer and more sophisticated opportunities to capture business intelligence and gain insight into learner behavior and course effectiveness. Highly interactive courses are more expensive to design and implement. Failure to follow evidence-based instructional design principles, however, not only reduces educational effectiveness but also means that meaningful opportunities for real-time learner performance and course evaluation are lost. 
         [0072]    Conclusions 
         [0073]    The supposedly unobserved nature of the learner experience in the online training and education environment affords unique possibilities for evaluation. Capturing the sequence, duration and timing of user interactions with course materials and learning opportunities in real-time permits the development of a learning behavioral profile of each individual, and interpretive metrics can be run on the performance of individuals, groups, and course content and structure. Collection and analysis of robust data sets permits greater intelligence, responsiveness and reactivity to be integrated into web-based courses. LiMS paves the way to fully evaluating students and courseware with data resources to support robust training development that is at once engaging, productive and fiscally valuable. 
         [0074]    Moreover, we believe that there is enormous potential for real-time intervention within a course that is “LiMS enabled”. As the observer of student interactions, LiMS data can provide the resources for a course to become a ‘coach’ or learning facilitator. By accessing captured student behavioral data the possibility exists to programmatically customize the online training environment in real-time, thereby adjusting the manner in which content is presented on screen or altering the functionality of a course. 
         [0075]    As has been noted, if the expense of online training cannot be validated “those investments in training will be more likely [sic] rejected by the organization&#39;s senior management.” 
         [0076]    LiMS provides important justification for the budgets necessary to build dynamic courses in supportive environments. It provides a solid and multidimensional route to a more effective demonstration of ROI by coupling its analytic and interpretive capabilities to well designed interactive courseware and allow educational managers to mine the resultant intelligence. Early adopters of LiMS are working with our programmers and interface designers to enhance the presentation of behavioral data, as we continue to refine the tool. 
         [0077]    Other objects of the invention and its particular features and advantages will become more apparent from consideration of the following drawings and accompanying detailed description. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0078]      FIG. 1  is a block diagram of an example system illustrating aspects of the invention. 
           [0079]      FIG. 2  is a block diagram of another example system illustrating aspects of the invention. 
           [0080]      FIG. 3  is a block diagram of a further example system illustrating aspects of the invention. 
           [0081]      FIG. 4  is a block diagram of still another example system illustrating aspects of the invention. 
           [0082]      FIG. 5  is a block diagram of still a further example system illustrating aspects of the invention. 
           [0083]      FIG. 6  is an illustration of an example reporting dialog according to aspects of the invention. 
           [0084]      FIG. 7  is an illustration of another example reporting dialog according to aspects of the invention. 
           [0085]      FIG. 8  illustrates another example reporting dialog according to aspects of the invention. 
           [0086]      FIG. 9  illustrates another example reporting dialog according to aspects of the invention. 
           [0087]      FIG. 10  illustrates another example reporting dialog according to aspects of the invention. 
           [0088]      FIG. 11  illustrates another example reporting dialog according to aspects of the invention. 
           [0089]      FIG. 12  illustrates another example reporting dialog according to aspects of the invention. 
           [0090]      FIG. 13  illustrates another example reporting dialog according to aspects of the invention. 
           [0091]      FIG. 14  illustrates an example course participant interaction dialog illustrating aspects of the invention. 
           [0092]      FIGS. 15 and 16  illustrate a further example system according to aspects of the invention 
           [0093]      FIGS. 17-27  illustrate example database tables according to aspects of the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0094]      FIG. 1  is a block diagram of an example system  100  illustrating aspects of the invention. System  100  is directed to presenting an interactive course of study to a user and analyzing the user&#39;s behavior with respect to the course of study using captured events. User interface  120  can communicate with an interactive learning application  110  by receiving and transmitting course content and interaction data  130 . For example, the application  110  can present content to user interface  120 , and the user can interact with the content using the user interface  120 . The user&#39;s interactions are transmitted back to the application  110 . System  100  may be implemented using various combinations of computer hardware and software as well as various computer communications networks as is known in the art. 
         [0095]    Content and interactions  130  are also acquired by Learner Interaction Monitoring System (LIMS)  170 . LIMS  170  includes a monitoring function  140 , an analyzing function  150 , and a reporting function  160 . 
         [0096]    In an example operation, monitoring function  140  acquires the content and stores attributes of the content. For example, if the content is a block of text, monitoring function  140  may store the number of words. Monitoring function  140  also acquires user interaction events with the content, for example, by recording the presentation of content to the user, mouse clicks on page elements such as buttons, checkboxes and lists, other mouse events, scrolling events, keystrokes, and other event objects, as well as a timestamp for each. This timestamp can reflect either an actual time, system time, or a time relative to other monitored events, and may be implemented as an event handler. Monitoring function  140  may store the content and event data in a database such as an SQL database. 
         [0097]    Analyzing function  150  analyzes the data acquired by the monitoring function  140 , and uses this information to generate behavioral observations about the user&#39;s interaction with the course material. For example, analyzing function  150  may assign a baseline reading rate for a passage based on the number of words stored by monitoring function  140 . Typically, this will be 300 words per minute for a native speaker of English in North America. Analyzing function  150  may also calculate the time the user spent reading the passage as the difference in timestamp values between the time at which the reading passage was presented to the user and the time at which the user clicked a button to move on from the reading passage using a mouse, for example. By comparing the time required to read the passage with the baseline reading rate for the passage, the analyzing function  150  can generate a behavioral observation. For example, if the reading time was less than the baseline, it may be observed that the reader is not devoting sufficient time to the task. 
         [0098]    In some applications, the various baselines used by analyzer  140 , and behavioral observations about the user derived from the baselines, can be adjusted in real time as data is acquired during the operation of system  100 . For example, if analyzer  140  initially determines that a user is devoting insufficient time to reading a passage because the user&#39;s reading completion time is less than the baseline, this determination can later be updated if the user&#39;s reading rates for later passages are consistent with the reading rate for this passage, and if the user&#39;s scores for quizzes subsequently taken by the user relating to the passages are consistent with a thorough understanding of the material. 
         [0099]    In this example case, a passage which the user was initially determined to have properly read for understanding based on comparison of the reading time to the baseline may later be determined to have been poorly designed or difficult for the user to understand. This may be due to the fact that even though this course participant read the passage in a time consistent with the baseline, the user&#39;s reading time was longer than the user&#39;s particular adjusted baseline reading rate, and therefore, the user was in fact spending more time than usual trying to understand the passage. 
         [0100]    Analysis function  140  can also assign a behavioral grade to the user at the end of the course based upon the behavioral observations made during the course. The behavioral grade reflects an assessment of the user&#39;s behavioral approach to the course materials. For example, the user can initially be assigned the highest possible grade by default, with grade deductions for behavioral observations which reflect that the user did not consistently read all of the text presented, did not access optional content where available, did not answer all questions correctly, did not access optional content that offered guidance to answering questions correctly, or did not adjust their behavioral approach during the course to positively impact their performance, for example. These observations are simply examples, and many other types of behavioral observations can be used to determine a behavioral grade. 
         [0101]    The analysis function may also keep track of information that is not behaviorally specific in this context, such as raw scoring data for quizzes. This information can then be captured in real time or obtained from other components of the system after the course is completed by the user. This information can be used by the analysis section in combination with other information to generate behaviorally specific information and/or observations. For example, a particular score on a quiz question may not be considered to be behaviorally specific in a given context, but if this information provides insight into the reasons for a user&#39;s reading time for a related passage of text, it can be used to distinguish whether a behavioral observation that either the user is struggling with the material, or that the passage of text is worded in such a way as to be difficult to understand is appropriate. 
         [0102]    Reporting function  160  receives behavioral observations and optionally a behavioral grade generated by analysis module  150 , and generates reports based on the received information. 
         [0103]    Reports may include a descriptive profile of the learner that is generated based on a course grade and the behavioral data. For example, the descriptive profile may include statements such as: “Student X rarely accesses optional data even when it might result in performance improvement.” This statement would be an appropriate behavioral observation if the reporting function  160  receives information from the analysis function  150  indicating that the user did not access the optional data when given the opportunity, and that the user subsequently did not correctly answer quiz questions relating to this material, for example. 
         [0104]    Optionally, the reports may contain some or all of the behavioral observations for a given user relating to course content  130 , for all courses taken, or for a subset of all courses taken, for example. The reports may also contain non-behaviorally specific information relating to quizzes taken, and/or other specific observations for the student. 
         [0105]    Further, the reports may optionally contain a comparison of behavioral observations for all users using the course content  130 , or a subset of all of the students compared. The reports may compare how much time each student spent taking a course, how much time each student spent taking a quiz, course or quiz grades, an overall course or user effectiveness rating, and other such comparisons. The comparison data may be presented as raw data or may be presented graphically using charts and diagrams. 
         [0106]    The reports may also show a success to failure rate of the users of a course overall and by elements of the course. For example, success rates can be displayed by course, chapter, or attempts for a given user. 
         [0107]    The reports may also provide a quiz overview displaying the performance of the students on a quiz in whole or in part, statistics for each question, derived effectiveness of the question based on score and behavioral observations of all users, and other observations. 
         [0108]    The following table lists example learning behaviors that can be observed by the system, and example ways in which the observations are made by the system, for example, by event capturing: 
         [0000]    
       
         
               
               
               
             
           
               
                   
               
               
                   
                 Based on scored and 
                   
               
               
                 Learning Behaviors 
                 weighted actions 
                 Measurement 
               
               
                   
               
             
             
               
                 Focus/Distraction 
                 Text Reading Approach 
                 Reading Speed (words per second) 
               
               
                   
                   
                 initially measured against 5.5 words 
               
               
                   
                   
                 per second standard. Subsequently 
               
               
                   
                   
                 measured against an Individualized 
               
               
                   
                   
                 Comprehension Standard (ICS). The 
               
               
                   
                   
                 ICS is based on a personal average 
               
               
                   
                   
                 aggregate reading rate directly 
               
               
                   
                   
                 correlating to personal 
               
               
                   
                   
                 topic/concept comprehension. The 
               
               
                   
                   
                 ICS establishes both a range for 
               
               
                   
                   
                 speed and similarity of approach 
               
               
                   
                   
                 (when experiencing text of near 
               
               
                   
                   
                 equal construction) and meeting 
               
               
                   
                   
                 the ICS indicates focus. 
               
               
                   
                 Navigation 
                 Courses are ideally constructed to 
               
               
                   
                   
                 present concepts and content to 
               
               
                   
                   
                 enable a scaffolding of knowledge - 
               
               
                   
                   
                 with an expectation of a sequential 
               
               
                   
                   
                 learning approach. Course 
               
               
                   
                   
                 navigation is thereby expected to be 
               
               
                   
                   
                 sequential (page one, page two, 
               
               
                   
                   
                 page three, etc.), with minimal 
               
               
                   
                   
                 intra-directional navigation (page 
               
               
                   
                   
                 one, page two, page three, page 
               
               
                   
                   
                 two). Intra-directional navigation 
               
               
                   
                   
                 indicates lack of focus/attention as 
               
               
                   
                   
                 well as a possible need to revisit 
               
               
                   
                   
                 information experienced earlier OR 
               
               
                   
                   
                 an issue in experiencing 
               
               
                   
                   
                 concepts/content in opposition to 
               
               
                   
                   
                 Instructional Design. 
               
               
                   
                 Web Browser 
                 eLearning courses are experienced 
               
               
                   
                   
                 through the web browser 
               
               
                   
                   
                 environment. Additional web 
               
               
                   
                   
                 windows (child windows) may be 
               
               
                   
                   
                 spawned from the course window 
               
               
                   
                   
                 (parent, but the learner is not 
               
               
                   
                   
                 expected to leave the browser 
               
               
                   
                   
                 environment unless they exit the 
               
               
                   
                   
                 course. Leaving the browser 
               
               
                   
                   
                 environment to engage other 
               
               
                   
                   
                 applications indicates multi-tasking 
               
               
                   
                   
                 rather than a preference to stay on- 
               
               
                   
                   
                 task. 
               
               
                   
                 Seat Completion 
                 Seat time indicates the length of 
               
               
                   
                   
                 time the learner takes to complete a 
               
               
                   
                   
                 single course. Courses ideally are 
               
               
                   
                   
                 developed to be completed in one 
               
               
                   
                   
                 “seating”. Multiple exits and 
               
               
                   
                   
                 entrances into a course (multiple 
               
               
                   
                   
                 seating experiences) indicate and 
               
               
                   
                   
                 inability to complete a course in a 
               
               
                   
                   
                 focused, time managed manner. 
               
               
                   
                 Task Completion 
                 During a course a learner is both 
               
               
                   
                   
                 required and requested to complete 
               
               
                   
                   
                 certain tasks. Completing tasks 
               
               
                   
                   
                 indicates a focus on bringing 
               
               
                   
                   
                 assignments to a close. Measured 
               
               
                   
                   
                 tasks include closing opened child 
               
               
                   
                   
                 browser windows, completing 
               
               
                   
                   
                 optional assessments (likert ratings) 
               
               
                   
                   
                 other survey style challenges (fill- 
               
               
                   
                   
                 in-blank). 
               
               
                   
                 Response Time 
                 Interaction Response Time (IRT) 
               
               
                   
                   
                 represents the optimal time 
               
               
                   
                   
                 required (as a range-of-time 
               
               
                   
                   
                 standard) to understand a 
               
               
                   
                   
                 challenge, make a decision and act 
               
               
                   
                   
                 on that decision. IRT is irrespective 
               
               
                   
                   
                 of assessment result (correct vs. 
               
               
                   
                   
                 incorrect), only indicating attention 
               
               
                   
                   
                 to prioritizing the experience of the 
               
               
                   
                   
                 challenge. IRT indicates issue 
               
               
                   
                   
                 comprehension and response 
               
               
                   
                   
                 selection. 
               
               
                 Inquisitive/Indifferent 
                 Optionals 
                 Courses ideally include optional 
               
               
                   
                   
                 elements requiring an interaction to 
               
               
                   
                   
                 reveal hidden content (invisible OR 
               
               
                   
                   
                 located “off screen”). Optionals can 
               
               
                   
                   
                 be accessed via hyperlinks which 
               
               
                   
                   
                 expose content in new windows, 
               
               
                   
                   
                 buttons that reveal documents or 
               
               
                   
                   
                 actioned elements (text or graphics) 
               
               
                   
                   
                 that expose singular OR grouped 
               
               
                   
                   
                 elements. Optionals are usually 
               
               
                   
                   
                 content or concepts that are 
               
               
                   
                   
                 presented in addition to the core 
               
               
                   
                   
                 content or concepts of the course, 
               
               
                   
                   
                 to enhance the learning experience 
               
               
                   
                   
                 or further clarify topic information. 
               
               
                   
                   
                 Optional selection indicates an 
               
               
                   
                   
                 interest in expanding 
               
               
                   
                   
                 comprehension beyond the minimal 
               
               
                   
                   
                 requirements. 
               
               
                   
                 Embedded Optionals 
                 Primary Optionals (POs) may 
               
               
                   
                   
                 include Embedded Optionals (EOs). 
               
               
                   
                   
                 The selection of a PO may reveal a 
               
               
                   
                   
                 group of elements which may 
               
               
                   
                   
                 include an EO (a hyperlink that, 
               
               
                   
                   
                 when selected, opens supplemental 
               
               
                   
                   
                 content in a new browser window 
               
               
                   
                   
                 (Child window). Selection of an EO 
               
               
                   
                   
                 indicates an increase in a weighted 
               
               
                   
                   
                 interest in comprehension. 
               
               
                   
                 Web Browser 
                 eLearning courses are experienced 
               
               
                   
                   
                 through the web browser 
               
               
                   
                   
                 environment. Additional web 
               
               
                   
                   
                 windows (child windows) may be 
               
               
                   
                   
                 spawned from the course window 
               
               
                   
                   
                 (parent, but the learner is not 
               
               
                   
                   
                 expected to leave the browser 
               
               
                   
                   
                 environment unless they exit the 
               
               
                   
                   
                 course. Leaving the browser 
               
               
                   
                   
                 environment to engage other 
               
               
                   
                   
                 applications indicates multi-tasking 
               
               
                   
                   
                 rather than a preference to stay on- 
               
               
                   
                   
                 task. 
               
               
                   
                 Audio 
                 When available - the positive 
               
               
                   
                   
                 engagement in controlling audio 
               
               
                   
                   
                 indicates an interest in concept or 
               
               
                   
                   
                 content clarification 
               
               
                   
                   
                 (comprehension). Positive audio 
               
               
                   
                   
                 control engagement is indentified 
               
               
                   
                   
                 by actions such as: Play, Pause, 
               
               
                   
                   
                 Replay, Reverse Scrub. 
               
               
                   
                 Video 
                 When available - the positive 
               
               
                   
                   
                 engagement in controlling video 
               
               
                   
                   
                 indicates an interest in concept or 
               
               
                   
                   
                 content clarification 
               
               
                   
                   
                 (comprehension). Positive video 
               
               
                   
                   
                 control engagement is indentified 
               
               
                   
                   
                 by actions such as: Play, Pause, 
               
               
                   
                   
                 Replay, Reverse Scrub. 
               
               
                   
                 Paige Turner 
                 If Paige Turner support/control is 
               
               
                   
                   
                 enabled, any positive engagement 
               
               
                   
                   
                 with this Training Education 
               
               
                   
                   
                 Assistive CHaracteR (TEACHR) 
               
               
                   
                   
                 indicates an interest in content or 
               
               
                   
                   
                 concept clarification. Positive 
               
               
                   
                   
                 engagement is identified as a 
               
               
                   
                   
                 request for help (summon PT to the 
               
               
                   
                   
                 web screen). 
               
               
                 Motivated/Disinterested 
                 Launch 
                 Courses usually have an “available 
               
               
                   
                   
                 window” during which they can be 
               
               
                   
                   
                 experienced by a learner (a 
               
               
                   
                   
                 beginning and an end date). 
               
               
                   
                   
                 Launching a course within the first 
               
               
                   
                   
                 quarter of the available window 
               
               
                   
                   
                 time indicates an interest in 
               
               
                   
                   
                 engaging the assigned training in a 
               
               
                   
                   
                 timely manner. 
               
               
                   
                 Completion 
                 Courses usually have an “available 
               
               
                   
                   
                 window” during which they can be 
               
               
                   
                   
                 experienced by a learner (a 
               
               
                   
                   
                 beginning and an end date). 
               
               
                   
                   
                 Completing a course within the first 
               
               
                   
                   
                 quarter of the available window 
               
               
                   
                   
                 time indicates an interest in 
               
               
                   
                   
                 finishing the assigned training in a 
               
               
                   
                   
                 timely manner. 
               
               
                   
                 Text Reading Approach 
                 Reading Speed (words per second) 
               
               
                   
                   
                 initially measured against 5.5 words 
               
               
                   
                   
                 per second standard. Subsequently 
               
               
                   
                   
                 measured against an Individualized 
               
               
                   
                   
                 Comprehension Standard (ICS). The 
               
               
                   
                   
                 ICS is based on a personal average 
               
               
                   
                   
                 aggregate reading rate directly 
               
               
                   
                   
                 correlating to personal 
               
               
                   
                   
                 topic/concept comprehension. The 
               
               
                   
                   
                 ICS establishes both a range for 
               
               
                   
                   
                 speed and similarity of approach 
               
               
                   
                   
                 (when experiencing text of near 
               
               
                   
                   
                 equal construction) and meeting 
               
               
                   
                   
                 the ICS indicates motivation to read 
               
               
                   
                   
                 for comprehension. 
               
               
                   
                 Optional Engagement 
                 A tendency to engage the majority 
               
               
                   
                   
                 of optional elements within a course 
               
               
                   
                   
                 indicates an increased level of 
               
               
                   
                   
                 interest in exploring possible routes 
               
               
                   
                   
                 of content or concept clarification 
               
               
                   
                   
                 or enhancement. Engagement does 
               
               
                   
                   
                 not equate comprehension. 
               
               
                   
                 Assessment 
                 Meeting the minimal requirements 
               
               
                   
                   
                 for passing an assessment indicates 
               
               
                   
                   
                 an interest to meet at least the 
               
               
                   
                   
                 minimal standard established for 
               
               
                   
                   
                 content or concept comprehension. 
               
               
                   
                   
                 Intrinsic motivation is indicated 
               
               
                   
                   
                 when passing an assessment in the 
               
               
                   
                   
                 first attempt. 
               
               
                   
                 Seat Time 
                 Successfully completing a course 
               
               
                   
                   
                 within the Targeted Comprehension 
               
               
                   
                   
                 Time (TCT) indicates a desire to 
               
               
                   
                   
                 finish an assigned task within an 
               
               
                   
                   
                 expected time frame while 
               
               
                   
                   
                 presenting the expected behaviors. 
               
               
                   
                   
                 TCT is established by establishing a 
               
               
                   
                   
                 minimum length of time required to 
               
               
                   
                   
                 complete all required tasks within 
               
               
                   
                   
                 the course. 
               
               
                 Confidence/Unsure 
                 Response Time 
                 Interaction Response Time (IRT) 
               
               
                   
                   
                 represents the optimal time 
               
               
                   
                   
                 required (as a range-of-time 
               
               
                   
                   
                 standard) to understand a 
               
               
                   
                   
                 challenge, make a decision and act 
               
               
                   
                   
                 on that decision. IRT is irrespective 
               
               
                   
                   
                 of assessment result (correct vs. 
               
               
                   
                   
                 incorrect), only indicating attention 
               
               
                   
                   
                 to prioritizing the experience of the 
               
               
                   
                   
                 challenge. IRT indicates issue 
               
               
                   
                   
                 comprehension and response 
               
               
                   
                   
                 selection. 
               
               
                   
                 Approach 
                 Challenges require us to identify a 
               
               
                   
                   
                 best course of action and then 
               
               
                   
                   
                 initiate that action within a specific 
               
               
                   
                   
                 time frame (see IRT above). 
               
               
                   
                   
                 Approach measures if the learner 
               
               
                   
                   
                 adheres to their initial course of 
               
               
                   
                   
                 action or, in reconsideration, 
               
               
                   
                   
                 modifies their initial response based 
               
               
                   
                   
                 on no additional information. 
               
               
                   
                   
                 Modification indicates 
               
               
                   
                   
                 reconsideration irrespective of 
               
               
                   
                   
                 challenge outcome (positive or 
               
               
                   
                   
                 negative) 
               
               
                   
                 Success 
                 Measured as meeting or surpassing 
               
               
                   
                   
                 the minimal requirements of an 
               
               
                   
                   
                 assessment or challenge on the first 
               
               
                   
                   
                 attempt. Indicates commitment and 
               
               
                   
                   
                 confidence. 
               
               
                   
                 Optional Engagement 
                 A tendency to engage the majority 
               
               
                   
                   
                 of optional elements within a course 
               
               
                   
                   
                 indicates an increased level of 
               
               
                   
                   
                 interest in exploring possible routes 
               
               
                   
                   
                 of content or concept clarification 
               
               
                   
                   
                 or enhancement. Engagement does 
               
               
                   
                   
                 not equate comprehension. 
               
               
                   
                 Hints 
                 Many times a challenge will include 
               
               
                   
                   
                 a “hint” which, when selected, will 
               
               
                   
                   
                 reveal information to assist the 
               
               
                   
                   
                 learner in successfully meeting the 
               
               
                   
                   
                 challenge. Confident learners will 
               
               
                   
                   
                 not select these optional hints. 
               
               
                 Comprehension/ 
                 Text Reading Approach 
                 Reading Speed (words per second) 
               
               
                 Misunderstanding 
                   
                 initially measured against 5.5 words 
               
               
                   
                   
                 per second standard. Subsequently 
               
               
                   
                   
                 measured against an Individualized 
               
               
                   
                   
                 Comprehension Standard (ICS). The 
               
               
                   
                   
                 ICS is based on a personal average 
               
               
                   
                   
                 aggregate reading rate directly 
               
               
                   
                   
                 correlating to personal 
               
               
                   
                   
                 topic/concept comprehension. The 
               
               
                   
                   
                 ICS establishes both a range for 
               
               
                   
                   
                 speed and similarity of approach 
               
               
                   
                   
                 (when experiencing text of near 
               
               
                   
                   
                 equal construction) and meeting 
               
               
                   
                   
                 the ICS indicates intent to read for 
               
               
                   
                   
                 comprehension. 
               
               
                   
                 Response Time 
                 Successful Interaction Response 
               
               
                   
                   
                 Time (SIRT) represents the optimal 
               
               
                   
                   
                 time required (as a range-of-time 
               
               
                   
                   
                 standard) to understand a 
               
               
                   
                   
                 challenge, make a CORRECT 
               
               
                   
                   
                 decision and act on that decision. 
               
               
                   
                   
                 SIRT IS result dependent (correct vs. 
               
               
                   
                   
                 incorrect). SIRT indicates issue 
               
               
                   
                   
                 comprehension and CORRECT 
               
               
                   
                   
                 response selection. 
               
               
                   
                 Success 
                 Measured as meeting or surpassing 
               
               
                   
                   
                 the minimal requirements of an 
               
               
                   
                   
                 assessment or challenge on the first 
               
               
                   
                   
                 attempt. Indicates commitment and 
               
               
                   
                   
                 confidence. 
               
               
                   
                 Seat Time 
                 Successfully completing a course 
               
               
                   
                   
                 within the Targeted Comprehension 
               
               
                   
                   
                 Time (TCT) indicates a desire to 
               
               
                   
                   
                 finish an assigned task within an 
               
               
                   
                   
                 expected time frame while 
               
               
                   
                   
                 presenting the expected behaviors. 
               
               
                   
                   
                 TCT is established by establishing a 
               
               
                   
                   
                 minimum length of time required to 
               
               
                   
                   
                 complete all required tasks within 
               
               
                   
                   
                 the course. 
               
               
                   
                 Discrimination 
                 Courses may include optional 
               
               
                   
                   
                 elements that require the learner to 
               
               
                   
                   
                 select them to display the hidden 
               
               
                   
                   
                 content. Some content may or may 
               
               
                   
                   
                 not present information the learner 
               
               
                   
                   
                 requires to assist in comprehension. 
               
               
                   
                   
                 Optional Content Discrimination 
               
               
                   
                   
                 (OCD) indicates a learner is able to 
               
               
                   
                   
                 discern which information is helpful 
               
               
                   
                   
                 and which is not required. Positive 
               
               
                   
                   
                 OCD is measured by a high variance 
               
               
                   
                   
                 in length of time optional content is 
               
               
                   
                   
                 open based on the time required to 
               
               
                   
                   
                 determine value. 
               
               
                   
               
             
          
         
       
     
         [0109]    LiMS  170  may optionally be implemented as a group of discrete components, modules, or functions. For example, the interactive learning application  110 , user interface  120 , monitoring function  140 , analyzing function  150 , and/or reporting function  150  may each be implemented as a different application, with one or more of the applications running on separate computers which are in communication with one another, or may be implemented as separate programs running on the same computer or as functions of the same program running on a computer, for example. 
         [0110]    Various permutations of the system topology which do not depart from the spirit of the invention will be clear to those having skill in the art. For example,  FIG. 2  shows an example system  200  which functions similarly to system  100 , except in that example system  200  is explicitly implemented as one discrete LiMS unit  270 . In this example, interactive learning system  210 , user interface  220 , monitoring function, analyzing function  250 , and reporting function  260 , as well as all communications among these units, are integrated into one discrete system. 
         [0111]      FIG. 3  is a block diagram of a system  300  which is similar to systems  100  and  200 , and illustrates an implementation of aspects of the invention in a typical Internet communications architecture. For example, interactive learning application  310  can be web based, and be served from a web server over a network  390 . Network  390  can be the Internet, a local area network, wide area network, mobile communications network, or any other suitable computer communications network. User interface  320  can be a computing device connected to network  390 , and may incorporate a web browser or other client interface which can interact with application  310  over network  390 . Content and interactions  330  transmitted from application  310  to user interface  320  over network  390  are captured by monitoring function  340  and utilized by analysis function  350  and reporting function  360  in a similar fashion as in systems  100  and  200  ( FIGS. 1 and 2 ). The monitoring function  340 , analysis function  350  and reporting function  360  are shown incorporated into a LiMS system  370  which communicates with the interactive learning application  310  and user interface  320  over network  390 , but it will be clear to those having skill in the art that various topological implementations are possible as further discussed herein. 
         [0112]      FIG. 4  illustrates an example system  400  similar to systems  100   200 , and  300  ( FIGS. 1 ,  2 , and  3 ) which incorporates a web browser plug-in  490 . 
         [0113]    Course server  410  is in communication with user computer  420  over network  480 , and presents course content  430  to the user of user computer  420  via web browser  425 . The user can interact with the course content  430  using web browser  425 , generating events  435 . 
         [0114]    Some or all of events  435  can optionally be transmitted to the course server  410  over network  480 . For example, if the user clicks a button in web browser  425  to advance to the next page of a reading, data corresponding to the mouse click is transmitted from the browser  425  to the course server  410  in order to signal completion and may prompt the course server  410  to transmit the next part of the course content  430 . In some implementations of system  400 , it may not be necessary to transmit events to the course server. 
         [0115]    LiMS plug-in  490  captures all events  435  and transmits event information, including event type, a timestamp for the event, and optionally other information relating to the event, to monitoring function  440  over network  480 . Optionally, plug-in  490  captures and forwards course content  430  from the browser to monitoring function  440  over network  480 . 
         [0116]    A plug-in is a software component or set of software components that adds specific abilities to a larger software application, and can enable customization of the functionality of an application. For example, plug-ins are commonly used in web browsers to play video, scan for viruses, and display new file types. Well-known plug-ins examples include Adobe Flash Player, QuickTime, and Microsoft Silverlight. The host application operates independently of the plug-in, making it possible for end-users to add and update plug-ins dynamically without needing to make changes to the host application. 
         [0117]    Plug-in  490  captures events relating to the user&#39;s interaction with course content  430  that is displayed to the user by the browser  425 . In implementations where plug-in  490  forwards events to the course server  410 , plug-in  490  may capture a more comprehensive set of events  435  than are transmitted to course server  410  over network  480 . For example, if the user hovers a mouse cursor over a selection button that is displayed as a part of course content  430 , and the user subsequently clicks the button, plug-in  490  may capture both mouse click event data and mouse hover event data. However because the hover data may not be of interest to the operation of course server  410 , only the mouse click data is transmitted to the course server. But if the complete set of event data, including both the mouse click data and the mouse hover data is of interest in analyzing the user&#39;s behavioral interaction with the course, the full set of data is captured by plug-in  490  and forwarded to monitoring function  440  over network  480 . 
         [0118]    The LiMS plug-in  490  can be implemented using JavaScript and/or a JavaScript library such as jQuery, and can be configured to capture all types of user interactions with the course content  430  via browser  425  as events. Typically, plug-in  490  will handle a given event by communicating its occurrence and a timestamp to monitor  470 . 
         [0119]    For example, when implemented as a web-page the course content  430  can be thought of as containing various objects, such as buttons. Each object can have various properties, such as whether or not it is pressed. If a user clicks the button with a mouse, this button click can be captured as an event, and stored with a timestamp. Further granularity is possible. For instance, separate events can be captured when the mouse button is pressed and when it is released. Other information can also be obtained relating to a button object, such as the amount of time a mouse cursor is hovered over the button even if it is not pressed. 
         [0120]    A non-exhaustive list of example events which can be captured by the plug-in  490  and transmitted to the monitor  470  along with a timestamp or other data are listed in the following table: 
         [0000]    
       
         
               
               
               
             
           
               
                   
               
               
                 Event: 
                 Object: 
                 Triggered When: 
               
               
                   
               
             
             
               
                 onAbort 
                 Image 
                 The loading of the 
               
               
                   
                   
                 image is cancelled. 
               
               
                 onBlur 
                 Button, Checkbox, 
                 The object in question 
               
               
                   
                 FileUpload, Layer, 
                 loses focus (e.g. by 
               
               
                   
                 Password, Radio, 
                 clicking outside it or 
               
               
                   
                 Reset, Select, 
                 pressing the TAB 
               
               
                   
                 Submit, Text, 
                 key). 
               
               
                   
                 TextArea, Window 
               
               
                 onChange 
                 FileUpload, Select, 
                 The data in the form 
               
               
                   
                 Text, TextArea 
                 element is changed 
               
               
                   
                   
                 by the user. 
               
               
                 onClick 
                 Button, Document, 
                 The object is clicked 
               
               
                   
                 Checkbox, Link, 
                 on. 
               
               
                   
                 Radio, Reset, Submit 
               
               
                 onDblClick 
                 Document, Link 
                 The object is double- 
               
               
                   
                   
                 clicked on. 
               
               
                 onDragDrop 
                 Window 
                 An icon is dragged 
               
               
                   
                   
                 and dropped into the 
               
               
                   
                   
                 browser. 
               
               
                 onError 
                 Image, Window 
                 A JavaScript error 
               
               
                   
                   
                 occurs. 
               
               
                 onFocus 
                 Button, Checkbox, 
                 The object in question 
               
               
                   
                 FileUpload, Layer, 
                 gains focus (e.g. by 
               
               
                   
                 Password, Radio, 
                 clicking on it or 
               
               
                   
                 Reset, Select, 
                 pressing the TAB 
               
               
                   
                 Submit, Text, 
                 key). 
               
               
                   
                 TextArea, Window 
               
               
                 onKeyDown 
                 Document, Image, 
                 The user presses a 
               
               
                   
                 Link, TextArea 
                 key. 
               
               
                 onKeyPress 
                 Document, Image, 
                 The user presses or 
               
               
                   
                 Link, TextArea 
                 holds down a key. 
               
               
                 onKeyUp 
                 Document, Image, 
                 The user releases a 
               
               
                   
                 Link, TextArea 
                 key. 
               
               
                 onLoad 
                 Image, Window 
                 The whole page has 
               
               
                   
                   
                 finished loading. 
               
               
                 onMouseDown 
                 Button, Document, 
                 The user presses a 
               
               
                   
                 Link 
                 mouse button. 
               
               
                 onMouseMove 
                 None 
                 The user moves the 
               
               
                   
                   
                 mouse. 
               
               
                 onMouseOut 
                 Image, Link 
                 The user moves the 
               
               
                   
                   
                 mouse away from the 
               
               
                   
                   
                 object. 
               
               
                 onMouseOver 
                 Image, Link 
                 The user moves the 
               
               
                   
                   
                 mouse over the 
               
               
                   
                   
                 object. 
               
               
                 onMouseUp 
                 Button, Document, 
                 The user releases a 
               
               
                   
                 Link 
                 mouse button. 
               
               
                 onMove 
                 Window 
                 The user moves the 
               
               
                   
                   
                 browser window or 
               
               
                   
                   
                 frame. 
               
               
                 onReset 
                 Form 
                 The user clicks the 
               
               
                   
                   
                 form&#39;s Reset button. 
               
               
                 onResize 
                 Window 
                 The user resizes the 
               
               
                   
                   
                 browser window or 
               
               
                   
                   
                 frame. 
               
               
                 onSelect 
                 Text, Textarea 
                 The user selects text 
               
               
                   
                   
                 within the field. 
               
               
                 onSubmit 
                 Form 
                 The user clicks the 
               
               
                   
                   
                 form&#39;s Submit button. 
               
               
                 onUnload 
                 Window 
                 The user leaves the 
               
               
                   
                   
                 page. 
               
               
                   
               
             
          
         
       
     
         [0121]    This list is not intended to be exhaustive, and those having skill in the art will understand that many other types of objects, events, and event handlers exist and can be used in the same way. For example, all HTML, JavaScript, jQuery, JScript, ActionScript, ECMAScript, VBScript, Java, DOM, events or those of their dialects as well as any other event promulgated under any W3C standard, or any other events usable for any application, web browser, interface, or API can be recorded along with a timestamp or other data. 
         [0122]    Monitoring function  440 , analyzing function  450 , and reporting function  460  each operate in a similar fashion to the corresponding elements in systems  100 ,  200 , and  300 , and while they are shown implemented as an analysis and reporting computer  470  that is in communication with plug-in  490  over network  480 , various other topologies will be evident to those having skill in the art as further explained herein. 
         [0123]      FIG. 5  illustrates another example system  500  which is substantially similar to system  400  ( FIG. 4 ) but illustrates a distributed topology where the course server  510 , user computer  520 , monitoring function  540 , analyzing function  550 , and reporting function  560  are each implemented discretely and are in communication over network  580 . Such an arrangement can arise when implementing the system modularly among different providers, or in a cloud computing environment, for example. It will be clear to those having skill in the art that various topologies including combinations of the discrete components illustrated in system  500  are possible without departing from the spirit of the invention. 
         [0124]    Referring to the various example architectures of  FIGS. 1-5 , it will be clear to those having skill in the art that different advantages can be obtained by implementing the various permutations of the elements of the systems that are possible, with the different elements of the system variously implemented as discrete or integrated components and occupying various positions in a system or network topology as desired to achieve different purposes. 
         [0125]    For example, a standalone system similar to system  200  in  FIG. 2  may be desired in order to provide an all-in-one system to a consumer who does not have access to a network or who does not wish to rely upon network connectivity to make use of the system. A networked system similar to systems  300  and  400  in  FIGS. 3 and 4  may be preferred in order to provide the course and monitoring system over the internet to a user under a subscriber model. Network systems similar to systems  300 ,  400 , and  500  in  FIGS. 3 ,  4  and  5  may be desired in order to provide LIMS functionality as an add-on service from a separate provider than the provider providing the course material. 
         [0126]    In alternative embodiments of the invention, the monitoring, analysis, or reporting functions can each also be implemented in the plug-in using client-side scripting, or in the web server using server-side scripting (not shown). 
         [0127]      FIG. 6  illustrates an example reporting dialog  600  according to aspects of the invention. Dialog  600  provides in interface to a reporting function which operates as described with respect to  FIGS. 1-5 . By interacting with dialog  600 , a user can select from a number of different reports to view which are generated based on information from an analysis function which operates as described with respect to  FIGS. 1-5 . 
         [0128]      FIG. 7  illustrates an example report  700  according to aspects of the invention. Report  700  provides a comparative behavioral analysis of two users with respect to a particular section of content. Report  700  relates that example user Andrew Belzer had a total time in content section of 155 seconds, while example user Norman Gentile had a total time in content section of 94 seconds. This information is displayed both textually and graphically in this example. 
         [0129]    The information related by report  700  illustrates that the reporting function can provide raw performance data for a user. For example, the information in report  700  may be generated in any of the systems described with respect to  FIGS. 1-5  by recording the time the content was displayed to the user using an event capture, and recording the time the content was completed in a similar fashion. The analysis function may simply calculate the difference between these timestamps to arrive at the total time in content section reported in report  700 . 
         [0130]    Other types of reports may be generated using this data. For example, if the content section is a passage of text, the text can be captured by the monitoring function, and a baseline reading time can be calculated for the content section. In this way, another report can be generated providing behavioral information for the example users. For example, if the analysis section calculates a baseline reading time of 100 seconds for the content section, a reported behavioral observation could indicate that Andrew Belzer was having difficulty reading the text, or that Norman Gentile was spending an appropriate amount of time reading the text. However, if the calculated baseline reading time were 150 seconds, the behavioral report would reflect that Andrew Belzer was spending an appropriate amount of time reading the passage, while Norman Gentile was not spending enough time reading the passage for adequate comprehension. 
         [0131]    However, if in subsequent content sections Norman Gentile correctly answers quiz questions relating to the subject matter while Andrew Belzer does not, the baseline can be adjusted automatically by the analysis section to reflect a more appropriate baseline for that particular content passage and participant audience. Behavioral observations that were made using the original baseline data can then be updated to reflect the new baseline where appropriate. This adaptive calibration can be made more robust by running the course for a larger sample size of course participants. Optionally, the adaptive calibration may only be enabled by the system when a cohort, cross-section, or totality of the course participants reaches a certain number. 
         [0132]    In addition to reporting behavioral observations after completion of the course by a user, behavioral observations can also be related to the user in real time. For instance, the system may notify Norman Gentile that he would gain more benefit from the course by spending more time reading when he moves on to a subsequent section of text in the course. Many other configurations of calculation and reporting are possible. 
         [0133]      FIG. 8  illustrates another example reporting dialog  800  according to aspects of the invention. Dialog  800  is similar to Dialog  600  ( FIG. 6 ) and provides in interface to reporting functionality. By interacting with dialog  800 , a user can select from a number of different reports to view which are generated based on information from an analysis function which operates as described with respect to  FIGS. 1-7 . 
         [0134]    Dialog  800  provides a summary overview of demographic information such as the number of course participants and a breakdown of the participants by age, sex, and the percentage of course participants determined to have attained a desired level of comprehension in a manner consistent with the disclosure herein. Additional information can be provided, such as a listing of the top performing students, an identification of the web browser used by each of the students. 
         [0135]      FIG. 9  illustrates another example reporting dialog  900  according to aspects of the invention, which provides information specifically directed to comprehension. Dialog  800  is similar to Dialog  800  ( FIG. 8 ) and provides in interface to reporting functionality. Dialog  900  may be accessed by selecting a link to this information in Dialog  800 , or by another suitable means. 
         [0136]    Dialog  900  reports measured learning behaviors of an example course participant. Dialog  900  provides a selection of learning behaviors which can be chosen for consideration, including Focus/Distraction, Inquisitive/Indifferent, Motivated/Disinterested, Confidence/Unsure, Comprehension/Misunderstanding, and Enter/Exit frequency. Optionally, dialog  900  can provide other choices.  FIG. 9  shows dialog  900  as displaying graphical data pertaining to the Comprehension/Misunderstanding learning behavior. In this example, the course user has been determined to have 14.9% level of comprehension of the course. The determination can have been made in a manner disclosed or substantially consistent with the information disclosed herein. 
         [0137]      FIG. 10  illustrates another example reporting dialog  1000  according to aspects of the invention, which provides information relevant to the course itself. Here, dialog  1000  reports an evaluation of the presentation of the course content based on an analysis of the data captured during administration of the course to one or more users, and substantially as described herein. Dialog  1000  reports an average calculated reading speed for all of the course materials, and also provides a breakdown of reading speeds for each individual portion of the course text. Because the reading speeds for the majority of sections of text were determined to be too fast in this example, reporting dialog  1000  reports the determination that the course seems to present content in a manner which challenges students to read within a target words-per-second range. Higher word-per-second reading times indicate text skimming by the user, while lower numbers may indicate that the text should be edited for clarity. 
         [0138]      FIG. 11  illustrates another example reporting dialog  1100  according to aspects of the invention which provides information relevant to the course itself. Here, dialog  1100  reports an evaluation of the usage and usability of optional content provided in the course. For example, dialog  1100  reports data on three types of optional content, where an average of 1.5 out of 4 available group content items were opened by users, zero out of 4 available window content items were opened by users, and zero out of 1 available text content items were opened by users. Dialog  1100  graphically reports an overall evaluation score as 87.5% negative and 12.5% positive based upon these observations. Dialog  1100  also reports the determination that learning opportunities such as hyperlinks, pop-ups or other hidden elements that require a learner to perform an action to experience are largely being ignored; that consideration should be given to further encourage the learner to interact with these elements as they tend to clarify or enhance the educational experience; and that in addition, it may be desirable to revisit the benefit of including these optional elements and consider removing them if they are not being used or if they are ineffective, in order to streamline the training. 
         [0139]    Presentation of the course content based on an analysis of the data captured during administration of the course to one or more users, and substantially as described herein. Dialog  1000  reports an average calculated reading speed for all of the course materials, and also provides a breakdown of reading speeds for each individual portion of the course text. Because the reading speeds for the majority of sections of text were determined to be too fast in this example, reporting dialog  1000  reports the determination that the course seems to present content in a manner which challenges students to read within a target words-per-second range. Higher word-per-second reading times indicate text skimming by the user, while lower numbers may indicate that the text should be edited for clarity. 
         [0140]      FIG. 12  illustrates another example reporting dialog  1200  according to aspects of the invention which provides information relevant to the course itself. Here, dialog  1200  reports an evaluation of test performance relating to the number of correct answers selected by course participants, and to the number of attempts made by course participants before selecting the correct answer. In this example, dialog  1200  reports an average number of attempts per question, which in this example is 1.25 over 4 questions. Dialog  1200  also reports a calculated overall test completion figure for a first score, here, reported as 75%, and a final score, here, reported as 100%, reported over 2 attempts. A graphical representation of an overall comprehension/non-comprehension score is also presented as 75% non-comprehension, and 25% comprehension. Dialog  1200  also reports a breakdown of the number of attempts required for each question to be answered correctly. Because the test performance was determined to be poor in this example, dialog  1200  reports the determination that a failing score indicates a poor understanding of concepts or content presented within the course; that multiple attempts at answering questions underscores the course participants&#39; challenge with comprehending the course topic. 
         [0141]      FIG. 13  illustrates another example reporting dialog  1300  according to aspects of the invention which provides a summary of information relevant to an example course participant. Here, dialog  1300  reports that the course participant had a poor reading speed score of 21.43%, a poor page revisit score of 71.43%, a poor content discrimination score of 11.11%, a poor test performance score of 25%, a good action conviction score of 75%, and a good enter/exit frequency score of 100%. A text based analysis is presented explaining the significance of each of these behavioral determinations. 
         [0142]    Here it is reported that based on the reading speed score, the course participant has not read text within the acceptable range. Reading above the range means the user has skimmed the text and their behavior does not indicate comprehension. Reading below the range indicates either a challenge in comprehension, or possible distraction while reading. 
         [0143]    Based on the page revisit score, the course participant seems to have difficulty in recalling information. Comprehension requires a true understanding of concepts or processes. The need to revisit previous pages indicates the course participant did not fully grasp content or concepts necessary to complete tasks which build upon previously encountered knowledge. 
         [0144]    Based on the content discrimination score, the course participant does not take advantage of learning opportunities that require them to perform an action to experience. Optional learning opportunities afford the student the possibility of enhancing their understanding of concepts or processes. Not electing to engage these may indicate the student understands the topic without the need for additional information or their level of interest may not match what is required for true comprehension. 
         [0145]    The test performance score indicates a poor understanding of concepts or content presented within the course. Multiple attempts at answering questions underscores the person&#39;s challenge with comprehending the course topic and can assist in reviewing specific challenges based on specific questions. According to some embodiments, a comprehension report similar to the dialog  900  ( FIG. 9 ) can be provided with further details regarding the course participant&#39;s comprehension performance. 
         [0146]    Based on the conviction of action score, the course participant is determined to show success in making decisions based on a true understanding of a challenge. Based on the enter/exit frequency score, the course participant is determined to show good time management skills, and the ability to minimize distractions and complete tasks within their assigned time frames. 
         [0147]    Other relevant text based observations are possible and will be evident to those having skill in the art by reference to this disclosure. 
         [0148]      FIG. 14  illustrates an example course participant interaction dialog  1400  illustrating aspects of the invention. Dialog  1400  shows an example page of an online course delivered to a participant, which can be monitored by a LiMS according to any of the systems and methods discussed herein. Dialog  1400  shows a passage of text  1410  which the course participant is expected to read and comprehend, as well as a number of auxiliary materials  1420 ,  1420 ′,  1420 ″ which the course participant can select for further learning. 
         [0149]    In example dialog  1400 , if the course participant selects auxiliary materials  1420  or  1420 ′, additional materials are displayed which may include additional text relating to text  1410 . Optionally, a TEACH® Training Enhancement Assistive Character™ such as the Paige Turner® construct  1420 ″ may be displayed in order to relate auxiliary information in the style of a live teacher. 
         [0150]    Paige Turner® construct  1420 ″ monitors and analyzes learner behaviors in real time, providing a path and process to modify these behaviors—much like a teacher observing a student and correcting his actions. Construct  1420 ″ monitors a student&#39;s actions in a more granular manner as it can record not only the result of the student action, but the sequential behaviors which lead up to the action, and has the ability to independently interrupt the student when the timing or sequencing of their behaviors indicates a challenge. 
         [0151]    Construct  1420 ″ can also be “summoned” to the screen by the learner to engage in a “dialog” to assist the learner should they encounter a challenge during the course—just like raising a hand in a classroom and asking for help. 
         [0152]    Dialog  1400  also includes navigation buttons  1430 ,  1430 ′,  1430 ″,  1430 ′″,  1430 ″″ which can control navigation from dialog  1400  to a previous or subsequent part of the online course, or can signal that the course participant is finished, or other similar navigational tasks. Any mouse click or other operation of these controls, e.g. by keyboard, can be captured, time-stamped, and used to analyze user behavior as further described herein. In addition, any other interaction with the browser can be captured, time-stamped, and used to analyze user behavior in this way. For example, the amount of time that a mouse cursor hovers over a particular control, portion of text, or an image in dialog  1400 , or the timing between click and release of the mouse button or key may be relevant in making behavioral observations relating to interest or engagement with the topic, decisiveness in answering, comprehension, or any other observation as further discussed herein. 
         [0153]      FIGS. 15 and 16  illustrate a further example system  1500  according to aspects of the invention. System  1500  includes a computing device having input elements, including a keyboard and mouse, and output elements, including a display screen and a running browser. The browser receives an eLearning course including HTML pages from an outside source, such as a web server. LiMS scripts parse each HTML page, and identify elements of each page. This data is transmitted to PHP scripts which parse and store the element data in a SQL database, optionally along with a time-stamp. PHP functions running live and in the background receive data from the SQL database and analyze behavioral data, reposting the analyzed behavioral data to database tables in the SQL database. This analyzed behavioral data is returned via PHP scripts to the LiMS scripts and are used to update the screen. 
         [0154]    The course participant can interact with the eLearning HTML pages using the input devices. In particular, the course participant interacts with objects in each page, which may include buttons, sliders, and the like, and as further discussed herein. LiMS scripts capture these interactions, and identify interaction attributes. This information can be used to update the screen. This information is also transmitted to PHP scripts which receive the interaction data, parse the interaction data, and store the interaction data in the SQL database. This data is also available to be returned to the LiMS scripts via the PHP scripts. 
         [0155]      FIG. 16  illustrates system  1500  with the HTML parsing functionality omitted for clarity of illustration. Optionally, system  1500  may be implemented without this functionality where all of the content of the eLearning course are known to the system apriori.  FIG. 16  also illustrates wherein a construct such as Paige Turner® is called by the LiMS scripts for display to the course participant, an intervention type is determined, video clips are called, and the behavior and engagement are reported to the LiMS script, which can parse and store this information in the SQL database. 
         [0156]      FIGS. 17-27  illustrate example database entries according to aspects of the invention. 
         [0157]      FIG. 17  illustrates an example database table which reflects a number of example page elements appearing on example pages of an eLearning course. Elements appearing in the table of  FIG. 17  include buttons, text, hyperlinks, and pop-ups. Many other elements can be used, as discussed herein. Each element is stored in the table along with a timestamp, position, type, and state. 
         [0158]      FIG. 18  illustrates an example database table which reflects a number of example event captures of interactions by course participant with example pages of an eLearning course. Events appearing in the table of  FIG. 18  are captured in the table with their respective timestamps as well as information relating to the element to which the event pertains, if applicable. Many other events can be used, as discussed herein 
         [0159]      FIG. 19  illustrates an example database table which reflects statistics for course participants with respect to given pages of the eLearning course. Statistics appearing in each line of the table of  FIG. 19  include the student ID, page ID, average time on page, average words read per second, average ratio of groups opened, average ratio of windows opened, average ratio of texts opened, and average question interaction ratio. Many other statistics can be used, as discussed herein 
         [0160]      FIG. 20  illustrates an example database table showing course participant statistics for questions presented in the eLearning course. Statistics appearing in each line of the table of  FIG. 20  include the student ID, question ID, number of attempts, first answer, timestamp for first answer, final answer, final result, whether hints were available for the question, and whether the hint was used for the first attempt. Many other statistics can be used, as discussed herein. 
         [0161]      FIG. 21  illustrates an example database table which shows information for pages presented in the eLearning course. Information appearing in each line of the table of  FIG. 21  includes the page ID, chapter ID, section ID, page name, total visible words on page, number of groups, number of windows, amount of text, number of questions, number of non-question elements, as well as a timestamp. Many other pieces of information can be used in the table as discussed herein. 
         [0162]      FIG. 22  illustrates an example database table which shows event captures by the LiMS during the eLearning course. Information appearing in each line of the table of  FIG. 22  includes the action ID, experience ID, element ID, behavior ID, element value, and a timestamp. 
         [0163]      FIG. 23  illustrates an example database table which contains information for questions appearing in the eLearning course. Information appearing in each line of the table of  FIG. 23  includes the question ID, page ID on which the question appears, question name, question type (multiple choice or true/false), text of the question, correct answer, number of hints available, and a timestamp. 
         [0164]      FIG. 24  illustrates an example database table which contains information for chapters appearing in the eLearning course. Information appearing in each line of the table of  FIG. 24  includes chapter ID, course ID, chapter type (content or test, for example), total pages in chapter, chapter name, and a timestamp. 
         [0165]      FIG. 25  illustrates an example database table which contains information for various interactions by the course participant with a page of the eLearning course. Information appearing in each line of the table of  FIG. 25  includes an experience ID, total time on page, a timestamp for the first movement of a mouse by the user, mouse coordinates, the previous page ID, number of words read per second, number of groups opened, number of windows opened, number of texts opened, number of questions interacted with, number of non-question elements interacted with, ratio of groups opened to groups not opened, ratio of windows opened to not opened, ratio of texts opened to not opened, and number of non-questions opened to not opened. 
         [0166]      FIG. 26  illustrates an example database including a number of tables used in implementing an example system according to the invention. Example tables track information relating to actions, answers, users, event captures, chapters, clients, courses, elements, events, experiences, pages, questions, sections, serials, elements, helpfiles, standards, status, student results, course aggregate behavior, statistics, and students, for example. Many other kinds of tables can be used to track relevant information as discussed herein.  FIG. 26  also illustrates an example table which contains information related to particular event types. Each line of the table includes an event ID and an event name. Types of events described in the table include the following: abort, beforeunload, blur, change, click, close, dblclick, error, focus, focusin, focusout, keydown, keypress, keyup, load, mousedown, mouseenter, mouseleave, mousemove, mouseout, mouseover, mouseup, pagehide, pageshow, reset, resize, scroll, select, submit, and unload events. Many other kinds of events can be tracked as further discussed herein. 
         [0167]      FIG. 27  illustrates another example database reflecting various tables stored therein, for operating a system according to the invention. 
         [0168]    Although the invention has been described with reference to a particular arrangement of parts, features and the like, these are not intended to exhaust all possible arrangements or features, and indeed many modifications and variations will be ascertainable to those of skill in the art.