Source: http://edutechwiki.unige.ch/en/Learning_analytics
Timestamp: 2019-04-24 01:58:59+00:00

Document:
One could define learning analytics as collection of methods that allow teachers and maybe the learners to understand what is going on. I.e. all sorts of data collection, transformation, analysis and visualization tools that allow to gain insight on participant's behaviors and productions (including discussion and reflections). The learning analytics community in their call for the 1st International Conference on learning analytics provides a more ambitious definition: “Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs”. This definition includes a clearly transformative perspective.
Educational data mining is a field that has similar interests, but that focuses more on research. In addition, favorite data gathering, aggregation/clustering, analysis and visualization techniques differ a bit.
The rapid development of “big data” methods and tools coincides with new management and measurement processes in corporations. The term “business intelligence” is used to describe this intersection of data and insight. When applied to the education sector, analytics fall into two broad sectors (Table 1): learning and academic.
Learning analytics (LA) is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs. Learning analytics are largely concerned with improving learner success.
Academic analytics is the improvement of organizational processes, workflows, resource allocation, and institutional measurement through the use of learner, academic, and institutional data. Academic analytics, akin to business analytics, are concerned with improving organizational effectiveness.
This definition of the society of learning analytics and the US Dept. of education report that we shall discuss below are political and like many other constructs in the education sciences "learning analytics" promises better education. We therefore conclude the introduction with the impression that learning analytics either can be seen (1) as tools that should be integrated into the learning environment and scenario with respect to specific teaching goals or (2) as a more general and bigger "framework" for doing "education intelligence". The latter also could include the former, but not necessarily so.
Large scale analytics (also called academic analytics) whose function is to improve institutions or even whole educational systems.
This chapter attempts to identify various strands of research and development and to discuss some of these in more detail.
Student and teacher surveys of various kinds, e.g. course assessment or exit questionnaires.
learning e-portfolios, i.e. students assemble productions and reflect upon these (use quite a lot in professional education).
Tracking tools in learning management systems.
Cockpits and scaffolding used in many CSCL tools.
Student modeling in artificial intelligence and education.
On a more individual basis analytics is related to Infotention and other knowledge/idea management concepts and tools.
Modern learning analytics (i.e. that is labelled as such) probably initially drew a lot of ideas from industry and its various attempts to deal with information, such as business intelligence, Quality management, Enterprise resource planning (ERP), Balanced score cards, marketing research and Web analytics. The common idea is to extract Key performance indicators from both structured and unstructured data and to improve decision making.
A related second important influence was a political will to articulate and measure quality standards in education leading to a culture of quantiative assessment (Oliver and Whelan, 2011; Hrabowski, Suess et al. 2011).
A third influence was and increasingly is prior research. According to Baker (2013), “Since the 1960s, methods for extracting useful information from large data sets, termed analytics or data mining , have played a key role in fields such as physics and biology. In the last few years, the same trend has emerged in educational research and practice, an area termed learning analytics (LA; Ferguson, 2012) or educational data mining (EDM; Baker & Yacef, 2009). In brief, these two research areas seek to find ways to make beneficial use of the increasing amounts of data available about learners in order to better understand the processes of learning and the social and motivational factors surrounding learning.” (Learning, Schooling, and Data Analytics, retrieved March 2014.
One difference between various approaches to learning analytics (in a wide sense) concerns the locus of data collection and data analysis, and on how big a scale (number of students, activities and time) it is done. In radical constructivist thought, students should be self-regulators and the role of the system is just provide enough tools for them as individuals or groups for understanding what is going on. In more teacher-oriented socio-constructivism, the teachers, in addition, need indicators in order to monitor orchestrations (activities in pedagogical scenarios). In other designs such as direct instruction, the system should provide detail performance data that are rather based on quantitative test data. Finally, powers above like schools or the educational systems would like to monitor various aspects.
Educational data mining (EDM) could be described as a research area that focuses more on data retrieval and analysis. I mainly targeted educational research using and developing all sorts of technology, e.g. information retrieval, clustering, classification, sequence mining, text mining, visualization, and social network analyis. Today, EDM also can be seen as "helper discipline" for more sophisticated learning analytics approaches.
Improvement of student models, i.e. models that are typically used in artificial intelligence and education systems such as intelligent tutoring systems.
Studying pedagogical support (both in learning software, and in other domains, such as collaborative learning behaviors), towards discovering which types of pedagogical support are most effective.
Empirical evidence to refine and extend educational theories and well-known educational phenomena, towards gaining deeper understanding of the key factors impacting learning, often with a view to design better learning system.
This overview paper also points out that relationship mining was dominant between 1995 and 2005, Prediction moved to the dominant position in 2008-2009. In addition, the authors noted an increase of data coming from the instrumentation of existing online courses as opposed to harvesting the researcher's-as-teacher's own data.
Fournier et al. (2011) quote Downs (2010) “There are different tools for measuring learning engagement, and most of them are quantificational. The obvious ones [measure] page access, time-on-task, successful submission of question results – things like that. Those are suitable for a basic level of assessment. You can tell whether students are actually doing something. That’s important in certain circumstances. But to think that constitutes analytics in any meaningful sense would be a gross oversimplification. [...] There is a whole set of approaches having to do with content analysis. The idea is to look at contributions in discussion forums, and to analyze the kind of contribution. Was it descriptive? Was it on-topic? Was it evaluative? Did it pose a question?”. For the later kind (btw. not just forums but all sorts of student productions), we do need text mining technology. In addition, Fournier point out that data mining technologies may have to be combined with semi-manual qualitative analysis, in particular social networking analysis.
The proposal starts somehow with the assumption that education continues to use rather simple tools like courseware or somewhat now popular web 2.0 tools like personal learning environments. In other words, the fog is identified as the problem and not the way education is designed. I.e. this proposal focuses on "intelligence" as opposed to "design". If you like, "learning design" could become more like a "General motors" approach (marketing oriented) than "Apple" (design-oriented). Related to that we also can identify the general assumption that "metrics" work, while in reality test-crazy systems like the US high-school education have a much lower performance than design-based systems like the Finnish one. We are not really familiar with the long history of business intelligence, management information systems but we are familiar with the most prominent critique, i.e. that such systems put a distance between decision makers and staff, kill communication lines and inhibit managers from thinking.
We would like to draw parallels with (1) the metadata community that spent a lot of time designing standards for describing documents and instead of working on how to create more expressive documents and understanding how people compose documents, (2) with business that spends energy on marketing and related business intelligence instead of designing better products, (3) with folks who believe in adaptive systems forgetting that learner control is central to deep learning and that most higher education is collaborative, (4) with the utter failure of intelligent tutoring systems trying to give control to the machine and (5) finally with the illusion of learning style. These negative remarks are not meant to say that this project should or must fail, but they are meant to state two things: The #1 issue is in education is not analytics, but designing good learning scenarios within appropriate learning environments (most are not). The #2 issue is massive long term funding. Such a system won't work before at least 50 man years over a 10 year period will be spent.
Somewhat it also is assumed that teachers don't know what is going on and that learners can't keep track of their progress or more precisely that teachers can't design scenarios that will help both teachers and students knowing what is going on. We mostly share that assumption, but would like to point out that knowledge tools do exist, e.g. knowledge forum, but these are never used. This also can be said with respect to CSCL tools that usually include scaffolding and meta-reflective tools. In other words, this proposal seems to imply that that education and students will remain simple, but "enhanced" with both teacher and student cockpits that sort of rely on either fuzzy data mined from social tools (SNSs, forums, etc.) or quantitative data from learning management systems.
Finally, this project raises deep privacy and innovation issues. Since analytics can be used for assessment, there will be attempts to create life-long scores. In addition, if students are required to play the analytics game in order to improve chances, this will be other blow to creativity. If educational systems formally adopt analytics, it opens the way for keeping education in line with "main-stream" e-learning, an approach designed for training simple subject matters (basic facts and skills) through reading, quizzing and chatting. This, because analytics will work fairly easily with simple stuff, e.g. scores, lists of buttons clicked, number of blog and forum posts, etc.
This being said, we find the SoLAR project interesting, but under the condition that it should not hamper use of creative learning designs and environments. Standardization should be enabling (e.g. like notation systems in music are) and not reduce choice.
Bienkowski et al. (2012) in their Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics (draft) prepared for the US Department of Education firstly defines Educational data mining. It is “ emerging as a research area with a suite of computational and psychological methods and research approaches for understanding how students learn. New computer-supported interactive learning methods and tools— intelligent tutoring systems, simulations, games—have opened up opportunities to collect and analyze student data, to discover patterns and trends in those data, and to make new discoveries and test hypotheses about how students learn. Data collected from online learning systems can be aggregated over large numbers of students and can contain many variables that data mining algorithms can explore for model building.” (p.9, draft).
We find it very suprising that the Bienkowski report argues that learning analytics do not address the development of new computational methods when in reality they most interesting learning analytics projects (e.g. Buckingham Shum & Crick, 2012 or Govaerts et al., 2010) exactly do that. In addition, we find that the report seems to missing one crucial issue: Enabling the learner and learner communities. 20 years of research in educational technology is missing (e.g. cognitive tools, learning with portfolioss, project-oriented learning and writing-to-learn). On the other hand, AI&Ed concepts that have been given up in the 1990's (and for good reasons) seem to re-emerge without sufficient grounding in contemporary learning theory and first principles of instructional design. Analytics that attempt to improve the learning process through awareness tools are maybe "small scale analytics" and definitely not predictive.
In a presentation, called Turning Learning into Numbers - A Learning Analytics Framework, Hendrik Drachsler and Wolfgang Geller presented the following global component framework of learning analytics. The framework also was presented in Hendrik Drachsler's lecture on recommender systems for learning (April 2012).
Such a framework helps thinking about the various "components" that make up the learning analytics topic.
Tanya Elias (2011) in her Learning Analytics: Definitions, Processes and Potential argues that learning analytics can be matched to Baker's (2007) knowledge continuum, which somewhat could be compared to learning level models, such as the Bloom taxonomy.
Wisdom - Use Knowledge to Establish and Achieve Goals.
Processes of Learning Analytics (Elias).
According to the author (Elias:11) learning analytics consist of computer, people, theory and organizations. “ Learning analytics uses of four types of technology resources to complete ongoing pattern of three-phase cycles aimed at the continual improvement of learning and teaching” (Elias, 2011:17).
This model has the merit to explicitly state that learning analytics is a complex iterative process. In particular, being able to put analytics to good use is not the same as producing reports or predictions. By the way, I wouldn't associate "knowledge" exclusively with prediction in the overview table. The latter refers to just one kind of academic knowledge. For participants (learners and teachers) it is much more important to create a deep understanding what "is going on". Also interesting is the "sharing" part.
In addition, we believe that interpretation of data should be negotiated, i.e. we also would add a sharing tab between "information processing" and "knowledge application". A simple example can illustrate that. A Mediawiki analytics tool such as StatMediaWiki can show that some students actively use the discussion pages and others do not. It turns out that those who don't can be either weak students or very good ones. The latter meet face to face on a regular basis. In other words, one has to present the data to students and discuss with them.
context analytics — mobile computing is transforming access to both people and content.
Work on a system is in progress.
Soller, Martinez, Jermann and Muehlenbrock (2004, 2005:Abstract) develop a collaboration management cycle framework that distinguishs between mirroring systems, which display basic actions to collaborators, metacognitive tools, which represent the state of interaction via a set of key indicators, and coaching systems, which offer advice based on an interpretation of those indicators.
Phase 1: The data collection phase involves observing and recording the interaction of users. Data are logged and stored for later processing.
Phase 2: Higher-level variables, termed indicators are computer to represent the current state of interaction. For example, an agreement indicator might be derived by comparing the problem solving actions of two or more students, or a symmetry indicator might result from a comparison of participation indicators.
Phase 3: The current state of interaction can then be compared to a desired model of interaction, i.e. a set of indicator values that describe productive and unproductive interaction states. “For instance, we might want learners to be verbose (i.e. to attain a high value on a verbosity indicator), to interact frequently (i.e. maintain a high value on a reciprocity indicator), and participate equally (i.e. to minimize the value on an asymmetry indicator).” (p. 7).
Phase 4: Finally, remedial actions might be proposed by the system if there are discrepancies.
Soller et al. (2005:) add a phase 5: “After exiting Phase 4, but before re-entering Phase 1 of the following collaboration management cycle, we pass through the evaluation phase. Here, we reconsider the question, “What is the final objective?”, and assess how well we have met our goals”. In other words, the "system" is analysed as a whole.
Mirroring tools automatically collect and aggregate data about the students’ interaction (phases 1 and 2 in Figure 1), and reflect this information back to the user with some kind of visualization. Locus of processing thus remains in the hand of learners or teachers.
Metacognitive tools display information about what the desired interaction might look like alongside a visualization of the current state of indicators (phases 1, 2 and 3 in the Figure), i.e. offer some additional insight to both learners and teachers.
Guiding systems perform all the phases in the collaboration management process and directly propose remedial actions to the learners. Details of the current state of interaction and the desired model may remain hidden.
Learners’ attention: what do learners focus on? What problems and questions they raised, what comments they made, what viewpoints they expressed etc.
learners’ rhetorical attitude to discourse contributions: With what and who do a learner agrees/disagrees? What ideas he supports? What data he questioned?
learning topics distribution: What are the hottest learning topics, by who they have been proposed and discussed?
learners’ social interactions: How do learners act within a discussion group? What are the relationships between learners?
Since "Analytics" is a term that can include any kind of purpose as well as any kind of activity to observe and analyze, the set of technologies is rather huge and being developped in very different contexts. Recent interest in learning analytics seems to be sparked by the growing importance of on-line marketing (including search engine optimization) as well as user experience design of e-commerce sites. However, as we tried to show in our short discussion of CSCL technology, mining student "traces" has a longer tradition.
The categories we describe below overlap to a certain extent.
Probably very close to data mining, but a slightly distinct community. In earlier publications, in particular in the AI&Ed tradition, papers did not use the term analytics but referred to "traces" and "student modeling". E.g. see the publications from the SILEX group at LIRIS (University of Lyon), e.g. Settouti (2009). Theses systems collect data from specially crafted log files (or databases) that belong to the application.
A similar technique would be the Firefox Greasemonkey add-on. When last checked, we didn't find any SNAPP-like tools, but a few dozen scripts that will modify or enhance Google Analytics, e.g. filter out referrers from social media sites.
A different technique (which can't be called client-side analytics) concerns web widgets that owners of web pages can insert. These then can collect data and send it to a server. A typical example is the little "Share" widget to the left that uses the well known addThis service. For example, one can imagine that teachers could require students of a class to share all visited pages on a common service.
Research in educational technology and related fields about learning, research in related fields that use education for data (e.g. computer science).
Most systems also provide data to the learner and the teacher in order to enhance learning and teaching processes.
Learning management systems usually implement IMS Content Packaging and SCORM 1.2 tracking, i.e. show individual or class statics about pages consulted (IMS CP) and quizzing performance (SCORM). In addition, they may track student paths through their own "organizations".
Educational workflow systems such as LAMS (Dalziel, 2003) allow to monitor student progress with respect to activity completion, time spent on task, etc. LAMS also can export student productions as portfolio and then allows to look at student's productions and performance within various tools.
Intelligent tutoring systems, gone out of favor in the 1990's but still alive, include by define analytics. However, its purpose is to enhance the individual learner's process by various means, e.g. diagnosing and adapting.
None of the software listed below comes even close to the goals of SoLAR framework. We roughly can distinguish between software that gathers (and optionally displays) data in very specific applications, software that mines the web and finally general purpose analysis/visualization software.
Ali, Hatala, Gašević & Jovanović (2011), Baker & Yacef (2009) and Asadi, Jovanović, Gašević & Hatala (2011) provide overviews of various learning analytics tools. We started looking a some of these systems. Some are not available and most others only seem work within very specific environments, although they may have been designed in a more general spirit. From what we understand right now (that is little), there exist currently very few interesting tools that could be used by teachers (or even somewhat tech-savy educational technologist). In particular, ready to use tools for innovative learning designs that engage students in various forms of writing seems to be almost totally missing. E.g. most creative teachers use wikis in one or another way and indeed could use a whole lot of analytics. Read more about that in the wiki metrics, rubrics and collaboration tools.
It will take me some time to get a clearer picture - Daniel K. Schneider 19:41, 15 March 2012 (CET).
A variety of tools made for other purposes is being used for educational analytics. Some of these have been developed for use in education. Below we list a few.
Web analytics tools are both used for Search engine optimization, Interaction design, user experience and usability studies.
Besides Google Analytics, there are many other tools. Most are commercial, but minimal services are often free. E.g. Bing WebMaster tools, Search Metrics, SEMRush, Yahoo site explorer, Userfly, Tynt, Kissmetrics, Mixpanel.
Pajek, is a program, for Windows, for analysis and visualization of large networks.
The ACODEA Framework (Automatic Classification of Online Discussions with Extracted Attributes) by Mu et al. (2012). This is not a tool, but a configuration and use scenario for the SIDE tool (Mayfield and Rosé, 2010a).
So-called awareness tools provide real-time analytics. According to Dourish and Belloti (1992), awareness is an understanding of the activities of others, which provides a context for your own activity. Gutwin and Greenberg (1996:209) define eleven elements for workspace awareness, for example: presence (who is participating in the activitiy ?), location (where are they working ?), activity level (how active are they in the workspace ?), actions (what are they doing ? what are their current activites and tasks ?), and changes (what changes are they making, and where ?) Note that "where" can refer both to a real and virtual location (e.g. a living room or a forum in a learning platform).
LightSide will be an online service for automated revision asistance and classroom support. It also is available as free software that can be download, but that must be trained by yourself.
Involver. This is a toolbox that includes: Conversation Suite to monitor the conversation across Facebook, Twitter and Google+; Application Suite to deploy apps; Visual Builder and Social Markup Language (SML™) for designing pages and social apps.
Rapid Miner is the most popular open source data mining tool. I can import data from many sources (also perform web scrapping), analyze with over 500 operators and visualize, etc.
These tools are combinations of page scrapping (content extraction), text summarization a comparison), sorting, and visualization. Their scope and ease of use seems to differ a lot.
social bookmarking, reference manager and Citation index services. These also provide quite a lot of data on what authors and other users produce and organize.
NetDraw a Windows program for visualizing social network data. (Borgatti, 2002).
Educational badges systems that are managed through some kind of server technology can be used to harvest information about learner achievements, paths, skills, etc.
For example, “Mozilla's Open Badges project is working to solve that problem, making it easy for any organization or learning community to issue, earn and display badges across the web. The result: recognizing 21st century skills, unlocking career and educational opportunities, and helping learners everywhere level up in their life and work.” (About, retrieved 15:37, 14 March 2012 (CET).
We don't have neither the time nor the resources to list tools developed for other domains (business analytics in particular). An interesting non-business initiative is Global pulse, a UN project. Part of Global Pulse’s approach is learning how to distill information from data so we can identify insights that can be useful in program planning and, in the long run, can contribute to shaping policy. Tools developed include for example "Twitter and perceptions of crisis-related stress" and "impact of financial crisis on primary schools".
By default, most so-called learning management systems have built-in tracking tools. In addition, there may be extra tools, either developed for research or available as additional modules in commercial systems. Such tools may help to improve traditional e-learning designs.
LOCO-Analyst “is implemented as an extension of Reload Content Packaging Editor, an open-source tool for creating courses compliant with the IMS Content Packaging (IMS CP) specification. By extending this tool with the functionalities of LOCO-Analyst, we have ensured that teachers effectively use the same tool for creating learning objects, receiving and viewing automatically generated feedback about their use, and modifying the learning objects accordingly.” (LOCO-Analyst, retrieved 14:06, 2 March 2012 (CET)).
As of March 2012, the systems works with iHelp Courses, an open-source standards-compliant LCMS developed at the University of Saskatchewan.
In LOCO-Analyst, “feedback about individual student was divided into four tab panels: Forums, Chats, Learning, and Annotations [...]. The Forums and Chats panels show student’s online interactions with other students during the learning process. The Learning panel presents information on student’s interaction with learning content. Finally, the Annotations panel provides feedback associated with the annotations (notes, comments, tags) created or used by a student during the learning process.” (Ali et al. 2001). For each of the four "areas" key data can then be consulted.
“The generation of feedback in LOCO-Analyst is based on analysis of the user tracking data. These analyses are based on the notion of Learning Object Context (LOC) which is about a student (or a group of students) interacting with a learning content by performing certain activity (e.g. reading, quizzing, chatting) with a particular purpose in mind. The purpose of LOC is to facilitate abstraction of relevant concepts from user-tracking data of various e-learning systems and tools.” (LOCO-Analyst, retrieved 14:06, 16 March 2012).
Jaillet (2005) presents a model with three indicators for measuring learning in the Acolad distance teaching platform. These indices are are based on Leontiev's activity theory that distinguishes three levels: Activities are high level and they are goal and needs driven. Activities are conducted using strategies and performing actions using tools (instruments). Actions are performed using operators. In the context of education, both actions and operations can be either individual or collaborative (social).
The study measures student activity (called meta-activity) through three dimensions: attendance, availability and involvement: All indicators are measured with respect to the most active student.
653(61;84;42)2629480 means that a student connected 61% of 653 (with respect to the student that has the highest connection), during 84% of 480 hours and that he completed 42% operations out of 2692 (max.).
It is not very clear from the paper whether the data leading to "meta-activity" indicators are supposed to measure actions or operations. The study also includes an other interesting idea.
Since Acolad uses a 2D spatial metaphor, i.e. 3D drawings, for various learning activities, one can guess from the student's location what they are working on. There is evidence from research in CSCW that people do tend to relocated in virtual environment. In other words, learning analytics should provide better data from environments that associate learning activities with a space.
The Blackboard Analytics for Learn™ module is advertized as “easy, self-service access to data that can help give you an enterprise-level perspective. Gain insight into user activity, course design and student performance data across departments and colleges — enabling you to improve your use of the Blackboard Learn™ platform in support of teaching, learning, and student success.” (retrieved April 5, 2012).
How are my students performing on learning standards over time?
How can I easily find students who aren’t engaged in their online courses?
Which tools are being used in courses the most? The least?
How many logins, page views, and other metrics have occurred over time?
Who are the most active instructors? Who are the least active?
What strategies to improve the quality of course design and instruction result in better student performance? Instructor training courses? Course reviews?
Quote: Teachers, students, study program managers and administrators need feedback about the status of the activities in online-courses. MOCLog will realize a monitoring system, that helps to analyze logfiles of the Moodle LMS more effectively and efficiently thus contributing to improve the quality of teaching and learning.
WCET Predictive Analytics Reporting (PAR) Framework. Aim: to conduct large-scale analyses of federated data sets within postsecondary institutions to better inform loss prevention and identify drivers related to student progression and completion. This project (as of April 2012) seems to be in planning stage.
Teacher ADVisor (TADV): uses LCMS tracking data to extract student, group, or class models (Kobsa, Dimitrova, & Boyle, 2005).
Student Inspector: analyzes and visualizes the log data of student activities in online learning environments in order to assist educators to learn more about their students (Zinn & Scheuer, 2007).
CourseViz: extracts information from WebCT.
ASTRO and BBStats, some kind of Blackboard plugins (?) that provide a dashboard.
Tools like Metrix (working with any mobile device) allow teachers to define a skill set and then enter student data that allow to "track, analyze and differentiate" students.
Some institutions did or will implement tracking tools that pull in data from several sources. Some tools are designed for both teachers and learnings, others are in the realm of academic analytics (as opposed to learning analytics).
The University of Western Sydney implemented a system with the following uses in mind (Scott, 2010).
To prove quality – e.g. data can be drawn from different quantitative and qualitative data bases and, after triangulation, can be used to prove the quality of what UWS is doing. Data analysis can use absolute criteria and/or relative (benchmarked) standards.
To improve quality – e.g. data can be used to identify key improvement priorities at the university, college, division, course and unit levels.
This framework uses about ten data sources, mostly surveys (student satisfaction, after graduation) and peformance reports. Interestingly, it includes time series, e.g. student satisfaction during, at exit and after obtaining a degree.
Towers (2010) describes Queensland University of Technology (QUT) annual Courses Performance Report as a combination of Individual Course Reports (ICRs), the Consolidated Courses Performance Report (CCPR), Underperforming Courses Status Update and the Strategic Faculty Courses Update (SFCU). “Individual Course Reports (ICRs) is to prompt an annual health check of each course’s performance, drawing upon course viability, quality of learning environment, and learning outcomes data. In total, data on 16 indicators are included in the ICRs.” (Towers, 2010: 119). The paper include an example of an 8-page report in the appendix.
Courses that aim at deep learning, e.g. applicable knowledge and/or higher order knowledge are usually taught in smaller classes most often uses software that is distinct from the ones used in so-called e-learning, e.g. wikis, learning e-portfolios, content management systems, blogs, webtops and combinations of these. In addition, some systems developed in educational technology research are being used. The latter often implement analytics for research, but often there is a teacher interface using the same data. Most creative power teaching, however, uses what we would call "street technology" and for these, analytics tool rarely exist. A good example would be the Enquiry Blog Builder we shortly describe below. Another example is StatMediaWiki, a tool that we use to learn about student contributions in the french version of this wiki. Most analytics tools we found so far, require some learner participation like tagging or filling in shorter or longer questionnaires.
Since project-oriented learning and related constructivist designs do rarely include "pages to view" and quizzes, simple quantitative data do not provide good enough analytics, although some quantitative data such as "number of edits", "number of connections", etc. are useful. Some teachers, in particular in the engineering sciences also use project management software that by definition include analytics. However these are less suited for writing, collaboration and creating communities of learning than typical out-of-the-box multi-purpose portalware.
The main issue of what we would call "small scale analytics" is not how to measure learning, but how to improve learning scenarios through tools that help both the learner and the teacher understand "what is going on" and therefore improve reflection.
Oliver and Whelan (2010) point out that “Many agree that current available data indicators - such as grades and completion rates - do not provide useable evidence in terms of learning achievements and employability (Goldstein, 2005; Norris, Baer et al. 2008)” and they argue that “qualitative evidence of learning is traditionally housed in student portfolio systems”. Such systems then should include mechanisms for self-management of learning that could feed into institutional learning analytics. At Curtin, to encourage students to reflect on and assess their own achievement of learning, the iPortfolio incorporates a self-rating tool based on Curtin's graduate attributes, “enabling the owner to collate evidence and reflections and assign themselves an overall star-rating based on Dreyfus and Dreyfus' Five-Stage Model of Adult Skill Acquisition (Dreyfus, 2004). The dynamic and aggregated results are available to the user: [as shown in the figure], the student can see a radar graph showing their self-rating in comparison with the aggregated ratings of their invited assessors (these could include peers, mentors, industry contacts, and so on).” (Oliver & Wheelan:2010).
This example as well as De Liddo et al. (2011) tagging system discussed above demonstrates a very simple principle that one could implement in any other environment: (Also) ask the learner.
The SAM (Student Activity Monitor) was developed in the context of the EU ROLE project. “To increase self-reflection and awareness among students and for teachers, we developed a tool that allows analysis of student activities with different visualizations”. Although SAM was developed in a project that focused on developing a kit for constructing personal learning environments it can be used in any other context where the learning process is largely driven by rather autonomous learning activities. SAM is implemented as Flex software application and it is documented in the sourceforge ROLE project wiki as Widget Student Activity Meter. It relies on a widget that users must use: the Widget CAM Widget that enables/disables user monitoring. Enabling the Monitoring provides learners with benefits regarding recommendation and self-reflection mechanisms.
The Cam dashboard is a an application that enables the visualization of usage data as a basis to detect changes in usage patterns. The purpose is to detect variations in the use of PLEs based on changes in usage patterns with widgets and services.
So far we didn't find any .... - Daniel K. Schneider 19:34, 11 April 2012 (CEST) (but maybe we will have to spend some more time searching or download the software kit).
EnquirySpiral - This widget provides a graphical display of the number of posts made in the first nine categories. A spiral of blobs appears over an image with each blob representing a category. The blobs are small and red when no posts have been made. They change to yellow for one or two posts, and green for three or more. In this way it is easy for the student to see how they are progressing, assuming the nine categories are well chosen.
EnquirySpider - This widget works in the same way as the EnquirySpiral, except that the blobs are arranged in a star shape. They are associated with seven categories. (from nine to sixteen so they don't conflict with the EnquirySpiral). The categories are intended to match with the Effective Lifelong Learning Inventory diagram.
(Enquiry Blog Builder, retrieved April 10 2012).
The kit also includes BlogBuilder allowing batch creation of blogs in one go with provided teacher and student names. Teachers who login will then see the a dashboard showing the progress of the students assigned to them. In other words, the teacher will get monitoring widgets that correspond to the three widgets above. These will show the student widgets.
Buckingham Shum and Deakin Crick (2012) describe a setup that collects learning dispositions through a questionnaire. Results then are shown to the students and teachers. “The inventory is a self-report web questionnaire comprising 72 items in the schools version and 75 in the adult version. It measures what learners say about themselves in a particular domain, at a particular point in time” (Preprint: page 4).
Students are shown a spider diagram of these dimensions. Since ELLI can be applied repetitively, a diagram also can display change, i.e. (hopefully) increase. Furthermore data can be aggregated across groups of learners in order to provide collective profiles for all dimensions or details about a specific dimension.
Cohere from Open University is a visual tool to create, connect and share Ideas. E.g. de Liddo et al. (2011) analyse quality of contribution to discourse as an indicator for meaningful learning.
concept map tools and other idea managers. Some of these run as services.
Small analytics (individual learners, teachers and the class) are not new, although we hope that more tools that will help us understanding what is going on will be available in the future. Such tools are mostly lacking for environments that engage students in rich activities and productions (as opposed to so-called e-learning that uses learning management systems). In project-oriented learning designs, we also should think about building tools into the scenario that let the students add data, e.g. "what did I contribute" and "why".
Big time analytics should carefully think about strategy, i.e. ways of thinking about improving education. In particular we believe it important that one should not automatically focus on alignment as in a balanced scorecard method and rather adopt a "learning organization" perspective, e.g. adopt some kind of expansive learning philosophy. Below is a picture from a talk by Stefano Mastrogiacomo that illustrates various way of thinking.
Of course, discussion about the aims of learning analytics already takes place, in particular ethical issues (refs?). However, we suggest to focus on the idea that Change should be related to learner and teacher empowerment as opposed to strengthening the already sometimes counterproductive education bureaucracy.
Learning analytics is an old topic in educational technology. Some CSCL research already was mentioned. Much older artificial intelligence and education always had some kind of analytics for both the learner and the teacher built-in. Adaptive systems have been produced by the dozen and failed to interest teachers. An educationally more light-weight variant of tutoring systems are so-called recommended systems. These may become more popular, although attention must be payed of not atttempting to lock students into a box. E.g. one could image adding 20% of "noise" (i.e. recommendation that are rather far as opposed to close). That's how genetic algorithms work.
New in learning analytics are the ability to track students across environments (e.g. all social services) and the interest for using analytics as evaluation tool az institution level and above. This leads to privacy issues but not just. Analytics can have intrinsic adverse effects. See the next item.
Analytics in other fields often had an adverse effect. In public policy, monitoring and reporting tool had the effect to blow up procedures, kill initiative, remove communication, etc. Analytics can (but must not) lead to absolute disasters both in terms of both cost and effectiveness. Reliance on key indicators and reporting can bring out the dark side of management: “toxic leadership, managerial worst practices, project failures and suffering at work.” (Stefano Mastrogiacomo) and increase the power of bureaucracy at the expense of individual creativity and communication. In education that could mean: Teachers will have to use mind crippling e-learning platforms (else the organization can't collect data), participants will learn how to produce data for the system (as opposed to produce meaning) and participants will talk less to each other (since time has to be spend on collecting and archiving data).
Why don't teachers ask more information from the learners (self-reporting)? Many tools allow students to tell what they did and in addition to reflect upon, and teachers can require it. For example, LAMS activities can be configured with a (mandatory) notebook to be filled in at the end. Wiki's such as this one include personal home pages where students can add links to their contributions. Actually, tools we found for project-oriented designs such as inquiry learning or portfolio construction do this.
Are there analytics that can distinguish between development of routine expertise, as opposed to adaptive expertise (new schemas that allow to respond to new situtations) ? (Alexander 2003, cited by Werner et al. 2013)).
Learning and Knowledge Analytics Bibliography by George Siemens (et al?).
LAK 2012. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. ACM, New York, NY, USA.
Learning and Knowledge Analytics, devoted to learning and knowledg. Managed by G. Simons.
Topic Learning Analytics at OU Netherlands. Includes an organisational vision/model of LA.
Learning Emergence Website of Simon Buckingham Shum. Critique of naive learning analytics dreams, researcher and co-author of very interesting "learning-process oriented software".
A Netvibes page made by G. Siemens pulls together various online productions.
Learning Analytics and Knowledge: LAK12. The course is full but you can consult blog posts, contents in wikispaces, etc.
By Simon Buckingham Shum, jan 2012.
Learning Analytics: Dream, Nightmare, or Fairydust?
what if we could observe all events in a learning-environment ? (Abelardo Pardo, April 2012).
Week 6: Tools, methods, and levels of learning analytics (Part of the Lak12 online course). This wiki page includes links to other lists of tools.
What We Don’t Know About Learning Analytics Posted on May 29, 2011 by Michael Feldstein.
Learning Analytics - Dream or Nightmare annotated slides by Simon Buckingham Shum, April 13, 2012.
Learning analytics: Starvation and telling us what we already know? Blog post by David Jones, April 2012.
What are the really useful analytics? David Jones, March 2012.
MMOC LAK12 ending. “I see the power of LA to give the interested learner data about self and others and the learning s/he wishes to accomplish.” Posted by C. Galas on April 11 2012.
3 Ways Educational Technology Tools Predict Student Success by Eric Horowitz, Jul 13, 2015.
Anjewierden, Anjo and Lilia Efimova. Understanding weblog communities through digital traces: a framework, a tool and an example. In “International Workshop on Community Informatics (COMINF 2006)”, Robert Meersman, Zahir Tari, and Pilar Herrero (eds.), pp. 279–289, LNCS 4277, Springer, Montpellier, 2006 (November).
Anjewierden, Anjo; Rogier Brussee and Lilia Efimova. Shared conceptualisations in weblogs, in ”BlogTalks 2.0: The European Conference on Weblogs”, Thomas N. Burg (ed.), Danube, University of Krems, 110–138, 2005.
Aviv, R., Erlich, Z., Ravid, G., & Geva, A. (2003). Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks, 7(3), 1-­‐23.
Baepler, P. M., Cynthia James. (2010). Academic Analytics and Data Mining in Higher Education. International Journal for the Scholarship of Teaching and Learning, 4(2).
Baker, R.S.J.d., de Carvalho, A. M. J. A. (2008) Labeling Student Behavior Faster and More Precisely with Text Replays. Proceedings of the 1st International Conference on Educational Data Mining, 38-47.
Bienkowski, Marie; Mingyu Feng and Barbara Means (2012). Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics, Department of Education’s, Office of Educational Technology, http://ctl2.sri.com/eframe/wp-content/uploads/2012/04/EDM-LA-Brief-Draft_4_10_12c.pdf PDF (Draft version April 12 2012). See also the this announcement at CCC.
Campbell, John P.; Peter B. DeBlois, and Diana G. Oblinger (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007), pp. 40–57, http://www.educause.edu/library/erm0742.
Carla Casilli (2011). Information visualization: a new visual language. 8 parts. Available at: http://carlacasilli.posterous.com/information-visualization-a-new-visual-langua-63895.
Coates, H. (2010). New directions in quality management. In C. S. Nair, L. Webster & P. Mertova (Eds), Leadership and management of quality in higher education. Oxford, UK: Chandos Publishing: 169-186.
Cooper, Adam. A Brief History of Analytics A Briefing Paper. CETIS Analytics Series. JISC CETIS, November 2012. http://publications.cetis.ac.uk/wp-content/uploads/2012/12/Analytics-Brief-History-Vol-1-No9.pdf.
Dawson, S. M., Erica. (2008). Investigating the application of IT generated data as an indicator of learning and teaching performance. Wollongong: Australian Learning and Teaching Council.
Dawson, S., Heathcote, L. and Poole, G. (2010), “Harnessing ICT Potential: The Adoption and Analysis of ICT Systems for Enhancing the Student Learning Experience,” The International Journal of Educational Management, 24(2), pp. 116-129.
Dawson, S., Heathcote, L. and Poole, G. (2010). Harnessing ICT potential: The adoption and analysis of ICT systems for enhancing the student learning experience, International Journal of Educational Management 24(2) pp. 116-128.
Dawson, Shane. 2010. 'Seeing' the Learning Community: An Exploration of the Development of a Resource for Monitoring Online Student Networking. British Journal of Educational Technology 41(5): 736-752.
Dawson,. (2010) Seeing the learning community. An exploration of the development of a resource for monitoring online student networking. British Journal of Educational Technology, 41(5), 736-752. doi:10.1111/j.1467-8535.2009.00970.x.
De Laat, M., Lally, V., Lipponen, L., & Simons, R.J. (2006). Analysing student engagement with learning and tutoring activities in networked learning communities: a multi-method approach. International Journal of Web Based Communities, 2(4), 394-412.
Downs, Stephen (2010). Collaboration, Analytics, and the LMS: A Conversation with Stephen Downes, Campus Technology, October 14, 2010, retrieved 18:37, 19 March 2012 (CET).
Drachsler,H.,et al.(2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning .1st Workshop on Recommender Systems in Technology Enhanced Learning(RecSysTEL@EC-TEL2010)September,28,2010, Barcelona, Spain.
Dron, J. and Anderson, T. (2009). On the design of collective applications, Proceedings of the 2009 International Conference on Computational Science and Engineering, Volume 04, pp. 368-374.
Duval, E.; Verbert, K. “On the role of technical standards for learning technologies.” IEEE Transactions on Learning Technologies 1, no. 4 (October 2008): 229-234. https://lirias.kuleuven.be/handle/123456789/234781.
Eckerson, W. W. (2006). Performance dashboards: Measuring, monitoring, and managing your business. Hoboken, New Jersey: John Wiley & Sons.
Ferguson, Rebecca (2012). The State Of Learning Analytics in 2012: A Review and Future Challenges. Technical Report KMI-12-01, Knowledge Media Institute, The Open University, UK. Abstract/PDF.
García-Saiz, D. & Zorilla Pantaleón, M.E. (2011). E-learning web miner: A data mining application to help instructors Involved in virtual courses. In M. Pechenizkiy et al. (Eds.), Proceedings of the 3rd Conference on Educational Data Mining 2011 (pp. 323–324). Eindhoven, The Netherlands.
Haythornthwaite, C. (2008). Learning relations and networks in web-based communities. International Journal of Web Based Communities, 4(2), 140-158.
Hendricks, M., Plantz, M.C., & Pritchard, K.J. (2008). Measuring outcomes of United Wayfunded programs: Expectations and reality. In J.G. Carman & K.A. Fredricks (Eds.), Nonprofits and evaluation. New Directions for Evaluation, 119, pp. 13-35.
Hijon R. and Carlos, R. (2006). E-learning platforms analysis and development of students tracking functionality, Proceedings of the 18th World Conference on Educational Multimedia,Hypermedia & Telecomunications pp. 2823-2828.
Leah P. Macfadyen and Shane Dawson, “Mining LMS Data to Develop an ‘Early Warning System’ for Educators: A Proof of Concept,” Computers & Education, vol. 54, no. 2 (2010), pp. 588–599.
Libby V. Morris, Catherine Finnegan, and Sz-Shyan Wu, “Tracking Student Behavior, Persistence, and Achievement in Online Courses,” The Internet and Higher Education, vol. 8, no. 3 (2005), pp. 221–231.
Macfadyen, L. P. and Sorenson. P. (2010) “Using LiMS (the Learner Interaction Monitoring System) to track online learner engagement and evaluate course design.” In Proceedings of the 3rd international conference on educational data mining (pp. 301–302), Pittsburgh, USA.
Macfayden, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning” system for educators: a proof of concept. Computers & Education, 54(2), 588–599.
This is an often cited text defining how EDM could be directly useful to educators.
Macfadyen, L. P., & Dawson, S. (2012). Numbers Are Not Enough. Why e-Learning Analytics Failed to Inform an Institutional Strategic Plan. Educational Technology & Society,15(3), 149–163. PDF.
Mazza R., and Dimitrova, V. (2004). Visualising student tracking data to support instructors in web-based distance education, 13th International World Wide Web conference, New York, NY, USA: ACM Press, pp. 154-161. PDF, retrieved 19:41, 15 March 2012 (CET).
Mazza, R. & V. Dimitrova. (2003) "CourseVis: Externalising Student Information to Facilitate Instructors in Distance Learning" in Proceedings of 11th International Conference on Artificial Intelligence in Education (AIED03), F. Verdejo and U. Hoppe (Eds), Sydney: IOS Press.
Mazza, R. B., Luca. (2007). Monitoring an Online Course with the GISMO Tool: A Case Study. Journal of Interactive Learning Research, 18(2), 15.
Mazza, R., & Dimitrova, V. (2007). CourseVis: a graphical student monitoring tool for supporting instructors in webbased distance courses. International Journal of Human-Computer Studies, 65(2), 125-139.
Merceron, A. & K. Yacef (2003). "A Web-based Tutoring Tool with Mining Facilities to Improve Learning and Teaching" in Proceedings of 11th International Conference on Artificial Intelligence in Education., F. Verdejo and U. Hoppe (Eds), pp 201-208, Sydney: IOS Press.
Merceron, A. and K. Yacef (2005b). Educational data mining: a case study. In C. K. Looi, G. McCalla, B. Bredeweg, and J. Breuker, editors, Proceedings of the 12th Conference on Artificial Intelligence in Education, pages 467-474, Amsterdam, The Netherlands, 2005. IOS Press.
Merceron, Agathe and Kalina Yacef (2005). TADA-Ed for Educational Data Mining, Interactive Multimedia Journal of Computer-Enhanced Learning (IMEJ), http://www.imej.wfu.edu/articles/2005/1/03/, retrieved 19:41, 15 March 2012 (CET).
Najjar. J; M. Wolpers, and E. Duval. Contextualized attention metadata. D-Lib Magazine, 13(9/10), Sept. 2007.
Paulsen, Michael B. and Kenneth A. Feldman, Exploring the dimensions of the scholarship of teaching and learning: Analytics for an emerging literature, New directions for Institutional Research 129, 21-36, DOI: 10.1002/ir.169. Not learning analytics, but interesting from a technical point of view.
Rath, A. S., D. Devaurs, and S. Lindstaedt. UICO: an ontology-based user interaction context model for automatic task detection on the computer desktop. In Proceedings of the 1st Workshop on Context, Information and Ontologies, CIAO ’09, pages 8:1—-8:10, New York, NY, USA, 2009. ACM.
Romero, C. , Ventura, S.N., & Garcia, E. (2008). Data mining in course management systems: Moodle case study and tutorial, Computers & Education, 51(1), 368-384.
Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications, 33(1), 135-146.
Scott, G. (2010). TILT: UWS Tracking and Improvement System for Learning and Teaching, University of Western Sydney. PDF, retrieved 12:45, 10 April 2012 (CEST).
Siemens George; Dragan Gasevic, Caroline Haythornthwaite, Shane Dawson, Simon Buckingham Shum, Rebecca Ferguson, Erik Duval, Katrien Verbert and Ryan S. J. d. Baker (2011). Open Learning Analytics: an integrated & modularized platform Proposal to design, implement and evaluate an open platform to integrate heterogeneous learning analytics techniques, Society for Learning Analytics Research. PDF, retrieved 22:02, 1 March 2012 (CET).
Sylvan, Elisabeth Amy . (2007). The Sharing of Wonderful Ideas: Influence and Interaction in Online Communities of Creators, PhD Thesis, Massachusetts Institute Of Technology. PDF. (Good reading if you are interested in analyzing online communities of creators (OCOC) with Network Analysis.
Ternier, S.; K. Verbert, G. Parra, B. Vandeputte, J. Klerkx, E. Duval, V. Ordonez, and X. Ochoa. The Ariadne Infrastructure for Managing and Storing Metadata. IEEE Internet Computing, 13(4):18–25, July 2009.
Towers, S., Alderman, L., Nielsen, S. & McLean, S. V. (2010). A risk-based approach to course quality assurance. Proceedings Australian Universities Quality Forum (AuQF2010). http://www.auqa.edu.au/files/publications/auqf_proceedings_2010.pdf, retrieved 13:37, 10 April 2012 (CEST).
Wolpers. M; J. Najjar, K. Verbert, and E. Duval. Tracking actual usage: the attention metadata approach. Educational Technology and Society, 10(3):106–121, 2007.
Zaiane, O.R. "Web Usage Mining for a Better Web-Based Learning Environment" in Proceedings of Conference on Advanced Technology for Education (CATE'01), pp 60-64, Banff, Alberta (2001).
Zhang, H. and Almeroth, K. (2010). Moodog: Tracking Student Activity in Online Course Management Systems. Journal of Interactive Learning Research, 21(3), 407-429. Chesapeake, VA: AACE. Retrieved October 5, 2010 from http://www.editlib.org.aupac.lib.athabascau.ca/p/32307.
The Journal of Educational Technology and Society Special Issue on Learning Analytics due in 2012.
American Behavioral Scientist Special Issue on Learning Analytics due in 2012.
Constantino-Gonzalez, M.A., Suthers, D.D. and Escamilla de los Santos, J.G. (2002). Coaching web-based collaborative learning based on problem solution differences and participation. International Journal of Artificial Intelligence in Education, 13, 263-299.
Dönmez, P., Rosé, C. P., Stegmann, K., Weinberger, A., & Fischer, F. (2005). Supporting CSCL with Automatic Corpus Analysis Technology. In T. Koschmann & D. Suthers & T. W. Chan (Eds.), Proceedings of the International Conference on Computer Support for Collaborative Learning (pp. 125-134). Taipeh, Taiwan.
Fischer, F., Bruhn, J., Gräsel, C. & Mandl, H. (2002). Fostering collaborative knowledge construction with visualization tools. Learning and Instruction, 12, 213-232.
Hutchins (1995). How a cockpit remembers its speeds. Cognitive Science, 19, 265-288.
Jermann, P., Soller, A., & Muehlenbrock, M. (2001). From Mirroring to Guiding: A Review of State of the Art Technology for Supporting Collaborative Learning. Proceedings of the First European Conference on Computer-Supported Collaborative Learning, Maastricht, The Netherlands, 324-331.
Khayat, N. Mock, M., Kindermann, J. (2011). Towards automatic behavior analysis of learners in a technology-enhanced learning environment. CSEDU 2011. May 6-8, 2011. Noordwijkerhout, the Netherlands.
Kolodner, J., & Guzdial, M. (1996). Effects with and of CSCL: Tracking learning in a new paradigm. In T. Koschmann (Ed.) CSCL: Theory and Practice of an Emerging Paradigm (pp. 307-320). Mahwah NJ: Lawrence Erlbaum Associates.
Kop, R., Fournier, H., & Sitlia, H. (2011). The value of learning analytics to networked learning on a personal learning environment. First International Conference on Learning Analytics and Knowledge 2011, Banff, Alberta, Canada, February 27–March 1, 2011.
Merceron, A. and K. Yacef. Interestingness measures for association rules in educational data. In Proceedings of Educational Data Mining Conference, pages 57-66, 2008.
Mühlenbrock, M., & Hoppe, U. (1999). Computer supported interaction analysis of group problem solving. In C. Hoadley & J. Roschelle (Eds.) Proceedings of the Conference on Computer Supported Collaborative Learning (CSCL-99) (pp. 398-405). Mahwah, NJ: Erlbaum.
Settouti, L.S.; N. Guin, V. Luengo, A. Mille. (2010). A Trace-Based Learner Modelling Framework for Technology-Enhanced Learning Systems. 1Oth IEEE International Conference on Advanced Learning Technologies, Sousse, Tunisia. pp. 73-77. ISBN 978-1-4244-7144-7.
Settouti, L. S; Y. Prié, J.C. Marty, A. Mille. (2009). A Trace-Based System for Technology-Enhanced Learning Systems Personalisation. 9th IEEE International Conference on Advanced Learning Technologies, Riga, Latvia.
Soller, A., & Lesgold, A. (2003). A computational approach to analyzing online knowledge sharing interaction. Proceedings of Artificial Intelligence in Education 2003, Sydney, Australia, 253-260.
Suthers, D. D., Ravi, V., Medina, R., Joseph, S., & Dwyer, N. (2008). Beyond threaded discussion: representational guidance in asynchronous collaborative learning environments. Computers & Education, 50, 1103-­‐1127.
Stegmann, K., & Fischer, F. (2011). Quantifying Qualities in Collaborative Knowledge Construction: The Analysis of Online Discussions. Analyzing Interactions in CSCL, 247–268. Springer.
Southavilay. V, K. Yacef, and R. A. Calvo. Analysis of collaborative writing processes using hidden markov models and semantic heuristics. In Submitted to Workshop on Semantic Aspects in Data Mining (SADM) at ICDM2010, 2010.
Wang Yi-Chia, Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., & Rosé, C. P. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning, 3(3), 237–271. doi: 10.1007/s11412-007-9034-0.
Weinbrenner, S., Engler, J., Wichmann, A., & Hoppe, U. (2010). Monitoring and analysing students' systematic behaviour - the SCY pedagogical agent framework. Paper presented at the ECTEL conference. Barcelona.
Most of this article is licensed under the Creative Commons Attribution-NonCommercial 3.0 License and you also must cite cite Penetrating the Fog: Analytics in Learning and Education and Elias, T. (2011) Learning Analytics: Definitions, Processes, Potential if you reuse parts from these sources, e.g. the figures.
Some figures are not licensed for reuse and are reproduced with permission. You can't reproduce these.
Page last modified 17:05, 29 March 2017.

References: V. 
 V. 
 V. 
 V. 
 V. 
 V. 
 V.