Analyzing and visualizing student work with BeSocratic

Today, most higher education institutions use broad learning management systems such as Blackboard, Moodle, or Instructure Canvas to aid in assessment. Additionally, specialized systems (such as Mastering software series, OWL, etc.) exist for individual disciples and courses. While these systems have shown improvements in student learning, the majority of questions they may ask fall into one of two categories: free response text-based questions or multiple-choice/matching questions. Free-response systems allow teachers to ask meaningful questions which require students to have deep understanding of the subject in order to answer correctly. More restrictive questions use simple multiple choice or matching questions. Research has suggested that multiple choice or matching questions cannot be used to properly assess deep knowledge on a subject [1,2]. The exercises often only involve memorization. Which system a teacher uses depends on the amount of time the teacher has to evaluate student solutions. Evaluating student answers from free response systems requires teachers to manually check each submission; this is too time-consuming for teachers to perform on a regular basis. Instead, teachers rely on using multiple-choice or matching questions that can be quickly or automatically evaluated. The ideal system combines the best parts of both types of systems. A teacher would be able to ask questions that require students to have a deep understanding of the material and reply in an intuitive free form manner. In this ideal system, student responses would be automatically evaluated and analyzed. Furthermore, the analyzing large amount of student data, especially free form data, can be a laborious and time-consuming task. Motivated by all of this research, we have developed a novel pedagogical system, BeSocratic, which focuses on evaluating visual representations to give students individualized and meaningful instruction and assessment. BeSocratic recognizes free-form student input and provides students with meaningful feedback to improve their problem solving ability. Furthermore, BeSocratic analyzes collected student data in a variety of ways allowing teachers to better manage and track progress within their classrooms. It is in this manner that BeSocratic is able to ask students meaningful questions requiring free-form input based on deep subject knowledge and provide automatic evaluation that is necessary for use in large classrooms. This poster will explain how to use BeSocratic to assess and analyze student work in a variety of ways. BeSocratic activities are composed of one or more activity steps. Each step is designed and developed using one or more modules. BeSocratic modules can be divided into two general categories: non-interactive and interactive. Non-interactive modules (which include text boxes, images, videos, and ink canvases) do not provide students with feedback. Interactive modules allow BeSocratic to pose free-form questions and are able to give automatic feedback to the student and to the teacher. Such modules include: SocraticGraphs, OrganicPad, and GraphPad. While some of the BeSocratic modules are targeted for use with a Tablet PC or touch-enabled device, it may be used with mouse and keyboards as well. The GraphPad module is designed specifically to help computer science students learn a variety of data structures. It uses ink recognition to convert student drawings into interactive nodes and edges that may be evaluated and analyzed. Using a solution data structure provided by the teacher, GraphPad checks for differences between a student's structure and the teacher's solution. GraphPad gives feedback based on the results of that comparison. GraphPad recognizes nodes, undirected and directed edges, and labels for nodes and edges. This simple design allows the construction of practically any graph structure that may be found in a computer science data structures class including: lists, stacks, trees, generic graphs, and finite state automata. GraphPad was originally created for desktop and laptop computers where it showed positive results in computer science data structure classrooms [3,4]. GraphPad has been ported to BeSocratic where it may be used on a wider variety of devices. Figure 1 shows a student who has drawn a binary search tree using the GraphPad module. Each action that a student makes in the system is recorded in a list. This list can then be reused to replay the student work. BeSocratic's ability to replay student work helps teachers gain deeper insight into the cognitive processes of students working on their assignments. Using the replay feature, teachers can identify where a student is having difficulty. By identifying errors in this fashion, teachers can refine their instruction to address the root of the mistakes. In addition, a teacher has the option to (anonymously) project a student's solution onto a screen and use it as a part of lecture. We have successfully used this technique in previous research teaching data structures with GraphPad [4]. In order to view a large amount of student sequence data at one time, BeSocratic displays each sequence in a compact form. When creating these sequence visualizations, BeSocratic first creates a list of the unique tags used in all of the selected submission sequences. Then it maps a unique color to each one of these unique tags so that the sequence tags can be represented by colored bars displayed in a list (see Figure 2). Tool-tips are implemented such that a the tag's description along with a snapshot of the student's replay appears when the mouse cursor hovers over a particular tag (colored bar). Using this compact representation, BeSocratic contains several clustering methods and visualizations to help teachers identify students who potentially need help. These methods include: spectral clustering, physically-based modeling, hierarchical clustering, and hidden Markov modeling. For example, BeSocratic is able to generate dendrograms based on the results of hierarchical clustering. We generate our hierarchical clustering using a "bottom-up" or agglomerative algorithm that initially starts with each sequence placed inside its own set. The algorithm repeatedly finds the two most similar sets (using a Levenstein distance) and places them inside their own set. This process repeats until there is one set left consisting of all of the sequences. Once finished, the algorithm has built a tree of sets that range from a root with all sequences to the bottom leaves with only one sequence each. An example of a dendrogram generated from student data is shown in Figure 3. Each student sequence is connected to each other through the tree. The height of the line from the bottom level of the tree corresponds to the dissimilarity between the two sets it joins. For example, notice how the root level is the furthest from the bottom level since the root connects the two most dissimilar sets. With our tool, you may move a line to 'pierce' the dendrogram. Clusters of sequences corresponding to the dendrograms branches that were pierced are shaded with the same color. Notice the 3 branches pierced and the resulting 3 clusters in Figure 3. Also note that building a dendrogram also orders the sequences such that similar sequences are grouped closer together. We have similar visualization for our other analysis methods. The goals of these techniques are to assist teachers in the analysis of their student work without requiring teachers to replay each student submission. The visualizations, such as the dendrogram shown above, can potentially be used to identify students who require assistance as well as the problems they are experiencing. Along with a poster describing BeSocratic, we will be performing live demonstrations of BeSocratic running on Tablet PCs and iPads. We will demonstrate to attendees how to use BeSocratic in their classrooms to assess, collect, and analyze student work. BeSocratic is an online tool and is available to use for free at BeSocratic.clemson. edu.