Making Meaning of Data: Exploring Representations of Classroom Activities from a First Year Engineering Course

Real-time, pedagogical feedback can be useful for instructors and graduate teaching assistants in assessing the effectiveness of their instructional activities. This is especially useful in first-year engineering classes, where laboratory and team activities may be more common. The G-RATE, Global Real-Time Assessment Tool for Teaching Enhancement, is a tool to provide researchbased feedback for instructors about their classroom interactions across four areas based on the “How People Learn” framework: knowledge-centeredness, community centeredness, learner-centeredness, and assessment-centeredness. The G-RATE provides a teaching profile across these categories, as well as a chronological profile of teaching interactions. Given the huge amounts of classroom data captured over large spans of time, the process of effectively representing the depth and breadth of classroom activities becomes messy. How can data be represented in a detailed, meaningful manner that can inform educators’ teaching practices? Using data collected among 7 instructors in a first-year engineering course at a large Midwestern university, the authors present an overview of data collection techniques used to inform faculty about their pedagogical instruction. Implications for translating this data into feedback are presented along with sample profiles of faculty instruction using the G-RATE. Introduction and G-RATE Background Pedagogical feedback can be used to help instructors and graduate TAs assess the effectiveness of their teaching methods. In response to the lack of such research-based assessment tools, the G-RATE (Global Real-time Assessment Tool for Teaching Enhancement) was developed in 2010 from the VaNTH Observation System, a classroom observation tool used to assess instruction in bioengineering classrooms at Vanderbilt, Northwestern, Texas, and Harvard/MI. The G-RATE provides multidimensional pedagogical feedback to instructors about their classroom interactions based on the “How People Learn” (HPL) framework, and can be used to assess instruction in a variety of contexts. HPL considers the extent to which classroom interactions represent four categories: knowledge-centeredness (e.g. how well students learn concepts to support understanding and the development of expertise in a domain), learnercenteredness (e.g. attentiveness to the previous knowledge and beliefs learners have), assessment-centeredness (e.g. providing opportunities for learner feedback), and communitycenteredness (e.g. supporting learner abilities to collaborate and learn from each other). Users of G-RATE can use one of five roles for specific functions during the collection and analysis of classroom interactions. The administrator function allows users to modify observation parameters prior to the start of a class or lab. The observer function records real-time instructional data as code strings during a class or lab. The student function assesses students via Likert scale survey items for formative or summative use for the class or lab. The instructor function allows instructors to explore their pedagogy after a class or lab via reflective items. Last, the researcher function compiles the data collected by the other G-RATE functions. Previous papers have traced the evolution of the G-RATE through its development and initial pilot test. Representing large quantities of the captured rich classroom data in ways meaningful to instructors and researchers has been an issue of note. In the following sections, we discuss issues in visualizing big data and in representing classroom data using the G-RATE. Literature Review A shift to focusing on analyzing large quantities of data has helped provide deep insights across a variety of fields, from medicine to transportation. The term “big data” is used to describe “things one can do at a large scale that cannot be done at a smaller one, to extract new insights or create new forms of value, in ways that change markets, organizations, the relationship between citizens and governments, and more (p. 6). ” In the context of education, the terms “learning analytics” and “academic analytics” are often used to describe learners and instructors in their contexts for optimizing learning and analytics to produce action at institutional, regional, and national/international levels. These statistical analytics have often been accompanied by the use of visual thinking to illustrate data patterns and insights, and these visualizations come with their own set of unique challenges based on the type of data visualized and the technique used. Keim described six categories of data that can be visualized: (1) one-dimensional data, (2) two-dimensional data, (3) multi-dimensional data, (4) text and hypertext, (5) hierarchies and graphs, and (6) algorithms and software. Displaying large quantities of these data types can be complex due to technical challenges in data storage and processing and because of challenges in displaying the rich amounts of information in ways that are meaningful without being misleading or overwhelming. Bresciani and Eppler classified these types of visualization disadvantages in Table 1. Visualizations may contain: (1) cognitive disadvantages that impede accurate interpretation of the data, (2) emotional disadvantages where the visualization may not attract the viewer, and (3) social disadvantages that may affect who views the visualization and how. These disadvantages may be designer-induced or user-induced. Designer-induced disadvantages relate to the creation and implementation of visualizations, while user-induced disadvantages relate issues with viewing and interpreting the visualizations. These disadvantages should be considered in the design and presentation of visual data in order to present the most effect display to an audience. To best represent data in ways that minimize these disadvantages, a large number of visualization techniques have been developed. Fields like computer graphics, computer science, industrial engineering, and statistics provide a vast amount of detailed information on the mathematic and algorithmic background of visualization; for the purposes of this paper, an overview of several visualization techniques are presented to introduce the reader to the types of visualizations considered for this study. Four visualization techniques that may be combined to produce specific visualization systems include: • Geometrically-transformed displays – transformations of multidimensional data sets are geometrically displayed through techniques like scatterplot matrices and parallel