DOI: 10.5061/dryad.k0p2ngfgj
title: Computing integrated activities scored for programming concepts

description: Educators across disciplines are implementing lessons and activities that
 integrate computing concepts into their curriculum to broaden
 participation in computing. Out of myriad important introductory computing
 skills, it is unknown which—and to what extent—these concepts are included
 in these integrated experiences, especially when compared to concepts
 commonly taught in introductory computer science courses. Thus, it is
 unclear how integrated computing activities serve the goal of broadening
 participation in computing. To address this deficit, we compiled a
 database of 81 integrated computing activities, constructed a framework of
 fundamental programming concepts, and scored each activity in the database
 for the presence of each concept. The dataset also includes different
 activity features, including discipline, programming language, student
 age, and duration of activity. 
description: Selection Criteria: Features and Limitations
 Non-CS Disciplinary Learning Objectives The
 first selection criterion for activities to include in the analysis was
 the inclusion of learning objectives in a discipline other than computing.
 No restrictions were placed on which other disciplines qualified, and we
 found activities from language arts, math, science, art, music, foreign
 language, history, social studies, and even spatial skill development for
 young children. One indirect benefit of requiring non-computing
 disciplinary learning objectives was that many included activities have
 substantive lesson plans. These lesson plans make the activities more
 accessible to teachers by including TPACK-related information, such as
 disciplinary learning objectives for the activity. As a result, the
 authors recognize the limitations of requiring non-computing learning
 objectives but also that it provides a level of authenticity and
 accessibility for the included activities. One of the
 major sources of computing integration activities affected by this
 requirement was the ScratchEd website. Scratch is a popular language for
 computing integration activities, aided by an extensive repository of
 student- and teacher-created projects that users are encouraged to remix
 into their own projects. The thousands of programs in this repository are
 of widely varying complexity and quality, and most of them are listed with
 a topic but without explicit learning objectives. To draw from this wealth
 of activities without comprehensively including projects, we identified
 lists of vetted computing integrated activities using Scratch to include
 in the analysis. These lists were "Integrated Scratch Programming in
 the Curriculum," "Scratch Projects Across the Curriculum,"
 "From Music to Math: Scratch Across Every Subject," and
 "Scratch Cross-Curricular Integration Guide." Similarly,
 resources related to the Snap! language had plentiful examples of projects
 across disciplines with limited explicit non-CS disciplinary learning
 objectives. Block-Based Programming
 Languages Because computing integration activities are
 becoming popular, an initial search revealed too many activities to score
 in one analysis. To narrow the scope of the analysis, the next selection
 criterion was that the activity had to use a block-based programming
 language. This criterion has benefits and limitations. One of the main
 benefits for the goal of the current analysis was that block-based
 activities include a range of concepts, regardless of their syntactic or
 semantic difficulty (Grover & Basu, 2017; Papadakis et al., 2014).
 This benefit means that concepts that best serve the activity can be
 included for learners with little to no programming experience (Weintrop
 & Wilensky, 2018). The associated limitation, however, was that
 concepts are also restricted by the blocks that are built into the
 language. Most popular languages use a low-floor, high-ceiling design that
 includes blocks for all concepts that would be taught in an introductory
 programming course, though (Grover, 2021; Weintrop & Wilensky,
 2015). Another limitation was that prominent, text-based integration
 activities, such as Bootstrap’s curricula in Algebra and Physics, are
 excluded. This selection criterion also notably
 excluded commonly used science simulation platforms, like NetLogo and
 PhET. These platforms include a large range of simulations for scientific
 phenomena and other models beyond science. While the simulations allow
 users to easily access the source code, the primary interface does not
 include the program used to create the simulation. In addition, the source
 code, except for some adapted NetLogo simulations, is text-based. Though
 the programs are heavily commented to make them understandable, they do
 not meet the inclusion criteria for the current dataset. More
 programming-centric and block-based options for scientific simulations,
 like StarLogo Nova, were included. Access
 Accessibility of the activities was the final criterion for
 inclusion. Following the accessibility criteria used by Lin and Weintrop
 (2021), we included activities only if they could be found online, were
 free of cost, did not require a physical device like robotics toolkits,
 and were updated recently enough that they ran on current versions of
 languages and operating systems. The requirement to be found online is not
 expected to substantially narrow the analysis because Lin and Weintrop
 found that 90% of block-based programming languages ran in a web browser.
 Exclusion for use of physical devices is a corollary to the requirement to
 be free of cost. We felt that these criteria would result in a dataset
 that had the broadest and most equitable applications because many public
 schools in low-income areas in the US cannot afford physical computing or
 robotics kits. Search Criteria
 Users need to recognize that the current dataset was based on a
 review of computing integration activities but not a
 <em>systematic</em> review. Unlike systematic literature
 reviews of scholarly work on a given keyword or topic area, there were no
 databases of indexed computing integration activities that span our
 inclusion criteria. Some repositories for certain languages exist, such as
 ScratchEd’s repository of Scratch projects and the Exploring Computational
 Thinking repository of Pencil Code and Python activities. However,
 computing integration activities are not published through a central
 organization, so they can be difficult to find. In lieu
 of a systematic review, we attempted to build a database that represented
 activities from a variety of disciplines, student ages, designers, and
 languages. To create this database, we included any activities that we
 were already aware of, such as Action Fractions, links from lists of
 computing integration activities, such as "Scratch Projects Across
 the Curriculum," links from CSforAll’s curriculum directory, and a
 general Google search for "‘integrated computing’ activities"
 and "‘computational thinking’ + programming" or
 "‘computational thinking’ + coding." We examined the first 100
 returns for these searches. However, many of the activities found through
 Google search were excluded based on our criteria, primarily for not
 including non-CS learning objectives. We included
 activities as whole units, whether they were single-class lessons or
 extended curricular units that included multiple lessons, like Coding as
 Another Language. Treating individual lessons from curricular units as
 individual activities would have created an over-representation of
 extended units (e.g., 72 lessons for the Kindergarten, 1st, and 2nd grade
 curricular units from Coding as Another Language instead of 3 activities).
 Our database included 81 activities from the following sources:
 • CANON Lab • Code.org’s CS
 Connections • Code.org’s Hour of Code
 • Coding as Another Language curriculum • CS+
 units from University of California San Diego •
 CSforALL’s Curriculum Repository (including 144 curricular units at the
 time of searching) • CT4Edu •
 Everyday Computing • Exploring Computational
 Thinking • Google search • Google’s
 CS First • Integrated computing activities from Georgia
 State University • Project GUTS •
 ScratchEd • The Tech Interactive •
 TVO Learn • UCL Scratch Maths We
 analyzed the distribution of these activities’ characteristics based on
 primary discipline, student age, programming language, and minimum time to
 complete. Based on discipline, we recognized that we had only two from
 history or social studies and searched for additional activities. While we
 found many projects on ScratchEd’s website, they did not meet the
 selection criteria. Required courses, including Language Arts, Math, and
 Science had a sufficient number of activities, matching their
 representation in the school day. We also had a wide range of activities
 based on student age and minimum time to complete, so we did not search
 for any additional activities based on these characteristics.
 To explore the representation in our database based on
 programming languages, we used the categories identified by Lin and
 Weintrop (2021) to ensure coverage of different types of block-based
 languages. The database has activities from Pencil Code (i.e., block-based
 implementation of a text-based language), Scratch (i.e., multimedia
 focused on animations and storytelling), AppLab (i.e., mobile app
 development), StarLogo Nova (i.e., simulations), and ScratchJr (i.e.,
 pre-reading language). We decided against requiring languages from Lin and
 Weintrop’s other categories for data science, physical computing, and
 task-specific languages because they did not match our inclusion criteria.
 We explored other common languages to include, like Alice, Snap!, and App
 Inventor, but we did not find activities that matched our
 criteria.
description: # Computing integrated activities scored for programming concepts
 [https://doi.org/10.5061/dryad.k0p2ngfgj](https://doi.org/10.5061/dryad.k0p2ngfgj) **Associated journal article:** Margulieux, L. E., Parker, M. C., Uzun, G. C., & Cohen, J. D. (2023). Levels of computing concepts used in computing integration activities across disciplines. *Journal of Technology and Teacher Education, 31*(2), 167-202. Waynesville, NC USA: Society for Information Technology & Teacher Education. [https://www.learntechlib.org/p/221815/](https://www.learntechlib.org/p/221815/) Please contact Lauren Margulieux, lmargulieux@gsu.edu, for additional information about this dataset. The listed computing integrated activities were scored for which programming concepts they included. A blank cell means the activity did not include the concept, and a 1 means it did. Activities were scored binarily, either they included the concept or not. The number of times the concept is used within an activity was not counted to avoid uneven representation across shorter or longer activities. ### Initial Programming Concept Framework Integrated computing activities are often theoretically framed in a CT framework, especially for science and math activities for which national standards (e.g., the Next Generation Science Standards and Common Core) explicitly include CT. Thus, we explored CT frameworks for the programming concepts that they include (Aho, 2012; Armoni, 2016; Barr & Stephenson, 2011; Brennan & Resnick, 2012; Denner et al., 2012; Denning, 2017; Grover & Pea, 2013; Lye & Koh, 2014; Palts & Pedaste, 2020; Tang et al., 2020; Weintrop et al., 2016; Wing, 2010; Yadav et al., 2014). As Tang et al. (2020) highlight in their systematic review of CT frameworks aimed at assessment, many of the frameworks focus on problem-solving competencies rather than programming concepts. The main exception to this trend is Brennan and Resnick’s (2012) framework, which includes a category for computational concepts that lists sequences, parallelism, loops, events, conditionals, operators, and data. We used these concepts as the initial foundation for our scoring scheme. In addition, Weintrop et al.’s (2016) definition of CT for math and science integration includes specific data practices, such as creating, manipulating, and visualizing data. We transformed these practices into programming practices, such as inputting variable values, using operators to calculate variables, and various visualization concepts. To supplement the CT frameworks, we also considered how different block-based programs grouped blocks into menus. These groupings were recently analyzed in Lin and Weintrop’s (2021) landscape report of block-based programming languages. Their analysis of 36 languages grouped blocks based on concepts. The conceptual groupings included: variables/data structures, logic operators, sprite/character appearance, operators (numerical, textual, color), customize block (i.e., function), movement, sound, sense/input, interacting with the physical/virtual environment, output, debug, comment, and extension. These categories were added to our initial conceptual framework. From this theoretical basis, we began an iterative scoring of computing integration activities. Our goal was to refine the framework based on concepts that were missing and differentiate large categories into more specific concepts, such as separating the concept of loops into *for* and *while* loops. ### Revisions to Programming Concept Framework Our full, revised programming concept framework can be found in Table 1, with definitions and examples for each concept that we considered when analyzing the integrated computing activities. When considering each concept definition, it is important to differentiate the role of the programmer and of the user, which can often be the same person in integrated activities. The programmer is the person creating the program by interacting with the blocks, such as initializing a variable. The user is the person interacting with the program as it runs, such as responding to prompts to change the value of variables. More details about the evolution of the framework can be found in Margulieux, L. E., Parker, M. C., *Uzun, G. C.*, & Cohen, J. D. (2023). Levels of computing concepts used in computing integration activities across disciplines. *Journal of Technology and Teacher Education, 31*(2), 167-202. Waynesville, NC USA: Society for Information Technology & Teacher Education. [https://www.learntechlib.org/p/221815/](https://www.learntechlib.org/p/221815/) **Table 1** *Definitions of programming concepts used in integrated computing activities with examples* | ***Concepts*** | ***Definitions*** | ***Examples*** | | :---------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------- | | \*\*Algorithms \*\*– nature of the program | | | | Sequence | The order of blocks was critical to correct execution | Defining the variables before using them to calculate other variables | | Parallelism | Used multiple code sequences in the program | Multiple sprites programmed with their own behaviors | | \*\*Variables (and Data Structures) \*\*– how data is stored or handled | | | | Variable | Used a variable | Assigned a string to the variable "adjective" and used that variable later in the program | | sub-concept: +=/change | Used a += or change block to calculate a new value for a variable | Increase the value of variables to visualize the Fibonacci sequence | | sub-concept: calculated variable | Used arithmetic operators or the value of other variables to calculate the value of the variable | Subtract the value of other angles from 180 to find the value of the last angle in a triangle | | List | Used a list to assign multiple values to a variable | Used a list to provide multiple options for the variable "adjective" | | **Operators –** how data is processed or analyzed | | | | Arithmetic | Used +, -, \*, /, or other operators to calculate numerical values | Divide a wavelength in half to draw the first half of the wave | | Relational | Used &lt;, &lt;=, &gt;, &gt;=, or == | Compare the value of different fractions | | Boolean/Logical | Used and, or, not, true, false, or other logical operators | Determine the quadrant of a graph based on the x- and y-axis value | | \*\*Loops – \*\*how code is repeated | | | | For loop | Used a loop that repeated for a specific number of times entered by the user or a forever loop that repeated infinitely | A program that repeats a sequence multiple times to draw a symmetrical shape | | Sub-concept: loop index | Used an index to count or control how many times to loop | An index within a loop that counts the number of times a loop has run | | While loop | Used a loop with a termination condition | Loops until a variable becomes negative | | sub-concept: nested loop | Used a loop within another structure, like a loop | Repeat a drawing sequence within a larger repeated sequence to draw symmetrical shapes | | \*\*Conditionals – \*\*how the program makes decisions | | | | If-then | Used a statement to determine whether a condition was met to determine whether code should be executed | Compare a variable to a predetermined value to determine whether to display a message | | If-else | Used a series of statements to determine which matched a given condition | Compare a variable to a predetermined value to determine which message to display | | sub-concept: nested conditional | Used a conditional within another structure, like a loop | Repeat a conditional statement for the changing value of a variable within a loop | | **Functions –** chunks of code that are easy to reuse | | | | Define/call | Called a function defined by the user | Define a function that draws a shape | | Parameter | Included a parameter as part of the function definition | Define a function that draws and shape using a parameter for the size of the shape | | **Visualization –** how components move around the screen | | | | Movement | Moved sprites or objects to visualize a process | Characters in a story move throughout the story | | sub-concept: pen up/down | Used the pen up or pen down blocks | Using pen up and down to differentiate between different shapes | | Cartesian | Used Cartesian coordinates to determine the location of a sprite or object | Start a character in the bottom left corner by setting their location to negative x and y values | | Angles | Changed the direction of sprites or objects using angles | Draw a triangle based on the angles of vertices | | \*\*Components – \*\*audiovisual components to be visualized | | | | Multimedia | Added a multimedia component to the program such as a sprite, sound, pen, or stamp | Add two characters and a background to a digital story | | sub-concept: button | Added a button for the user to click to trigger an event | Add a button that says "Click here when ready" | | sub-concept: counter | Added a counter for the number of times an event has occurred | Count how many times a sprite has touched another sprite | | Properties | Changed the default properties | Change the color of a pen | | \*\*Output – \*\*the program communicating with the user | | | | String | Displayed a string entered by the programmer | A character is programmed to say a line | | Variable | Displayed the value of a variable, either string or numeric | The program prints the final value of a variable | | String and Variable | Displayed a string and value of a variable in the same output statement | A character says “And you picked”, + a variable selected by the user | | \*\*Input – \*\*the user communicating with the program | | | | Event | Triggered the execution of code based on the action of the user, sensor, or internal feature | A character in a story won’t talk until the user clicks them or a message is broadcasted | | Variable: String | The user entered a string into the program | A poem generator asks for a verb | | Variable: Numerical | The programmer entered a starting variable value or asks the user for input | The user picks a number | | Cleaning/transforming | Used a cleaning or transforming process on user-inputted data | Change all text to lowercase | ## Scoring Activities for Concepts Activities were scored by two of three raters for whether they included a concept. We decided to score activities for whether they included a concept rather than the number of times a concept is used. Our goal was to determine which concepts are more commonly used across computing integration activities, and scoring for the number of times a concept was used in an activity might have skewed the results based on incidental features of the activities. For example, if an activity used five variables, the conceptual knowledge required is not necessarily different than if an activity used two variables. Thus, scoring for times a concept was used was more likely to result in over-representation of the concept rather than information about which concepts were most commonly used in computing integration. The raters were three CS education researchers. Two have a Ph.D. in areas related to CS education, one from educational psychology and one from CS and have worked and published in computing education for at least five years. The other has a Master’s degree in CS and is earning a Ph.D. in instructional technology with an emphasis on CS education. Each activity was scored by at least one rater with a degree in education and at least one rater with a degree in computer science. To determine interrater reliability, we used Cohen’s Kappa. Cohen’s Kappa is appropriate to compare binary data from two raters, such as whether a concept is used in an activity. Cohen’s Kappa compares the observed agreement, in this case 96%, to the chance of agreement, calculated from the base rate of each binary option from each rater. For example, if rater one marked 80% of concepts as present in each activity on average while rater two marked 20% of concepts as present, then the chance of agreement would be low. A higher Kappa indicates higher agreement, with 0.81 to 1.00 indicating nearly absolute agreement (McHugh, 2012). For the current study, Kappa was 0.92, indicating nearly absolute agreement.