Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

Promoting Scientific Literacy and Effective Citizenship through Standards 

Education 

I. Introduction 

The Center for Integrative Studies in General Sciences assessment team conducts the 

assessment and evaluation of the learning gains made by students enrolled in CISGS courses; our 

data collection includes surveys and student artifacts such as, mid-term and final examinations. 

Electronic surveys are collected at the start and end of the course, the objective is to understand 

participants’ perceptions and awareness about standards and standardization processes. This 

report summarizes the data collection for the target courses in Spring 2018. We present a 

summary of the analyses of student artifacts (mid-term and final exams). 

Table 1. Data Summary for Spring 2018 

Course Data Source Status 

ISB 201- section 01 

Pre- and post-survey N=160 

Mid-term and final exams N=15 

ISB 201- sections 02 & 04 

Pre- and post-survey N=320 

Mid-term and final exams N=30 

ISP 203A- sections 01 & 03 

Pre- and post-survey N= 378 

Mid-term and final exams N=30 

Our project goal was to enhance undergraduate students’ understanding of standards and 

standardization processes. To achieve this goal, we developed standards education modules to 

help students achieve the following standard-related learning goals: 

• Understand what standards are and the processes involved in their development. 

o Relative to stakeholders 

o Relative to the role of science 

• Explain and predict the implications (ethical, economic, social, etc.) associated with the 

development, implementation and sustained application of standards. 

The modules were implemented in three Integrated Studies Biology (ISB) courses during Fall of 

2017 (pilot phase) and Spring 2018. Figure 1 summarizes the instructional implementation. 

1 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

Standards Modules Implementation 

ISP 203A 

ISB 201 

ISB 201 

Figure 1. The left column highlights common elements of instruction; course-specific themes are 

included in the boxes. 

II. Summary Analyses: Midterm and Final Exams 

The focus of the target courses was environmental science and environmental justice, but each 

course approached and emphasized the subjects using different perspectives. Students enrolled 

in the target courses completed midterm and final examinations. These exams were common 

across the courses. Appendix 1 includes a summary of the questions for each of the exams. Table 

2 shows the exam questions and their alignment to the standard-related learning goals. 

It is important to note that the objective of these analyses was not to determine the 

“correctness” of the responses, rather to determine if there was increased awareness and 

understanding relative to standards and standardization processes. We looked for changes in the 

responses (from midterm to final) that demonstrated added complexity in relation to standards 

and standardization processes. 

Table 2. Alignment between the exam questions and the standard learning goals. 

Standard-related Learning 

Goals 

Midterm Question* Final Question* 

Description of standards Describe what technical 

standards are AND provide 

an example of a technical 

standard related to water 

quality (ISP) or related to 

pesticides (ISB201) 

If you were able to implement a 

Technical Standard to combat [water 

quality concern (ISP) or climate 

change (ISB)] as head of the Michigan 

Department of Environmental Quality 

(MDEQ), what would that standard 

be? 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

Standard-related Learning 

Goals 

Midterm Question* Final Question* 

Role of science in the Describe or explain the role of Clearly explain the science that would 

development of standards science in the development of 

[water ISP or pesticide (ISB)] 

standard 

justify your decision in adopting / 

developing this specific standard and 

how it would target [water quality 

concern (ISP) or climate change (ISB)] 

Role of stakeholders in the Describe three stakeholder List and clearly describe THREE 

development of standards groups associated with 

[course-chosen] standard 

significant Stakeholders with diverse 

perspectives that would be 

influenced/impacted and why they 

would care. 

Ethical implications 

associated with standards 

and standardization 

processes 

Describe how implementing or 

NOT implementing technical 

standards can have 

ethical implications for certain 

groups in society 

Explain and predict two implications 

(ethical, economic, social, etc.) 

associated with the development and 

implementation of that Standard. 

* When relevant, the distinction between the courses (ISP and ISB) is noted in brackets. 

A. Coding Process 

Our coding process was designed to help us determine instances where students augmented 

their ideas and used more complex arguments in their responses (i.e. incorporating more 

elements related to standards and standardization processes). 

For each of the standard-related learning goals, we categorized the exam responses based on 

their level of complexity, going from those that offered a basic/general response to more 

complex responses that included additional information related to standards. Each response was 

assigned a numerical code from 0 to 3 (Table 3). 

Table 3. Criteria for Response Complexity 

Criteria (numerical code) 

Vague response, unrelated to the question, or “I don’t know (0=Too vague) 

Provides a general/basic description. (1=Basic) 

Mentions types of systems/processes likely to be influenced by standards. 

(2=Medium) 

Mentions potential implications for people & environment, and/or mentions the 

potential effects of using the standard. (2=Medium) 

Response incorporates several aspects of standards and standardization processes. 

(3=High) 

To illustrate the coding process, table 4 presents one exemplar question from the midterm 

(related to description of standards) and one exemplar question from the final exam (related to 

3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

the role of science) and shows exemplar student responses with the assigned numerical codes 

indicated in parenthesis. 

Table 4. Exemplar Responses for Midterm and Final Exams 

Midterm* (numerical code) Final** (numerical code) 

“Technical standards are the standards that “By controlling and decreasing waste going into 

determine the quality of water” (0); bodies of water, we can protect the water quality. 

“It can't be toxic especially for the food than [sic] Less harmful waste in lakes the better the lake." (1); 

people can eat" (0); "Methane and CO2 both are a byproduct of cars, by 

“Technical standards are instructions to a certain making cars hybrid less methane and CO2 would be 

item, a way to use it or how to do something. It emitted into the air, reducing greenhouse gases." (1); 

given [sic] instructions on the accuracy of "CO2 has a high GWP, which means it can absorb a 

something. An example would be a pipe that lot of heat, which will lead to warmer temperature. 

transforms water has instructions to go with it to Trees pull in CO2 during the photosynthesis. Having a 

test the accuracy of the water” (1); tree/plant a tree in your yard would help remove 

"Technical standards are rules (laws) for quality CO2 from the atmosphere, less CO2 in the 

to ensure a certain level. One example is limiting atmosphere leads to less warmer climates." (2); 

presence of lead in water to a certain (low) "fertilizers are a big problem for water quality/ 

amount of ppm."(1); marine life. Run off or fertilizers culturally eutropicate 

"A pesticide used in lakes in the United States is [sic] water + polluting our waters acting like natural 

causing fish to turn from male to female, and vice fertilizers to algae causing an increase in blooming. 

versa. Europe has found and banned high usage As algae blooms it takes up a lot of O2 leading to 

of the pesticide, to preserve wildlife." (2); hypoxia cause marine life to die off creating a 

"One standard associated with pesticides is that positive feedback loop" (2); 

bottles of pesticides give specific directions for "I would need scientists to conduct water surveys 

how much to spray and where in order to and experiments in order to see how much point 

minimize pesticide drift (which is when pesticides source pollution there is in our water, where it's 

end up in places they were not meant to, like mainly coming from and how much of a reduction 

residential areas, and could possibly cause harm would be beneficial but still realistic for the 

to people, animals, and the environment)." (3); companies polluting. This would help reduce the 

"Technical standards are often voluntary contamination of Michigan’s drinking water."(3); 

measures companies + organizations adopt in "Snow has the potential to reflect about 95% of light 

order to meet safety requirements, streamline back into space, avoiding being trapped into the 

production, as well as a slew of other reasons. An atmosphere, as where black asphalt reflects about 

example of a technical standard related to water 3% - 5%. Reflecting 95% of light (heat) 'into space is 

quality is that the EPA has standards for what much better for keeping temperatures consistent 

levels of lead, copper, etc. can be in drinking than absorbing 95% of light (heat), where it will be 

water in order for it to be considered safe for trapped. More so by increasing reflectivity, we are 

consumption."(3) not removing GH gasses from the atmosphere but 

decreasing the amounts of heat in the atmosphere 

able to be trapped by high GWP chems [sic]." (3) 

* Midterm exam question: Describe what technical standards are AND provide an example of a technical 

standard related to water quality (ISP) or related to pesticides (ISB201). ** Final exam question: Clearly 

explain the science that would justify your decision in adopting/ developing this specific standard and 

how it would target [water quality concern (ISP) or climate change (ISB)]. 

4 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

B. Results and Discussion 

In this section we present the results from the analyses of the midterm and final exams. We 

collected a random sample of student exams from each section across the courses (table 1). 

For each of the standard-related learning goals: Description of standards; Role of science; Role of 

stakeholders; and Ethical implications (see Table 2), we calculated the frequencies of the 

response complexity criteria (table 3). 0 = Too vague; 1 = Basic; 2 = Medium; and 3 = High 

As depicted in Figure 1 for the four standard-related learning goals students’ responses largely 

fell into the basic and medium complexity categories. The “description of standards” learning 

goal has the highest percentage of responses under the medium complexity category (46%). For 

all standard-related learning goals a small percentage of the responses fell under the high 

complexity category with “ethical implications” and “role of science” learning goals showing 5% 

of the responses in this category. 

Midterm Exam Response Complexity 

60 

0 

10 

20 

30 

40 

50

Percent responses (all courses) 

Description of 

standards Role of science Role of stakeholders 

Ethical 

implications 

Too vague 10 11 27 16 

Basic 41 45 42 55 

Medium 46 39 28 23 

High 3 5 3 5 

Figure 1. Midterm exam (summative all courses). The percentages were calculated based on the 

total responses for each of the standard-related learning goals (Ns= 59; 75; 45; and 74 for each 

respective standard-related learning goal left to right). 

5 
 

 

 

 

 

 

 

 

 

 

 

 

Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

The results for the final exam (Figure 2), show that for all the standard-related learning goals 

there are no responses under the “too vague” category as compared to the midterm 

percentages for that category. In addition, for the “role of science” and the “role of 

stakeholders” standard-related learning goals the results indicate higher percentages of 

responses shifting towards the medium and high response complexity categories. 

Final Exam Response Complexity 

80 

Percent responses (all courses) 

70 

60 

50 

40 

30 

20 

10 

0 

Description of Role of Ethical Role of science standards stakeholders implications 

Too vague 0 0 0 0 

Basic 68 20 38 42 

Medium 23 66 57 47 

High 9 14 5 11 

Figure 2. Final exam (summative all courses). The percentages were calculated based on the total 

responses for each of the standard-related learning goals (Ns= 75; 74; 74; and 73 for each 

respective standard-related learning goal left to right). 

To determine changes in response complexity from midterm to final examinations we compared 

the results for each of the standard-related learning goals (Figures 3a to 3d). Across the learning 

goals there is a gain in response complexity from the midterm to the final. 

6 
 

 

 

 

 

 

Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

Figure 3 A - D. Comparison between midterm and final exams across the standard-related 

learning goals. 

Taken together these results indicate that for all of the standard-related learning goals there 

seems to be a shift in the distribution of the complexity of responses towards more complex 

responses when comparing the midterm and the final exams. This is also apparent in the 

response examples that we included in table 4. 

We are in the process of writing a paper to submit to the Journal of College Science Teaching. In 

this publication we will compile the results of all the data analyses including surveys, student 

artifacts and instructional modules. 

7 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Project Evaluation Status Summary NIST Grant 

Prepared by: Dr. Claudia Elena Vergara December 2019 

Appendix 1 

The content for the examinations used in the assessment is summarized in the table below. 

ISB201_01 

- Midterm: 

o Identify and describe 2 distinct Standards that are likely in place relative to the 

use and application of pesticides in this situation. For each, explain the possible 

role that science would have played in its development. For each, identify 3 

distinct stakeholders who would likely have hoped to provide input for the 

determination of that specific standard and an indication as to why it would be of 

interest to them. 

o Relative to the creation of standards in this scenario, what ethical concerns might 

exist for the way standards are developed or implemented? 

ISB201_02 & 04 

- Midterm: 

o Describe one specific standard associated with pesticides, list/briefly describe 

three stake holder groups associated with that specific standard 

o Explain the role of science in the development of that standard 

o Describe one specific 'ethical' concern associated with the way this standard was 

developed or implemented. 

ISP203A_01 & 03 

- Midterm: 

o Describe what technical standards are AND provide an example of a technical 

standard related to water quality. 

o Describe the role of science in the development of technical standards 

o Describe how implementing or NOT implementing technical standards can have 

ethical implications for certain groups in society. 

Final Exam: The questions were common across the three courses. Students were asked: 

1) If you were able to implement a single Technical Standard to combat a [water quality 

concern (ISP203A_01&03); climate change for ISB201 all sections] as head of the 

Michigan Department of Environmental Quality (MDEQ), what would that standard be? 

2) Clearly explain the science that would justify your decision in adopting / developing this 

specific standard and how it would target your [course-specific, see bracket above] 

concern. 

3) List and clearly describe THREE significant Stakeholders with diverse perspectives that 

would be influenced / impacted and why they would care. 

4) Explain and predict two implications (ethical, economic, social, etc.) associated with the 

development and implementation of that Standard. 

8