# Sequential Writing Assignments to Critically Evaluate Primary Scientific Literature

Suzanne Wood

University of Toronto

In the later years of undergraduate study, students read, process, and evaluate primary literature within specific fields of study. Shifting from textbooks to the vast amounts of peer-reviewed current literature can be difficult for students. This article details an innovative approach to helping students successfully make this transition through a series of sequential assignments based on the levels of increasing cognitive complexity in Bloom's taxonomy. These assignments are designed to be flexible enough to use in fields throughout the sciences and beyond, while allowing instructors to tailor these assignments to meet the needs of their particular students.

Dans les dernières années de leurs études de premier cycle, les étudiants procèdent à la lecture, à l'assimilation et à l'examen des principaux travaux dans des domaines d'études particuliers. Le passage des manuels d'apprentissage à la masse profuse des travaux de recherche actuels évalués par les pairs peut être difficile pour les étudiants. Dans notre article, nous présentons une approche novatrice visant à aider les étudiants à réussir cette transition grâce à un ensemble de devoirs dont la succession répond à l'échelle de complexité cognitive selon la taxinomie de Bloom. Ces devoirs sont conçus de manière flexible pour un usage dans différents domaines scientifiques et ailleurs, ce qui permet aux enseignants de les adapter selon les besoins particuliers de leurs étudiants.

We hope to provide our undergraduate students with skills that will be useful to them in both post-graduate studies as well as the workforce, in addition to helping them develop a strong fact base in a specific discipline of study. In the early years of undergraduate education, we tend to focus on bolstering that fact base using packaged and contextualized information that is found in textbooks. In later years, we ask our students to shift to primary literature, analyzing information in a critical manner that models the type of reasoning we hope students maintain long after graduation. However, this transition can be challenging for students. Primary literature tends to be laden with jargon that is discipline specific, rather than incorporating a writing style aimed at a general

undergraduate audience, as we find in textbooks. Students may also find the sheer volume of primary literature available to be daunting and may have difficulties selecting which resources to trust and incorporate into their own work.

As instructors of students facing these challenges, we should be asking ourselves how to work through this informational transition to enhance our learners' critical thinking skills that, while practiced in the classroom, will translate to critical evaluation of ideas outside of the classroom. For example: How are students to evaluate the disciplinary importance and validity of a study? Can students be taught to discern important disciplinary findings (i.e., findings that become integrated in future textbooks) and the findings that will rarely be

CELT
COLLECTED ESSAYS
ON LEARNING
AND TEACHING
2020 CELT Vol. XIII
celt.uwindsor.ca
www.stlhe.ca

<figure><img src="image_1.png" /></figure>
discussed again? Most importantly: Is there a way we
can help guide each of our students to this level of
understanding?

Previous Methods to Bolster
Undergraduate Critical Reading

Building undergraduate critical reading skills is an ongoing challenge within the pedagogical community. As a result, several methods have been developed to guide students through reading original literature with varying degrees of scaffolding and emphasis on different aspects of the articles. Some particularly useful methods include Figure Facts (Round & Campbell, 2013), CREATE (Hoskins et al., 2007), structured reading questions (Van Camp & Van Camp, 2013), and *New York Times* style writing assignments (Brownell et al., 2013).

Figure Facts is a method that directs students to fill out a detailed table, which consists of questions that focus on the figures of a scientific journal article (Round & Campbell, 2013). The emphasis of this technique is to help students move away from reading journal articles as they might read textbooks by emphasizing the text over the figures. This more closely mimics the style of discussion of original research in graduate school and beyond, in which “going through the figures” is proxy for discussing the entire paper. An advantage of this method is that students increased the time spent examining and taking notes about the figures, which resulted in students being more skilled in interpreting data and feeling less frustrated when reading primary research articles. A difficulty with this technique is that it may be difficult to generalize to fields outside of the specific sciences in which data figures are the key method for conveying data in original research (compared to social sciences, e.g.). This also leaves unaddressed the struggles students can face when reading other sections of papers, such as the introduction and discussion.

Another method involves designing a course
to revolve around a series of journal article
assignments (Hoskins, Stevens, & Nehm, 2007). The
CREATE (consider, read, elucidate hypotheses,
analyze and interpret the data and think of the next
experiment) method follows the path of a single line

of research throughout the term, allowing students to
both practice interpreting original research while
simultaneously developing a better appreciation for
the way science progresses. There are clear benefits
to this method, as students demonstrated
improvement on pre- and post-course tests of critical
thinking about data and reported generally positive
feelings about the experience (Hoskins et al., 2007).
Another version of CREATE was shown to reduce
the number of students who failed to complete a
general education neuroscience course (Bodnar et al.,
2016). The major hindrance to generalizability is that
this technique necessitates reading journal articles
that all revolve around a single topic. This may be
limiting for instructors whose courses range in topics
more broadly.

General critical reading assignments have
also been incorporated in the classroom, with
questions focused on locating the thesis of a work
and listing supporting evidence. One study examined
performance on pre- and post-course tests of critical
reading skills after a series of structured reading
questions, and found improvement, with students
reporting they found the reading questions helpful
(Van Camp & Van Camp, 2013). As the content of
the tests were not counterbalanced, and there was no
control group for comparison, the findings of this
study are difficult to interpret. However, the principle
of repeated practice of critical reading throughout a
term is sound, and this work can be viewed as another
example for educators to draw from.

Another course used frequent writing assignments and in-class discussions to enhance scientific understanding (Brownell et al., 2013). The writing assignments were modeled after the style of *New York Times* summaries, with repeated rounds of drafts and feedback built in throughout the term. Pre- and post-course questionnaires revealed that this method enhanced students’ confidence in comprehending and communicating about scientific research. The concern with this method is its scalability, which is dependent upon departmental resources and support from graders or teaching assistants. However, alternate versions of this assignment with fewer rounds of feedback may also provide beneficial results and could be used in larger classes with fewer resources.
In sum, there are a number of outstanding resources already developed to assist with primary literature comprehension. However, these resources are not always a good fit due to practical limitations such as the generalizability across disciplines and time constraints that affect the number of assignments that can be included within a course. The new method presented here is a series of assignments designed to be flexible in numerous contexts (e.g., student experience level, class size, discipline) and can be used progressively or in isolation.

## Bloom's Taxonomy and Critical Reading

These assignments model the levels of increasing cognitive complexity in Bloom's taxonomy (Anderson & Krathwohl, 2001) starting from basic comprehension and continuing through to the creation of new ideas in order to create a method that is useful for a range of skill levels. Bloom's was used as a framework to design these assignments because it is a model we use frequently to envision how our learners progress from novice to expert levels of understanding. Thus, it is an empirical question as to whether or not the level of difficulty is truly increasing along the progression of assignments.

Categorizing the assignments by their relevance to the different levels of Bloom's taxonomy also allows an instructor to quickly gauge which assignments might be most appropriate (see Table 1). In this way, if an instructor's course were to consist primarily of novice psychology students (e.g., lower-division, or first- or second-year courses), the instructor may choose to use the “Understand” and “Apply” assignments, and forego the rest. Likewise, if an instructor were to teach more advanced students, they may choose to start with the “Analyze” assignment, followed by the “Evaluate”, and/or “Create” assignments, while skipping the “Understand” and “Apply” components. A mixture is likewise possible, in which an instructor has their students provide a summary of an article (“Understand”) followed by a critique (“Evaluate”).

For example, in one of my third-year courses, I start out by modeling how to write a summary in class, emphasizing the distillation of

important concepts and the challenge of writing succinctly. Students are then left to write a summary and an article comparison (“Analyze”) for one of several assigned journal articles before the midterm. The assignment is due before the class in which the article is discussed, so that students may better showcase their own level of understanding and analysis in their written assignments. Having multiple articles to choose from as well as multiple due dates allows students to pick an article to analyze that they find interesting, or, alternately, pick an article to analyze whose due date is during a week that is lighter in terms of coursework in their other classes. I highly recommend this kind of small intervention that allows more flexibility to our students. They appreciate the choice, and I appreciate the ability to grade a large number of assignments over the course of several weeks rather than all at once. My students receive feedback on their work and use it to guide their summaries and critiques due during the second half of the term.

## Description of Assignments

### Understand

One of the first challenges for students learning how to read and distil primary literature is understanding the major implications of the work. Students may initially focus too much on details such as the individual references cited that, while relevant, are not central to the point of the paper. This assignment is a first step at challenging students to understand the importance of a paper. They are required to cut through jargon and succinctly summarize the main points of an entire paper. The skills students develop with this first assignment provide a solid foundation upon which to develop more sophisticated analytic abilities.

### Apply

The next challenge for our students to overcome is applying information they have learned to a novel situation. The key to this skill is to think about the deep structure of a problem that was addressed in a paper and to use the solution in a new situation. The level of similarity between the “old” and “new” problem can vary to match your students’ needs. Many instructors are already incorporating this idea
<caption>Table 1</caption>

*Overview of the series of assignments, with pedagogical goals and typical errors detailed*

<table><thead><tr><td>Bloom's Level</td><td>Assignment</td><td>Pedagogical Goal</td><td>Common Pitfalls</td></tr></thead><tbody><tr><td>Understand</td><td>High-level summary of article</td><td>Distil what information is most important</td><td>Too detailed; missing "big picture"</td></tr><tr><td>Apply</td><td>New independent variable, same method</td><td>Use the methods of a paper to predict the outcome of a similar experiment</td><td>Inability to generalize knowledge when applied to a new context</td></tr><tr><td>Analyze</td><td>Compare two journal articles</td><td>Uncover relevant differences and form argument about what makes one paper stronger</td><td>Focus on irrelevant differences; opinion about "stronger" paper unsubstantiated</td></tr><tr><td>Evaluate</td><td>Critique a single journal article</td><td>Pinpoint important weaknesses and/or strengths</td><td>Focus on unimportant weaknesses or strengths; focuses only on either strengths or weaknesses (not both)</td></tr><tr><td>Create</td><td>Research proposal; provide part of a study (e.g., data, methods) and students write the rest</td><td>Generate new ideas incorporating theories and methods learned in journal articles</td><td>Lack of individual thought; modeled too closely to previous articles provided</td></tr></tbody></table>

into their tests, but using this as a writing assignment could allow a more detailed, thoughtful response from students.

## Analyze

Once students are able to understand and apply knowledge presented in primary literature, our next challenge is to help students recognize that they have enough knowledge and skill to form an opinion about the quality of original research. Before jumping into a critique of a single paper, a structured comparison of two papers can help train students to identify qualities of papers that help determine how "strong" or "good" they are. This concept of the strength of a

paper can be challenging to convey to students but is worth spending time exploring. A strong paper addresses an important topic, has excellent argumentation and methods throughout, and is ultimately more likely to be referenced and discussed in the future. Instead of first asking students to determine whether or not a paper is strong, we can ask if one paper is stronger than another. For this skill to be developed, the two papers do not need to be directly related. Assessing which paper better motivates its study in the introduction, or which paper better matches its techniques to its questions of interest are decisions students can make about two papers on disparate topics.
## Evaluate

While the comparison (Analyze) builds up the skills necessary to identify weaker versus stronger pieces of research, the critique allows the students to take a broader view and judge whether a single article is weak or strong, overall. I sometimes tell my students to think of the critique as a comparison between one paper and all other papers out there, or the Platonic ideal of a paper. This is the skill we hope students practice when distilling the large amounts of previously published work in their literature reviews or research proposals. This skill will also serve them well after graduation when determining the value of novel research findings reported in popular media and elsewhere. I highly recommend building in some steps before the critique, as many students initially feel as though they are not qualified to critique a published paper.

## Create

Finally, after students are able to successfully critique previously published studies, they are ready to create their own. This skill can be practiced either through a more traditional research proposal or by giving students a section of a paper (e.g., the methods) and asking them to write up the remaining sections. Excelling in this skill involves a detailed understanding of the previous work in a field, as well as an understanding of what makes a “good” paper. Without a solid foundation, students are apt to submit proposals for replications or very mild modifications on experiments already discussed in class. Creativity is something even seasoned scientists work on cultivating throughout their careers. Using the preceding assignments to develop fundamental skills in understanding and critiquing research will assist our students to develop their own, novel research ideas.

## Conclusions

Most educators are hoping to assist their students in developing critical thinking skills that can be applied both in and out of the classroom. In the short term,

these skills can help students succeed in their undergraduate programs. In the long term, these skills can help students become better informed citizens. Developing these skills is a multifaceted process that should take place in stages throughout the years of postsecondary education. Using Bloom's Taxonomy, the assignments described here model one way of incorporating these skills throughout undergraduate coursework. If one or more pieces of these assignments sound like they could be useful additions to your classroom, please refer to the *Society for the Teaching of Psychology* website (http://teachpsych.org) in 2020 for the full set of assignment descriptions and rubrics.

## References

Anderson, L. W., & Krathwohl, D. R. (2001). *A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives*. New York: Longman.

Bodnar, R. J., Rotella, F. M., Loiacono, I., Coke, T., Olsson, K., Barrientos, A., ... Stellar, J. R. (2016). "C.R.E.A.T.E."-ing Unique Primary-Source Research Paper Assignments for a Pleasure and Pain Course Teaching Neuroscientific Principles in a Large General Education Undergraduate Course. *Journal of Undergraduate Neuroscience Education: JUNE: A Publication of FUN, Faculty for Undergraduate Neuroscience, 14*(2), A104-10. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/27385918%0Ahttp://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4917340

Brownell, S. E., Price, J. V., & Steinman, L. (2013). A writing-intensive course improves biology undergraduates' perception and confidence of their abilities to read scientific literature and communicate science. *AJP: Advances in Physiology Education, 37*(1), 70-79. https://doi.org/10.1152/advan.00138.2012

Hoskins, S. G., Stevens, L. M., & Nehm, R. H. (2007). Selective use of the primary literature
transforms the classroom into a virtual
laboratory. *Genetics*, **176**(3), 1381–1389.
https://doi.org/10.1534/genetics.107.0711
83

Round, J. E., & Campbell, A. M. (2013). Figure facts: Encouraging undergraduates to take a data-centered approach to reading primary literature. *CBE Life Sciences Education, 12*(1), 39–46. https://doi.org/10.1187/cbe.11-07-0057

Van Camp, D., & Van Camp, W. (2013). Using content reading assignments in a psychology course to teach critical reading skills. *Journal of the Scholarship of Teaching and Learning, 13*(1), 86–99.

**Appendix A: Survey**

**Sequential Writing Assignments to Critically Evaluate Primary Literature**

Collected Essays on Learning and Teaching

Sample Guided Article Comparison Guidelines

Here are the basic article comparison requirements:

* No more than 3 pages (double-spaced)

* Article 1 is compared with one other article before Exam 1 (Articles 2, 3 or 4)

* The comparison will be made up of a brief (1 paragraph) summary, as well as a comparison section

* Assignment submitted via Canvas before the class in which we discuss the article

The following two sections make up the comparison
assignment (marked out of 20 pts):

SUMMARY (1 paragraph; 5 pts)

* Written about the new article (not Article 1)

* Context in which study was run / why this study is important

* Hypothesis

* Methods (keep it high level)

* Results (keep it high level)

* Conclusions / implications / societal benefits (if applicable)

COMPARISON (15 pts)

In comparison to Article 1:

1. What are the motivations and/or hypotheses for the different studies? Do they both seem like valid, important questions to be addressed, or does one of the studies seem more important to you? Why or why not?

2. Do both studies involve subjects and methods that adequately address the stated motivation/hypothesis for the study? Does one study do this better than the other? Are the findings in each study equally generalizable? Why or why not?

3. Are the claims made by the authors supported by their data in both articles? Are weaknesses and/or null findings explained sufficiently in both articles? Are the results tied back to the big picture (in terms of what was brought up in the introduction and/or future directions) in both articles? Does one article do this better than the other? Why or why not?

Look for the finalized assignments on the Society for the
Teaching of Psychology website under Teaching Resources in
2020: http://teachpsych.org

See Table 2 on the following page for the Sample
Guided Article Comparison Rubric

Acknowledgements

The *Society for the Teaching of Psychology* organization's
Instructional Resource Award provided financial
support for this project

Biography

Suzanne Wood is an Assistant Professor, Teaching Stream in the Psychology Department at the University of Toronto. She teaches behavioural neuroscience courses and mentors undergraduates on a wide range of projects.
<caption>Table 2</caption>

Sample Guided Article Comparison Rubric

<table><thead><tr><th>Criteria</th><th>Exceeds Expectations (100%)</th><th>(80%)</th><th>Meets Expectations (60%)</th><th>(40%)</th><th>Below Expectations (20%)</th></tr></thead><tbody><tr><td>Summary (5)</td><td>Succinct, accurate summary that describes only the most relevant points</td><td>Accurate summary that hits the major points, but with a bit too much detail</td><td>Summary is accurate, but either too long or misses important point(s)</td><td>Accurate summary, but too long and misses important point(s)</td><td>Inaccurate summary, too long, and misses major points of the study</td></tr><tr><td>Comparison: Motivation/ Hypotheses (2)</td><td>Argument is clear, relevant and directly based upon accurate summary of authors' points</td><td>--</td><td>Accurate summary of authors' points, opinion expressed is unclear</td><td>--</td><td>Inaccurate summary of authors' points, no opinion expressed</td></tr><tr><td>Comparison: Methods (8)</td><td>Accurate, thorough summary of both articles' methods, interesting and important comparisons made</td><td>Accurate, thorough summary of both articles' methods, some comparisons made but not all relevant</td><td>Accurate summary of both articles' methods, overstated strengths and/or weaknesses provided</td><td>Mostly accurate summary of both articles' methods, only either strengths or weaknesses stated</td><td>Inaccurate, incomplete summary of both articles' methods, no valid comparisons made</td></tr><tr><td>Comparison: Discussion (5)</td><td>Accurate, thorough summary of both articles' discussion sections, interesting and important comparisons made</td><td>Accurate, thorough summary of important points of both articles' discussion sections, some important comparisons made but not all relevant</td><td>Accurate summary of both articles' discussion sections, some comparisons made, critical comparisons are missing</td><td>Accurate summary of both articles' discussion sections, comparisons are few and/or consistently weak</td><td>Inaccurate, incomplete summary of both articles' discussion sections, no valid comparisons made</td></tr></tbody></table>