Practice-Oriented Paper
Evaluating the Higher-Order Thinking
Skills Used Within a Student-Made
Reading Quiz Exchange
Jeffrey Martin
Momoyama Gakuin University
Question-making within L2 language coursework is an effective springboard for L2 learners
to practice L2 form construction and engage in higher-order thinking skills. Such inquiry
and critical thinking can be guided through a taxonomy for learning, which crossed six
revised levels of cognitive processing from Bloom's Taxonomy and the four levels of Webb’s
Depth-of-Knowledge. This paper outlines two stages where a group of learners assessedand created reading comprehension questions under this framework. The preliminaryawareness-raising stage had classmates work together to identify the cognitive processes andknowledge dimensions of comprehension and discussion questions from a critical reading
textbook. Then, the main project had learners create their own reading quizzes using news
articles of interests accessed from a level-appropriate website for L2 English learners. The
analysis was on the range of questions produced by the students and their reflections about
their ability to do so. Students reported benefits to their L2 reading and increased awarenessof the thinking skills needed for question-making. The results suggested that this studentmade quiz exchange can serve as a reference to help teachers engage students in higher-orderforms of inquiry.

Interpreting and formulating questions in a second language (L2) involves bothsyntactic and semantic processing that can pose challenges to L2 learners’ successin achieving communicative aims. Language teachers can craft student-centered,motivating language coursework to develop learners’ capacity to handle forms
of inquiry, both in regard to constructions of grammar (Larsen-Freeman et al.,2016) and the higher-order thinking skills called upon by them. Broadly adopted
within educational psychology, the revised taxonomy of educational objectives
45

Martinby Anderson and Krathwohl (2001) is a framework for assessing and exercisinga range of higher-order cognition by students. This framework crossed six
revised levels of cognitive processing from Bloom's Taxonomy (1956) and thefour levels of Webb’s Depth-of-Knowledge (1997, 1999) to create a matrix fordelineating learning objectives. These categories of cognizing are remembering,understanding, applying, analyzing, evaluating, and creating. The objects of theseactions are forms of knowledge categorized as factual, conceptual, procedural,and metacognitive. This framework of twenty-four combinations can guide L2
learners to notice the cognitive actions and objects of knowledge that range
within a rigorous set of questions that they may encounter or create.
This paper outlines an attempt to use this framework with a group of
L2 learners of English. The aim was to learn about the range and rigor of thequestions these students could create. The procedure was in two stages. First,
an initial awareness-raising stage had students assess questions from a reading
textbook in terms of cognition and knowledge. In the second stage, the students
created, revised, and exchanged reading quizzes in a multistep process. The
design aimed to meet the learners’ innate need for learner autonomy, reward forcompetency, and interpersonal relatedness (Deci & Ryan, 1985). The resulting
student-centered process yielded a range of inquiry between the students. The
findings of this paper centered on the tallying of these qualities from across thequizzes they created. The rigor of thinking required by the students’ questions
are presented in a density plot similar to that of Hess et al. (2009). The resultspresent a snapshot of the rigor entailed in the student-made quizzes and indicate
the directions teachers can take to broaden students’ engagement in higher-order
thinking.
For L2 learning, organizing questions along levels of complexity can form a
guide for learners to explore expanding levels of inquiry. The framework proposed
in 1956 by Bloom and colleagues was developed to provide this organization
for such learning objectives within education. The development of Bloom’s
taxonomy began from 1948 when test makers aimed to facilitate a categoricalsystem for creating and comparing multiple-choice items for comprehensive
examinations. From question items to lesson plans and curriculum standards,
46

Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

the original taxonomy and its revision by Anderson and Krathwohl (2001) havesince been applied extensively throughout the fields of education. Their work
cited Bloom’s call for possible revisions to the taxonomy as ideas about education
evolve. The revised taxonomy has since received empirical support regarding
both question item development in assessment and standards development
within educational policy (Näsström, 2009). In this paper, Anderson and
Krathwohl’s revision of Bloom’s taxonomy is applied again to the developmentof question-making. This application is student-centered and the educational
objective is to develop L2 learners’ ability to read L2 texts and to formulate a
range of thoughtful questions.
The Cognitive Process and Knowledge Dimensions
Anderson and Krathwohl’s (2001) framework consists of actions of thought that
include remembering, understanding, applying, analyzing, evaluating, and creating.
This concise list of verbs represents a continuum of increasing complex actions
theorized to form a cognitive process dimension. For instance, the continuum
ranges from recognizing and recalling (under remembering) to hypothesizing
and constructing (under creating) (Anderson & Krathwohl, 2001). The objects
acted upon under this framework were categories of “knowledge that students
are expected to acquire or construct” (2001, pp. 4–5). These four levels were
the factual, conceptual, procedural, and metacognitive. Of note, this distinction
between knowledge types was formulated earlier as depth-of-knowledge by Webb
(1997, 1999) within the field of mathematics education. The resulting matrix ofthe cognitive dimension and the knowledge dimension can describe the intentionof questions in ways such as recalling a fact, inferring a procedure, or critiquing a
concept. Anderson and Krathwohl (2001) acknowledged that distinctions along
the two dimensions are not always clear-cut, but that the general hierarchy begins
with lower-order categories which are subsumed by, and provide the foundation
for, higher-order thinking skills.
Text Difficulty and Student Choice
An appropriate grading of reading texts helps ensure that students can focus on
the texts’ content and the making of reading comprehension questions. Nassaji
47

Martin(2003) found the lower-level process of word recognition and the relation of
texts to personal schema to be the determining barrier to effective reading by
less-skilled L2 readers. Furthermore, L2 readers can maintain a focus on meaning
while reading, uninhibited by vocabulary knowledge, if an estimated 98% ofa text’s lexis is known (Hu & Nation, 2000; Schmitt et al., 2011). Thus, usinglevel-appropriate texts is ideal for L2 learners if they are to focus on meaning andengage well in a quiz-making project among peers. Nonetheless, there are tradeoffs between accuracy and feasibility when estimating L2 readers’ vocabularyknowledge (Gyllstad et al., 2020). For this collaborative lower-stakes classroom
procedure, learner choice among acceptable material was prioritized to ensure
student-centered learning.
To help ensure reasonably appropriate text difficulty and student choice,students can freely browse news articles of interest on Breaking News English(www.breakingnewsenglish.com), which is a free online resource that lists
current news stories at different lexical grades meant for L2 readers. Users of thewebsite typically have three levels of lexical difficulty to choose from for eachnews article. This choice of difficulty and topic can foster student engagement
and be a reasonable way for appropriate text difficulty to be self-adjusting.
Facilitating Interaction
Having students make and exchange quizzes upends the teacher’s central role ofassessor and allows students to take on this empowering role. To spark intrinsic
motivation and perseverance to develop L2 reading skills, student exchange
can aim to tap language learners’ innate need for learner autonomy, reward forcompetency, and interpersonal relatedness (Deci & Ryan, 1985). These aspectsof motivation are also key to L2 reading (Komiyama & McMorris, 2017).
Regarding Japanese EFL learners, Kikuchi and Sakai (2009) found that they canfail to accept the value of a task when there is little or no interaction or choiceprovided by a teacher. Furthermore, Agawa (2020) also outlined how students
felt more capable when they helped each other complete tasks. Tanaka andHiromori (2007) found satisfaction in group work among peers but also thatstudents who are typically less motivated can particularly benefit from a sense of48

Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

belonging. Thus, quiz development can be done within groups in a class while
the quiz exchange can be between individual classmates from other groups.
Additionally, anonymity of exchange of the final quizzes puts all participants on
the same footing on the day of exchange, which can lessen social pressure (Cheng& Tsai, 2012).
Teachers can set a questioning format that is approachable and challenging.
The rigor of thought needed to create questions is often different to the rigor
of answering them. For instance, multiple-choice questioning eases the taskof completing and grading quizzes, but it raises the challenge of quiz creation.
In such a case, students must understand their text, create their questions, andcreate compelling distractor options. In the field of reading comprehension, Day
and Park (2005) presented six types of comprehension questioning from literalto personal response with formats including open-ended and multiple-choicequestions. It would be troublesome for students to mark open-ended questions
and the most felicitous quiz method would involve multiple-choice quizzes
automatically marked via a web-based quizzing platform. Additionally, selecting
the multiple-choice question format reduces complexity of procedure forstudents. It allows them to focus on higher-order questioning for their chosennews articles within a common framework where they make individual choices
regarding content. In sum, design choices by teachers can provide for students anunderstandable, peer-supported, and student-centered procedure that provides
unique challenges to each student.
The Present Study
This study aims to critically evaluate the range of higher-order thinking skills
evoked by the reading comprehension questions created by a group of L2 learners.
The main project was a student-made reading quiz exchange that spanned weeks
of an English course focused on reading skills and fluency. The primary focusof the study is on tallying the evaluations of the questions in the way Hess et al.
(2009) tallied the learning objectives of classroom activities. This analysis is on
how the students’ questions aligned along the revised levels of Bloom’s taxonomy(the cognitive process dimension) and Webb’s depth-of-knowledge (the49Martinknowledge dimension). Interest was also in what students reported about theirreading and question-making abilities. The study sought to answer the following
research questions:
1. To what extent will the questions on each student’s quiz range the higher
levels of the cognitive process and knowledge dimensions of Anderson
and Krathwohl’s (2001) revised taxonomy?
2. What realizations or instances of learning will the students self-report
after completing the project?

MethodParticipants
Twenty L2 learners of English from a university in the Kanto region of Japan
participated in this project. These L2 learners comprised an intact group of
students taking a language course with a focus on L2 reading skills. Seventeen of
the participants spoke Japanese as a first language; two spoke Chinese and one
spoke Korean as a first language. The proficiency level among the students in thiscourse was representative of the students in the language program, largely CEFR
A1 to A2 (TOEIC scores generally between 200 and 400) with a few individuals
approaching CEFR B1 level (TOEIC scores around 550). The reading course
was conducted online via Zoom, a peer-to-peer software used for synchronous
online education. I was the teacher of this intact group of students.
Materials and Platforms
The initial stage made use of the textbook Asian Issues 1: Practice in Critical
Readings (Graham-Marr, 2018). This textbook provides varied forms ofcomprehension and critical thinking questions for a selection of reading passages
related to topics from within a number of Asian cultures. The main project madeuse of Google Forms, a web-based survey and quiz administration application.
Google Forms was enabled as part of each student’s university Gmail account.
The students accessed www.breakingnewsenglish.com (Banville, 2021) andselected their individual news articles about which they made their reading
quizzes. Typically, three levels of word difficulty were available for each article.
50

Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

The students were to make six multiple choice questions and at least onequestion each needed to be of the following four types outlined in Table 1. These
question forms were informed by research in the field, such as Day and Park
(2005). The set is simplified to four types for the sake of the L2 learners so as not
to overcomplicate the procedure. The setting of four question types also helped
ensure a base level of variety in their question making.
Procedure
The students engaged in an initial awareness-raising stage and then the main
quiz exchange project stage. The first stage introduced the idea that there aredifferences in mental rigor entailed by different questions. The class of students as
a whole and in groups discussed the placement of questions from several units inthe critical reading textbook. The sample of questions in the Appendix gives anidea of how the textbook was related to the cognitive processes and knowledge
dimensions of the taxonomy. This practice in the first stage was introduced
gradually over the initial weeks of the course. It was done collaboratively tostimulate class discussion, group work, and independent thought about the topic
of questioning and higher-order thinking skills.
The main project, the 2nd stage, followed a formal sequence over the latter six
weeks of the term (one session per week). Each student had some weeks prior tosettle on a news article chosen on the Breaking News English website. They alsochose the vocabulary difficulty level of their articles from the website’s options.
From the start of the main project, the students were assigned to groups of four
Table 1
Description of required question types
Types of expectations
Type 1 – Detail question

Providing a detail directly mentioned in the passage

Type 2 – Vocabulary question Deriving the meaning of a word based on surrounding context
Type 3 – Inference question

Making inference about the content based on details

Type 4 – NOT question

Distinguishing which option is NOT true or NOT relevant

51

Martinpeople who worked together each week over the six weeks. During Session 1 andfor homework, students worked independently and with groupmates to initiallycraft questions for their individual news article. Students were instructed on
how to utilize keywords from their chosen texts. They were also reminded of theinference and critical thinking that had been covered in the course textbook.
Session 2 was for discussion and revision. The researcher came to each group
to discuss what levels of high-order thinking were being called upon by thequestions they were crafting. To maintain a student-centered focus, the researcher
did not overly direct the students to change their questions to attend to different
levels of Anderson and Krathwohl’s (2001) framework. The exception was whenquestions were merely about recalling factual information or when distractor
options needed clarification. The emphasis was on letting the student create
their own questions and to keep the role of assessor centered among the students.
The instructor gave some help with grammatical constructions, but groups wereencouraged to settle on corrections through group discussion.
Session 3 was for setting up their quizzes on Google Forms. A template
Google Form was provided, the students each created their own copy, andsubsequently they set the basic settings for their quiz. Each student then sent the
researcher an editor link. The researcher organized and viewed all 20 quizzes,
and then provided feedback to individual students. Time for formatting andcontent revisions continued on their Google Forms on Session 4 and after.
Before Session 6, the students finalized their quizzes, and the researcher arranged
the allotments of Google Form links to be distributed to students in the class.
Classmates were to take quizzes via these shared links. Names were removed
from the quizzes. Session 6 was exchange day and each student received eightlinks to quizzes made by students from other groups. They did not receive links
to their own quiz or quizzes made by their group mates. Within an 80-minuteperiod of Session 6, each student completed their eight quizzes. Additional quiz
links were available to students who finished with time to spare.
Following exchange day, students wrote their reflections about the
project via a Google Form. Questions on this form included, “What was yourimpression about this quiz-making and exchange process?”, “What do you think
52

Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

you learned?”, and “Was anything a surprise to you?” Given that this project was
one of many activities in this reading course, this collection of qualitative data
sufficed given the time restrictions.

Analysis

The primary focus of this study was to evaluate the rigor of the questions producedby the L2 learners within this quiz-exchange project. The secondary focus wason instances of learning by students self-reported at the end of the project. Eacharticle and set of questions from each student-made quiz was read and evaluatedagainst the crossing of levels for cognitive processing and depth-of-knowledge.
This evaluation process mirrored the textbook work shown in the Appendix. The
evaluation was done directly by the researcher and one other English teacher who
was familiar with the project. This analysis of the quizzes was done as a team
where each question was discussed and assessed. Because the group of learners
was small, and as Anderson and Krathwohl (2001) note, the distinctions along
the two dimensions are not always clear-cut, it was most feasible to rely on thisteamed approach for question evaluation.
The findings of this paper centered on the tallying of these qualities fromacross the quizzes made by the group of 20 students. The rigor of thinking
required by the students’ questions were presented using a density plot, agraphical technique used in Hess et al. (2009). The resulting diagram illustrates
the general center of gravity of questioning within the two-dimensional matrix.
Additional graphic features were added to the presentation, such as the weight
point of each student’s quiz and a general trend line of those data points.
Qualitative data consisted of the students’ self-reported reflections about the
project.

Results and Discussion

All students completed the steps of the process on time and all 20 students
participated in the exchange. On the exchange day, a number of students
requested and received links to more quizzes after completing their eight assigned
quizzes. During this quiz-making and exchange project and upon its completion,
53

Martinit was clear to the researcher that this approach was well-received by the group of
students.
Questions along the cognitive process and knowledge
dimensions
In answering RQ1, the students’ questions were largely judged to entail
remembering facts and understanding concepts along the taxonomy (see Figure1). Regarding the cognitive process dimension, 36.8% and 57.0% of questions
were attributed to the actions of remembering and understanding. Regarding theknowledge dimension, 47.4% and 46.5% of questions were attributed to factualand conceptual objects of knowledge. A smaller percentage of questions wereconsidered to be more rigorous and a selection is presented here. For example,
an article about a prison with a mice problem described how many staff andprisoners were relocated. The question, “How many people were moved from theprison?” asked the test-taker to apply factual information by recalling two facts
and carrying out the action of combining them into one figure. While seemingly
not too difficult, more is built into this question than merely recalling a single
point of fact. From another quiz, the question, “What is the most relevant word?”,had test-takers ascribe a single word to encompass the whole article. This was
judged to be analyzing concepts of the article by way of differentiating and finding
coherence, both higher-order actions of thought than understanding alone. These
related verbs are outlined in Anderson and Krathwohl (2001).
The extent that higher-order thinking skills were used appeared to be similar
to the degree found among the lesson plans researched by Hess et al. (2009). Theirresults can provide a reference point and help frame expectations for students
more generally. However, deeper comparisons to Hess et al. (2009) are not madesince it was a study of the learning objectives of lesson plan designs for younger
learners studying language arts and mathematics. Nonetheless, their informativeuse of density plots was utilized for this study (Figure 1). Additionally, the centersof gravity of the students are overlaid on the two-dimensional matrix along withthe general trendline. The purpose of this graphical presentation is to easily
grasp the range of thinking skills built into the students’ questions. The small
54

Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

Figure 1. Density plot.
Note. This density plot indicates the range of the cognitive rigor of the students’question-making. The blue dots indicate the averaged position of a quiz’s
questions made by each individual student. Some dots overlap. The white dottedline represents a simple trendline centered for the group of students. Percentages
do not always sum to 100% due to rounding error.

55

Martinsample size is not sufficient for statistical analysis, but as a descriptive analysis,
the general baseline levels for the cognitive process and knowledge dimensions
for the 20 students could be ascertained.
Student learning
Concerning RQ2, many students reported learning how to better comprehendand more effectively make inferences when reading, and they referenced
terminology and strategies covered with the critical reading textbook. Regarding
the main quiz-making and exchange project, students reported how the practiceof effective reading was necessitated by the process of making quiz questions fortheir chosen texts. Examples of this outcome were evident within the students’
written feedback. One student mentioned the importance of keywording.
“I felt it was important to look for keywords when reading articles and to use
skimming when reading sentences. Especially when I was making a quiz, I
noticed that if I knew the keywords, the sentences would be easier to read.”
Another student discussed the challenges of creating questions that call on
readers to infer answers and use higher-order thinking skills.
“Types 1, 2 and 4 are not hard to make but Type 3 is so hard to make. Because
Type 3 need you to guess the thought of writer before you read the whole text
and give the real answer. The answer is not wrote in the text and I am not thewriter, so I may give the wrong correct answer.”
Another student said that the process of making a quiz was challenging but
the process helped develop the ability to use test-taking strategies.
“I thought about the questions by referring to the questions in the class. It wasa little difficult for me to make the quiz. But I had a good time working on
it. I learned that there are some patterns in the quiz. I was able to solve otherquestions more easily.”This feedback suggested that the project was successful in helping students
to actively engage in reading texts. Moreover, the students’ attention was brought
to the grammatical construction and word choice of questioning. Many students
took on this linguistic challenge and made many well-formulated and interestingquestions. These findings form a reference point for language teachers to guide56Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

students in their L2 reading and to engage them in higher levels of questionmaking.
Future considerations
The abilities of these students to make reading quizzes was captured in thisstudy, but some limitations were identified. The reading units from the textbook
included questions that covered the higher ranges of Anderson and Krathwohl’s(2001) taxonomy. But the textbook contained open and closed questions, andthe open questions tended to more flexibly tap the highest ranges by calling forin-depth answers. Nonetheless, allowing students to make open-ended questions
within this quiz-exchange design could invite three potential problems: overly
simplistic question-making, prolonged administration of the quiz exchange, andan infeasible structure for quizzes to be consistently constructed and marked.
Using multiple-choice questions made it feasible to run the project, but additional
practice in explicitly relating higher-order open questioning of the textbook to
the multiple-choice question format could be beneficial. Nonetheless, there is abalance to keep by teachers between giving too much instruction and maintaining
a student-centered learning environment.
Making multiple-choice questions with appropriate distracter options
for high-stakes testing is a challenging specialty (see Eckes, 2015; McNamara,
1996), but such a level of multiple-choice question-making was not neededin such a project. The student-centered environment was maintained byprioritizing learner autonomy to choose texts and make questions. A sense of
relatedness was intended by placing students in groups to support each other.
Anonymity during the final quiz exchange was intended to help students rely ongroup mates without worry of critique from the whole class. Work in the course
textbook and the choosing of reading difficulty for their news articles was aimed
to help students feel competent. The outcomes of these intentions can serve as an
example to reference by teachers designing similar classroom tasks.

Conclusion

A group of L2 learners engaged with comprehension and discussion questions
from a critical reading textbook in relation to a cognitive processing and knowledge
57Martindimension framework. The students’ main objective was to create their own
reading quizzes for news articles of their choice from a news website appropriate
for L2 English learners. The students’ quiz questions called upon a modest
range of higher-order thinking skills. Moreover, the student's written reflections
indicated improved awareness of reading strategies as well as the ability to makedistinctions between the thinking skills outlined by the framework. The majorityof their quiz questions required actions of thought more demanding than merely
recalling facts. Questions about understanding concepts were common and somequestions called for analysis. Increasing this higher range of thinking skills may
be reached by providing more extensive examples of the multiple-choice question
format but to an extent that would not detract from a student-centered learning
environment. From the descriptive figures and the written reflections, in sum,
the outcomes of this study indicated that a student-centered quiz-making andexchange procedure like this can be a fruitful part of an L2 language course.

References

Agawa, T. (2020). A qualitative examination of relatedness needs in Japanese
EFL classrooms and task motivation. Journal for the Psychology of Language
Learning, 2(1), 6–25. https://doi.org/10.52598/jpll/2/1/2
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching,
and assessing: A revision of Bloom’s taxonomy of educational objectives.
Longman.
Bloom, B. S. (1956). Taxonomy of educational objectives: The classificationof educational goals, by a committee of college and university examiners.
Handbook 1: Cognitive domain. Longman.
Cheng, K.-H., & Tsai, C.-C. (2012). Students’ interpersonal perspectives
on, conceptions of and approaches to learning in online peer assessment.
Australasian Journal of Educational Technology, 28(4), 599–618. https://doi.org/10.14742/ajet.830
Day, R. R., & Park, J. (2005). Developing reading comprehension questions.
Reading in a Foreign Language, 17(1), 60–73.
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in58

Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

human behavior. Plenum Press.
Eckes, T. (2015). Introduction to many-facet Rasch measurement. Peter Lang.
Graham-Marr, A. (2018). Asian Issues 1: Practice in Critical Reading. Abax ELT
Publishers. http://abax.co.jp/product.php?id=191
Gyllstad, H., McLean, S., & Stewart, J. (2021). Using confidence intervals todetermine adequate item sample sizes for vocabulary tests: An essential
but overlooked practice. Language Testing, 38(4), 558–579. https://doi.
org/10.1177/0265532220979562
Hess, K. K., Jones, B. S., Carlock, D., & Walkup, J. R. (2009). Cognitive rigor:
Blending the strengths of Bloom’s taxonomy and Webb’s depth of knowledge toenhance classroom-level processes.
Hu, M., & Nation, I. S. P. (2000). Unknown vocabulary density and reading
comprehension. Reading in a Foreign Language, 13(1), 403–430. https://doi.org/10.26686/wgtn.12560354
Kikuchi, K., & Sakai, H. (2009). Japanese learners’ demotivation to study
English: A survey study. JALT Journal, 31(2), 183–204. https://doi.
org/10.37546/JALTJJ31.2
Komiyama, R., & McMorris, A. (2017). Examining international students’motivation to read in English from a self-determination theory perspective.
The CATESOL Journal, 29(2), 61–80.
Larsen-Freeman, D., Celce-Murcia, M., Frodesen, J., White, B., & Williams, H.
A. (2016). The grammar book: Form, meaning, and use for English language
teachers (3rd ed.). National Geographic Learning.
McNamara, T. F. (1996). Measuring second language performance. Longman.
Nassaji, H. (2003). Higher-level and lower-level text processing skills in
advanced ESL reading comprehension. The Modern Language Journal,87(2), 261–276. https://doi.org/10.1111/1540-4781.00189
Näsström, G. (2009). Interpretation of standards with Bloom’s revised
taxonomy: A comparison of teachers and assessment experts. International
Journal of Research & Method in Education, 32(1), 39–51. https://doi.
org/10.1080/17437270902749262
Schmitt, N., Jiang, X., & Grabe, W. (2011). The percentage of words known in
59

Martina text and reading comprehension. The Modern Language Journal, 95(1),26–43. https://doi.org/10.1111/j.1540-4781.2011.01146.x
Webb, N. (1997). Criteria for alignment of expectations and assessments inmathematics and science education. In Research Monograph No. 8. Councilof Chief State School Officers.
Webb, N. (1999). Alignment of science and mathematics standards andassessments in four states. In Research Monograph No. 18. Council of Chief
State School Officers.

Author Bio

Jeffrey Martin has taught EFL learners in a variety of settings within secondary
and tertiary education. He has an M.S.Ed. from Temple University, Japan Campus.
Some of his research interests include materials design, language assessment,
vocabulary learning, and the study of L2 listening. jeffmjp@gmail.com
Received: October 29, 2021
Accepted: September 27, 2022

60

Evaluating the Higher-Order Thinking Skills, OCJSI 4, pages 45-61

Appendix

For Stage 1, a sampling of questions from one unit in Asian Issues 1, plus theircategorization

Understanding the Story Section (one of the three questions in this subsection)#1. What is the best title for this article? [D](A) Four Seasons (B) Five Seasons (C) Hot and Humid (D) The Rainy Season
Critical Reading Section (two selected questions from this subsection)
Cause → effects#2. [cold air meeting warm air] → the rainy season#3. rainy season brings water → [rice grows]
Critical Thinking Section (two selected questions from this subsection)
Which are facts and which are opinions? Write F for fact, O for opinion.
#4. [F] The rainy season lasts for a month or more.
#5. [O] The four seasons are an important part of the Japanese character.
Representation of analysis of questions resulting from class discussions.
Knowledge Dimension (objects of learning)
factualconceptual

procedural#1#2, #3

metacognitive

rememberCognitive
Process
Dimension(actions ofthought)

understandapply
analyze#4#5evaluate
createNote: This sampling of questions is from subsections of “Plum Rain – Rainy
Season” in Unit 3. When Stage 1 was actually done, a majority of the questions
from a given unit were discussed and assessed. For this sample of questions, thecorrect answers are presented in brackets for reference.
61