Short Research Papers

Reliance on Quizlet for Content andAssessment

Stephen Paton and Alexander Cameron
Fukuoka University
When emergency remote teaching was implemented in Semester 1, 2020, course content
and assessment schedules needed to be quickly amended. Having previously found Quizlet,
a popular digital flashcards mobile app and website, to be an effective and enjoyable
vocabulary learning tool, both authors independently included it in their online courses.
Using Quizlet, teachers can assign vocabulary sets for students to learn, and for teachers to
track student progress via a teacher account on the Quizlet website. In the emergency online
teaching situation, this seemed an ideal tool by which content could be delivered and uponwhich assessment could be based. Throughout the semester, the majority of students carriedout their assigned vocabulary study, but questions arose about the degree to which students
were benefiting from these tasks and the extent to which they may or may not enjoy them.
There was also the possibility that some students might be “gaming” the system. A survey (N= 386) was administered to help clarify whether relying so heavily upon Quizlet for online
class content and assessment had been a good decision. The survey results suggested that thestudents enjoyed using Quizlet, that they perceived benefits in doing so, and that they took
the work seriously, and therefore that utilizing Quizlet had been a good approach to take.
2020年の前期に緊急遠隔授業が実施された際、コース内容と評価スケジュールを早急に修正する必要があった。デジタルフラッシュカードのモバイルアプリとウェブ
サイトで人気のあるQuizletが、効果的で楽しい語彙学習ツールであることを認識していた 著者2人は、独自に遠隔授業にQuizletを取り入れた。Quizletを使用することで、教師は生徒が学習する単語リストを割り当て、Quizletのウェブサイト上の教師アカウントを介して生徒の学習状況を確認することができる。緊急時のオンライン授業では、コンテンツを提供し、それに基づいて評価を行うことができる理想的なツールだと思った。学期を通して、大半の学生は与えられた単語学習を行っていたが、学生がこれらのタスクからどの程度効果が出ているのか、楽しんでいるのかいないのかという疑問が生じた。また、一部の学生がシステムを不正行為している可能性もあった。そこで、オンライン授業のコンテンツと評価を多大にQuizletに頼っ90Reliance on Quizlet, OCJSI 3, pages 90-102
たことが正しかったかどうかを明らかにするために、アンケート調査を実施した（N =386）。アンケートの結果、学生はQuizletを使うことを楽しみ、メリットを感じ、真剣に取り組んでいることがわかり、Quizletの活用は良いアプローチであったと考えられた。

In early 2020, the COVID-19 pandemic forced teachers to implement emergency
remote teaching in lieu of physical classes. Revised syllabi and assessment criteria
had to be quickly decided upon. Complicating the rush towards alternate plans
and approaches was the question of whether students would have and be able
to use computers in their homes for online classes. Computer ownership andliteracy was suddenly a major stumbling block. Decisions regarding content anddelivery were further complicated by issues such as textbook availability. Amid
the confusion over what resources would be accessible, it was assumed that most,
if not all, students would possess a smartphone or touchscreen device, and thusthat tasks which could be facilitated on mobile devices were preferable.
Through discussions of the many decisions that needed to be made as thesemester began, the authors each decided that their revised syllabi would include
digital flashcard study using the smartphone app Quizlet (https://quizlet.com/
mission). Each had used Quizlet before, both in personal learning (for example,
Japanese vocabulary, kanji, etc.) and in administering vocabulary teaching
programs. Both were confident that Quizlet could be used to deliver distancelearning content and provide a reliable form of assessment.
Quizlet is a well-established and well-regarded teaching and learningtechnology. Dizon (2016) and Montaner-Villalba (2019) both found thatsignificant vocabulary gains for Spanish learners could be attained using Quizlet.
In a Japanese-English study using Quizlet as well as another digital flashcard
platform, Dizon and Tang (2017) found no significant difference in score
outcomes between digital flashcard study vs paper flashcard study. They did
find, however, that students tended to prefer digital flashcards. Similarly, Davieand Hilber (2015) found no significant gain in vocabulary test scores over more
traditional paper methods amongst university students of business English, but
their participants reported positive perceptions of Quizlet. Chien (2015), in a91

Paton & Cameronstudy of three digital flashcard platforms including Quizlet, reported positive
attitudes by university students studying English regarding digital flashcard
activities. Lander's (2016) finding that Quizlet met with "resounding approval"(p. 258) from the majority of students, suggested to the authors that we could
have confidence in it as we headed into an unfamiliar teaching situation.
During the semester of emergency remote teaching, the authors each used
Quizlet in various ways, using different word lists of different lengths. One
used sets between 15 and 30 words drawn from course textbook chapters andmaterials being used for each lesson. The other took a more general approach to
vocabulary development and created much longer word sets that drew upon the
New General Services List, a high-frequency word list of 2,800 words (Browne
et al., 2013). Similarly, however, each teacher set weekly tasks and milestones
that were to be met as the semester progressed.
The mobile app features five activities through which assigned words canbe learned and reinforced. A screenshot of each activity is shown in Figure 1.
The first, called Learn, requires that students either manually select the correct
meaning of a word that is displayed from four choices or enter it by typing. If a
mistake is made, the correct answer is shown, and that word reappears relatively
sooner as the learner progresses through their list of words. Once a word has
been correctly selected or entered twice, it is deemed to have been “learned”.
The Flashcards activity allows students to turn a digital flashcard over between
the target word and its meaning and swipe it away either to the right if it isconsidered “known” or to the left if it is not. The words sent to the left comeup again once the deck has been cleared. Thus, learners can identify and focus
on the words they have not yet mastered. The Write activity shows the learner
the translation of a target word or term, in response to which they have to type
the target word or meaning. Some allowance is given for typing or spelling
errors. This activity is complete when all the words have been correctly typed.
The Match activity displays twelve tiles, six of them showing target words and six
showing their meanings. In a timed session, the learner has to tap a tile followedby its partner. If they match, the cards disappear and the game ends when all tiles
have been removed. Finally, the Test activity facilitates a test according to the92Reliance on Quizlet, OCJSI 3, pages 90-102

learners’ choice of settings, involving either typing, multiple choice questions, ortrue/false questions. Learners can also select how many items to include in their
test.
When students complete an activity, their having done so is indicated on theteacher view in Quizlet’s website by a green checkmark for that activity. Thus,
each student’s progress and completion is easy to track and record. During thesemester, each of the authors required students to complete four of the fiveactivities for each week’s word set. This, it was assumed, would be a reliableindicator that students were engaging with the materials, developing theirunderstanding, and demonstrating comprehension of the vocabulary items.
As the semester progressed, most students seemed to be meeting the weekly
deadlines, and score updates were regularly provided to them. It appeared thatthe students were successfully engaging with the assigned vocabulary lists.
However, as the semester progressed, the authors each began to question whether
actual learning was taking place. There was no way of perceiving whether thecompletion of activities represented actual acquisition, either long-term or shortterm. In a regular classroom setting, weekly paper-based quizzes would be used
to determine this, but the absence of that option caused considerable uncertainty
about whether the Quizlet vocabulary study was bringing about any benefit forthe students. Also, the opportunity to “game the system” was a concern, giventhat there was nothing to stop students simply swiping every card to the rightin the Flashcards activity without applying effort towards actual learning, andFigure 1. The Quizlet mobile app activities - Learn, Flashcards, Write, Match, and Test.

93

Paton & Cameron

thereby earning one of their four required green checkmarks for doing so.
Similarly, students could also assign themselves a one-question “test”, answer it
correctly, and gain a green checkmark for completing the Test activity. Concerns
arose over whether too much reliance had been placed on Quizlet. Furthermore,
towards the end of the semester, decisions regarding the year’s second semester
revised syllabi had to be made, including whether it would be wise to administera similar program of vocabulary study again.
At this point, in order to answer the many questions that had arisen, an
anonymous survey was administered. The survey was designed to help findanswers to the following research questions:
1. Is simply monitoring the completion of tasks on Quizlet a valid andappropriate tool of online assessment?
2. Are students rushing through some of the activities without applying
effort to learning the words? If so, to what extent?
3. Are students utilizing Quizlet as intended for actual learning, or merely
working through activities without any perceived benefit?

MethodThe survey questions were translated into Japanese by a native speaker ofJapanese. The surveys were administered in the final weeks of the semester via
a Google Form with a link provided to students. Participants (N = 386) wereenrolled in various first-year and second-year English courses at several Japanese
universities. These students were studying different majors, such as engineering,
English, law, and commerce. Their courses had different foci, such as reading andwriting, listening and speaking, and TOEIC. Approximately 80% were first-year
students, with the remainder being second-year students. The survey included
an introduction stating that all answers were anonymous, that completion of
the survey was voluntary, and that participation indicated informed consent for
their answers to contribute to the teachers’ research project. Respondents wereemphatically encouraged to give honest answers. The following questions wereasked with multiple-choice answer options. (Each question also included an
opportunity for students to give comments in either Japanese or English.)94

Reliance on Quizlet, OCJSI 3, pages 90-102

1. Overall, how much did you enjoy using Quizlet?
2. Approximately how much time did you spend each week on the Quizlet
tasks?
3. What percentage of the words were you already familiar with?
4. Was reviewing words beneficial to you?
5. Did you feel your vocabulary knowledge grew as a result of Quizlet
activities?
6. Which activity was best for you for learning words?
7. Rush A) Did you often rush through some words and activities?
8. Rush B) If so, why did you rush through? (This question required awritten response).

Results

We were interested in general attitudes across all students rather than attitudes
that might be specific to their majors, so we analyzed all results as one dataset.
Overall, the results confirmed that utilizing Quizlet in this way had been
worthwhile and appropriate, with many positive and affirmative findings.
A majority of students reported positive engagement with the Quizlet app.
More than two-thirds (71.3%) reported that they “enjoyed” or “enjoyed a lot”,
while just over one quarter (27.4%) reported not liking much or “disliking” it(Figure 2). These results suggest that a large majority of students enjoyed using
Quizlet for vocabulary learning and review.
In answer to the second survey question, approximately 20% of students(Figure 2) reported spending less than 20 minutes per week carrying out theirassigned Quizlet homework, whereas a similar number spent 60 minutes ormore (Figure 3). The remaining 60% fell somewhere in between, spending
between 20–60 minutes per week, suggesting that this was an appropriate selfstudy workload.
In regards to how much of the vocabulary the students felt “familiar” with,roughly half of the students reported prior familiarity with 50% or more of thewords they were assigned to study. Furthermore, around a quarter reported
familiarity with 75% or more of the words (Figure 4). This suggests that, for95

Paton & Cameron

Figure 2. Responses to Question 1: Overall, how much did you enjoy using Quizlet?

Figure 3. Responses to Question 2: Approximately how much time did you spend each week on
the Quizlet tasks?

many students, time spent on Quizlet involved some learning of new words but
also involved a significant amount of time reviewing and reinforcing knowledgeof previously learned words. Any concern that this proportion of words being
already familiar might render the task of working through them again pointless
was abated by the responses to the following question.
Almost 90% of students reported that reviewing familiar words was“beneficial” or “very beneficial”. This suggests that participants found Quizlet
96Reliance on Quizlet, OCJSI 3, pages 90-102

Figure 4. Responses to Question 3: What percentage of the words were you already familiar with?

beneficial for both acquiring new words and reviewing previously acquired
words. It also indicates that many considered the Quizlet program generally to beof learning value and worthwhile. As with the “familiarity” aspect, “beneficial”
also requires some clarification, though, and was at least partially answeredby the following question, on whether the students felt that their vocabularyknowledge grew as a result of completing the Quizlet activities.
In answer to question 5, on whether students felt that their vocabularyknowledge had grown, a clear majority of students (88%) reported “yes” or “verymuch”, compared to 12% that said “no” or “not really” (Figure 6). This further
clarifies the previous question regarding whether the benefit might be derived
from a perceived increase in vocabulary knowledge.
When asked about which specific Quizlet activity they felt was best forlearning words, the Learn section yielded the highest rating with around 40%
of student responses, with the Write section at around a third (Figure 7). It isinteresting to note that Match, a more game-like and presumably fun activity,
was preferred by only 8.1% or respondents as beneficial to vocabulary learning.
Also, the Flashcards activity, which allows a user to self-test but does not test
their accuracy, was reported by only 9.7% as best for learning words.
Finally, a two-part question was asked regarding rushing, by which we meant97

Paton & Cameron

Figure 5. Responses to Question 4: Was reviewing words beneficial to you?

Figure 6. Responses to Question 5: Did you feel your vocabulary knowledge grew as a result of
Quizlet activities?

quickly performing whatever actions could be taken in order to attain the green
checkmark in the least possible time or with the least possible effort. Almost
two-thirds of students reported “rarely” or “never” rushing an activity (Figure8). Around one in five sometimes rushed through, 9.3% reported regularly or
often rushing through to complete an activity, and 7.5% did not respond. This
indicates that the majority of students (83.2%) were motivated to complete tasks
98

Reliance on Quizlet, OCJSI 3, pages 90-102

Figure 7. Responses to Question 6: Which activity was best for you for learning words?

properly most or all of the time. It also indicates a generally positive attitude to
the vocabulary-study program as a whole.
When asked why they rushed through tasks, 69.5% of students did not
respond, suggesting that most of these students were from the 63.5% who “rarely”
or “never” rushed an activity. Of the 117 who responded to this question, 84
stated that they rushed because “I already knew the words”, and 33 admitted that
“I just simply wanted to finish” the task. This would suggest that fewer than 1%

Figure 8. Responses to Question 7. Did you often rush through some words and activities?

99Paton & Cameronof students were “gaming” the system, although it is possible that the true figure
might be higher.
Based on these responses, it appeared that many students were motivated toincrease their vocabulary knowledge, that they valued the activities in Quizlet,
and that they meaningfully allotted time to completing the assigned tasks.

Discussion

The survey results suggest that relying upon Quizlet as a component of course
content and as a means of assessment was justified. A majority of students
reported enjoying engaging with the app and also reported feeling that it was
beneficial for not only learning new vocabulary but also for reviewing vocabularythat they had previously acquired. The students’ preference for the Learn andWrite activities implies that they differentiated between a sense of fun and actual
learning value, and favored the latter. It also appears that most students completed
tasks thoroughly, rather than taking advantage of shortcuts. This all suggests thatstudents were taking the learning opportunity seriously.
Most students allocated 20–60 minutes per week to Quizlet study. A small
portion spent less than 20 minutes per week, but it is unclear whether this wasdue to disinterest, large study workloads, or prior familiarity with the word sets.
Just over a fifth of all students reported spending more than one hour each week
to complete the Quizlet tasks. Given how easily the activities could be completed
without diligent effort, this suggests that they were taking the time to learn new
words and successfully complete the tasks.
As the authors had varying class types, a follow-up study could further
differentiate student attitudes according to English-class type and course major.
Further, the notion of self-reported familiarity with some vocabulary could be
followed up in greater depth, focusing upon the extent to which students arefamiliar with words that they might have considered themselves to be reviewing
rather than learning. When students reported familiarity with words, it was
unclear whether they were reporting deep vocabulary knowledge of a word or
simple recognition from previous encounters. A pre- and post-test might reveal
more about student vocabulary knowledge, but this was not an option at the100Reliance on Quizlet, OCJSI 3, pages 90-102

onset of this study.
A subsequent study could also follow up on why students might favor certain
activities over others. Indeed, student preferences towards more demanding
activities, such as Learn and Write over the more game-like Match, was one of
the more surprising results and warrants further study.
Overall, utilizing Quizlet for content and assessment appears to have been agood decision. Quizlet can be recommended to teachers who wish to include a
vocabulary-learning component in online courses.

References

Browne, C., Culligan, B., & Phillips, J. (2013). The New General Service List
Project. http://www.newgeneralservicelist.org
Chien, C.-W. (2015). Analysis the effectiveness of three online vocabulary
flashcard websites on L2 learners’ level of lexical knowledge. English
Language Teaching, 8(5), 111–121. doi:10.5539/elt.v8n5p111
Davie, N., & Hilber, T. (2015, Mar 14–16). Mobile-assisted language learning:Student attitudes to using smartphones to learn English vocabulary[Conference session]. 11th International Conference on Mobile Learning,
Madeira, Portugal.
Dizon, G. (2016). Quizlet in the EFL classroom: Enhancing academic
vocabulary acquisition of Japanese university students. In Teaching English
with Technology, 16(2), 40–56. http://www.tewtjournal.org
Dizon, G., & Tang, D. (2017). Comparing the efficacy of digital flashcards
versus paper flashcards to improve receptive and productive L2 vocabulary.
The EUROCALL Review, 25(1), 3–15. https://doi.org/10.4995/
eurocall.2017.6964
Lander, B. (2016). Quizlet: what the students think – a qualitative data
analysis. In S. Papadima- Sophocleous, L. Bradley & S. Thouësny (Eds.),
CALL communities and culture – short papers from EUROCALL 2016
(pp. 254–259). Research-publishing.net. doi.org/10.14705/rpnet.2016.
eurocall2016.571
Montaner-Villalba, S. (2019). The use of Quizlet to enhance vocabulary in the101

Paton & Cameron

English language classroom. In F. Meunier, J. Van de Vyver, L. Bradley & S.
Thouësny (Eds.), CALL and complexity – short papers from EUROCALL
2019 (pp. 304–309). Research-publishing.net. doi.org/10.14705/
rpnet.2019.38.1027

Author Bios

Stephen Paton (Fukuoka University) began teaching English in Australia before
coming to Japan in 2008. He is the editor of CUE Circular and president of Fukuoka
JALT. home@stevepaton.com
Alexander Cameron teaches English at Fukuoka University. His research interests
include e-learning, visual word recognition and vocabulary acquisition. cameron@fukuoka-u.ac.jp
Received: September 17, 2020
Accepted: November 5, 2021

102