Language Learning and Educational Linguistics 2017-2018
言語学習と教育言語学 2017 年度版

CEFR-J Self-assessment with Japanese First-year University Students
Kevin Mueller††Tokyo International University 1-13-1 Matoba-kita, Kawagoe, Saitama, 350-1197 Japan
E-mail: kmueller@tiu.ac.jp†
Abstract

This study, undertaken with 257 Japanese first-year university students at a Japanese university,
examines students’ self-assessment of their ability based on a portion of the CEFR-J can-do list.

It

was conducted at the outset of the year with students who are English Language Communication
majors to examine their self-assessment of English ability prior to the beginning of university
English courses.

These students are enrolled in a program that has them take nine hours ofcompulsory English courses each week, primarily with non-Japanese nationality faculty members.
The findings reveal that, in general, more students stated they could do the framework proficiency
skills at the lower end of the scale (in the A0 to A1 CEFR bands) while fewer self-assessed
themselves as able to perform at the higher end of the scale.

This study noted a few outlying

outcomes, most notably, that more students responded they could perform at a higher level than at alower level according to the CEFR-J rubric in two instances, and these apparent discrepancies are
explained herein. Moreover, while the overall study has a high reliability coefficient as measured
by the Cronbach alpha and the Mokken scale score, which measure the ranking order that a
participant who answered positively to a more difficult question is assumed to answer an easierquestion appropriately, was rather low.

These reliability factors are elucidated as well.

The paper

concludes with further considerations for future can-do self-assessment research employing thecomplete CEFR-J with Japanese university students.
Keywords

CEFR，CEFR-J，self-assessment，self-evaluation

日本人大学 1 年生の CEFR-J 自己査定
ミューラー ケビン†
†東京国際大学埼玉県川越市的場北 1-13-1〒350-1197
E-mail: †kmueller@tiu.ac.jp

概要
日本で英語を学ぶ 257 名の大学 1 年生を対象に、本研究では、CEFR-J の can-do リストに基づき、学生の自分の能力に対する自己査定に関して調査を行なった。この調査は、英語コミュニケーションを専攻する新入生が、学生が大学での英語の学びを始める前の段階で、自分の英語能力に関する自己査定を調べるために入学時に実施された。これらの学生は主に日本人以外の教員が担当する 1 週間あたり 9 時間の必修授業を受けるプログラムに登録していた。全体的に本調査を通して、多くの学生が 尺度の下位となる CEFR の A0 から A1 の範囲で文章構造能力に関して出来たと述べる一方で、高い尺度で力を発揮できたと自己査定した者は少なかった。さらに加えると、この研究では、とりわけ CEFR-J のルーブリックにおける 2 つの項目において、多く
Kevin Mueller，”CEFR-J Self-assessment with Japanese First-year University Students,“ 言語学習と教育言語学 2017 年度版,pp.85-96,
日本英語育学会・日本教育言語学会合同編集委員会編集, 早稲田大学情報教育研究所発行, 2018 年 3 月 31 日.
Copyright © 2017-18 by Kevin Mueller. All rights reserved.

の学生が下位のレベルよりも上位のレベルで力が発揮できるという、特異的な通常ではあり得ない結果も示しており、この明らかなる矛盾点を本論文で明らかにしている。さらに本調査全体を通して、より難しい質問に正答できる対象者は、より易しい質問には答えられるであろうとの想定のもとに順位を測定する Cronbach alpha および Mokken 尺度法によって測定された高信頼係
数はかなり低かった。これらの信頼度因子も明瞭に説明した。本論文では、一貫した CEFR-J 調査を日本人大学生に用い、今後
can-do 自己査定研究により重点を置くことが肝要であるとの結論を得ることができた。

キーワード

CEFR，CEFR-J，self-assessment，self-evaluation

The first, A p p e n d i x C , p r o v i d e s s e l f - a s s e s s m e n t

1. Introduction

statements
In 2001, after more than twenty years of research,

as

well

as

language

tests

and

feedback; this system is in place for learners,

the Council of Europe published The Common

primarily

those

who

are

European Framework of Reference for Languages

studyingone

thefollowing(CEFR) in both English and French.

of

independently
fourteen

It is now

European languages: Danish, Dutch, English,

published in forty languages, including JapaneseFinnish, French, German, Greek, Icelandic,

andIrish, Italian, Norwegian, Po rtuguese, Spani sh,wascoherent“designed
andtoprovidecomprehensiveatransparent,

basis

for

the

and Swedish [5].

The second, Appendix D,

elaboration of language syllabuses and curriculum

seeks to outline “a set of performance -related

guidelines, the design of teaching and learning

scales, describing what learners can actuallymaterials, and the assessment of foreign l anguage

do in the foreign language ” [6] .

proficiency. It is used in Europe but also in other“can do” descrip tors for each of the six levels.

continents" [ 1 ] .

These “can do” descript ors were created by the

The CEFR outlines foreign language ability at six
discrete levels: A1, A2, B1, B2, C1, and C2. In

A s s o c i a t i o n o f L a n g u a g e Te s t e r s i n E u r o p e( A LT E ) .
Whileaddition, it denotes three “plus” stages (A2+, B1+,
B2+).

With a foundation based on firsthand

There are

the

CEFR

was

designedfortheEuropean language learning context, it has

experimentation with second language learners andexpandedextensive

all-purpose, broad construct for understanding

input

from

linguists,

educators

andthroughout
ability

the
the

as

an

researchers, the framework, according to its writers,

languagemakes it possible: “t o e s t a b l i s h l e a r n i n g a n d

studying, instruction, and eva luation.

teaching objectives , to review curricula , toanddesign teaching materials and, to provide aeight-yearbasis for recognizing language qualifications

outside of Japan, implementing the CEFR willthus facilitating educational and occupational

be a key instigator in the transformation ofm o b i l i t y ” [2].

EnglishNegishi

from

world

[7]

argue

periodofeducati on

inperspectives

that,

based

analyses

Japan.

ofTo n ooninside

Nagai

an
and

and

The CEFR document is far more comprehensive thanO’Dwyer [8] concur with these researchers andmerely containing a collection of “can -do” statements thatclaim that the implementation of the CEFR

outline discernable language skills that a learner is able torubric in Japan has been generally beneficial

proficiently do.

forThis paper will concentrate primarily on

languageeducation

inJapan.

Three

these “can-do” aspects of the CEFR in both it original

examples can be shown to illustrate how the CEFR

form and the CEFR-J [3] (a framework conceived andhas been applied in Japan at the governmental le vel.

organized solely for the English learning and teachingFirst, Fenelly [9] points out that the December 13,

context in Japan) [4].

2013“English

Education

ReformPlan(Council ofcorresponding to Globalization” document explicitly

Europe), two appendices focus on “can -do” statements.

mentions CEFR levels noting that junior highWithinthe original

CEFR document86

school English education be at the A1-A2 levels andpresenthigh school instruction be at the B1 -B2 levels.

self-measurement practices. Fifth,

Second, t h e M i n i s t r y o f E d u c a t i o n , C u l t u r e ,have opportunities to engage in gauging theirSports,Science,

and

with

a

varietyofstudents(MEXT)o w n p r o f i c i e n c y. F i n a l l y, s t u d e n t s c a n c a r r yreleased a document with five propositions andthese self-assessment skills with them as they

clear

for

continue studying once a course is completed.

that incorporates

As a result of being involved in self -assessment,

guidelines

worldwideforTe c h n o l o g ystudents

improvingcommunication

Englishmeasuring students’ proficiency with a can-dothelanguagei n v e n t o r y, a s c i t e d b y To n o a n d N e g i s h i [ 1 0 ] .

participant in the learning proc ess.

To n o a n d N e g i s h i m a i n t a i n t h a t t h i s i n v e n t o r y
is

by

the

aforementioned

inspiredtwo

researchers

Kyokai(NHK),

Nihon

Hoso

CEFR.

learner

is

can

be

activeWhile being beneficial, self-assessment

Third,

thedoes have a number of weaknesses.

state

the

to

Dunning,

Heath

and

Suls

According[14] ,

a

few

Japan

psychological factors are at work that result in

Broadcasting Corporation, which is, in part,

defective self-assessments, and they sort them

funded

into

by

the

Japanesetheangovernment,

has

two

main

implemented the CEFR rubric for their foreign

self-assessment

language textbooks, and television and radio

usually

programs.

information

At the governmental level , various methods

dotypes.
maynot

F i r s t l y,

occur

have

and

the

peoplenecessary

to

make

a

do

not

take

they
is

unknown.

correct
intoo f i n c o r p o r a t i n g C E F R a r e o c c u r r i n g ; h o w e v e r,

consideration

N e g i s h i ’s r e s e a r c h [ 11 ] r e v e a l e d e i g h t y p e r c e n t

inaccurateof university students in Japan are in the A or

because people do not pay close attention to

B bands.

significant

Thus, the original CEFR A and B

what

becauseall

requiredappraisal,

inaccurate

self-assessment
and

often

advantageous

learners [12]; A1, A2, B1, B2 in the original

drawback of self-assessments is that students

were divided up into nine groups (A1.1, A1.2,

can deliberately lie when assessing their own

A1.3, A2.1, A2.2, B1.1, B1.2, B2.1, B2.2).

skills [15].
More

have.

information

which

addition, a Pre -A1 stage was incorporated;dooccurslevels were altered to more discretely classify

In

they

S e c o n d l y,pertinent

Anothertothisrelateds t u d y,an

L2therefore, the CEFR -J has the resulting twelve

learning context, Blanche and Merino [16] , in alevels:meta-analysisof[Pre-A1], [A1.1, A1.2, A1.3], [A2.1, A2.2] , [B1.1,

self-assessment,

foundB1.2], [B2.1, B2.2], [C1] [C2]self-assessment stretched from being rather
accuratetoverymeta-analysis,

2. Self-assessment Validity

theaccuracythataaccurate.

Ross

[17]

In

From a pedagogical perspective, there are aofexplanationsforencouraging

ano thercorroborated

findings of Blanche and Merino.
numberofl ea rne r ’s

the

H o w e v e r,

there are a number of caveats that require
explanation.

KrugerandDunning[18]

language learners to conduct self -assessment.

recognized that people with higher proficiency

In a language learning context, Oskarsson [13]tendedtorateelucidates

proficient

while

six

reasonswhyal e a r n e r ’s

themselves
less

as

being

proficientlessstudents

self-assessment is helpful. First, it provides

t e n d e d t o r a t e t h e m s e l v e s m o r e h i g h l y.

theresearchlearnerwithproficiency

thatexperience
then

in

aids

appraisinginlearning

doneonJapaneseIn

universitys t u d e n t s ’ E n g l i s h w r i t i n g a b i l i t y, M a t s u n o [ 1 9 ]development. Second, learners and instructors

evaluated

gain

students’

self -assessment

and

of

instructor assessment and found this tendencyd i s t i n g u i s h i n g d e g r e e s o f p r o f i c i e n c y. T h i r d , i tthat Kruger and Dunning noted as well: morestimulates

proficient

students

a b i l i t y.

Matsunoan

increasedalearnerlearning objectives.

appreciation
to

focus

on

further

Fourth, a teacher can87

underestimatedposits

that

this

their
“was

probablycausedbythetendencyofmanyJapanese to display a degree of modesty ” [20].

Instrument

In addition to these factors, genderappears to also possibly skew self -assessment
results.

Inadetermined

meta-analysis

The survey was conducted in Japanese andadministered

onlineusing

GoogleForms.

affects

Participants were required to indicate “yes” or “no”
to indicate whether they can or cannot do acontent being examined.

particularweregender[21]

self-assessment depending on the subject or
females

that

study

In the language arts,

shown

to

have

higherskill

statements.

on40

CEFR -J

can-doThe 40 can-do statements were takens e l f - e f f i c a c y b e l i e f s ; h o w e v e r, i n t h e f i e l d o f

fromlanguage arts when st udents of comparable

reading;

proficiency

production; and writing) from four of the CEFR -J

levelswere

analyzed,

males

all

based

overestimated themselves more than females

levels:

[22].

five

skillsspeaking:
A1.2,(understanding:spokenA2.1,

listening,interaction,

B1.1,

and

spoken

B2.1.

The

There are various methods for students

statements were randomized (see Appendices 1 andto assess their proficiency themselves in the L2

2 for the original Japanese questions and translated

context, and one study by Bachman and Palmer

English). The rationale for randomizing statements[23] found that the most accurate question typewas

asked learners about the perceived difficulty

progression of difficulty inherent in t he CEFR-J

with certain facets of English as an L2.

statements.

They

to

have

participantsbe“blind”

totheThese particular levels were selected

noted that the least effective type of questionas

was the “can-do” question.

International University placed, to varying degrees,

Students’ self-assessments cannot guarantee
of

true

incoming

in

these

freshmenlevelsonstudents

anatin-houseTokyo

placement

objective

measurements

a b i l i t y.

paper-based test formulated on CEFR standards

H o w e v e r,

to compensate for some o f thesewhich was administered at the end of March 2017,

deficiencies Ross notes that instructors “canprior to the commencement of 2017 spring term

strengthen reliability through such strategies

classes.

as engaging students in rubric construction”[24] and providing learners with training .

4. Results and Discussion

The present report is a summary of thefindings of aparticular self -assessment of

A s F i g u r e 1 . a n d Ta b l e s 1 a n d 2 r e v e a l t h a t

skills outlined in the CEFR -J by a first-year

there were more positive responses at the A1

Japanese university students.

and A2 bands.

only

the

findings

shortcomings

and

but

possible

It notes not

Fewer positive responses wer ealsomentions

found at the B1 and B2 bands.

futureresearchand seemingly contradictory exception is thealong these lines.

One notablespoken production skill category in the A1.2
and

3. MethodA2.2

levels.

Here,

more

positive

responses for the more advanced level (A2.2)
were given than for the more basic level (A1.2).

Participants

There may be two reasons for this.
question

257 students in first-year English as a foreign

on

the

(either “yes” or “no”).

voluntarily

participatedmoreParticipants

were

departments:
Internationalin

Language
Relations,

and

this
three

study.
different

Communication,
Economics.

questionnaireasksparticipants to respond in a binary fashion

language courses of Tokyo International University
fromOne, eachdiffuseor

subtleStudents may have
responses

to

th e

questions asked and thus a Li kert scale typeformat may have been more appropriate.

In

The

addition, the two questions at the A1.2 level

survey was administered at the beginning of April( Q u e s t i o n 25. I can express simple opinions related

2017.

to limited, familiar topics, using simple words and88basic phrases in a restricted range of sentence

B1 category question does not explicitly or implicitly

structures, provided I can prepare my speech inconnote a public speech as the language is vague (“Iadvance.

can talk about….”).

Question

29.

I

can

give

simpledescriptions e.g. of everyday object, using simple
words and basic phrases in a restricted range ofTa b l e

sentence structures, provided I can prepar e my

Positive1:

Raw

speech in advance.) could be perceived as more

According to Skill Group

Responses

NumbertoofParticipants’

CEFR-J

Questionschallenging as the A2.1 questions (Question 26. I
can introduce myself including my hobbies and
CEFR-J Levelabilities, using a series of simple phrases andListeningsentences. Question 30. I can give a brief talk about

A1.2

470

familiar

my

A2.1

405

neighborhood) supported by visual aids such as

B1.1

238

photos, pictures, and maps, using a series of simple

B2.1

130

topics(e.g.

myphrases and sentences.).

schoolandA possible rationale forthe perceived level of difficulty could be thatparticipants

concentrated

on

the

CEFR-J Levelphrasing“restricted range of sentence structures” in the A1.2
questions. What is meant by this expression is open
to interpretation and each participant may beunclearasto“restricted

howrange”.

to

gauge

the

meaning

Thus,

these

of

Reading

A1.2

445

A2.1

419

B1.1

230

B2.1

83

modifyingSpokenstatements in the questions at the A1.2 level may be

CEFR-J Level

perceived by participants as being more difficult.

Interaction

A1.2

432

second spoken production question (questions 29

A2.1

379

through 32. See Appendix 1 and 2 ).

Here, the

B1.1

239

largestresponded

B2.1

125
SpokenAnother apparent anomaly can be seen in thenumberpositively

to

of

the

students
B1.1

level

(161)

statement(154

CEFR-J Level

participants stated “yes” for the A1 level statement

Production

and 149 stated “yes” to the A2 level statement).

A1.2

323

Here, two possible explanations can be provided for

A2.1

370

this seeming contradiction.

Firstly, in the A1 and

B1.1

256

A2 category questions, modifying phrases related to

B2.1

62

grammatical structure (“restricted range of sentence

CEFR-J Level

Wr i t i n g

structures” in question 29 [A1] and “series of simple

A1.2

449

phrases and sentences” in question 30 [A2]) which

A2.1

358

do not exist in the B1 question may cause Japanese

B1.1

181

students

B2.1

80

who

are

particularly

sensitivetogrammatical points when speaking to not respond in
the positive.

* Note: The number of students at a given

A second reason for the apparent

contradiction is the A1 and A2 level questions could

CEFR-Jbe perceived as connoting that the respondent will

responded “yes” to CEFR -J can-do questions

give a public speech; the two questions explicitly

exceeds the total number of participants. The

state: “…provided I can prepare my speech innumber at students at given CEFR -J level oradvance” (question 29) and “I can give a brief talk…”

above is more than the number of participants(question 30).

The B1 category question does not

because students at the A2 or higher level

explicitly or implicitly connote a public speech The

should, in principle, respond “yes” to questions

89

level

or

above

and

those

who

at the A1.2 level; this would also be true for theindividual respondents for each skill category”

A2 and B1 levels for B1 and B2 students[27].

r e s p e c t i v e l y.

d a t a i s Runnels’ [28].

Another study with similar reliability
When calculating data on

the reliability of the entire CEFR-J A level can-dostatements

with

590

first-

andsecond-year

Japanese university students in Runnels’ study,
acrossallstatementsastrongreliabilityasmeasured by Cronbach’s alpha 0.944 scale wasfound.

Listening
Figure 1: Relationship among raw number ofself-ratings in five skill categories
Reading
Williams and Andrade, in a study of 243
Japanese university students studying English in
Japan, found that “ a n x i e t y w a s o f t e n a s s o c i a t e d
with

tasks

involvingothers” [25].

speak ing

in

front

of

Therefore, anxiety related toresponses.
The

measureparticipants’ofinternal

self-ratings

consistency

among

the

five

of

the

number,higher

the

the

more

statisticalreliability. A reliability coefficient of 0.7000 or
higher is deemed “acceptable” in the majority ofsocial science studies.

The coefficient in thispresent study

themirrors

finding of

250

227

173

(1)

(2)

(3)

74 (4)

220

178

(5)

(6)

65 (7)

56 (8)

227

222

77

30

(9)

(10)

( 11 )

(12)

218

197

153

53

(13)

(14)

(15)

(16)

216

188

142

63

(17)

(18)

(19)

(20)

216

191

97

62

(22)

(23)

(24)

169

221

95

24

(25)

(26)

(27)

(28)

Spoken

154

149

161

38

Production

(29)

(30)

(31)

(32)

228

170

98

46

(33)

(34)

(35)

(36)

221

188

83

34

(37)

(38)

(39)

(40)

the raw responses of individual students wasThis scale is from 0 to 1.0 and

B2.1

(21)

groups in this study was statistically reliable whenalpha was 0.9062.

B1.1

Spokenskill

Wr i t i n gcompared with each skill category as the Cronbach

A2.1

Interaction

giving a more formal public speech rather thansimply talking about a topic could have skewed

A1.2

Ta b l e 2 : R a w N u m b e r o f P a r t i c i p a n t s ’ P o s i t i v e
Responses to Each Question Along CEFRJ Skill
Groups (question number n oted in parentheses)internal

consistency that Tokeshi and Gao [26] had.

Intheir study that the C r o n b a c h ’ s a l p h a v a l u eamong the five skill categories was 0.872 when
they calculated “the average self -ratings of
CEFR-J
Level

Cronbach alpha

A1.2

0.744

A2.1

0.684

B1.1

0.799

B2.1

0.853

Ta b l e 3 .

90

Cronbach Alpha for CEFR -J Levels

As

shown

in

Ta b l e

3.,

the

A2.1

level

item at a higher level implies that “yes” should

questions had a few anomalies and they may

beexplain the relatively lower Cronbach alphaH-values

coefficient in comparison with the other levels.

unacceptable,

A Mokken Scale analysis for the A1.2, A2.1,

strong reliability [32].

B1.1,

and

B2.1

statements

as

group

[29]

researchundertook

at

all

lower

beneath
and

the

level
0.3

scoresquestions.).

threshold

over

0.6

are

denoteswasc o n d u c t e d ( r e s u l t s d i s p l a y e d i n Ta b l e 4 . b e l o w ) .
Runnels’

notedsimilar

Table 4. Mokken Scales Coefficients for Each
CEFR-J level Examined

analysis and she pointed out that “Mokken
scaling is a statistical technique that assumes

CEFR-J Level

H-coefficientthe order of difficulty of items is not the same

A1.2

0.394

a c r o s s a p o p u l a t i o n ( v a n S c h u u r, 2 0 0 3 ) a n d i t

A2.1

0.387

provides a measure of reliability by identifying

B1.1

0.376

items

B2.1

0.485

for

which

Guttmanpatterningiso c c u r r i n g a t h i g h e r r a t e s ( M o l e n a a r, 1 9 9 7 ;S i j t s m a & M o l e n a a r, 2 0 0 2 ) ” [ 3 0 ] .

Guttmanscalingtests

isutilizedinsurveys,orquestionnaires having binary items (“yes” or“no” as in the one employed in the present
study).

Ina

Guttmanscale,

items

arearranged in order so that if a participant
answers “yes” to an B2.1 level question, theyshould also respond “yes” to lower proficiency
items (e.g. at the A1.2, A2.1, and B1.1 levels).
Thus, stating “yes” to an item at a higher level
implies that “yes” should be noted at all lower
level questions.

Runnels [31]writes that

Mokken scaling generates a ratio that displays
theperceived

difficulty

of

each

can-dostatement according to the self -rated ability ofeach participant and the degree to which alargernumberofhigher

proficiency

participants perceived the can -do statement
more

challenging.

producedisThe

calledstatistic
the

that

is

coefficientofhomogeneity (H or H-value), and it reveals the
structure of answers for each statement by
means

of

item

limitations.

Thus,

this

coefficient notes the reliability scale for eachcan-do statement and exposes the degree that a
G u t t m a n m o d e l c a n b e s e e n f o r e a c h a n s w e r.
Coefficients of homogeneity range from 0 to 1.0.
A higher H-score correlates with an elementt h a t m e a s u r e s m o r e i n s y n c w i t h G u t t m a n ’s
proposition (i.e. items are arranged in order sothat if a respondent answers “yes” to an B2.1
level ques tion, they should also respond “yes”
to lower proficiency items; stating “yes” to an

Two hypotheses for relatively lower h -coefficientresults (all are on the lower cusp of acceptable results of
0.3 as aforementioned). Firstly, in contrast to Runnels’
study [33] where participants responded to the five A
levels in the CEFR-J (A1.1, A1.2, A1.3, A2.1, A2.2), thepresent study took a larger view and examined a range
from of levels from A1.2 to B2.1.

As previously noted,

conspicuous incongruities in participants’ an swers to two
A2.1 can-do statements relative to A1.2 and B1.2 queries
may have skewed reliability.

Secondly, the present study

only had respondents select from binary options (“yes” or“no”).

Having a more restricted field of options, while

producing more discrete data, constrains and compels
participants in an either-or situation which may not truly
reflect participants’ ability. In a similar study, Runnels’[34] utilized a five category Likert-scale response form
(strongly disagree to strong agree).
breadthofresponses

possibly

Providing more

reduces

the

error

co-efficient and diffuses the data.
A number of weaknesses with this study must
be detailed.

Firstly, the entire 110 CEFR -J can-do

list was not employed and thus while an overview ofthe participants’ self-ratings can be recognized, acompletepictureofthegroup’s

self-assessment cannot be seen.

CEFR -J

This would be a

step for a further study. The rationale for thislimited survey was twofold; first, at the start of an
academic year appropriating time for freshmen
students

toquestionnaire

participate
islimitedinaandvoluntary -based
therefore

theresearcher decided to use an abbreviated version,and second, a t t h e o u t s e t o f t h e a c a d e m i c y e a r,
the students in question sat for an in -house91

placement test that was developed along CEFR
lines.

confounded certain respondents.

The purpose of this test was to stream

students into sections so that students would

6. Conclusion

be with students at similar proficiency levels.
This in-house placement test was designed
based

on

the

Cambridge

KET

examination(geared for A1 and A2 learners).

FollowingThis study offers introductory and narrow
results on the CEFR-J can-do statements andmeasurement methods.

W h i l e t h e consistency

numerous pilot trials and follow -up analysis

of participants’ self-ratings among the five skill

and modifications, the in-house test revealed

groups was statistically reliable as measured by themeasureable reliability to the KET test (0.753

Cronbach

Cronbachalpha).

Thus,

administering

alpha

when

the

raw

responsesofaindividual students was compared with each skill

questionnaire with que ries from the A1, A2, B1,

category, there were a number of areas where theand B2 levels was deemed suitable.

CEFR-JA second

scalesappearedtorequireadditional

drawback of this study was the binary response

investigation as to appropriateness and reliability.

options (yes/no).

The findings reveal, as measured by the Mokken

inaLikert

Wi t h m o r e n u a n c e d o p t i o n s

scale

(e.g.

“strongly

agree,/

scale h-coefficient, that internal reliability was

somewhat agree / somewhat disagree / strongly

ratherdisagree”) may have provided more telling andweakness were commented on and further research

informative data.

T h i r d l y, c e r t a i n t e r m s i n

on CEFR-J can-do statements can take measured

the CEFR-J can-do list may be unfamiliar tosteps to possibly decrease self -rating reliability

students

andconcerns.

As theinternal

reliability

majority of participants in this study wer eshouldbefirst-year Japanese university students who

weakness of this present study of using binary

entered

and

thus

cause

confusion

therefore result in misleading data.

theuniversity

Further

explanations

studiesofundertaken,

the

for

investigating
CEFR -J

noting

one

this

the

statements
particularquestions

option responses; employing a Li kert-scale format
may reduce low Mokken scale coefficient results.

necessarilyFor

In addition, the limitations of self -rating, notably

e x a m p l e , q u e s t i o n 16 states “I can understand inwith Japanese university students’ perceptions ofdetailortheir English skill, and the possible gap with actual

reports written for my own field of work, provided I

proficiency as measured on the CEFR -J requires

can reread difficult sections.”

more examination.

specifications,toApril,

possible

pertaining to a work environment would not
pertainin

weak;

respondents.

instruction

manuals,

Students in their

One possible method to link

first year of studies at university in Japan mostparticipants’ self-assessments with more objective

likely would not be exposed to interacting withproficiency could be analyzing these self -rating

English in such a fashion and therefore m ay be at adata points alongside KET scores (synced with theloss as to how to respond.

original CEFR, not the CEFR-J).

the

study

speakers”

revolveswhich

A fourth weakness ofaroundarisesathroughout the questionnaire.

the

term

numberof“nativetimes

Acknowledgement This work was supported by

The entire notion ofTokyo International University Special Educational“native speaker” and its relevance to English

Research Grant.

proficiency is a different topic (e.g. English as
lingua franca); however, germane to this study isthat numerous participants may not have had
opportunities to interact with “native speakers”.
Thus, a question such as number 24, “I can discuss
abstract topics, provided they are within my terms
of knowledge, my interests, and my experience,
althoughIsometimes

cannotcontributetodiscussions between native speakers” may have文

献[1] European Framework of Reference for Lan guages,
Common European Framework of Reference:
Learning,
teaching,
assessment(CEFR).
Retrieved
August2,
2017
fromhttps://www.coe.int/en/web/common-european-fr
amework-reference-languages/[2]CouncilofEurope,CommonEuropeanframework of reference for languages: Learning,teaching, assessment. Strasbourg: Council ofEurope / Cambridge University Press, 2011.
[3]N. Nagai,and F. O’Dwyer, The actual andpotential impacts of the CEFR on language

92

education in Japan, Synergies Europe 6,
pp.141-152, 2011.
[4]Y.Tono,and M. Negishi, M., The CEFR-J:
Adapting the CEFR for English Language
Teaching in Japan. Framework & Language
Portfolio SIG Newsletter, 8, pp.5-12, 2012.
[5]CouncilofEurope,CommonEuropeanframework of reference for languages: Learning,teaching, assessment. Strasbourg: Council ofEurope / Cambridge University Press, p. 256,2011.
[6]CouncilofEurope,CommonEuropeanframework of reference for languages: Learning,teaching, assessment. Strasbourg: Council ofEurope / Cambridge University Press, p. 244,
2011.
[7]Y.Tono
andM.
Negishi,The
CEFR-J:
Adapting the CEFR for English Language
Teaching in Japan. Framework & Language
Portfolio SIG Newsletter, 8, pp.5-12, 2012.
[8] N. Nagai,and F. O’Dwyer, The actual andpotential impacts of the CEFR on language
education in Japan, Synergies Europe 6,
pp.141-152, 2011.
[ 9 ] M . G . F e n n e l l y, T h e I n f l u e n c e o f C E F R o n
English Education in Japan, Bulletin of
Shikoku University, 46, pp. 109–122, 2016.
[10] Y.Tono,and M. Negishi, The CEFR-J:
Adapting the CEFR for English Language
Teaching in Japan. Framework & Language
Portfolio SIG Newsletter, 8, pp.5-12, 2012.
[11] M. Negishi, The Development of the CEFR-J:
Where we are, where we are going, i n New
perspectives for foreign language teaching inhigher education: Exploring the possibilities ofapplication of CEFR, ed. N. Tomimori, M.
Furihatat, K. Haida, N. Kurosawa, & M.
Negishi, pp. 105-116, Tokyo WOLSEC, Tokyo
University of Foreign Studies, 2012.
[12] M. Negishi, M,. CEFR-J Kaihatsu no Keii [TheDevelopment Process of the CEFR -J], ARCLE
Review, vol. 5, issue 3, 37-52, 2011.
[13] Oskarsson, M., Approaches to self-assessmentin
foreignlanguagelearning, London: Pergamon, 1980.
[14] D.Dunning, C. Heat and J.M. Suls, Flawed
Self-Assessment. Psychological Science in the
PublicInterest ,5(3),
pp.
69-106.
doi:10.1111/j.1529-1006.2004.00018.x, 2004.
[ 1 5 ] D . G a r d n e r a n d L . M i l l e r, L . , E s t a b l i s h i n g
Self-Access: From Theory to Practice.
Cambridge University P ress, Cambridge ,1999.
[16] P. Blanche and B.J. Merino, Self-assessment offoreign- language skills: Implications forteachers
andresearchers. Language
Learning,39(3),
pp.
313-338.
doi:10.1111/j.1467-1770.1989.tb00595.x, 1989.
[17] S.Ross, Self-assessment in second language
testing: a meta-analysis and analysis ofexperiential factors. Language Testing,15 (1),pp.1-20.
doi:10.1177/026553229801500101 ,1998.
[18] J. Kruger and D.Dunning, Unskilled andunaware of it: How difficulties in recognizing
one’s own incompetence lead to inflated
self-assessments. Journal of Personality andSocial
Psychology,77(6),pp.
1121-1134.
doi:10.1037//0022-3514.77.6.1121, 1999.
[19]
S.
Matsuno,Self-,
peer-,
andteacher-assessments in Japanese university
EFL
writingclassrooms. Language
Testing,26(1),pp.
75-100.
doi:10.1177/0265532208097337, 1999.
[20]
S.
Matsuno,Self-,
peer-,
andteacher-assessments in Japanese university

EFL
writingclassrooms. Language
Testing,26(1),pp.
75-100.
doi:10.1177/0265532208097337, 1999.
[21] K.Denies and R. Janssen, Country and genderdifferences in the functioning of CEFR -based
can-do statements as a tool for self -assessing
Englishproficiency. Language
Assessment
Quarterly, 13(3),
pp.
251-276.
doi:10.1080/15434303.2016.1212055 , 2016.
[22] F. Parajes, Gender and perceived self-efficacy
in self-regulated learning. Theory Into Practice,41(2),
pp.
116-125.
http://dx.doi.org/10.1207/s15430421tip410
2_8, 2002.
[ 2 3 ] L.F. Bachman and A.S.Palmer,The constructvalidation of self-ratings of communicative
language ability. Language Testing,6 (1),pp.14-29.
doi:10.1177/026553228900600104 ,1999.
[ 2 4 ] J . A . R o s s , T h e r e l i a b i l i t y, v a l i d i t y, a n dutilityofself-assessment.
Practical
Assessment, Research, and Evaluation,
11 ( 1 0 ) , p p . 1 - 1 3 , 2 0 0 6 .
[25] K.Williams and M.Andrade,Foreign Language
Learning Anxiety in Japanese EFL University
Classes: Causes, Coping, and Locus of Control,
Electronic Journal of Foreign Language
Teaching, 5(2), pp.181–191, 2008.
[26] M. Tokeshi and L. Gao,CEFR-J Based Survey
forJapanese
University
Students ,The
European Conference on Education 2015
Official Conference Proceedings, pp. 433 -447,
2015.
[27] M. Tokeshi and L. Gao,CEFR-J Based Survey
forJapanese
University
Students,
The
European Conference on Education 2015
Official Conference Proceedings, pp. 433 -447,
2015.
[28] J. Runnels, An exploratory reliability andcontent analysis of the CEFR-Japan’s A-Level
can-do statements, JALT Journal 36(1), pp.
69-89, 2014.
Appendix 1
CEFR-J Can-do questionnaire (Japanese version)(Retrieved

from

http://www.tufs.ac.jp/ts/personal/tonolab/cefr-j/engl
ish/download.html)
1. 趣 味 や ス ポ ー ツ 、 部 活 動 な ど の 身 近 な ト ピ ッ ク にする短い話を、ゆっくりはっきりと話されれば、理解することができる。
2. ゆ っ く り は っ き り と 放 送 さ れ れ ば 、公 共 の 乗 り 物 や
駅や空港の短い簡潔なアナウンスを理解することができる。
3. 外 国 の 行 事 や 習 慣 な ど に 関 す る 説 明 の 概 要 を 、ゆ っ
くりはっきりと話されれば、理解することができる。
4. 自 然 な 速 さ の 標 準 的 な 英 語 で 話 さ れ て い れ ば 、テ レ
ビ番組や映画の母語話者同士の会話の要点を理解できる。
5. 日 常 生 活 の 身 近 な ト ピ ッ ク に つ い て の 話 を 、ゆ っ く
りはっきりと話されれば、場所や時間等の具体的な情報を聞きとることができる。
6. 学 校 の 宿 題 、旅 行 の 日 程 な ど の 明 確 で 具 体 的 な 事 実
を、はっきりとなじみのある発音で指示されれば、要点を理解することができる。

93

7. 自 分 の 周 り で 話 さ れ て い る 少 し 長 め の 議 論 で も 、はっきりとなじみのある発音であれば、その要点を理解することができる。
8. ト ピ ッ ク が 身 近 で あ れ ば 、長 い 話 や 複 雑 な 議 論 の 流れを理解することができる。

身近なトピックについて、簡単な語や基礎的な句を限られた構文を用い、簡単な意見を言うことができる。
26. 一 連 の 簡 単 な 語 句 や 文 を 使 っ て 、 自 分 の 趣 味 や 特
技に触れながら自己紹介をすることができる。

9. 簡 単 な ポ ス タ ー や 招 待 状 等 の 日 常 生 活 で 使 わ れ る
非 常 に 短 い 簡 単 な 文 章 を 読 み 、理 解 す る こ と が で き る 。

27. 使 え る 語 句 や 表 現 を 繋 い で 、 自 分 の 経 験 や 夢 、 希
望を順序だて、話しを広げながら、ある程度詳しく語ることができる。

10. 簡 単 な 語 を 用 い て 書 か れ た 人 物 描 写 、場 所 の 説 明 、
日常生活や文化の紹介などの、説明文を理解することができる。

28. あ る 視 点 に 賛 成 ま た は 反 対 の 理 由 や 代 替 案 な ど をあげて、事前に用意されたプレゼンテーションを聴衆の 前 で 流 暢 に 行 う こ と が で き 、一 連 の 質 問 に も あ る 程
度流暢に対応ができる。

11. 学 習 を 目 的 と し て 書 か れ た 新 聞 や 雑 誌 の 記 事 の 要
点を理解することができる。
12. 現 代 の 問 題 な ど 一 般 的 関 心 の 高 い ト ピ ッ ク を 扱 った文章を、辞書を使わずに読み、複数の視点の相違点や共通点を比較しながら読むことができる。
13. 身 近 な 人 か ら の 携 帯 メ ー ル な ど に よ る 、 旅 の 思 い出などが書かれた非常に短い簡単な近況報告を理解することができる。

29. 前 も っ て 発 話 す る こ と を 用 意 し た 上 で 、 日 常 生 活の物事を、簡単な語や基礎的な句を限られた構文を用
い、簡単に描写することができる。
30. 写 真 や 絵 、地 図 な ど の 視 覚 的 補 助 を 利 用 し な が ら 、一連の簡単な句や文を使って、身近なトピック（学校や地域など）について短い話をすることができる。

14. 簡 単 な 語 を 用 い て 書 か れ た 短 い 物 語 や 伝 記 な ど を理解することができる。

31. 自 分 の 考 え を 事 前 に 準 備 し て 、 メ モ の 助 け が あ れば、聞き手を混乱させないように、馴染みのあるトピックや自分に関心のある事柄について語ることができる。

15. ゲ ー ム の や り 方 、 申 込 書 の 記 入 の し か た 、 も の の組み立て方など、簡潔に書かれた手順を理解することができる。

32. デ ィ ベ ー ト な ど で 、 そ の ト ピ ッ ク が 関 心 の あ る 分
野のものであれば、論拠を並べ自分の主張を明確に述べることができる。

16. 難 し い 部 分 を 読 み 返 す こ と が で き れ ば 、 自 分 の 専
門分野の報告書・仕様書・操作マニュアルなどを、詳細に理解することができる。

33. 簡 単 な 語 や 基 礎 的 な 表 現 を 用 い て 、身 近 な こ と（ 好
き嫌い、家族、学校生活など）について短い文章を書くことができる。

17. 基 本 的 な 語 や 言 い 回 し を 使 っ て 日 常 の や り と り（何ができるかできないかや色についてのやりとりなど）において単純に応答することができる。

34. 日 常 的 ・ 個 人 的 な 内 容 で あ れ ば 、 招 待 状 、 私 的 な
手紙、メモ、メッセージなどを簡単な英語で書くことができる。

18. 順 序 を 表 す 表 現 で あ る first, then, next な ど の つ
なぎ言葉や「右に曲がって」や「まっすぐ行って」な
どの基本的な表現を使って、単純な道案内をすることができる。

35. 自 分 に 直 接 関 わ り の あ る 環 境 （ 学 校 、 職 場 、 地 域など）での出来事を、身近な状況で使われる語彙・文法を用いて、ある程度まとまりのあるかたちで、描写することができる。

19. 身 近 な ト ピ ッ ク （ 学 校 ・ 趣 味 ・ 将 来 の 希 望 ） に つ
いて、簡単な英語を幅広く使って意見を表明し、情報を交換することができる。

36. 自 分 の 専 門 分 野 で あ れ ば 、 メ ー ル や フ ァ ッ ク ス 、ビジネス・レターなどのビジネス文書を、感情の度合いをある程度含め、かつ用途に合った適切な文体で、書くことができる。

20. あ る 程 度 な じ み の あ る ト ピ ッ ク な ら ば 、 新 聞 ・ イ
ンターネットで読んだり、テレビで見たニュースの要点について議論することができる。
21. ス ポ ー ツ や 食 べ 物 な ど の 好 き 嫌 い な ど の と て も な
じ み の あ る ト ピ ッ ク に 関 し て 、は っ き り 話 さ れ れ ば 限
られたレパートリーを使って、簡単な意見交換をする
ことができる。
22. 補 助 と な る 絵 や も の を 用 い て 、 基 本 的 な 情 報 を 伝え、また、簡単な意見交換をすることができる。
23. 個 人 的 に 関 心 の あ る 具 体 的 な ト ピ ッ ク に つ い て 、
簡単な英語を多様に用いて、社交的な会話を続けることができる。

37. 簡 単 な 語 や 基 礎 的 な 表 現 を 用 い て 、 メ ッ セ ー ジ カード（誕生日カードなど）や身近な事柄についての短いメモなどを書ける。
38. 文 と 文 を and, but, because な ど の 簡 単 な 接 続 詞でつなげるような書き方であれば、基礎的・具体的な語彙、簡単な句や文を使った簡単な英語で、日記や写真、事物の説明文などのまとまりのある文章を書くことができる。
39. 身 近 な 状 況 で 使 わ れ る 語 彙 ・ 文 法 を 用 い れ ば 、 筋
道を立てて、作業の手順などを示す説明文を書くことができる。

24. 母 語 話 者 同 士 の 議 論 に 加 わ れ な い こ と も あ る が 、自分が学んだトピックや自分の興味や経験の範囲内のトピックなら、抽象的なトピックであっても、議論できる。

40. 旅 行 記 や 自 分 史 、 身 近 な エ ピ ソ ー ド な ど の 物 語 文について何か自分が知っていれば、多くの情報源から統合して情報や議論を整理しながら、それに対する自分の考えの根拠を示しつつ、ある程度の結束性のあるエッセイやレポートなどを、幅 広い語彙や複雑な文構造をある程度使って、書くことができる。

25. 前 も っ て 発 話 す る こ と を 用 意 し た 上 で 、 限 ら れ た

Appendix

94

2

CEFR-J

Can-doquestionnaire(English version)(Retrieved fromhttp://www.tufs.ac.jp/ts/personal/tonolab/cefr-j/engl
ish/download.html)

17. I can respond simply in basic, everyday
interactions such as talking about what I can/cannot
do or describing colour, using a limited repertoire of
expressions.

1. I can understand short conversations aboutfamiliar topics (e.g. hobbies, sports, club activities),
provided they are delivered in slow and clear speech.

18. I can give simple directions from place to place,using basic expressions such as "turn right" and "gostraight" along with sequencers such as first, then,and next.

2. I can understand short, simple announcements
e.g. on public transport or in stations or airport s,provided they are delivered slowly and clearly.
3. I can understand the gist of explanations ofcultural practices and customs that are unfamiliar
to me, provided they are delivered in slow and clear
speech involving rephrasing and repetition.
4. I can understand the main points of aconversation between native speakers in television
programmes and in films, provided they aredelivered at normal speed and in standard English.
5. I can catch concrete information (e.g. places andtimes) on familiar topics encountered in everyday
life, provided it is delivered in slow and clear
speech.

19. I can express opinions and exchange information
about familiar topics (e.g. school, hobbies, hopes forthe future), using a wide range of simple English.
20. I can discuss the main points of news stories I
have read about in the newspapers/ on the internet
or watched on TV, provided the topic is reasonably
familiar to me.
21. I can exchange simple opinions about very
familiar topics such as likes and for sports, foods,
etc., using a limited repertoire of expressions,provided that people speak clearly.
22. I can get across basic information and exchange
simple opinions, using pictures or objects to helpme.

6. I can understand the main points ofstraightforward factual messages (e.g. a school
assignment, a travel itinerary), provided speech isclearly articulated in a famili ar accent.

23. I can maintain a social conversation about
concrete topics of personal interest, using a wide
range of simple English.

7. I can understand the main points of extended
discussions around me, provided speech is clearly
articulated and in a familiar accent.

24. I can discuss abstract topics, provided they ar e
within my terms of knowledge, interests, and myexperience, although I sometimes cannot contributeto discussions between native speakers.

8. I can follow extended speech and complex lines ofargument provided the topic is reasonably familiar.
9. I can understand very short, simple, everyday
texts, such as simple posters and invitation cards.
10. I can understand explanatory texts describing
people, places, everyday life, andculture, etc.,
written in simple words.

25. I can express simple opinions related to limited,familiar topics, using simple words and basic
phrases in a restricted range of sentence structures,provided I can prepare my speech in advan ce.
26. I can introduce myself including my hobbies andabilities, using a series of simple phrases andsentences.

11. I can understand the main points of English
newspaper and magazine articles adapted for
educational purposes.

27. I can talk in some detail about my experiences,
hopes and dreams, expanding on what I say by
joining together words, phrases and expres sions I
can readily use to make longer contributions.

12. I can read texts dealing with topics of generalinterest, such as current affairs, without consulting
a dictionary, and can compare differences andsimilarities between multiple points of view.

28. I can give a prepared presentation withreasonable fluency, stating reasons for agreement or
disagreement or alternative proposals, and cananswer a series of questions.

13. I can understand very short reports of recent
events such as text messages from friends' or
relatives', describing travel memories, etc.

29. I can give simple descriptions e.g. of everyday
object, using simple words and basic phrases in a
restricted range of sentence structures, provided I
can prepare my speech in advance.

14. I can understand short narratives
biographies written in simple words.

and

15. I can understand clearly written instructions(e.g. for playing games, for filling in a form, forassembling things).
16. I can understand in detail specifications,
instruction manuals, or reports written
for myown field of work, provided I can r eread difficult
sections.

30. I can give a brief talk about familiar topics (e.g.
my school and my neighborhood) by visual aids suchas photos, pictures, and maps, using a series ofsimple phrases and sentences.
31. I can talk about familiar topics and other topics
of personal interest, without causing confusion to
the listeners, provided I can prepare my ideas inadvance and use brief notes to help me.
32. I can develop an argument clearly in a debate by95

providing evidence, provided the topic is of personal
interest.
33. I can write short texts about matters of personal
relevance (e.g. likes and dislikes, , and school l ife),
using simple words and basic expressions.
34. I can write invitations, personal letters, memos,and messages, in simple English, provided they areabout routine, personal matters.
35. I can write a description of substantial length
about events taking place in my immediate
environment (e.g. school, workplace, local area),using familiar vocabulary and grammar.
36. I can write business documents (e.g. e - mail, fax,
business letters), conveying degrees of emotion, in astyle appropriate to the purpos e, provided they are
in my professional field.
37. I can write message cards (e.g. birthday cards)
and short memos about events of personal relevance,
using simple words and basic expressions.
38. I can write texts of some length (e.g. diary
entries, explanations of photos and events) in simple
English, using basic, concrete vocabulary and simple
phrases and sentences, linking sentences w ith
simple connectives like and, but , and because .
39. I can write coherent instructions telling people
how to do things, with vocabulary and grammar of
immediate relevance.
40. I can write reasonably coherent essays andreports using a wide range of vocabulary andcomplexsentence
structures,
synthesisinginformation and arguments from a number ofsources, provided I know something about the
topics.

96