鳥取環境大学紀要第12号（2014．３）pp.25-31

英語の自由作文における学生の文法精度に関する長期的な考察
A Longitudinal Study of Students’ Grammatical Accuracy during
Their Production of Unscripted English

ベゴール・ベッティーナ

ゼノス・トレメイン

BEGOLE Bettina, XENOS Tremain

要旨：この研究は本学の英語強化プログラム内で行われている、時間制限を設けた自由作文において、学生が表現する文法精度の進捗度を調査したものである。この自由作文の訓練の主な目的は、学生が自らを素早くスムーズに表現することであるが、そこから彼らの文法精度の概観を見ることもできる。一学期のプログラムの間では、これらの作文課題における精度は予測したほどには進歩しなかった。進捗度比較には、学期前後に行われる英検スコアを用い、統計上（p=0.00005）、総合的な進歩が見られたが、文法セク
ションでは見られなかった。一貫性のある文法精度を習得し、自然に使えるようになるには、他の能力よりも時間がかかることが自由作文と英検スコアに反映されていると思われる。
【キーワード】書く、能力、評価
Abstract： In this study the authors examine progress in grammatical accuracy expressed in timed freecompositions (“free writing”) over the course of the Intensive English program at Tottori University ofEnvironmental Studies. The primary purpose of these writing exercises is to increase students’ ability to express
themselves quickly and smoothly, but they also provide a snapshot of students’ grammatical accuracy. Gains inaccuracy evinced by these writing assignments were lower than expected during the course of the program. Preand post-test Eiken scores were also examined in order to compare progress as measured by a standardized test,
and it was ascertained that while overall progress was statistically significant (p=0.00005), gains in the grammar
section of the test were not. The authors hypothesize that consistent grammatical accuracy requires more time tobecome internalized and automatic than other skills, and that this is reflected both in free writing and Eiken scores.
【Keywords】writing, competence, assessment

１．IE and Free Writing

program is the use of free writing, a ten-minute activity

The Intensive English program at Tottori University

in which students are asked to quickly write all theirof Environmental Studies is currently a three-semester

thoughts on an assigned topic without attention toprogram designed to improve the English ability ofgrammatical or orthographic accuracy and without thefirst- and second-year students. It is comprised ofaid of dictionaries. It is expected that at a certain level

Reading and Writing, Grammar and Translation, andof overall English competence, students will come to

Speaking and Listening. This improvement is expected

produce writing that is both fluent and grammaticallyto appear in improved Eiken scores.

accurate without conscious attention to grammar; that

One of the essential components of the current

is, that rules of expression will become internalized― 25 ―

CW6_T1134_08_D03_J.indd

25

2014/04/02

14:54:04

鳥取環境大学紀要

第12号

and automatic. Free writing was chosen because it

３．Methodallows students to produce connected discourse,

Free writing was done at strategic intervals

allowing the authors to examine a specific marker ofthroughout the course of each semester at a ratio of

discourse competence, i.e., grammatical accuracy.

approximately one free writing session to every threeSamples of student free writing in textbooks and onclass periods, with some variation by class group level

notebook paper were collected and analyzed in thisand teaching style. Broughton et al（1978）define freestudy in order to evaluate both the effectiveness ofwriting as free compositions in that “only the title isfree writing in fostering accuracy, and the utility ofprovided, and everything else is done by the student.”

this type of statistical analysis for measuring changes

The IE program dictates the following rules for freein students’ English proficiency, in both volume ofwriting:

writing and accuracy, over the course of the program.
☆ Write about the topic

Hymes（1972）advanced the notion of communicativecompetence, which encompassed the four areas of☆ Write everything you are thinking.

linguistic, sociolinguistic, discourse and strategic☆ Write without stopping your pen.

competencies. The authors of the present study expect☆ Relax and write as much as possible.

that all four of these would develop in tandem as☆ Use a pen.

second-language study advances; that is, they should
support each other as the student’s competence★ Don’t stop writing.

improves, with fluency and grammatical accuracy★ Don’t erase or cross out anything.

improving together.

★ Don’t worry about grammar.
★ Don’t worry about mistakes.

２．Motivation★ Don’t let your pen stop moving.
★ Don’t use a dictionary.

In order to evaluate the overall effectiveness of theIntensive English（hereafter, IE）program, search forpossible shortcomings and improve the program, the（Reprinted from Welcome to Intensive English and Yourauthors hoped to find to what extent students’ overall

Reading and Writing Class, 2012）English ability has improved by the end of the threesemester program. The IE program already employs

IE students are taught paragraph format andreading speed and comprehension tests to document

specific rules for composition, such as avoiding the use

improvements in these facets of second-languageof coordinating conjunctions to begin sentences, but（hereafter, L2）progress, but does not have a means

outside of adherence to format rules, students areof evaluating production. If patterns of usage anddiscouraged from editing their compositions during theexpression and error ratios could be demonstrated,time allotted for free writing, consistent with thefree writing analysis could prove useful in measuring

distinction between fluency and accuracy indicated byprogress and isolating areas for improvement.

Broughton et al（1978）and the premise that error

With a clear idea of student strengths andweaknesses, teachers can make more informed

correction should be discouraged during fluencybuilding exercises.

decisions concerning adjustment of syllabi andStudents in the free writing analysis portion of thiscurricula in response to specific areas to be improved.

study were divided into three groups based on theirAs an example, relatively high rates of native-language

pre-test Eiken scores（see Table 1）. Students took the（hereafter, L1）substitution in some class groups may

STEP Institutional Placement Test B（英語能力判定テ

mean these groups would benefit from more

スト B）in early April, 2012. Test B is designed forvocabulary-enhancing activities than other groups.

students expected to score in the Eiken test range of

Likewise, collocation misuse may signal weaknesses inlevel 3 to level 2. According to STEP（STEP, 2013）

linguistic competence.

these placement test scores accurately correlate with― 26 ―

CW6_T1134_08_D03_J.indd

26

2014/04/02

14:54:04

ベゴール

ゼノス

英語の自由作文における学生の文法精度に関する長期的な考察

Table 1：Group Divisions by Eiken Score

particularly difficult skill for students to master. The

Eiken Institutional
Placement Test
Score Range

Eiken level（STEP, 2013）

authors postulate that automaticity in appropriate

low269-408（n=47）

STEP 5～4

intermediate311-408（n=48）

STEP 4～3high408-632（n=52）

STEP 3～2

Group level

conjunction use should be fairly late to develop.
The degree to which each of the above categories oferror was present was measured for change over thecourse of three semesters and compared between
group levels, with an eye to determining to what
extent improvement had been made and, ultimately, todetermine the utility of this type of analysis as a

the standard Eiken STEP scores.
These groups represented both the faculties ofdiagnostic tool.

Business Administration and Environmental Studies.
Data from students not present for all free writing

４．Initial Hypothesis

sessions were omitted from the analysis. Initial

The authors had hoped to find significant

considerations for markers of accuracy included theimprovement in length of text students were able todegree to which collocations were used in the manner

produce in ten minutes concurrent with a simultaneous

used by native speakers （such as the use of the

reduction in rate of errors as a proportion of each text.

unnatural collocation, “write a picture”）and errors in

It was also assumed that a quantitative analysis ofconnotation and denotation of vocabulary, but it was

student errors versus length of text produced would

determined that evaluation of these is open to a degree

demonstrate concrete differences between classes ofof subjective interpretation（Suzuki, 1978）.As a result,different academic levels, as well as steady progress

the authors chose to focus on grammatical accuracy.

over the course of the program.

Several types of grammatical errors were observed,
including misinterpretation of part of speech（“a very

５．Free Writing Analysis and Interpretation

good remember”）, subject-verb disagreement（“myFigure 1 below illustrates the variance in averagebrother have a dog”）and lack of congruence in thenumber of words counted in student free writing by

use of pronouns（“He’s name is Kenji”）. However, asgroup and arranged by topic. Classes generally covered

these mistakes are purely grammatical and students

a different topic each week. The data below reflect

were specifically instructed not to “worry about
grammar,” it cannot be said that these are necessarily
the indicators of lack of grammatical awareness. In
other words, it cannot be said that reflection on, andcorrection of, compositions by the students would notdemonstrate a degree of grammatical knowledge, butrather, these types of errors in unscripted writingshow that grammar rules have not been internalized
to the degree that they became automatic.
This study focused on two types of errors theauthors believe can measure English competence
because their analysis is objective and easily
quantifiable: 1）correctness in use of coordinating
conjunctions, and 2）incidence of L1 substitution in
relation to total word count. It should be emphasizedthat the authors’ experience in teaching IE courses has
demonstrated that grammatically and contextually
appropriate use of coordinating conjunctions is a

Figure 1：Average Number of Words by Group― 27 ―

CW6_T1134_08_D03_J.indd

27

2014/04/02

14:54:04

鳥取環境大学紀要

第12号

only the five topics done by all class groups throughout

Table 3：Coordinating conjunction use in July, 2013

the three-semester program and counts only those

Number of
Ratio ofattempts tocorrect% useduse coordiconjunction
correctly
natinguse to length
conjunctionsof writing

students who were present for every session.
The surprising conclusion from our analysis of thetwo markers of accuracy in free writing was theabsence of statistically significant change over thecourse of three semesters. In cases where the number

Low

109

72%

0.074

of errors increased, it is possible that this was theIntermediate

26

62%

0.039

result of attempts to use vocabulary or grammar

High

107

63%

0.055

presently beyond students’ abilities, in which casemore error may not necessarily indicate a lack ofsignificant gain in either accuracy or amount. As noted

progress.

in the context of TOEIC study（Saegusa, 1985）andothers, students with lower starting levels are

５‒１

Mistakes in the use of logical connectors

appear most difficult to eradicate. It was repeatedly

generally able to make more rapid gains, so this result
could perhaps be expected.

emphasized to students that coordinating conjunctions
Incidence of L1 substitution includes items

should never be used to begin sentences. One of the

５–２

challenges in the IE program has been securing awritten in Japanese script（kanji, hiragana and katakana）

consistent standard for which rules to focus on andas well as romanized Japanese such as, “My motherteach and which to downplay or ignore. One that wework in tetsudou company,” and, “She is genki.”

have endeavored to focus on consistently is the

Figure 2 shows a decrease in L1 substitution, but

protocol for avoiding beginning sentences with the
conjunctions and, but, or, so, and because. These fiveitems have proved the most difficult for students toremember to use in the way taught, though it was
hoped that extensive free writing concurrent with anemphasis on this protocol would reduce the incidence
of incorrect use of these conjunctions.
This type of error was repeatedly commented on byteachers during class explanations and in corrections
given in writing in the context of other assignments,
which were evaluated for adherence to format asprescribed in the official curriculum.
It can be seen from these Tables 2 and 3 that with
one exception, the lower-level students, there was no
Table 2：Coordinating Conjunction Use in April 2012

Group

Number of
Ratio ofattempts tocorrect% useduse coordiconjunction
correctly
natinguse to length
conjunctionsof writing

Low

37

62%

0.056

Intermediate

44

53%

0.050

High

77

61%

0.051

Figure 2：L1 Substitution by Topic― 28 ―

CW6_T1134_08_D03_J.indd

28

2014/04/02

14:54:04

ベゴール

ゼノス

英語の自由作文における学生の文法精度に関する長期的な考察

not in a consistent pattern. As noted above, it isin standardized test scores. Saegusa（1985）found thatpossible that specific topics induced students to draw

the number of hours of study required to gain one pointon their native language to a greater degree thanon the TOEIC, for example, increased as students’ scores

others. The lower groups appear to have had moreincreased; with a TOEIC score of 250, for example, only

difficulty using English to express concepts relating toan hour of additional study was needed to gain one

school, fashion and entertainment, while thepoint, whereas a starting score of 850 required anintermediate group demonstrated a remarkably highadditional four hours of study. It may be estimated thatrate of L1 substitution while also producing

by the time of entry into the TUES IE program,significantly less text（see Figure 1）. These results

students have undergone six years of public schoolingwere counterintuitive.

with between three and four classes per week. However,when students took the Eiken pre-test（英語能力判定テ

５–３

It was expected that instructors adhered toスト B）, it was found that students in this study had

and enforced the guidelines for free writing as

lower scores than would be expected after six years ofprescribed in the curriculum, including keeping

study. Although MEXT has set a goal of Eiken Level 3

sessions to ten minutes, determining that students didby the end of middle school, approximately two-thirds of

not use dictionaries, etc., and indeed, adherence tostudents in this class had scores below this at the timethese strictures must be as strict as possible to yieldof admission to university. The authors found thisthe most accurate results. Rigid control of when andsomewhat surprising.

how many times free writing was done, however,

The authors were also disappointed to find thatwould preclude the flexibility necessary to adjust

there were not larger changes in students’ ability tosyllabi to meet particular class needs, and so there isuse grammar accurately and produce fluent English

some variation in sample size among different groups.

（as measured by volume）after three semesters of

In addition, among the three markers of accuracy westudy in the IE program.

have chosen to evaluate, there may be differences

The IE program began in 2001 as a three-times-a-among individual instructors in terms of the degree toweek program for first and second year students. In

which they reinforced these aspects of writing,2005, the second year became an elective class forparticularly in the case of coordinating conjunctions.

some departments. In 2010 the second year classes

Ferris（2002）explores the lack of consistency in error

were further reduced to twice a week. In academic

correction made by different instructors at different

year 2012, the program was cut from four semesters

times and concludes that problems with the specificto three; that is, second year students only study

process of correction undercut student progress inEnglish in first semester. The authors believed that aaccuracy. Citing a previous study of a large number ofcomparison of data from students taking the four-

student texts in which errors were marked bysemester program might show a greater improvement

teachers, Ferris finds a significant lack of accuracy inin accuracy and fluency than that found in the three-

the teachers themselves as well as a small but

semester group. To explore this, Eiken pre- and post-

significant proportion of unnecessary correction, whichtest scores for the last group of students to take themay interfere with more necessary correction. Indeed,four-semester program and the scores for the firstthe authors here find varying degrees of emphasis on

students to complete the three-semester program were

the specific markers of accuracy selected for thisanalyzed.
In the four-semester program, students received 240

analysis.

hours of instruction, compared to a maximum of 180 in
６．Test Scores Analysis and Interpretation

the three semesters; three koma per week for the first

From the outset there were concerns over theand second semesters, and two koma per week for thepresence of any statistically verifiable correlation

third semester. The authors postulated that one cause

between progress evinced by free writing and increase

for the lower-than-expected improvement was simply― 29 ―

CW6_T1134_08_D03_J.indd

29

2014/04/02

14:54:04

鳥取環境大学紀要

第12号

Table 6：STEP Results for Three-Semester Program

Table 4：Overview of STEP Test（STEP, 2013）
No ofproblems

Test B

4/2012pre-test

7/2013post-test
416.77

Vocabulary & Grammar

30

Mean（M）

387.32

Sentence Structure Cloze（Grammar）

5

α

0.05

T

9.75

two-tail T value

1.97

p（T≤t）two-tail2.22 x 10-19

Reading
Listening

Short conversation（Q&A）

5

Longer reading passages

10

Short conversation questions

15

Longer sentence questions

15

total

80

Looking at these results it can be said with 95%

a lack of sufficient class hours in the courses taken by

confidence that the four-semester IE program did

students completing the three-semester program. To

result in a statistically significant improvement in

explore this premise, data from the final year of theoverall test scores. However, looking at each test

four-semester program were compared with the data

component it was determined that the results for thefrom the first year of the three-semester program.

vocabulary and grammar section did not increase

Data were obtained from the Eiken Institutionalsignificantly.

Placement Test B. This test is a multiple-choice

Next, examination of the results for 2012 intake

standardized test comprised of four sections:students taking the same test in the first year of the

vocabulary and grammar, sentence structure（multiple

t h r e e - s e m e s t e r p r o g r a m （ n = 2 6 5 ）, d a t a w e r echoice cloze-style）, reading, and listening as seen in

determined as outlined in Table 6.
Again, p supports the hypothesis that the IE

Table 4 below.
In order to see if students made progress inprogram improves students’ English as measured bygrammatical accuracy and/or fluency as measured bythis particular test. In this case, all test sections except

a standardized test, Eiken pre- and post-test scores forthe sentence structure cloze problems showed

the four-semester program students（April 2011 andsignificant improvement. It seems that in fact the

February 2013）were compared with pre- and post test

three-semester program can be said to be effective,scores for three semester program students（April

although the authors were disappointed to find less2012 and July 2013）. The three-semester program

improvement in accuracy and fluency in students’ freestudents’ reading section scores showed the largest

writing.

gain, increasing 13.6% at the 95% confidence level, but

The test score data from the two groups reveal agrammatical accuracy as measured by this test was

23.0 point gain among the four-semester students anddisappointing. Tables 5 and 6 below analyze data froma 29.45 point gain among the three-semester students.

these two groups, beginning with pre- and post- test

It could be postulated that the three-semester program

results（n=131）for 2011 intake students in the final

implies greater gains than the four-semester; however,year of the four-semester program（Table 5）.

neither group displayed statistically significant

Table 5：STEP Results for Four-Semester Program

progress in grammatical accuracy.
Consistent with Saegusa’s（1985）findings that “the4/2011pre-test

2/2013post-test

difficulty of raising a TOEIC score is considerably

Mean
（M）

351.1

374.1

α

0.05

end,” it might be inferred that the relatively higher

T

4.79

two-tail T value

1.98

p（T≤t）two-tail4.4 x 10-6

greater at the upper end of the scale than at the lower
academic level of the 2012 intake students influenced
slower progressive gains. Wolfe-Quintero et al（1998）
examined extensive samples of L2 compositions andfound that automaticity of grammatical competence― 30 ―

CW6_T1134_08_D03_J.indd30

2014/04/02

14:54:05

ベゴール

ゼノス

was particularly slow to develop. The authors of the

英語の自由作文における学生の文法精度に関する長期的な考察

write, and by extension, speak English.

present study postulate that internalization ofBoth types of analyses—that of changes in markers

grammatical rules（automaticity）is one of the lastof accuracy evident in free writing over time, andcomponents of competence to develop; for this reason

statistical changes in standardized test scores—may

the authors expect that a greater length of time thanbe used in conjunction to isolate specific areas of

the three semesters under study may be required to

student progress and weakness. It is hoped thatevince substantial gains in this area. The second

further research will reveal more pronounced patterns

objective in this study, “to evaluate the utility of thisin these areas and assist IE teachers in developing

type of statistical analysis,” requires more data, i.e., astrategies to further boost student progress.

larger sample size, measured over a longer time.
Sources
７．Conclusion

１）Begole. B.（2012）Welcome to Intensive English andYour Reading and Writing Class. , in-house textbook,Statistical analysis of certain markers of accuracy in95.

student free writing indicated no great improvement
in student English accuracyover the course of three２）Broughton, G., Brumfit, C., Flavell, R., Hill, P. andsemesters, but the apparent absence of measurable

Pincas, A. （1978） Teaching English as a Foreign

progress was disproved by the results of standardized

Language. London: Routledge, 118, 130-31.

test scores. Although Eiken scores demonstrated３）英語能力判定テスト pamphlet, page 8 of 24（URL：

significantgains at the end of the three-semester

2013年８月26日現在）

program, similar gains were not found in fluency or

http://www.eiken.or.jp/test/booklet/

grammatical accuracy expressed in student free４）Ferris, D. R.（2002）Treatment of Error in Second

writing. In consideration of the incongruence of theLanguage Student Writing. Ann Arbor: U ofMichigan, 12-13.

two different sets of data, the authors tentatively
postulate that four semesters of study may lead to an５ ）H y m e s , D . H . （1 9 7 2 ） “ O n C o m m u n i c a t i v eincrease in grammatical accuracy as expressed in freeCompetence.” In Pride, J. B. and Holmes, J.（eds）

composition independent of that shown by theSociolinguistics: Selected Readings. Harmondsworth:standardized tests. In addition, with reference to thisPenguin, 269-93.

incongruence in results between Eiken and the free６）Saegusa, Y. （1985） “Prediction of English

writing analysis, as noted above, the method ofProficiency Progress.” Musashino English andanalysis of grammatical accuracy in free writing seems

American Literature, Vol. 18. Tokyo: Musashino

to require additional refinement before the authors will

Women’s University.

be satisfied. One goal of English education in Japan is７）STEP（2013）“Eiken Grades.” Eiken Foundation of

to foster human resources capable of functioning inJapan（URL：2013年８月26日現在）

English; in other words, capable of using English in a

http://stepeiken.org/grades

practical way. Although the significant Eiken test８）Suzuki, T. （1978） Words in Context. Tokyo:
Kodansha, 14-24.

score gains are laudable, and in particular the rather
dramatic improvement in both groups’ reading

９）Wolfe-Quintero, K., Inagaki, S., and Kim, I.（1998）

comprehension shows that students are indeed making

Second Language Development in Writing: Measures ofprogress, the lack of improvement in production of

Fluency Accuracy, and Complexity. Manoa: U ofEnglish means that students have not yet reached the

Hawai’I, 13-33.

point of being able to use English. The authors believethat because of the greater amount of time required（受付日2013年８月26日

受理日2013年12月27日）

for grammar to become internalized and automatic,
four semesters of study would be beneficial in making
the transition from “test English” to the ability to― 31 ―

CW6_T1134_08_D03_J.indd

31

2014/04/02

14:54:05