61

## English Language Screening for Scientifi  c Staff at Delft University of Technology

## Abstract

Delft  University  of  Technology  (DUT)  screened  her  (non-native  English)  scientifi  c  staff  on  their  level  of  English profi  ciency  in  the  academic year of 2006/2007. In this paper this large scale operation, involving planning, policy decisions, assessment means, advice and training are discussed. Since 2005 all the master programmes at DUT have been taught in English and since 3 years ago DUT has been an offi  cially bilingual university with around 5,500 master students and 1,100 international students in the year 2008. Therefore, results are framed against the background of becoming an international university.

## 1. Introduction

An increasing number of higher education programmes in the Netherlands are offering Englishmedium instruction. Delft University of Technology contributes to this trend by offering all her master programmes in English. When these programmes started it was generally assumed that the language profi  ciency of the non-native speaking lecturers was adequate for English-medium instruction (Klaassen 2001, Hellekjaer/Westergaard 2003). However, an increasing number of students complained about the lecturers' English profi  ciency (AAG, 2005, Ljosland 2008), which made administrators and university authorities aware of the need to assess the English language command of the lecturers.

Lecturers' English Language profi  ciency tends to be a much debated issue, as it is, according to some researchers, at the core of lecturing quality in English-medium higher education (Vinke 2005, Hellekjaer-this issue, Rasanen 2005). Vinke (1995) found that the lecturing quality of nonnative speaking lecturers decreased when teaching in English. She specifi  cally found that lecturers had a more limited vocabulary, and experienced a decrease in redundancy 1 , clarity and accuracy of expression. Additionally, these defi  ciencies were likely to cause a decrease in speech rate and expressiveness and an inability to go beyond the scope of their prepared presentation. Klaassen (2001) found clarity to be the most important construct for student perception of lectures as opposed to structure and interaction, whereas in a mother tongue lecture all these three constructs were of relevance to the student perception. The clarity construct was defi  ned as maintaining a lively argument, keeping eye contact, using supportive gestures, enthusiasm, concrete and specifi  c examples in appropriate context, visual support and redundancy. Furthermore, she concluded from student interviews and observations that clarity is a precondition for interaction to take place. If this precondition was not met, interaction between students and lecturers decreased (Klaassen 2001).

Clarity can help to sustain student understanding. Crawford Camiciottoli's results (2005) proved that native speakers adapt their speech to non-native students by slowing down their speech rate, being more redundant, using more elicitation markers and personal pronouns and references to the local culture to be more comprehensible. Morell (2004) found that interactive lectures are characterised by a greater amount of elicitation markers, personal pronouns, questions and negotiation rounds. On top of clarity aspects such as redundancy, lowering the speech rate, providing examples in appropriate context, these native speakers use question asking, personal pronouns and elicitation markers to make their class interactive. These might be skills missing at a lower level of language profi  ciency, and might entail that with an improved vocabulary and better language profi  ciency, lecturers will be able to re-establish interaction with students. Jordan (1997) points out that with increased interaction students' understanding may improve due to the option of negotiating meaning.

1 Vinke  uses  the  word  'redundancy'  to  mean  the  ability  to  paraphrase  or  explain  ideas/or  concepts  in  different words.

* Renate G. Klaassen & Madeleine Bos TU Delft, OC FOCUS Jaffalaan 9a 2628 RD Delft r.g.klaassen@tudelft.nl - M.H.P.C.Bos@tudelft.nl

These studies certainly suggest that student complaints should be taken seriously. Nonetheless, it remains unclear, whether the students' complaints are due to a low level of language profi  ciency or fl  uency among the non-native speaking lecturers, or whether these complaints are due to the students' own lack of language profi  ciency. For example, Airey (2009) found that while second year students were able to describe disciplinary concepts, fi  rst year students were unable to do so in English. This raises the question of how students can interact if they cannot even describe the concepts they have questions about. General language ability may not be a signifi  cant indicator of academic performance, yet disciplinary lexis can have a signifi  cant impact on an individual's development of deeper understanding (Turner 2006).

When a switch in language has taken place, students typically need more time to process the incoming information from the lecture itself, while engaging in note taking and processing visuals at the same time (Flowerdew/Miller 1997). In the same line, Hellekjear (this issue) shows that Norwegian students experience diffi  culties taking notes while listening to lecturers, distinguishing the meaning of words and understanding unfamiliar vocabulary. Airey (2009), points out that lecturers should be informed about students' needs and about what they can do to make lectures more accessible such as allowing questions after the lecture or being refl  ective when introducing new material in a lecture. In addition, they should also know what students can do themselves to improve their learning of English-medium instructions such as reading material before the lecture and discussing the lecture afterwards.

An advisory committee named 'English as a second language' at Delft University of Technology recommended testing the lecturers' level of English profi  ciency in order to identify language planning  policy  issues,  and  to  facilitate  staff  language  development.  It  was,  furthermore,  expected that constructive image building could take place on the basis of positive results from the test as well as that appropriate actions could be taken to improve the language profi  ciency of the staff and to improve student understanding. Therefore, the Board of Directors of Delft University of Technology decided to screen the English language profi  ciency of all scientifi  c staff (around 1,300) to be able to provide a representative picture of their English language profi  ciency.

A preliminary survey in 2005 (Klaassen 2005), comprising 13 universities in the Netherlands showed that very few institutions in the Netherlands had carried out mandatory English language assessment of scientifi  c teaching staff. The exception was Eindhoven University, which had done so at a single faculty. At present, Eindhoven University has also implemented mandatory screening, while seven universities (Leiden, Erasmus, Groningen, Utrecht, Radboud, Maastricht and Twente) have optional screening of English Language profi  ciency. The remainder did not respond to the survey or had no screening in place.

The  fi  rst  test  round  at  Delft  University  of  Technology  started  during  the  academic  year 2006/2007, and the university has since 2006 screened about 1,600 staff members. Scientifi  c staff, meaning professors, associate and assistant professors and lecturers, were tested on a mandatory basis, PhD students and support personnel with educational tasks were invited to take part on a voluntary basis. The present paper will focus on the results of the group that was tested on a mandatory basis.

Testing such a large number of scientifi  c staff, with no precedent in the Netherlands, and on a mandatory basis, turned out to be an explorative and diffi  cult process. Furthermore, a policy had to be developed to deal with the test results and set targets for the University, a testing organisation for a large number of testees had to be set up, and a valid and reliable testing method had to be found. At the same time, testing such a large number of staff provided a once in a lifetime opportunity for carrying out research on lecturing language profi  ciency on a large scale. The central focus of attention while carrying out the testing operation and the research was:

- What is the average level of scientifi  c staff's English Language profi  ciency at Delft University · of Technology?
- Is the International English Language Testing System (henceforward IELTS) level 6.5/ C1 · level of the Common European Frame of Reference (CEFR) requirement realistic?

In addition to these central questions, a number of assumptions which were held prior to testing, were considered. These assumptions were the following:

- The  Design  departments  will  score  higher  on  English  language  profi  ciency  than  the · engineering and science departments. Delft University consists of 8 departments with quite a varied character in technological approach. These departments have been clustered in Science departments, where fundamental research takes place, Engineering departments, in which an engineering approach is key to generating technological solutions for construction/product development, and the Design departments in which the design approach is central to product development/technical solutions.  Science  departments  include  Applied  Physics  (AP),  Electrical Engineering, Mathematical and Computer Sciences (EEMCS) and Technology Policy and Management  (TPM) 2 .  Engineering  departments  are  Civil  Engineering  and  Geosciences (CE), Mechanical, Maritime and Material Sciences (3ME) and Aerospace Engineering (AE). Design departments are Architecture (A) and Industrial design (ID). When the emphasis is on linguistic output, it is often assumed that the more technological the disciplinary fi  eld, the fewer linguistic skills are needed to explain the subject matter (as it would contain more formulae, etc.). If this is true, the more design-oriented departments will be more verbose in their course programmes, which will require a higher language level. (Smid et al. 1995).
- The younger the staff (and job position) the better the English Language profi  ciency; · Traditionally older staff have grown up learning English with the grammar-translation method instead of current communicative pedagogical strategies (van Els et al., 1984). The younger generation may thus be assumed to have learned English at a higher level, at least for oral purposes (Ellis, 1992).
- The test results on the QPT are higher than on the oral assessment · ; the QPT consists of grammar, reading and listening skills. The oral profi  ciency skills consist of listening and oral communication. Due to less frequent practice in oral communication, the grammar, reading and listening skills are likely to be better developed than the oral profi  ciency skills.
- The Institute of Reference is Leiden University where a similar test has been run for 800 · lecturers, before testing started at Delft University of technology. It is expected the normal distribution in Leiden will be indicative for the results that may be realised at Delft University of technology; the normal distribution on the test results is based on these 800 voluntarily tested lecturers in Leiden University shown in the following graph; C2= 15%, C1= 30%, B2= 35%, B1= 15%, A2/A1 = 5%

2 TPM is in the offi  cial  documents of Delft University of Technology, a design department. The position of this department in the triade of science, engineering and design is issue of internal debate. For the screening of English Language profi  ciency, in this paper, it is a part of the science departments.

Figure 1. Institute of Reference, Leiden University, CEFR levels of 800 tested staff

<!-- image -->

## 1.1.  Organising the Testing Process

The point of departure for determining a language policy, and deciding what levels of English profi  ciency  were  necessary,  was  a  provisionary  decision  made  by  the  3TU  Federation,  which comprises the three technical Universities in the Netherlands: Delft University of Technology, University of Twente and Eindhoven University of Technology (internal language policy document, 2006, offi  cial source not available). Their decision was to require staff to have an English profi  ciency level of IELTS 7.5. Although Delft University of Technology acknowledges the federal decision, it has not made a defi  nite decision on the language profi  ciency requirements.

Table 1. Profi  ciency levels

| Table 1   |
|-----------|

Previously,  Klaassen  (2005),  found  the  average  requirement  for  staff  across  the  Netherlands ranged from IELTS 6.0 - 8.0, CEFR B2 - C1, or Oxford Quick Placement test achieving at least 60% of the test. (For a comparison of language levels of the IELTS/CEF Profi  ciency levels see Table 1. Based on these fi  ndings and the wish to wait for the test results, Delft formulated the requirements that teaching staff must be one step ahead of the students' language level to be able to address the students adequately and to provide teaching in English. There is no empirical basis for this assumption; instead, it has been based on pragmatic reasoning. Consequently, Delft has decided that the staff's level of language profi  ciency will always have to be one level higher than the level of language profi  ciency of the students at a particular programme level. Table 3 summarises the ideal situation.

| CEFR English level of students   | Stage of study                                                | Min. CEFR English level of lecturers   |
|----------------------------------|---------------------------------------------------------------|----------------------------------------|
| C2                               | Final level Master student in English language and literature | C2                                     |
| C1                               | Final level Master student                                    | C2                                     |
| B2                               | Final level Bachelor student/ Entry level Master programme    | C1                                     |
| B1/B2                            | Final level secondary school/entry level Bachelor             | B2                                     |
| A1+A2                            | Not relevant for University level teaching/studying           |                                        |

Table 2. CEFR levels required from students and staff

There are two points of departure in determining the appropriate language level, the fi  rst being bottom up and the second top down; (1) on average secondary school pupils enter the university with a B1/B2 level of English (Hest/Staatsen 2002). (2) Leiden University claims that their students' English language profi  ciency at the end of the English Language and Literature degree programme was, and is, at the C2 level.

At Delft University of Technology the Bachelor programmes are required to offer 2 courses in the form of English medium instruction. The remainder of the courses are offered in Dutch. Additionally, many of the supportive course materials are in English. It is therefore assumed that at the end of the Bachelor level the students will still be more or less at a B2 level. Since students will have become familiar with the disciplinary discourse in their Bachelor-level study, and will have followed all their courses through English-medium instruction during the Master-level, they will hopefully be moving towards a C1 level during this stage. From the top end, it is expected that Delfts' graduate students are at one level lower in language profi  ciency than students studying English language and literature as their major.

## 2. Methods Section

In this section I will describe the tested sample, the test construction and validity, as well as the response rates.

## 2.1.  Sample

The tested population comprises professors, associate professors, assistant professors and lecturers. The Dutch University system works with a scientifi  c staff fl  ow also called the function profi  ling system, indicating the tenure track and level of salary of scientifi  c personnel. Note that assistant professors and higher (non-lecturers) have a doctorate degree, as indicated below.

Table 3. Job profi  les according to the University profi  ling system

<!-- image -->

In total 1,400 scientifi  c staff members were invited in the fi  rst round and 600 in the second round of testing. The numbers of staff with a particular position have been indicated above in the table. Around 20% of professors, 18% of associate professors, 32% of assistant professors and around 29% of lecturers 3 participated in the test.

Staff could either request exemption or cancellation for a relevant reason, and/or reschedule due to particular circumstances. Cancellations took place for several reasons, such as forthcoming retirement, long term illness, sabbatical, maternity leave, guest lectureships, unoffi  cial exemption on the basis of nationality or other reasons. All these cases were offi  cially ratifi  ed by the dean of the faculty. In some cases the test could be rescheduled. Staff could also ask for exemption based on the following criteria:

- IELTS test score of 7.5 A.
- Paper-based TOEFL score of 625 B.
- Master degree in English language and literature C.
- D.  A passport from a so called First language country (UK/USA/Canada/Australia/New Zealand) and documents showing that they had followed primary and secondary education in the fi  rst language country

## 2.2.  The Test

According to Klaassen (2005), while most universities in the Netherlands use locally developed tests, those universities using an offi  cial test use either the IELTS, Dialang or the Oxford Quick Placement test (QPT). At Leiden University a substantial number of lecturers (N=800) had already been tested on a voluntary basis with Dialang and a locally developed oral test based on the CEFR. As Leiden University already had substantial experience, Delft asked them to assist with the testing activities at Delft University of Technology. The assessors were all language specialists from Leiden University. In practice, it was not feasible to test with Dialang any longer, so both

3 The attrition rate of personnel requires continous testing. The results presented in this paper are therefore of a momentary nature.

N=227

Leiden and Delft decided to continue with QPT to assess the English language profi  ciency of the scientifi  c staff. The test used consisted of a 30-minute diagnostic computer test and a 30 minute oral language test. For the computer test the Oxford Quick Placement Test (QPT) was used. This test assesses reading, listening and grammar, and provides an indication of the level of language profi  ciency of the participant. However, emphasis was placed on the oral assessment in which the active language profi  ciency skills were tested.

The oral assessment always took place with an interviewer and a language assessor. The assessor used criteria derived from the Common European Framework of Reference (European Council 2001) to establish the level of language profi  ciency of the participant particularly in a teaching environment. The validity of authentic performance assessment, imitating the actual job stituation as close as possible, is considered a valid predictor of the real life performance of scientifi  c staff.

The assessment criteria for the oral assessment were derived from the 'can do' statements in the Common European Frame of Reference for language, learning, teaching and assessment, developed by the Council of Europe and published by Cambridge University Press in 2001.

- spoken fl  uency · 4 , consisting of vocabulary control, grammar and phonology; e.g. Can express him/herself  fl  uently  and  spontaneously,  almost  effortlessly.  Only  a  conceptually  diffi  cult subject can hinder a natural, smooth fl  ow of language (C1- level)
- addressing audiences · ; 5 e.g. Can give a clear, well-structured presentation of a complex subject, expanding and supporting points of view at some length with subsidiary points, reasons and relevant examples. Can handle interjections well, responding spontaneously and almost effortlessly (C1- level)
- coherence & cohesion · ; 6 e.g Can produce clear, smoothly fl  owing, well-structured speech, showing controlled use of organisational patterns, connectors and cohesive devices. (C1level)
- discussion & meetings; · 7  e.g. Can easily keep up with the debate, even on abstract, complex unfamiliar  topics.  Can  argue  a  formal  position  convincingly,  responding  to  questions  and comments and answering complex lines  of  counter  argument  fl  uently,  spontaneously  and appropriately. (C1-level)

These 'can do' criteria are in a rubric in the CEFR Document and they were used in the scoring form shown below.

Table 4. Scoring form of the Delft test

|                        | A1   | A2   | B1   | B2   | C1   | C2   |
|------------------------|------|------|------|------|------|------|
| Addressing Audiences   |      |      |      |      |      |      |
| Discussion & Meetings  |      |      |      |      |      |      |
| Coherence and Cohesion |      |      |      |      |      |      |
| Spoken Fluency         |      |      |      |      |      |      |
| Vocabulary Control     |      |      |      |      |      |      |
| Vocabulary Range       |      |      |      |      |      |      |
| Phonological Control   |      |      |      |      |      |      |
| Result ORAL assessment |      |      |      |      |      |      |
| Result QPT             |      |      |      |      |      |      |
| Overall Result         |      |      |      |      |      |      |

4 spoken fl  uency p.129, consisting of subcategories; vocabulary control/range p.112, grammar, phonology p.117

5 addressing audiences p. 60

6 coherence & cohesion p. 125

7 discussion & meetings p.78

Using these criteria from the Common European Framework of Reference for rating, during a 30 minute oral test the staff were required to:

- give an introductory talk, -
- give a 5 minute presentation, -
- describe and explain on the basis of a picture a trend in their disciplinary fi  eld and interact -with the interviewer,
- conduct a role play between a student and a lecturer. -

The 5 minute presentation could be prepared. All the other parts consisted of spontaneous production. Based on both the QPT result and the oral assessment, the participants would be informed about their level of language profi  ciency and given a recommendation with respect to language training courses which could be followed to improve their profi  ciency. The fi  nal grading results were weighted as 1:2, QPT: oral test. In other words, the oral test results had more value on the fi  nal results. It is assumed that a lecturer will need 180 hours of training to go from a B2 level (English Admission Course I) to a C1 level (Profi  ciency course I). Similarly, a lecturer would need a minimum of 180 hours to go from C1 to C2 level. It was decided not to train teachers who had a B1 level of language profi  ciency. Instead, staff with a B1 score were and are redirected to outside institutions. Those with A2/A1 levels are excluded from teaching in English-medium instruction altogether.

## 2.3.  Response Rates

Starting in June 2007, (N=) 1,592 staff members across 8 faculties were invited to the language profi  ciency test. Of these, 77% per cent (N=1,226) passed, while 43 staff members had received offi  cial exemption and 63 staff members had requested offi  cial postponement on the basis of illness or other reasons. The remainder of 23% (N= 379), still need to take the test. Response rates per faculty are as follows:

Table 5. Response rates per faculty

| Science                        | N =   | Engineering                            |   N= | Design            |   N= |
|--------------------------------|-------|----------------------------------------|------|-------------------|------|
| Technology, Policy Management, | & 198 | Aerospace Engineering                  |  106 | Architecture      |  333 |
| Applied physics                | 179   | Mechanical, Maritime Materials Science |  157 | Industrial Design |  174 |
| EEMCS                          | 229   | Civil Engineering                      |  195 |                   |      |

The reliability of this test cannot be calculated on the basis of the dataset since it consisted of fi  nal results only. Insight into individual scores on the 'can do' statements (criteria) have not been registered in any database, as no permission was given to study these data in detail.

## 3. Results

In  this  section  the  assumptions  pointed  out  in  the  introduction  will  be  discussed  one  by  one. First we will look at the average language profi  ciency per department and the distributions on the CEFR profi  ciency levels. Second, we will be considering the averages for job position and the distribution across job positions on the CEFR profi  ciency levels. Third, we will look at the scores difference between the QPT and Oral Assessment from the fi  rst cohort tested in 2006/2007. Fourth, we will briefl  y discuss the difference with the Institute of Reference the University of Leiden, and, fi  nally, we will discuss policy targets for language profi  ciency at DUT. It is expected that Mean of CERlevel

7

6,6-

6,4-

6,2- the results will provide insight into the language profi  ciency of the scientifi  c staff and the research questions made in the introduction.

## 3.1.  The Science, Engineering and Design Disciplines

Considering the results across the departments, we have found average results for each department ranging between a level of C1-II and C1- III. Within this range the average English Language Profi  ciency levels in Industrial Design and Architecture are the lowest across the departments and Aerospace and TPM, the highest. Contrary to our expectations, it is not the design disciplines which score the highest in English language profi  ciency but the Science and Engineering departments.

material

sciences

Figure 2. Means across departments (the CEFR level 6 in this fi  gure is C1-II, 7 = C1-III and 8 is C2 and not included in this fi  gure)

<!-- image -->

When we look at the disciplines of Design, Science and Engineering, we fi  nd the following distributions across the departments. For the Design disciplines we fi  nd marginal differences in the distribution. The average CEFR levels for Architecture were 6.53 and for Industrial Design 6.15. Despite the marginal difference in scores we found a signifi  cant (p>.04) difference. When we look at the distribution we fi  nd 4% at B2 level in Industrial Design that can be found at the C2 level in the Architecture department.

Table 6. Design disciplines % CEFR levels

| CEFR levels            | B2   | C1   | C2   |
|------------------------|------|------|------|
| Architecture (A)       | 19%  | 65%  | 18%  |
| Industrial Design (ID) | 21%  | 64%  | 14%  |

The more science oriented departments, EEMCS and Applied Physics, are somewhat higher with an extreme high for TPM in this fi  gure, of which 35% of the population are at a C2 level.

Table 7. Science Disciplines % CEFR levels

| CEFR levels          | B2   | C1   | C2   |
|----------------------|------|------|------|
| EEMCS                | 12%  | 57%  | 29%  |
| Applied Physics (AP) | 15%  | 57%  | 23%  |
| TPM                  | 8%   | 50%  | 35%  |

Signifi  cant differences at the p>.01 level are found between the average scores of TPM (M= 7.07) and Applied Physics (M=6.64) and signifi  cant difference 0.03 between the average scores of TPM (M=7.07) and EEMCS (M=6.7). TPM clearly is a positive outlier within the science disciplines.

The Engineering departments, Civil Engineering, Material Sciences and Aerospace Engineering are other outliers. In this case we fi  nd Aerospace Engineering has a much higher level of English language profi  ciency than the other two Engineering departments, Civil Engineering and Material Sciences, which have means of 6.7 and 6.6 respectively. Signifi  cant differences (mu= 0.00) are found both between Civil Engineering and Aerospace (M=7.45) and Material Science and Aerospace Engineering. The distributions are listed in Table 8.

Table 8. Engineering Disciplines % CEFR levels

| CEFR levels            | B2   | C1   | C2   |
|------------------------|------|------|------|
| Material Science (3ME) | 14%  | 60%  | 25%  |
| Civil Engineering (CE) | 14%  | 57%  | 22%  |
| Aerospace Engineering  | 3%   | 52%  | 41%  |

As can be seen, the Sciences disciplines and Engineering disciplines have language levels comparable to the outliers TPM and Aerospace Engineering. Aerospace and TPM tend to have a more international and scientifi  cally oriented department, which may explain the high levels of English language profi  ciency. Contrary to assumptions, it is also clear that the English language level is signifi  cantly lower in the design discipline. However, this might be infl  uenced by the composition of appointed staff and English language profi  ciency level across staff in the respective faculties. If we do a chi-square test on department and CEFR level controlled for by job title, it indeed appears that the signifi  cant difference between departments disappears. As displayed in Figure 2, this could only be found at the assistant professor level. Furthermore, the mean CEFR level scores across job profi  les (Note that 6,5 is the overall average, 6 being C1-II and 7 being C1-III), indicate that the relationship between department and CEFR level is likely to be determined by the job position of the scientifi  c staff.

## 3.2.  CEFR Level According to Job Profi  le

Indeed, our suspicion that job position is more indicative of language level is confi  rmed by another cross tabulation, with CEFR level and job position as potentially related variables and departments as control variable. Indeed the chi square scores are signifi  cant beyond the .05 level for all departments except for the Aerospace and Material Science department. If we take a closer look at the average (mean) English Language profi  ciency scores according to job position, we can state that the higher the level of appointment, professor vs assistant professor, the higher the level of language profi  ciency. Figure 2 shows the average profi  ciency scores for job position. The overall average is 6.5 between a C1-II level (at 6) and C1-II level (at 7). Staff with a doctorate are all above average in terms of language profi  ciency.

Mean of CEFRlevel

7.

5,5

7,384

7,087

6,875

Means for Job position

6,926

6,809

<!-- image -->

6,664

Figure 3. Means of CEFR levels across job profi  les (Note that 6,5 is the overall average, 6 being C1-II and 7 being C1-III)

An Independent sample t-test shows a signifi  cant difference of p>.02 between Professor 1 and Assistant Professor 1 levels. Similarly, there is a signifi  cant difference of p>.01 between Professor 2 and Assistant Professor 2. Staff at the lecturer level have on average lower levels of language profi  ciency. The quality of education may, therefore, suffer due to the lower language profi  ciency levels of appointed teaching staff. It is of interest to state that the more design oriented the faculty is, the more likely it is that they are working with professionals from the disciplinary fi  eld. These professionals are typically appointed as lecturing staff for only 1 or 2 days. Consequently, many of the professional staff appointed as lecturer may have a lower language profi  ciency. In comparison, scientifi  cally oriented faculty with permanent research staff will show higher average language profi  ciency levels. This might explain the signifi  cant difference in language level across departments. Note, however, that the differences are marginal as they are all still within a C1 profi  ciency range.

## 3.3.  Oral Assessment vs. QPT

In a previous study (Klaassen et al. 2010) we studied a subsample from the present population (n= 920), in which 23% of the entire population at the time scored a C2 level. We have also looked at the percentages of the sample population scoring C1 or C2 etc at the QPT computer test and the Oral assessment. Table 10 below demonstrates the differences in score on the QPT and the Oral Assessment (O) expressed in percentages of the population of the subsample. For example, at the department of Applied Physics (AP) 77% of the participants (N= 126) of the department scored a C2, whereas only 20% scored a C2 on the Oral test (O). If we look at another department e.g. Architecture (A) we fi  nd that 53% of 106 scored C2 on the QPT and only 17% C2 on the oral assessment. This appeared to be a consistent trend; i.e. the level on the computer test QPT is on average much higher than the results achieved in the oral test interview.

Table 9. Oxford-QPT results and Oral results across disciplines (on the left is science, on the right is engineering, and in the middle is design)

<!-- image -->

|            | AP   |    | TPM   |    | EEMCS   |    | AE   |    | CE   |    | 3ME   |    | A   |    | ID   |    | Total   |     |
|------------|------|----|-------|----|---------|----|------|----|------|----|-------|----|-----|----|------|----|---------|-----|
| level CEFR | %    | %  | %     |    | %       | %  | %    | %  | %    |    | %     | %  | %   |    | %    | %  | %       |     |
|            | QPT  | O  | QPT   | O  | QPT     | O  | QPT  | O  | QPT  | O  | QPT   | O  | QPT | O  | QPT  | O  |         | N=  |
| C2         | 77   | 20 | 73    | 35 | 65      | 26 | 65   | 23 | 56   | 26 | 60    | 23 | 53  | 17 | 53   | 12 | 23      | 212 |
| C1         | 14   | 64 | 20    | 52 | 19      | 60 | 27   | 68 | 27   | 60 | 21    | 60 | 22  | 61 | 34   | 65 | 59      | 546 |
| B2         | 6    | 15 | 6     | 12 | 13      | 19 | 7    | 10 | 13   | 14 | 16    | 16 | 16  | 22 | 10   | 22 | 17      | 155 |
| B1         | 3    | 1  | 0     | 0  | 3       | 1  | 2    | 0  | 4    | 1  | 2     | 1  | 8   | 0  | 3    | 1  | 1       | 6   |
| A2/ A1     | 0    | 0  | 1     | 0  | 0       | 0  | 0    | 0  | 0    | 0  | 1     | 0  | 1   | 1  | 1    | 0  | 0       | 1   |
| TotN =     | 126  |    | 113   |    | 144     |    | 60   |    | 10   |    | 104   |    | 106 |    | 101  |    |         | 920 |

The discrepancies between the QPT and Oral Assessment amount to 40% achieving a lower level on oral /listening skills language profi  ciency (the oral test), as opposed to the QPT at the C2 level, 35% for the C1 level and 5% at the B2. If we assume that the QPT represents the writing and reading skills, and the oral assessment represents the oral and listening skills, we fi  nd a dramatic drop in language profi  ciency for these specifi  c skills. Similar results were found at Leiden University with 800 lecturers who did the Dialang and the same oral test (verbal communication). Apparently, it is much harder to activate oral communication skills in a foreign language. In the fi  nal recommendation the Oral Assessment was given double the weighting of the QPT. This also meant the language profi  ciency levels were lowered for the entire population.

## 3.4.  Institute of Reference

The Institute of Reference (Leiden University) shows that on average the language level at Delft University on the same test is higher: B1= 1% in Delft vs. B1= 15% in Leiden, B2 = 15% in Delft vs. 35% in Leiden, 59% at C1 in Delft vs. 30%, C2 26% in Delft vs. 15% in Leiden. Whether this means that Delft University of Technology has on average a better language level than Leiden University, which is a more language, arts, social science and medical science oriented University, is diffi  cult to determine. The tested staff at Leiden University participated on a voluntary basis, which may have attracted staff who were insecure about their language level, and may thus not give a representative picture of the reality. It would certainly be of interest to fi  nd out whether this indicates a trend or whether it can be explained by other variables.

Figure 4. CEFR Level percentages at Leiden University vs. DUT

<!-- image -->

100% T

90% -

80%

70% -

60% -

50% -

40% -

30% -

20% -

10% -

0% +

BK

## 3.5.  Policy Targets

Doel 2013

90% min C1

Doel 2010

80% min C1

Doel 2010

70% min C1-I1I

Doel 2013

50% min C2

The Board of Directors at Delft University decided that 80% of the lecturers were to attain a C1 level in 2010, and that 90% should reach the C1 level before September 2013. Furthermore, 50% were to reach the C2 level in 2013 8 . In addition, the board has indicated that language profi  ciency is: · C1 · C1-III/

- to  be  an  obligatory part of the annual job appraisal process of those lecturers obtaining a · lower score than C1;
- to be a minimum entry requirement for job applicants who, within a time span of 3 years · from their appointment, were to qualify at a C1 or C2 level (Final Report, English Language Assessment Programme, 2007).

It has been found that 73% of the scientifi  c staff (Prof, assoc. Prof, assist. Prof) have been tested on language profi  ciency. Of the tested staff 80% of the lecturers were to attain a C1 level in 2010, and 90% should attain a C1 level before September 2013. Furthermore, 50% should reach the C2 level in 2013. If we look at the fi  gure below we see the percentages on the left hand side. The bars represent the different departments. The bottom line shows the 2013 target of 50% at C2 level. C2 is the bottom blue colour. We can see that a lot of training still needs to take place, particularly at the Design departments, Architecture (BK in Dutch/A in English) and Industrial Design (IO/ID). The top 80% line indicates the 2010 target of 80% minimally at C1 level. This target is almost reached as the yellow and purple parts show in the graph. The top light blue represents staff that did not pass or have not participated yet.

Figure 5. Achieved policy targets across departments 9

<!-- image -->

## 3.6.  Discussion

To sum up, the majority of the scientifi  c staff have a language profi  ciency level at C1 of the Common European Framework of Reference. This level is roughly equivalent to an IELTS Level of 6.5/7.0. At present, the IELTS 7.5 (minimum C2) recommended profi  ciency level by 3TU Federation Committee is rather ambitious, as only 21% of the entire scientifi  c staff included in the test achieve a C2 level. The C2 level group is distributed across function level: Professors 26%, Associate Professors 28%, Assistant Professors 37% and Lecturers 9%, which means that of the scientifi  c staff at large, about 79% still do not come close to this level. DUT has therefore decided to set the cut-off point at IELTS 7.0. Hopefully, this means that the targets for 2013 where 50% of the staff are to achieve a C2 level and 90% a C1 level will be still feasible. It can be concluded that a substantial time/effort investment will be required of the scientifi  c staff to reach IELTS 7.5 level goal.

8 Ratifi  ed in the 'Concept Decision Board of Directors with respect to English Language test november 2006 (offi  cial minutes of internal meetings are not available for public use)

9 This fi  gure could not be edited, therefore a translation of terminology is included BK=A, CiTG (CE), EWI =EEMCS, IO=ID, LR=AE, 3ME =3ME, TBM=TPM, TNW= AP, Niet behaald = did not pass, Doel = target

Language profi  ciency levels do not differ across faculties when corrected for by job profi  le. As mentioned above, it was also found that the higher the position of a person in the organisation, the higher the language profi  ciency level. Furthermore, oral and listening language profi  ciency tend to be less developed as opposed to the reading and writing skills of the scientifi  c staff. Finally, in comparison with the humanistic University of Leiden, the language profi  ciency level seems to be higher at Delft University of Technology. Since this may be due to selection factors, further research is required.

The whole screening operation was a result of complaints of students about the poor level of language profi  ciency of their lecturers. As the test results were rather reasonable, no more complaints have been made by students for a long time. Contrary to the expectations of the Board of Directors, however, student complaints have resumed, despite the effort to raise the profi  ciency levels of the scientifi  c staff. Therefore, the question remains whether the language profi  ciency of staff is the central issue or whether it is the lack of student profi  ciency. Hellekjaer (this issue) suggests that student language profi  ciency plays a crucial role in the successful implementation of EM instruction. However, teaching English to a large amount of students is beyond our engineering curricula. Another hunch is that adapting the English level to the students and opening for clarifi  cation questions might help student understanding. As we do not know the source of students' complaints, it is hard to tackle the problem, and we have the following alternatives:

- Assess students' language profi  ciency. 1.
- Discontinue the screening operation and have confi  dence in the tenure track of staff, which 2. guarantees improvement of language profi  ciency while moving up to a better job position.
- Work with permanent staff in research positions. 3.
- Distribute  management  responsibilities  more  equally  across  scientifi  c  staff,  so  that  also 4. professors with a high language skills get the opportunity to teach the students.

## 4. Conclusion

We started out with these central questions:

- What is the average level of scientifi  c staff's English Language profi  ciency at Delft University · of Technology?
- Is the International English Language Testing System (henceforward IELTS) level 6.5/ C1 · level of the Common European Frame of Reference (CEFR) requirement realistic?

The conclusion is that a fi  rst screening gives a representative picture of the language profi  ciency level of the scientifi  c staff at Delft University of Technology. On average the language profi  ciency level is C1 (55%). Nevertheless, a large percentage of scientifi  c staff members are not achieving a C2 level , which means that establishing this goal of C2 language profi  ciency would certainly not be feasible in the near future. The assumptions were that job position and disciplinary context would determine the level of language profi  ciency. Indeed the average language profi  ciency level in this study became higher when the job position of the scientifi  c staff improved from phd to assistant, associate and professor. This may mean that language profi  ciency is primarily infl  uenced by the exposure and frequent use of the language in appropriate (disciplinary) contexts, and is not determined by the nature of the discipline (more or less technical).

Unless a more systematic approach is used to learn and develop disciplinary lexis and academic discourse of both lecturers and students at an early stage of their career, the language profi  ciency for academic purposes is unlikely to improve. In that case lecturers and students are likely to remain critical of English-medium instruction in the educational context. A C1 level is certainly feasible at a maintenance level. However, Delft university needs to work in order to improve the situation if it wants to become a university with a truly international profi  le.

At present, students see the need for high levels of academic English profi  ciency. And if we are to prepare our local students for global citizenship we have to offer them the opportunity to listen to lecturers with a C2 level of English and provide the opportunity to acquire the languages at an acceptable level themselves. How to achieve this objective depends on the particular context, vision and ambitions of any university within the European Union.

## References

- AAG 2005: [online]. http://www.aag.tudelft.nl/verkiezingssite (accessed 2005).
- Council of Europe 2001: Language Policy Division, Common European Framework of Reference for language, learning, teaching and testing. Cambridge University Press. Also available via [online] www.uk.cambridge.org/elt, http:// www.coe.int/t/dg4/linguistic/Source/Framework\_EN.pdf (accessed 2005).
- Bos, Madeleine/Roubos,Tim/Klaassen, Renate, G./van den Boogaard, Gerard 2007: Final Report; English Language Assessment Programme, Internal report.
- Crawford Camiciotolli, Belinda 2005: Adjusting a business lecture for an international audience: a case study. In English for Specifi  c Purposes 24, 183-199.
- Dafouz, Emma/Nunez, Begona 2007: Analysing stance in a CLIL University Context; Non-Native Speaker Use of Personal Pronouns and Modal Verbs, In Coyle, D./Beardsmore, H. Baetens, Research on Content and Language Integrated Learning (CLIL), International Journal of Bilingual Education and Bilingualism , Vol. 10:5, 647-663.
- Ellis, Rod 1992: Instructed Second Language Acquisition . Basil Blackwell.
- Els,Theo/Bongaerts, Theo/Extra, Guus/van Os, Charles/Jansen-van Dieten, Anne-Mieke 1993: Applied Linguistics and the Learning and Teaching of Foreign Languages . Edward Arnold.
- Flowerdew John/Miller, Lindsay 1997: The teaching of Academic Listening Comprehension and the Question of Authenticity. In English for Specifi  c Purposes 16 (1), 27-46.
- Hellekjaer, Glenn O./Westergaard, Marit, R. 2003: An exploratory survey of content learning through English at Nordic Universities. In Van Leeuwen, Ch./Wilkinson, R., Multilingual Approaches in University Education; Challenges and Practices . Uitgeverij valkhof Pers Nijmegen, 65-81.
- Hest, Erna /Staatsen, Francis 2002: Dutch Language Qualifi  cations in a European Perspective. Source Cito Website [online]. http://www.cito.com/research\_and\_development/participation\_international\_research/cefr.aspx (accessed 1 June 2010).
- Jordan, R.R. 1997: English for Academic Purposes; a guide and resource book for teachers . Cambridge: Cambridge University Press.
- Klaassen, Renate G. 2001: The International University Curriculum; Challenge in English-medium Engineering Education . Thela Thesis (dissertation).
- Morell, Teresa 2004: Interactive lecture discourse for university EFL students. In English for Specifi  c Purposes 23, 325-338.
- Räsänen, Anne 2005: Pursuing quality through institutional policies. Paper presented at BIMU 05 conference Bi- and multilingual  universities:  Challenges  and  future  prospects .  Helsinki  1-3  September,  2005.  [online]  http://www. palmenia.helsinki.fi  /congress/bilingual2005/program.asp (accessed 2005).
- Smid, Henk J./Verweij, A./Jochems, Wim/Snippe, Joke (eds.) 1995: Studiesucces van buitenlandse studenten; studievoortgang in het technische wetenschappelijk onderwijs geanalyseerd. In Tijdschrift voor Onderwijs Research 20 (2), 165-177.
- Turner, Yvonne: Refl  ecting on the Relationship Between Standardised Admissions, Academic Expectations and Diverse Student Cohorts in Postgraduate Taught Business and Management Programmes. In International Journal of Management Education 16 (1), 7-13 [online] . http://www.heacademy.ac.uk/assets/bmaf/documents/publications/ IJME/Vol61/editorial\_essay.pdf (accessed May 2010).
- Uhlenbeck, Annemarie, M. 2002: The Development of an Assessment Procedure for Beginning Teachers of English as a Foreign Language . Leiden University, ICLON Graduate School of Education (dissertation).
- Vinke, Diane A. 1995: English as the Medium of Instruction in Dutch Engineering Education. Delft University Press, (doctoral dissertation).

76