A Key Actor in EMI METU English Proficiency Exam: Design, Implementation & Validation Aim The School of Foreign Languages (SFL) initiated a Program Evaluation and Renewal study to investigate the effectiveness of the English Language education curricula at its two departments based on a large scale needs analysis of all relevant stakeholders which involved the evaluation and validation of METU English Proficiency Exam, integral to EMI at METU. Design This research was designed as a mixed methods study. Our approach towards the qualitative / quantitative dilemma was pragmatic. Pragmatism views these two methods as compatible (Howe, 1988, in Teddlie & Tashakkori, 2009), and therefore finds it productive to combine these methods to study phenomena especially in social sciences. Hence, the pragmatic paradigm was the framework of this study. Needs Analysis Data Analysis English Proficiency Examination EPE Revisited- OPERATIONS Design Identifying Target Language Use tasks, task characteristics, theoretical constructs Operationalization Developing test task specifications & test blueprint Administration Alpha & Beta testing Piloting Field-testing Final revision First official version of new EPE Results: Communicative Needs of EMI students Benchmarking, Piloting & External Validity To validate EPE, an experimental pretest-posttest design (N=211) was used to compare EPE scores with an international standardized test, TOEFL IBT, a gatekeeper in many EMI universities. There was a significant positive correlation between EPE and the pretest (r = 0,89, n = 102, p = 0.070); between the two versions of the TOEFL IBT (r = 0,89, n = 102, p = 0.00); and between EPE and the posttest (r = 0.85, n = 115, p = 0.044). EPE cut scores were based on this study. Listening: Field, 2012 Reading: Urqhart & Weir, !998; Khalifa & Weir, 2009 Writing: Shaw & Weir, 2008 Reliability & Internal Validity Predictive Validity & Stakeholder Perception Stakeholders Qualitative / Quantitative Instr. 2612 DBE Ss Surveys / Focus- group interviews 2205 MLD Ss Surveys / Focus- group interviews 133 SFL Instructors Surveys / Interviews 66 Faculty Members Interviews 40 Graduates in 19 Firms Interviews 14 Employers in 12 Firms Interviews Document Analysis of student essays, exams, lab reports from all faculties Communicative Activity Required Skill Listening -listening in-class -assigned listening Very deep attention understanding instructions Medium to deep attention following lectures /discussions Reading -background reading -intensive reading search reading reading longer texts careful reading understanding instructions analytical/critical reading interpreting visuals Writing -writing in -class -assigned writing note-taking writing responses to exam projects, reports, papers fluency & coherence Speaking -spontaneous speech -planned speech communicating with instructors asking & answering Q’s delivering presentations fluency in speech Grammar & Vocabulary basic structures - wide range academic lexis • For quantitative data, descriptive and inferential statistics were employed. Depending on the results of tests of normality, either ANOVA/ t-Test or Kruskall- Wallis and Mann-Whitney U tests were used. • As for qualitative data, the recordings of interviews were analyzed by MaxQDA 11. Miles and Hubermann Qualitative Data Analysis Framework was used (1994). Theoretical Framework of Test Design Sociocognitive Framework, Weir, 2005 The relationship between, TOEFL Pre-and Po EPEsttests Reliability: Cronbach’s alpha, which measures the internal consistency of a group of items was found to be (α= .81). This alpha coefficient is high (≥70 ) in terms of testing. Item Discrimination: The Rpbi mean for the MC items was found to be Rpbi .33, meaning that the exam discriminated well between higher and lower proficiency levels. Items with Rpbi in the .30 - .49 range are considered strong discriminators. Item difficulty: P values for the scored items show a desired distribution of the P (.4