EFFECTS OF ITEM DISCLOSURE ON TOEFL PERFORMANCE

New legislation requires that standardized tests be disclosed after they have been administered. According to procedures followed to date, TOEFL forms previously used in International or Special Center administrations have been provided to institutions for use in the Institutional testing program. In light of the new legislation, alternative procedures need to be explored. One alternative is to combine items from previously used forms, possibly including ones that have been disclosed, into new forms for use in the Institutional program. Before initiating such a plan it is necessary to know how TOEFL performance would be affected if candidates had access to some of the items before administration of a test containing those items. The present study addressed this question through an experimental approach. A number of specially constructed TOEFL forms, here called “disclosed forms,” were made available to foreign students in intensive English language programs. Later the students were administered a special TOEFL consisting of items from those forms and a TOEFL consisting of undisclosed items. Effects of item disclosure were determined by contrasting performance on these two tests. The students were given copies of the disclosed forms, and their language instructors devoted class time to discussing a portion of the items in those forms. The study was thus expected to show the effect that might be produced if, as might happen in reality, students were tutored on disclosed items in test-preparation courses. Separate analysis of test performance for items that had been disclosed but not discussed in class would show the extent to which students benefit from disclosed items when left on their own initiative to study them. An additional variable in the study was the size of the disclosed item pool. It was hypothesized that, if students must cover a large number of test forms in order to be exposed to all the items that will appear on a later test, they will be less likely to benefit from disclosure than if they need only cover a small number of forms. To test this hypothesis, items to appear on the test were spread through six disclosed forms for students in eight institutions and through twelve forms for students in eight other institutions. A significant disclosure effect was observed: Scores for the test containing disclosed items were greater than those for the test containing undisclosed items, indicating that the students studied the disclosed forms and recalled specific items from them. The effect of disclosure was greater for students who received six disclosed forms than for those who received twelve forms, confirming the hypothesis presented above. The effect for students receiving the larger number of forms was significant, nevertheless. Student questionnaire data suggested that the time spent studying the disclosed material was relatively unaffected by the number of forms given the students. Additional analyses indicated disclosure effects for items in forms that were not discussed in class as well as for items that were discussed. Thus, the effect of disclosure was not due solely to the class experience, as the students apparently studied the disclosed forms on their own initiative. A disclosure effect was observed for each of three main language groups and for every item type, which attests to the robustness of this effect.