RESEARCH IN SUPPORT OF CURRICULUM DEVELOPMENT IN A SELECTED SERVICE SCHOOL OF AIR UNIVERSITY*
Authors: Frederiksen Norman O. , Saunders David R. , Wand Barbara 
Published: August 2014

This report describes the preparation of test materials which were designed to aid in the evaluation of instruction in the Command and Staff School of Air University. A major purpose of this course is to increase the administrative proficiency of field grade Air Force officers. Test materials for evaluation of instruction are desirable in order to support improvement in curriculum planning and instructional procedures.

The test which was developed is a situational test which requires the examinee to play four roles: Commanding Officer, Director of Materiel, Director of Personnel, and Director of Operations of a fictitious composite wing. The examinee is provided with background information about the wing and the AF base where it is located, suitable material from the “files,” and the contents of an in‐basket which is appropriate to each role (hence the name, In‐Basket Test). The in‐basket contains letters, memoranda, and other documents which embody problems aimed at eliciting behavior relevant to the objectives of instruction in the course. The examinee's task is to write the letters and memoranda which he would write if he were actually on the job, to sign letters prepared for him, or to take other suitable action. Scoring of these products is intended to provide information which would be useful to the Air University in evaluating the instruction in the Command and Staff School. Full instructions for administering and scoring the test are provided in the Appendixes.

This report also presents the results of a tryout of the test. The purpose of this tryout was to investigate some of the statistical properties of the test, student reaction to the test, and the adequacy of the instructions and procedures for administration. An entire class in the Command and Staff School took the test in the third week of the course.

The results indicate that the In‐Basket Test can be scored with a reasonably high degree of reliability, but that the present form of the test is low in content reliability. There is evidence, however, that the content reliability could be improved by reassigning scoring weights to the responses; by eliminating poor items and revising others; by increasing the homogeneity of the items through use of techniques such as factor analysis.

No criterion of validity was available, but correlations with other measures are presented which seem to indicate that the test is not measuring the same things measured either by the ACE or by the CSS final course grade. There is also some evidence that the relationship of the in‐basket role to the student's area of greatest experience in the Air Force is a factor of minor importance in influencing his score. The opinions expressed by students in a post‐test inquiry indicated a substantial acceptance of the test materials for purposes involving comparisons of groups.

It is concluded from analysis of the data obtained from the tryout that the pretest form of the test is suitable for certain administrative purposes, such as comparisons of groups before and after training, but not for purposes of individual assessment. It is recommended that steps be taken to improve its reliability before the test is widely used. Suggestions are presented for extending the range of information obtained from the test, and for increasing its reliability through reassessment of scoring weights, improvement in the homogeneity of the subtests, revision of items, and changes in the scoring procedure.