In addition, the items need to include a variety of listening tasks, such as comprehension of the main ideas of a conversation or mini-lecture, recognition and recall of details of a conversation, identification of specific words and phrases used in a passage, and so forth. Reliability refers to the precision and consistency of scores derived from a test instrument. General factors influencing reliability include, for example, whether instructions for the examinees are clear and explicit, or whether the examinees are sufficiently familiar with the format of the CAT before taking it. Situational factors include those related to the testing environment, such as noise level or whether headphones are provided. Individual factors include transient factors, such as the physical and psychological health of the test takers, and stable factors, such as examinees' experience with similar tests.
Validity refers to whether a test actually measures what it purports to measure.
It relates to the appropriacy of the inferences made on the basis of the test scores. CAT developers and users need to examine issues related to each of these types of validity. Examinees should be given the time and opportunity to become thoroughly familiar with both the computer and the testing system. An orientation to the structure and types of items they will encounter during the CAT should also be required for all examinees. The practice items should be equivalent in structure and content to those contained in the item bank.