And given that national data collection efforts that use teacher surveys to describe teaching practices are in their infancy, researchers and policymakers want to know how much faith they can have in this type of data. Studies that have investigated the reliability and validity of using surveys to gather information on teaching practices have produced both encouraging and discouraging findings. The reliability of a survey describes whether its use in repeated trials will yield the same results. Low reliability could be the result of teachers finding the questions difficult to interpret or inaccurately recalling what they do in their classrooms. But knowing that an instrument is reliable does not justify the assumption that it is valid.
Validity describes the extent to which an instrument accurately measures the phenomena of interest.
To date, efforts to evaluate the reliability of the TFS items on teaching practices have raised questions but not resolved them. Though the analyses used in the report try to account for this, the authors note that the findings should be interpreted with caution. Aptitude tests always consist of multiple questions that measure an underlying characteristic, such as mathematics ability. Tracking the cost of only one product, such as canned soup, would not provide an accurate or informative picture of inflation. And answering one algebra question would not provide an accurate measure of mathematics aptitude.