Predictive validity involves estimating a future criterion. Concurrent validity relates to current status on a criterion. Key Issues in Criterion-Related Validity Generalizability should be demonstrated through cross validation. Carefully consider the adequacy of criterion measures . There are usually multiple ways to represent a criterion, rarely is one perfect.
Small samples have correlation coefficients that tend to bounce around a great deal.
Consider the impact of range restrictions on predictive accuracy, and differential accuracy. As range and variability of scores of a test shrink, so does the possible predictive power of the test. This is most relevant for tests measuring theoretical abstractions. The test and the definition of what is measured are intimately linked. The level of construct validity is based on how well the theory works and how well the test measures the abstraction in question.