A further complexity is introduced by the fact that social policy reforms both reflect and shape public attitudes. Changes in eligibility that are intended to affect demographic behavior may also affect the incentives of both clients and program staff to report events, as well as the ability of the system simply to process reports. With complicated program reforms, a great load is put on administrative data systems, which may become less reliable as sources of data for evaluations. The welfare reform debate does not end with the passage of federal legislation, nor even with the first round of changes at the state level. But this potential cannot be fully exploited now, according to Moffitt, because the results of academic studies are so mixed, and no one can say with confidence why the results are mixed.
Studies differ in the datasets analyzed and in the models used to overcome confounding effects.
Another problem in conducting research on welfare is the difficulty in deter- mining what policies are in place. Lacking that information, analysts cannot use the national survey data effectively for analyses of impact in an era of ever-widening policy variation. Trying to figure out exactly what rules prevailed in a particular location at a given time is an arduous task, as Bumpass could testify, and he urged the U.S. Department of Health and Human Services to maintain a county-level database available to all as a way to facilitate useful research. This problem of uncertainty in matching survey data with the welfare rules applying at a particular place and time will only become worse as state programs diverge even more under the 1996 act. Since reforms can now be introduced by states without any requirement to collect data and report to the U.S. Department of Health and Human Services, there will be no federal agency from which, in theory, researchers could collect the information needed for a geocoded dataset of welfare rules.