Evaluation of the NFIP
       
Performance Assessment and Evaluation Measures for Periodic Use by the National Flood Insurance Program


Prepared as part of the 2001 - 2006 Evaluation of 
the National Flood Insurance Program


Ted R. Miller
Elizabeth Langston
Valerie Nelkin


Pacific Institute for Research and Evaluation
11710 Beltsville Dr., Suite 125
Calverton, MD 20705


October 2006


EVALUATION OF THE NATIONAL FLOOD INSURANCE PROGRAM 
This Evaluation comprises a series of reports prepared by the American Institutes for Research (AIR) and selected subcontractors under a contract managed by AIR. These reports assess questions identified and prioritized by a steering committee about the National Flood Insurance Program. Individual reports will be posted on the FEMA website as they are finalized. The website URL is http://www.fema.gov/business/nfip/nfipeval.shtm. The reports in the Evaluation are

The Evaluation of the National Flood Insurance Program - Final Report  
American Institutes for Research and NFIP Evaluation Working Group


Assessing the Adequacy of the National Flood Insurance Program's 1 Percent Flood Standard. Galloway, Baecher, Plasencia, Coulton, Louthain, and Bagha, Water Policy Collaborative, University of Maryland.

The Role of Actuarial Soundness in the National Flood Insurance Program. Bingham, Charron, Kirschner, Messick, and Sabade, Deloitte Consulting.

Costs and Consequences of Flooding and the Impact of the National Flood Insurance Program. Sarmiento and Miller, Pacific Institute of Research and Evaluation.

Developmental and Environmental Impacts of the National Flood Insurance Program: A Review of Literature. Rosenbaum, American Institutes for Research.

The Developmental and Environmental Impact of the National Flood Insurance Program: A Summary Research Report. Rosenbaum and Boulware, American Institutes for Research.

An Evaluation of Compliance with the National Flood Insurance Program Part A: Achieving Community Compliance. Monday, Grill, Esformes, Eng, Kinney, and Shapiro, American Institutes for Research. 

An Evaluation of Compliance with the National Flood Insurance Program Part B: Are Minimum Building Requirements Being Met? Mathis and Nicholson, Dewberry.

Evaluation of the National Flood Insurance Program's Building Standards. Jones, Coulbourne, Marshall, and Rogers, Christopher Jones and Associates.
       
Managing Future Development Conditions in the National Flood Insurance Program. Blais, Nguyen, Tate, Dogan, and Petrow, ABSG Consulting; and Mifflin and Jones. 

The National Flood Insurance Program's Mandatory Purchase Requirement: Policies, Processes and Stakeholders. Tobin and Calfee, American Institutes for Research.

The National Flood Insurance Program's Market Penetration Rate: Estimates and Policy Implications. Dixon, Clancy, Seabury, and Overton, RAND Corporation.

Performance Assessment and Evaluation Measures for Periodic Use by the National Flood Insurance Program. Miller, Langston, and Nelkin, Pacific Institute of Research and Evaluation.
       
State Roles and Responsibilities in the National Flood Insurance Program. Mittler, Morgan, Shapiro, and Grill, American Institutes for Research.

      The research described in this report was funded with Federal funds from the Federal Emergency Management Agency under contract # 282-98-0029 and under subcontract to the American Institutes for Research. The content of this publication does not necessarily reflect the views or policies of the Federal Emergency Management Agency, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.
      
      A 30-year-old not-for-profit corporation headquartered in Calverton, MD, the Pacific Institute for Research and Evaluation specializes in basic science studies of the causes and origins of risky and anti-social behaviors, as well as the evaluation of policies and programs designed to reduce the incidence of risk-taking and mitigate its consequences. The hallmark of PIRE's activities is a proactive stance that promotes conceptually sound, scientifically supported, culturally sensitive prevention, mitigation, and loss compensation practices.
      
      Established in 1946, with headquarters in Washington, D.C., the American Institutes for Research (AIR) is an independent, nonpartisan not-for-profit organization that conducts behavioral and social science research on important social issues and delivers technical assistance both domestically and internationally in the areas of health, education, and workforce productivity. 
      

AUTHORS

      Ted R. Miller, Principal Research Scientist and Director, Public Services Research Institute, Pacific Institute for Research and Evaluation directed the design of the Evaluation of the National Flood Insurance Program, as well as PIRE's work on the Evaluation itself. He has led more than 175 studies, including 75 economic analyses. Dr. Miller has 29 years of experience analyzing property and casualty insurance issues for the Federal government.. He has been involved with flood insurance intermittently since the 1970s when he evaluated the capability and experiences of small communities in implementing the NFIP and associated building code and inspection systems. His recent research primarily probes risk levels, the costs of drowning and other preventable injury incidents, and the benefits of prevention from societal, employer, government, and insurer perspectives. He spent 20 years analyzing housing rehabilitation, low-income housing, economic development, and public finance issues. He received a Ph.D. in Regional Science (spatial economics), a Masters in City and Regional Planning, and a M.S. in Operations Research from the University of Pennsylvania, as well as a B.S. in Engineering from Case-Western Reserve University.
      
      Elizabeth A. Langston, a Senior Project Manager at the Pacific Institute for Research and Evaluation has 17 years of evaluation research experience. Much of her work has been in the area of policing and traffic safety, including research in impaired driving, field sobriety testing, community policing, and evaluation of law enforcement training programs. Ms. Langston began her career at PIRE and returned there in 2002. In between, she served as Executive Director of the Center for Criminal Justice Studies, the research arm of the Fraternal Order of Police. At the Urban Institute, she managed the National Evaluation of the Community Oriented Policing Services (COPS) Program. Ms. Langston received a M.A. in Applied Psychology from the University of Arkansas at Little Rock and a B.A. in International Relations from Hendrix College.
      
      Valerie S. Nelkin, Senior Evaluation Director at the Pacific Institute for Research and Evaluation, was the task manager for the study of performance measures, part of the Evaluation of the National Flood Insurance Program. She is a senior public health and social science analyst and manager at PIRE. Before joining PIRE, Ms. Nelkin was Vice President of a management consulting firm, and then President of her own health and human services consulting firm. She has extensive research and evaluation experience in public health, human services, education, and safety programs, and has authored many publications and reports in these fields. Ms. Nelkin is skilled in program and policy evaluation, data collection, and qualitative analysis. She has conducted program evaluations in numerous culturally diverse communities across the country. Ms. Nelkin has served on a number of national public health task forces and has given presentations at many professional conferences. She earned a Master's degree in Human Services and Social Planning from Antioch University and a B.A. in Spanish & English; Education from the University of Missouri.

TABLE OF CONTENTS
1	EXECUTIVE SUMMARY	 3
2	BACKGROUND	4
2.1	Methods	 4
2.2	Study Challenges	 5
2.3	Comments on NFIP Data Collection and Analysis	 6
3	SUGGESTED PERFORMANCE MEASURES BY TOPIC	 7
3.1	Category 1: The Occupancy and Use of Floodplains	 7
3.1.1	Issues and Measures	 8
3.2	The Costs and Consequences of Flooding	 10
3.2.1	Issues and Measures	 10
3.3	Insurance Rating and Indemnity Functions	 12
3.3.1	Issues and Measures	 12
3.4	Floodplain Management and Enforcement	 16
3.4.1	Issues and Measures	 17
3.5	Hazard Identification and Risk Assessment (Mapping)	 21
3.5.1	Issues and Measures	 21
3.6	Communications and Marketing	 22
3.6.1	Issues and Measures	 22
4	CLASSIFICATION OF THE MEASURES BY NFIP GOAL	 25
5	CONCLUSION	 27
6	REFERENCES	 29
APPENDIX A. OVERALL LOGIC MODEL OF THE 
NATIONAL FLOOD INSURANCE PROGRAM	 3


ACKNOWLEDGEMENTS

      The authors are grateful to many individuals for their assistance, comments, and insights. This includes Claudia Murphy, Mike Robinson, Marc Shapiro, Tom Hayes, Larry Larson, Dennis Mileti, Frank Reilly, and Rich Tobin. We also thank those we interviewed during this study and the many members of the evaluation team who provided written recommendations on measures to include in this report. The responsibility for the final content of the report, however, rests with the authors.


1 EXECUTIVE SUMMARY
      This study evaluates the performance assessment and evaluation measures routinely used by the National Flood Insurance Program (NFIP). It recommends continued collection of the measures that FEMA currently uses. Those measures, however, are limited in number and scope. The study recommends additional measures that would better inform stakeholders and policymakers about the NFIP's costs, benefits, and accomplishments and allow an ongoing assessment of the extent to which its implementation contributes to its goals. 
      
      This report is based on interviews with 15 NFIP managers and stakeholders selected by the NFIP staff overseeing the project, coupled with input from the AIR evaluation team. The stakeholders included staff from the Government Accounting Office and the Congressional Research Service, a state flood insurance coordinator, and a representative of the insurance industry. Many of the measures recommended were collected in the evaluation and warrant monitoring. Others were identified in the evaluation design but not collected during the evaluation due to resource limitations. Still others respond to quality problems with management data and other information deficiencies uncovered during the evaluation.
      
      Among the 27 groups of measures recommended, two address the occupancy and use of floodplains, four examine the costs and consequences of flooding, seven cover insurance rating and indemnity functions, eight address floodplain management and enforcement, three examine hazard identification and risk assessment, and the remaining three cover communications and marketing. The highest priority additions to current measures are:
      
* Enhanced tracking of the number and nature of properties in Special Flood Hazard Areas (SFHAs) and the NFIP penetration rate among those properties
* Better tracking of unresolved problems surfaced during CACs/CAVs
* Monitoring to detect communities with concentrations of submit-to-rate properties that are non-compliant with local building codes and standards
* Annually using the HAZUS simulation model to measure the reduction in the costs and consequences of floods during the past year that resulted from NFIP mitigation efforts
* Improving information on progress with flood hazard mapping

      A number of data enhancements through new data collection would support cost-effective performance measure development and provide data that improve program management. The most desirable enhancements appear to be:
* Requiring agents writing a NFIP policy to record the same information about structure value and the existence of a mortgage that they would record in writing a homeowner's policy
* Promptly entering all Community Assistance Calls and Community Assistance Visits (CACs/CAVs) into the Community Information System
* Improving the flood depth and structure elevation data recorded by claims adjusters
* Improving and accelerating the flow of claims information to local officials charged with making substantial damage determinations

2 BACKGROUND
      This study evaluates the performance assessment and evaluation measures routinely used by the National Flood Insurance Program (NFIP) and recommends improvements. Performance assessment measures are indicators used to monitor program functioning. Evaluation measures are indicators used to record how a program functions and what it achieves, with the intent of documenting successes, detecting problems, and suggesting improvements. This report is concerned solely with measures that should be used on a periodic or ongoing basis, not with measures collected once to answer questions about program operations, outcomes, and impacts. The study addresses two related questions.

1. What measures does FEMA use to assess and evaluate the NFIP's performance regarding its efforts to affect: (a) the occupancy and use of floodplains; (b) the costs and consequences of flooding; (c) insurance rating and indemnity functions; (d) floodplain management and enforcement; (e) hazard identification and risk assessment; and (f) communications and marketing?

2. Are there alternative measures of performance that would better inform stakeholders and policymakers about the NFIP's costs, benefits, and accomplishments and allow an ongoing assessment of the extent to which its implementation contributes to its ultimate goals, namely: a) decreased risk of flood losses; b) reduced costs and adverse consequences of flooding; c) reduced demands and expectations for federal disaster assistance after floods; and d) the restoration and preservation of the natural and beneficial values of floodplains?
      
      In any organization, useful performance measures are necessary to monitor program performance and to guide program management. As a part of the Evaluation of the National Flood Insurance Program (NFIP), this study explored the measures used to assess and evaluate the NFIP's performance with respect to six areas of interest. The Evaluation will consist of many special studies to address questions of interest. This sub-task of the evaluation is a cross-cutting study, designed to explore alternative performance measures which are available currently or could be made available on an on-going basis rather than as part of a special study. 
      
2.1 Methods
      At FEMA's request, this sub-study focused on performance measures derived from key informant interviews and from the American Institutes for Research (AIR) evaluation effort. Using a structured open-ended discussion guide, face-to-face or by telephone, the Pacific Institute for Research and Evaluation (PIRE) interviewed 15 key informants including management staff from FEMA headquarters and selected regional offices, FEMA's Office of Inspector General, the Government Accounting Office, the Congressional Research Service, a state flood insurance coordinator, and a representative of the insurance industry. The list of key informants was provided by the NFIP staff overseeing the study. All of those recommended for interviews participated in the study. Additionally, PIRE informally interviewed all six senior members of the AIR evaluation design team.
      
      The interviews took approximately one hour. To encourage openness among the respondents, we agreed to report the responses anonymously in this report.
      
      In the interviews, PIRE first asked respondents of their role within the NFIP. Respondents were asked to identify what NFIP/FEMA performance and monitoring information they received, who collects the information, how often the information is collected, and who receives the information (e.g., internal use, Government Performance Results Act (GPRA) measure, reported to the Office of Management and Budget (OMB), reported to Congress). We also asked respondents to consider a wish list of useful information not currently reported or collected, and followed up on any suggestions by asking about accessibility and basic cost issues, and what priority they would put on collecting or assembling this information. 
      
Specifically, the respondents were asked: 
1. What is the nature of your contact with NFIP/FEMA? 
2. Do you receive or track any data that can be used as an indicator of NFIP performance?
3. In what form or format do you receive the data/measures (e.g. electronic, paper, maps, oral reports)?
4. How accurate and complete are the data? 
5. What data do you think are needed that are not available now? 
6. How could this information be collected?
7. If you had this information, would it help you in setting policy, reporting, or something else?

      Results from the 21 completed interviews were entered into a matrix listing the performance measures, associated issues, current and planned availability, and suggestions for alternate or additional measures. The responses in this matrix were classified by the six areas of interest and then analyzed and condensed.
      
      We subsequently sent all ten key members of the AIR evaluation team a written request for input. They were asked to identify information, especially information collected during their NFIP evaluation projects that FEMA should gather periodically to assess NFIP performance or monitor NFIP-related efforts including mitigation activities. Five of those polled recommended measures. The request for input probed for suggested data sources, data collection frequency, and internal and external audiences for the measures recommended.
      
2.2 Study Challenges
      This study offered multiple challenges. Although the evaluation plan was sub-divided into many different areas, this project was cross-cutting: it explored all areas of the evaluation matrix. This led to a broad but moderately deep exposure to all areas of the NFIP program.
      
      The study also intersected with both existing performance measures and the on-going evaluation in ways that were confusing to the respondents. In the document "Design for the Evaluation of the National Flood Insurance Program", January 2002, the evaluation logic model (See Appendix A) provides a matrix which lists the goals, objectives, inputs, activities, outputs, target population and outcome/impacts of the NFIP. The outcome/impacts column provides a total of 77 evaluation measures for the seven areas of the evaluation. Our task was to find performance measures which had not previously been identified and to focus on evaluation measures that should be used on a recurring basis.
      
      Three problems made data collection challenging. First, outside of the studies being conducted for the evaluation, the evaluation measures used by the agency are buried in an uncatalogued diversity of documents, some of which we surely failed to find. Second, most respondents had been interviewed multiple times as part of the NFIP evaluation process. Perhaps because they had been thinking about the broader NFIP evaluation, they often posed larger questions and raised issues that would best be handled as an additional one-time sub-study rather than through periodic measurement. For example, many people noted that data were needed on market penetration. Respondents all noted that the number of structures in a given flood zone is an important part of a penetration calculation, but this information is not readily available. This issue is studied as part of the NFIP evaluation in a sophisticated study that combines data from the biennial survey, property tax rolls (parcel by parcel), flood zone maps matched to addresses, census data, and BureauNet data (see Dixon et al., 2006). The respondents did not mention these potential sources of information on the issue, which is not surprising since much time and effort were necessary to match them up on a one-time basis. The present study asked about a proxy for estimating market penetration could be used on an on-going basis. Unfortunately, none was suggested other than routinely updating and expanding the analysis undertaken during the evaluation. Third, the initial project design proved too wide-reaching - reviewing hundreds of reports in the hopes of finding performance measures was not feasible within the project budget. After discussions with AIR and FEMA, we narrowed our search to information from our interviews plus documents. These three challenges forced us to rely more heavily than we anticipated on our own research design skills in order to produce a recommended set of measures that were feasible to collect and covered the broad range of relevant topics.
      
2.3 Comments on NFIP Data Collection and Analysis
      The interviews revealed that the NFIP is lacking some of the data that it needs to monitor performance. The interviewees pointed out that budget and burden considerations mean FEMA cannot comprehensively monitor all activities and outcomes in all areas of the program; some important measures cannot be affordably collected on a routine basis; some potential sources of data have high error rates; some tools are outdated; and the NFIP's data tracking technology is in transition.
      One of the most frequently mentioned needs was the desire for more accurate and recent maps. Accurate mapping is important for risk identification, reducing the number of complaints about maps, and improving the ability to estimate market penetration. A major mapping update project is progressing, and respondents looked forward to better data, but also with the knowledge that map updates would cause their own problems, since structures change ratings as maps change.

3 SUGGESTED PERFORMANCE MEASURES BY TOPIC
      Here we present a summary of interviews with fifteen key NFIP staff or steering committee members, as well as informal discussions with six additional AIR team members and our July 2005 survey of the AIR evaluation team. The open-ended interviews encouraged respondents to suggest performance measures. Because the respondents had varied jobs and responsibilities within the NFIP, we encouraged them to speak specifically about their own areas of expertise. 
      
      We supplemented the interview findings with measures gleaned from the design for the broader NFIP evaluation, selected NFIP publications, and our own observations. The comments are grouped into the six areas of inquiry used in the national evaluation's design. We used these as an organizational tool to present the interview and other findings, grouping responses into categories. The six areas are:
(1) Occupancy and use of floodplains
(2) Costs and consequences of flooding
(3) Insurance rating and indemnity functions
(4) Floodplain management and enforcement
(5) Hazard identification and risk assessment (mapping)
(6) Communications and marketing.

A closing section arrays the recommended performance measures by NFIP program goal.
      
      Suggested performance measures are introduced in the text then, displayed in a grid format. Each grid lists a measure group number and title, the assessment objectives, the recommended and current use of the measure, the measures, data sources and frequency of reporting data, the level of analysis, (indicating the geographic areas for analysis), cost (classified as low, moderate or high), quality/limitations/issues, and internal only/external (which indicates whether the data is for use outside of the agency, e.g., by flood plain managers or Congress, or solely for internal use by NFIP staff). 
      
3.1 Category 1: The Occupancy and Use of Floodplains
      The nature of occupancy and proper use of floodplains is one of the central, but most divergent issues that underlie the NFIP and its goals. These goals are not clearly understood by all stakeholders involved in NFIP activities. Some respondents indicated that the main goal of the NFIP is reducing flood damage by discouraging building in flood prone areas. Others felt that the most important goal was not discouraging new housing in flood zones, but establishing and assuring compliance with floodplain building codes and ordinances designed to prevent damage in a 1 percent probability flood. Others thought that the main NFIP goal was reducing taxpayer liability for flood damage in flood-prone areas. Many respondents pointed out that the stated goals in this area were unclear. The GPRA Blueprint of FY 2000-2006 indeed lists as an objective to "More clearly define the role of insurance in mitigation."  
      
      In fact, the primary stated purposes of the 1968 Act creating the NFIP were to "Through insurance, better indemnify individuals for flood losses that created personal hardships and economic distress; reduce future flood damages through State and community floodplain management regulations; and reduce Federal expenditures for disaster assistance and flood control" (FEMA 2002). As discussed in greater depth in another Evaluation report, Design for the Evaluation of the National Flood Insurance Program, to provide standardization across all components of the Evaluation, FEMA and the Evaluation team came to a consensus opinion on the goals of the NFIP, agreeing that the following four goals be used as the basis for evaluation of the NFIP:
      1)	Decrease the risk of flood losses;
      2)	Reduce the costs and adverse consequences of flooding;
      3)	Reduce demands and expectations for Federal disaster assistance after floods; and
      4)	Restore and preserve the natural and beneficial values of floodplains.

3.1.1 Issues and Measures
      Based on the interview data and our observations during the evaluation, the following measurement activities are desirable:
      
* Continuing collection of biennial survey data on the number of structures in SFHAs for each community participating in the NFIP, coupled with analysis of the trend in structure census over time by community characteristics. Concerns about biennial survey accuracy suggested the need to strengthen non-respondent follow-up, periodically validate a sample of responses, and provide better feedback to communities that allows them to see how biennial survey data are used to increase their stake in survey accuracy. The recommended feedback is described more fully in Area 3, insurance rating and indemnity functions
* Adding information about the percentage of structures in the community that are post-FIRM to the biennial survey or using data on the percentage of insured structures that are post-FIRM as a proxy.
* Decennial breakdown of biennial survey data examining type of structures in the flood plain (residences, businesses, other) and how they change over time. This effort would draw primarily on Census data and be conducted for regionally representative samples of coastal, riverine, and other floodprone communities nationwide. The decennial residential census data already loaded into HAZUS may be helpful in the estimation process, coupled with data extraction from property tax rolls. These estimates are needed to understand how many properties in SFHAs are eligible for NFIP coverage and what percentage has obtained it. They also will support monitoring of the change in the number of structures in SFHAs over time.
* Monitoring the benefits of floods in a sample of natural areas that were developed as green space or crop areas (using a case study approach).
* Estimating effectiveness of the NFIP in terms of lives saved. No one suggested a practical way to estimate this measure.
      
      Tables 1-2 (and similar tables in subsequent sections) provide details about the recommended performance measures. Each table includes:
      
* A title that briefly identifies its topic area
* An assessment objective, the purpose of providing measures on this topic
* An indication of whether the topic is a recommended data enhancement that will permit development of more accurate performance assessment and evaluation measures, a measure recommended for regular use as a core monitoring measure, or an exploratory measure that may not prove useful or currently is too costly to collect frequently, accompanied  by information on how extensively the measure has been used
* The relevant measure or measures recommended to fulfill the assessment objective
* Data sources for the measure(s) and a recommended frequency of collection; sometimes, notably in Table 1, this information is differentiated between measures
* The level (and nature) of the analysis, e.g., whether data should be collected by community, from a sample or all communities, and any statistical modeling required
* Cost of routine assessment, a loose classification into high, moderate, or low based on the amount of new data collection required, the cost to access any required secondary data sources, and the analytic effort required once the data are collected
* Quality, limitations, issues, which summarizes challenges and concerns about the recommended measures and data sources and sometimes records FEMA actions to improve data quality or accessibility
* Internal use only/external, which indicates which measures incorporate confidential information and therefore should not be publicly available

TABLE 1. Number of Structures at Risk of Flooding
Assessment Objectives	Provide a program exposure measure
Data Enhancement	
Measures	A) Number of structures in SFHAs by community
B) Structure breakdown by type (single family residential, multi-family residential, commercial, other)
C) % of post-FIRM structures
Data Sources and Frequency	A) Existing biennial community survey
B) Decennial census, possibly using HAZUS
C) Enhanced biennial community survey or the NFIP policy data loaded in CIS
Level of Analysis	By community and aggregated to state and national levels. 
Cost	Moderate for measure A, large for B, moderate to large for biennial survey portion of C, minimal for the CIS portion of C
Quality/Limitations/Issues	Data quality in the existing survey has proven to be mixed, with some communities providing accurate counts, others merely making a rough guess, and a third group not reporting. FEMA should develop a statistically sound method to adjust the estimates for non-reporting. FEMA also needs to incentivize accurate reporting, perhaps by sending useful community-specific analyses back to communities based on the exposure data they provide
Internal Use Only/External	External


TABLE 2. Environmental Impact of Designating Flood-Prone Areas as Green Space 
Assessment Objectives	Assess the benefits of floods in natural areas that were developed as green space or crop areas.
Exploratory Analysis	
Measures	Still remain to be determined 
Data Sources and Frequency	On-going case studies, possibly performed by local academics
Level of Analysis	Modest sample of areas
Cost	Moderate
Quality/Limitations/Issues	The benefits in areas designated for protection may differ from benefits that would result from designating other lightly developed flood-prone areas
Internal Use Only/External	External

3.2 The Costs and Consequences of Flooding
      The interview respondents offered virtually no input on this topic area. In our view, the key issues in this area relate to the financial impacts of flood insurance and NFIP-induced mitigation. Those impacts, in turn, shape the NFIP's impact on development in Special Flood Hazard Areas (SFHAs) and in floodplains generally. 
      
3.2.1 Issues and Measures
      In Sarmiento and Miller (2006), the NFIP Evaluation applied the HAZUS simulation model to assess the annual cost savings resulting from flood insurance and related mitigation efforts. The model also assessed savings to government and the impact on development costs in the SFHA. Table 3 suggests continuing to exercise the HAZUS model. When the 2010 Census is released, its information on structures and selected demographics should be loaded into HAZUS, along with other updates, and Sarmiento and Miller's analysis should be updated. 
      We recommend experimenting with a further set of HAZUS runs that may be appropriate to run annually. These runs would cover all floods that were declared as disasters, plus any other floods above a selected size or damage threshold that were not confined to sparsely populated areas. An initial simulation would produce a damage estimate that should resemble actual damages, allowing verification that the model reproduced reality reasonably well. (Estimate quality may be much better across the portfolio of the year's floods than it is for any individual flood.) Additional simulations should be run to estimate what damages and government costs would have been if no NFIP-induced mitigation had occurred. This assessment would inform decision-makers, the press, and the public of the important contribution that NFIP mitigation makes in controlling flood damage and place NFIP mitigation expenditures and annual claims payments in perspective.
      One measure reported to Congress annually is the annual dollar reduction in flood damages resulting from NFIP mitigation requirements and mitigation efforts (Table 4). This measure is calculated as the difference in mean damages to Pre vs. Post FIRM buildings times the number of compliant Post-FIRM buildings  Those who calculate it feel that it is a conservative estimate. HAZUS simulations conducted as part of the national evaluation (Sarmiento and Miller 2006) support that view and provide a confirmatory independent validation of the dollar reduction. This measure clearly should continue to be used.

TABLE 3. NFIP's Impact on the Costs and Consequences of Flooding
Assessment Objectives	Determine what cost savings result from flood insurance and related mitigation efforts. Estimate how NFIP coverage affects the distribution of payers for flood losses (e.g. taxpayers, homeowners, NFIP).
Core Analysis	Successfully collected in the national evaluation
Measures	Flood damages and NFIP-induced cost savings in SFHAs
Data Sources and Frequency	HAZUS simulation modeling supplemented by NFIP and SBA data on flood compensation. Decennial update of 2006 study as new residential Census data become available
Level of Analysis	For a representative sample of affected floodplains and for the U.S. in aggregate
Cost	High
Quality/Limitations/Issues	Lengthy computer time needed to calculate results may limit the number of flood plains that practically can be incorporated into the analysis, reducing representativeness
Internal Use Only/External	External
      
TABLE 4. NFIP's Impact on Annual Flood Losses
Assessment Objectives	Determine the reduction in annual flood losses that results from NFIP mitigation efforts
Core Analysis	GPRA performance measure currently in use
Measures	Difference in average damages per pre-FIRM and post-FIRM building times the number of compliant Post-FIRM buildings
Data Sources and Frequency	Annual, based on FEMA data.
Level of Analysis	National
Cost	Low
Quality/Limitations/Issues	HAZUS simulations would be costly, but could provide estimates for individual floodplains experiencing major floods in the past year and a more refined national estimate
Internal Use Only/External	External, reported to Congress
      
      A related measure of interest, shown in Table 5, is the impact of the NFIP on Federal costs due to flooding. A practical way to compute this measure is to exercise the HAZUS flood loss simulation model.
      
TABLE 5. NFIP's Impact on Federal Flood Costs
Assessment Objectives	Determine the annual reduction in Federal spending that results from NFIP coverage and associated mitigation efforts
Core Analysis	Analyzed using HAZUS in the national evaluation
Measures	Difference in Federal expenditure per pre-FIRM versus post-FIRM building in a sample of declared flood disaster areas times estimated number of post-FIRM  structures involved in the relevant floods 
Data Sources and Frequency	Annual, based on FEMA and SBA data.
Level of Analysis	National
Cost	Medium
Quality/Limitations/Issues	HAZUS simulations could provide estimates for floodplains experiencing major floods in the past year and a more refined national estimate. A lower cost estimate would be possible if the government started collecting Pre- or Post-FIRM status on buildings receiving disaster assistance following floods.
Internal Use Only/External	External, reported to Congress
      
      After-the-fact qualitative case-study accounts of the effects of the NFIP on property losses during flood disasters have almost never been collected in a form designed for public consumption. That means the NFIP fails to capture positive publicity and create greater awareness of the value of mitigation and of flood insurance coverage. Each year, as Table 6 details, we recommend that the NFIP write three-page case studies of three to nine floods for a lay audience. These case studies should describe the floodplain affected, the flood, what the losses were, what losses were avoided through mitigation, how much better residents with flood insurance fared, and when relevant, how pre-FIRM structures and Community Rating System (CRS) efforts affected the losses. If the case studies are informed by NFIP, SBA and National Weather Service data, HAZUS runs, and information from FEMA's flood program coordinators for the state/region, they will not require OMB clearance. We believe these case studies will prove valuable in community and legislative education and create positive publicity around the value of buying flood insurance.

TABLE 6. Detailed Benefits of the NFIP 
Assessment Objectives	For specific, generally large floods, determine what were the losses, what losses were avoided through mitigation, how much better did residents with flood insurance fare, and how pre-FIRM structures and CRS efforts affected the losses.
Core Analysis	Never has been collected
Measures	Two-page case studies that summarize flood loss, payout for those with flood insurance, uncompensated costs
Data Sources and Frequency	Three to nine annually, depending on how many large floods occurred. HAZUS simulations supplemented by NFIP, SBA, and National Weather Service data, and by interviews with FEMA personnel involved in handling FEMA's response to or paying claims from the flood. Retrospective HAZUS simulations of damages might be desirable as well or might provide an adequate basis for the case studies if calibrated with NFIP loss experience in the flood event
Level of Analysis	Community or floodplain
Cost	Moderate
Quality/Limitations/Issues	The case studies will not be representative
Internal Use Only/External	External

3.3 Insurance Rating and Indemnity Functions
      Insurance rating and indemnity functions examine the activities of the NFIP as an insurer. This area is broad and far-reaching. It encompasses the business aspects of the program, from risk assessment to underwriting to loss adjustment, from financial soundness and administrative efficiency to subsidy and repetitive loss control.
      
3.3.1 Issues and Measures
      The assessment needs respondents mentioned related to insurance rating and indemnity functions include: 
* Repetitive loss properties and subsidies (location of repetitive loss properties, number of times flooded, costs, information on the subsidies [those built before 1975 or the date of the initial FIRM for the community, whichever is later]
* Distribution of loss mitigation efforts for repetitive loss properties (buy-outs, razing structures, etc.)
* Insurance ratings (accuracy, fairness)
* Insurance administration and customer satisfaction
* Claim processing time/cost/number of errors
* Income to expense ratio and 
* Profit or deficit (recognizing that the NFIP is intended to break even)
      For this category we combined the measures suggested by respondents with our analysis of the Annual Actuarial Rate Review data to propose the measures listed in Tables 7-8.
      
TABLE 7. Continue and Marginally Expand the Bureau Net Data Base on Insurance-in-Force and the Associated National Actuarial Rate Report
Assessment Objectives	Maintain and report data about individual insured structures 
Data Enhancement	
Measures	Existing Bureau Net data and actuarial report, with two other items that are known to the selling agent collected about each property and reported in aggregate, namely
(1) Value of insured structure
(2) Whether a mortgage exists
Data Sources and Frequency	WYO companies, NFIP direct contractor
Level of Analysis	National, regional, community
Cost	High (largely already incurred)
Quality/Limitations/Issues	The two added data fields capture data that responsible selling agents have on hand
Internal Use Only/External	BureauNet: internal, external on approved request in non-identified form
Annual actuarial rate report: external

TABLE 8. Insurer Compliance
Assessment Objectives	Monitor how quickly and accurately policies are sold and claims are processed
Core Analysis
Exploratory Analysis	Claims processing time has been a GPRA measure in some years
All other information currently is collected but is not aggregated and analyzed
Measures	Application processing time
Claims processing time
% of policies correctly rated
Nature and frequency of policy errors
Data Sources and Frequency	Biennial WYO Compliance Audits, NFIP Operations Review (every 2-3 years)
NFIP records
Level of Analysis	By WYO company
Cost	Moderate, since the data exist already
Quality/Limitations/Issues	Accuracy of policy rating will be based on sample data
Internal Use Only/External	Internal

      A long-standing NFIP problem is that local building inspectors rarely make substantial damage declarations when issuing permits for the repair of flood-damaged structures below BFE, meaning they miss the opportunity to force buildings into compliance and invite repetitive loss problems. Respondents suggested that substantial damage declarations, although still relatively political, might increase if local permit officials got immediate property-specific data on claims costs. The NFIP's heavy reliance on WYO companies' claims adjusters, however, means that damage information is not centralized quickly, which poses a formidable hurdle to providing it locally in a timely manner. Table 9 describes these information needs and the broader information needs around repetitive loss properties.

TABLE 9. Repetitive Loss and Substantial Damage
Assessment Objectives	To document NFIP losses on repetitive loss structures
To provide timely information that prompts an increase in the number of flood-damaged properties that local governments declare have suffered substantial damage and therefore must be elevated to BFE if they are repaired
Core Analysis	The first three measures are collected and occasionally have been analyzed
Data collection for the final two measures might be too costly and burdensome
Measures	% of repetitive loss claims & claims dollars by zone & whether Pre-FIRM
% of repetitive loss claims that generate substantial damage declarations 
Trends per repetitive loss property in claim size & in time between claims 
Distribution of days between claim filing and provision of insurance loss data to the local floodplain manager
Properties with substantial damage declarations by days to notification of damage level to the floodplain manager and level of damage relative to policy limits
Data Sources and Frequency	Actuarial Info System
Level of Analysis	National, Statewide, and Community wide
Cost	Moderate, since the data exist already
Quality/Limitations/Issues	This information is politically sensitive. Flood claim data are held by multiple WYO companies across the U.S. and do not reach the NFIP quickly. Current contact information on floodplain managers often is lacking. Privacy issues also discourage release of information on personally identified flood claims. A demonstration effort might be desirable to determine if timely claims cost information prompts substantial damage declarations.
Internal Use Only/External	Internal
      
      Additional performance measures are needed that relate to market penetration. Tables 10-13 detail the needs in this area. They suggest tracking several market penetration measures over time - biennial estimates of the number of structures in SFHAs (see Table 1), the number of compliant and non-compliant structures, the number of structures that are insured by compliance category, the policy renewal rate, and the reasons that policies are not renewed. Similar data are desirable about structures in floodprone areas outside SFHAs but are less practical to collect. 
      
TABLE 10. NFIP Penetration: A Basic Set of Research Measures to Collect
Assessment Objectives	Track NFIP penetration in flood-prone areas. Monitor policy turnover.
Core Analysis

Exploratory Analysis	The first measure currently is only feasible to collect accurately through an occasional, costly sampling using the methods from the national evaluation
The other measures have not been collected and may not prove interesting enough to collect more than once
Measures	Policies per 1000 structures in SFHAs
Policies dropped by years since coverage commenced
Ratio of new policies to discontinued policies
Data Sources and Frequency	Number of policies sold and dropped are available from the actuarial information system. 
Estimated number of structures is available from the Biennial Survey and the decennial Census 
Level of Analysis	National and State-wide. Because all communities do not provide data on number of structures, some weighting will be necessary to obtain national and state rates. 
Cost	The cost is modest, as the data are already collected at the community level. 
Quality/Limitations/Issues	The Biennial Survey data on number of structures available are limited to communities which respond to the survey. The Biennial Survey data are not always accurate as they rely on multiple communities to record and submit data. Also, survey data do not distinguish businesses from multi-family residential structures or from industrial, public, or farm buildings (all are lumped together as "other"). 
Internal Use Only/External	External

Perhaps every five years, FEMA also should run a regression analysis that uses the community market penetration data to determine which factors best predict high penetration among factors such as proximity to an ocean, feet above sea-level, the year maps were updated, etc., plus data from a periodic survey of a sample of property owners. This effort will build a body of knowledge about how to increase penetration and detect shifts in the marketplace over time.

TABLE 11. NFIP Penetration: An Expensive, Ideal Set of Research Measures to Collect
Assessment Objectives	Understand how penetration varies spatially and how premium subsidy influences it
Data Enhancement	Long-term objective, probably not affordable presently
Measures	% of households buying policies
Pre-FIRM properties by zone
Other properties in SFHAs by zone
Other properties in mapped 500-year flood zones
Properties not in mapped 500-year flood zones
Data Sources and Frequency	Linked map determination company and tax assessment data, plus geocoded policy data, every 5 years
Level of Analysis	National probability sample
Cost	High
Quality/Limitations/Issues	Affordable sample size will restrict representativeness. Data by state are desirable but prohibitively expensive. Digitized FIRMS and other technological advances may reduce future data cost.
Internal Use Only/External	External

TABLE 12. Reasons for Non-Renewal
Assessment Objectives	Gain insight into ways to increase policyholder retention and into failure to maintain mandated coverage
Data Enhancement; Exploratory Analysis	Has not been collected
Measures	Reasons for discontinuation. Mortgage status.
Data Sources and Frequency	Survey of a sample of policyholders who discontinued coverage, every third year
Level of Analysis	National
Cost	High
Quality/Limitations/Issues	Will require OMB clearance.
Internal Use Only/External	Largely internal

TABLE 13. Predictors of NFIP Penetration
Assessment Objectives	Learn how to increase penetration. Monitor trends 
Exploratory Analysis	Has not been conducted
Measures	Community and property factors that significantly predict penetration (isolated by regression analysis)
Data Sources and Frequency	Measures 11 and 12 plus community census data
Level of Analysis	National, coastal vs riverine
Cost	High
Quality/Limitations/Issues	The limitations of the input data apply. Correlation does not guarantee a causal relationship.
Internal Use Only/External	External

3.4 Floodplain Management and Enforcement
      The goals listed in the Design for the Evaluation of the National Flood Insurance Program under "Floodplain Management Assistance, Monitoring and Enforcement" are to:
1. Ensure that communities properly implement their floodplain management ordinances in compliance with NFIP requirements; 
2. Avoid the risks that are present in floodplains;
3. Minimize the impacts of flood risks when they cannot be avoided; and 
4. Mitigate the impacts of damages when they occur.
      
      Floodplain management is not done directly through the NFIP but through states and communities. As a pre-requisite for making flood insurance available to residents of a community that contains a SFHA, FEMA requires that the community have a floodplain management ordinance stating that new or substantially damaged or improved residential structures in the SFHA must be elevated to or above base flood elevation (BFE) and that non residential structures be elevated or floodproofed to the BFE. The community also must have a permitting and inspection system that implements the mitigation ordinance.
      
      Community compliance with flood regulations was mentioned by many of the respondents. Two studies were conducted during the evaluation on the extent to which communities were meeting their compliance requirements. One study deals with the adequacy of community compliance processes (Monday et al., 2006) and the second whether minimum building requirements are being met (Mathis and Nicholson, 2006). Data sources for the first study included the Community Information System (CIS) containing results of the CACs/CAVs (about 85% are computerized), paper files on uncomputerized CACs/CAVs, FEMA policy guidance, and interviews with regional office staff and managers. The second study used data collected from FIRMS and FISs, National Spatial Reference System, Policy in Force Database, local elevation certificates, flood proofing information, records of variance, and a physical survey of 125 sample structures in ten geographic clusters. One respondent suggested monitoring the compliance rate over time as a measure of success. As discussed in Monday et al., (2006), however, FEMA and the NFIP must remedy completeness and accuracy problems with CIS data collection before this measure will be sufficiently accurate to provide useful information. See Monday et al., (2006) for further suggestions regarding collection of further compliance data. 
      
      The NFIP has various incentive programs to encourage additional floodplain management. Of relevance is the Community Rating System (CRS), a voluntary incentive program that encourages community floodplain management activities that exceed the minimum NFIP requirements. Through the CRS, flood insurance premium rates are discounted in compliant communities. The discounts are intended to reflect the reduced flood risk resulting from community actions which meet three goals of the CRS: (1) reduce flood losses; (2) facilitate accurate insurance rating; and (3) promote the awareness of flood insurance. Over 1000 communities are in the program representing over 66% of the NFIP's policy base. As a GPRA measure, FEMA tracks one CRS incentive item, the percentage of the national population that is covered by approved local mitigation plans.
      
      As reported in the Community Rating System: Effectiveness and Other Issues, FEMA's Inspector General examined the effectiveness of CRS as a tool to improve local policies and practices related to floodplain management. Overall, the Inspector General's report finds that CRS is a disciplined and well-defined program in terms of its guidelines, requirements, and rating processes and procedures. However, the report states that FEMA could enhance the effectiveness of CRS by: (1) performing Community Assistance Visits (CAVs) in all CRS communities, (2) marketing CRS to communities having greater exposure to the NFIP, (3) providing credit for increasing flood insurance coverage in a community, and (4) providing CRS coordinators with access to claims data.
       
3.4.1 Issues and Measures
      Tables 14-21 detail the measures recommended in this area. The issues they address include:
      
* The NFIP - CIS (Community Information System) should be improved. The CIS provides information on the Community Rating System, floodplain management, CACs/CAVs, map status, letter of map changes, insurance in force, and claims experience, as well as community demographics. Although only 85 percent of CAVs are entered into the computer, these data can be analyzed. (Recent changes that encourage States to enter most CAV data directly into the computer are expected to reduce missing data.) Our evaluation staff also recommended using CIS to record all contacts and visits (not just CACs and CAVs) between communities and FEMA or state floodplain management staff. 
* CIS reports are needed on the proportion of communities in a region that were evaluated using CACs/CAVs, on problems detected, and on time to resolution of violations detected during CAVs and time since detection for unresolved violations for all NFIP communities. 
* Basic management data about community staffing are missing. Recommendations include requiring communities to provide contact information for their floodplain manager, and collecting information about who received Emergency Management Institute and similar training and who is a certified floodplain manager.
* Compliance with the basic NFIP regulations and compliance with CRS incentive items that the community has put in place should continue to be monitored by community and should be reported across communities within each state.
* Building standards need to be evaluated as changes are implemented and compliance data improve. Are buildings actually safer when built to specifications? FEMA operates Mitigation Assessment Teams (MAT Teams) that evaluate and report on the performance of buildings after disasters. A tremendous amount has been learned from these reports and they have had a major influence on building practices. These investigation data are far more useful than general claims reports which provide little or no information on why a building did not perform other than that it flooded.
* A supplemental way to address this is to look at loss history and insurance history of losses on structures that comply with NFIP mitigation regulations. Unfortunately, such an investigation undertaken as part of the national evaluation (Jones et al., 2006) was severely constrained by requested, but not mandated data that are missing in most claims adjustment reports. To remedy this data problem, FEMA needs to find means and incentives for time-pressed claims adjusters in the field to accurately record flood zone, BFE, building value, and flood conditions including flood depth relative to ground and relative to lowest floor. The analysis of standards merits update as better data become available and as standards evolve.
* Elevation rating accuracy warrants monitoring: How accurate were the determinations? Who made them? What proportion of elevations were correct to within 6 inches? What situations/locations/companies account for the highest error rates? Although costly, spot monitoring will help to assure new buildings built on floodplains are elevated and generally built to NFIP requirements. A sample of new buildings should be verified - using elevation certificates and field checks. The elevation study undertaken by Dewberry as part of the national evaluation (Mathis and Nicholson 2006) provides a model for these assessments. 
* The use, processing, and outcomes of submit-to-rate requests for properties that received variances permitting non-compliance with the community's flood protection code and standards provide insight into compliance with floodplain management requirements.
* 
TABLE 14. Mitigation Planning
Assessment Objectives	Monitor the extent of mitigation planning
Core Measure	GRPA Measure
Measures	Percentage of the national population that is covered by approved local mitigation plans
Data Sources and Frequency	Community Rating System, annually
FEMA approved Pre-Disaster Mitigation Plans
Level of Analysis	Region, State
Cost	Low
Quality/Limitations/Issues	Ignores difference in plan quality (beyond meeting the minimum requirements for approval) and implementation/adherence
Internal Use Only/External	External

TABLE 15. Community Information System (CIS)
Assessment Objectives	Maintain basic data on status, problems, and contacts as well as the results of CAVs/CACs by community 
Data Enhancement	
Measures	CAV/CAC Results:
Problems with the community's floodplain management regulations 
Problems with the community administration and enforcement procedures Engineering or other problems with the maps or Flood Insurance Study 
Other problems in the community's floodplain management program
Problems with the Biennial report data 
Programmatic issue or problems identified
Potential violations of the community floodplain management regulations
Remedial actions been taken 
Notes: (long hand notes from CAV)
Follow-up information

Add CIS fields to record informal contacts (other than CACs and CAVs) between communities and FEMA or state floodplain management staff
Data Sources and Frequency	CIS, annual summary
Level of Analysis	National, regional, community
Cost	Moderate
Quality/Limitations/Issues	Enlisting regional offices in processing their paper entries could encourage computer entry of future data. Currently, 15% of CACs and CAVs are not entered, with the missing data problems concentrated in a small number of states and regional offices. 
Internal Use Only/External	Primarily internal


TABLE 16. CAC and CAV Activity and Outcomes
Assessment Objectives	Monitor CAC/CAV coverage and progress in resolving problems identified
Data Enhancement	
Measures	Percentage of communities and of insured structures evaluated using CACs/CAVs 
Types of communities most likely to get evaluated
Problems detected
Time to resolution of violations detected during CAVs
Time since identification for unresolved violations. 
Data Sources and Frequency	CIS, annual, save a quarterly list of open violations
Level of Analysis	Region, State
Cost	Large currently due to information that has not been entered into CIS; low once CIS data entry requirements are enforced
Quality/Limitations/Issues	Enlisting regional offices in processing their paper entries could encourage computer entry of future data
Internal Use Only/External	Internal

TABLE 17. Floodplain Management Staff
Assessment Objectives	Maintain information on community floodplain managers and their qualifications
Data Enhancement	
Measures	Floodplain manager contact information
% time devoted to flood plain management
Whether certified as a flood plain manager
Attendance at emergency management training
Data Sources and Frequency	Add contact information to Biennial Survey
Link managers' names to FEMA Emergency Management Institute training records annually
In their annual report on CAP SSSE activities, require states to provide the names of people they train 
Link managers' names to certification records annually
Level of Analysis	National, State, Community
Cost	Moderate 
Quality/Limitations/Issues	Linking names will be challenging when flood plain managers move to different communities
Internal Use Only/External	External

TABLE 18. NFIP Compliance
Assessment Objectives	Monitor community compliance with the basic NFIP regulations and with CRS incentive items
Core Analysis	Community data currently are collected by ISO and entered in CIS, but they are not currently aggregated and reported by state
Measures	Communities with compliance problems
Variation across states in community compliance with NFIP requirements including compliance with CRS incentive items that communities are credited with having
States that are performing the best in terms of floodplain management
States achieving the highest increases in compliance and floodplain management 
Data Sources and Frequency	CIS, biennially; ISO community visits
Level of Analysis	State
Cost	Moderate
Quality/Limitations/Issues	Affordability may dictate restricting the assessment to communities with automated CIS records or recent ISO visits
Internal Use Only/External	Internal, including state coordinators


TABLE 19. Evaluation of Building Standards
Assessment Objectives	To determine if building standards actually improve building performance in floods
Exploratory Analysis	Analyzed in the national evaluation
Measures	Loss history and insurance history of losses of structures built to different standards
Investigation by Mitigation Assessment Teams (MAT Teams).
Data Sources and Frequency	Team assessments as construction issues arise or standards and practices change. Regression analyses as data availability, data quality improvements, and revised standards warrant.
Level of Analysis	Assessment of narrow issues; national regressions
Cost	Moderate
Quality/Limitations/Issues	The national evaluation revealed such a statistical investigation currently is crippled by requested, but not mandated data that are missing in most claims adjustment reports. To remedy this, FEMA needs to find means and incentives for time-pressed claims adjusters in the field to accurately record flood zone, BFE, building value, and flood conditions.
Internal Use Only/External	External

TABLE 20. Elevation Data Accuracy 
Assessment Objectives	Monitor elevation accuracy
Exploratory Analysis	Analyzed in the national evaluation; may not be a large enough problem to warrant continued measurement
Measures	How accurate is the determination? Who made the determination? What proportion of elevations are correct to within 6 inches? What situations/locations/companies account for the highest error rates? 
Data Sources and Frequency	A sample of new buildings should be audited - using elevation certificates and field checks. Every 5-10 years.
Level of Analysis	State, community, survey company
Cost	Very high
Quality/Limitations/Issues	Affordability constrains validation efforts to a small number of communities
Internal Use Only/External	Internal

TABLE 21. Submit to Rate Requests and their Processing and Outcomes 
Assessment Objectives	Monitor the frequency of submit to rate requests for non-compliant structures (which indicate that construction was permitted that did not comply with the flood protection provisions of the community building code and ordinance) and the rate surcharges. Detect communities that are lax in code enforcement and other potential problems with submit-to-rate policies.
Core Analysis	Has not been analyzed previously
Measures	Number of submit-to-rate requests
Communities with a high percentage of new policies that are submit-to-rate
Mean rate surcharge
Non-continuation rate and loss experience for submit-to-rate properties versus other properties in comparable communities
Data Sources and Frequency	NFIP records
Level of Analysis	National, state, and community
Cost	Low
Quality/Limitations/Issues	Comparisons of claims experience ideally should be confined to the communities housing the submit-to-rate properties, but if these properties cluster, other communities might have to be included in order to get sufficient properties that comply with standards to support a comparison. Some submit-to-rates are for compliant buildings and will need to be separated out.
Internal Use Only/External	Primarily internal

3.5 Hazard Identification and Risk Assessment (Mapping)
      Because flood hazard conditions change over time due to natural and manmade changes in watersheds and floodplains, FEMA has an ongoing program to update flood maps for floodprone communities. However, flood map update needs are increasing and Federal funds are limited. Thus a significant portion of the approximately 100,000-panel flood map inventory is outdated and 60% are not digital (FEMA 2006). To reverse this trend, FEMA designed the Map Modernization Program to modernize the flood map inventory. The plan outlined the steps necessary to update FEMA's flood maps for the nation to digital format, provide updated flood information, and streamline FEMA's operations in raising public awareness of the importance of the maps and responding to requests to revise them. Since that time, the plan has continually evolved as new products, processes, and technical specifications have been developed and implemented. 
      
      FEMA's Map Modernization Program plans to produce digital flood maps that cover 92% of the U.S, population and 65% of its land area by 2009, with 75% of the mapped stream miles meeting "the Floodplain Boundary Standard, meaning that the floodplain boundaries on the maps are drawn using the best available topographic data. This covers 80 percent of the population. Of the stream miles mapped, 30 percent will be based on new, updated, or validated engineering analysis, covering 40 percent of the population." 
      
      Respondents said that the remapping initiative will cause major changes. Updated maps will add homes to SFHAs and remove others and affect the number of communities with SFHAs. With increased development, the amount of area in floodplains is expected to increase. Changing the requirements for in-place properties will bring a new set of challenges, especially given that they will have grandfathered insurance rates. Thus, bringing these new homes under the NFIP umbrella will have financial implications and implications for the program's progress toward actuarial soundness. 
      
3.5.1 Issues and Measures
      The suggested performance measures would record progress toward the targeted coverage levels. Notably, the most readily understandable of these progress measures, the percentage of maps that have been digitized, is sensibly one of the NFIP GPRA measures. In addition to the progress measures, measures are needed to track and manage the cost of mapping and to document cost-sharing of mapping efforts. Tables 22-24 summarize the recommended measures.
      
      The number of complaints about a map also was suggested as a performance measure. Although complaints about a map might indicate errors, however, they instead could indicate that a map is being used effectively to deny inappropriate development. Therefore we rejected this suggestion.
      

TABLE 22. Efficiency in Mapping 
Assessment Objectives	Monitor and promote more cost-efficient spending to produce flood hazard boundary maps.
Core Analysis	Mean cost is analyzed currently but the distribution of costs and the time trend in costs are not
Measures	Mean, decile distribution, and standard deviation of cost to produce a new or updated map (separately for those with and without a digital data)
Data Sources and Frequency	FEMA records, annually
Level of Analysis	National, by mapping contractor and region
Cost	Low
Quality/Limitations/Issues	Variations in the size of the area mapped and the nature of the flood hazard may require further categorization.
Internal Use Only/External	Internal

TABLE 23. Map Cost Sharing
Assessment Objectives	Document the extent of cost sharing in the mapping effort
Core Analysis	Currently analyzed
Measures	Percentage of maps created/revised in partnership with a state or community, percentage of total mapping costs borne by partners, percentage of mapping costs for shared-cost maps borne by partners
Data Sources and Frequency	NFIP records, annually
Level of Analysis	National and regional
Cost	Low
Quality/Limitations/Issues	Trends must be interpreted in the context of state government financial conditions
Internal Use Only/External	External
      
TABLE 24. Map Update Progress
Assessment Objectives	Monitor unmet needs for maps and the rate that needs are being met
Core Analysis	The first two sets of measures currently are tracked
The remaining measures have not been analyzed
Measures	Percentage of maps digitized (a NFIP GRPA Measure)
Percentage of population and of stream-miles covered by digital maps
Age distribution of maps by rate of development
Time from designation as a remapping priority to completion of revised maps ready for public comment
Time after a 100-year or larger flood event with damage above a set dollar threshold until a map is spot-checked
Data Sources and Frequency	Map inventory, mapping backlog, biennial reports and decennial Census
Level of Analysis	National, regional
Cost	Moderate
Quality/Limitations/Issues	Assessment done by an uninvested third party is highly desirable
Internal Use Only/External	External

3.6 Communications and Marketing
Communications about flood risks and marketing of flood insurance have been studied extensively. Here, however, our focus is on measurements that should be repeated over time.

3.6.1 Issues and Measures
      As Tables 25-27 show, suggested measures related to communications and marketing include:
* Post-Buy Reports - Reports documenting purchased advertising are submitted by media coordinators. A template for these reports is needed to standardize the reporting. This will facilitate comparisons over time. Respondents complained that historically reports often were incomplete. 
* Gross Rating Points (GRP) - (Reach) times (Frequency). This figure could be gathered from the media coordinators. It would enable NFIP to compare various outreach efforts such as 15 second vs. 30 second ads. It could also be benchmarked against similar campaigns, both FEMA campaigns, and compared to insurance industry campaigns. This information is used mainly with paid media; when PSAs are used, the GRP information is generally not available. 
* Ad Cost per Gross Rating Point - the cost per advertisement per gross rating point provides a unit of comparison when considering alternatives in media purchases.
* Return on Marketing Investment: For every dollar spent on advertising, what is the increase in policies sold?
* Marketing methods - Routine reporting is needed tracking how Write Your Own companies are marketing flood insurance.
* Demographic Models of Purchasers - Cover America identified three groups of voluntary flood insurance purchasers labeled "financial planners", "extra protectors" and "home improvers. FEMA needs to periodically update its demographic model of these purchasers to detect changes in the mix of non-mandatory policy holders and adjust its marketing accordingly.
* Public Attitude Toward Flood Risk - what is known about flood coverage and its impacts, how vulnerable do people feel to flood loss, what arguments for flood insurance are most convincing, etc., and shifts in these attitudes over time. 

TABLE 25. Nature and Outcomes of Paid NFIP Advertising
Assessment Objectives	Track advertising purchased, exposure obtained, and cost-effectiveness 
Core Analysis	Has been conducted in conjunction with prior advertising campaigns
Measures	Standardized post-buy report including
* Market
* Gross rating points, reach, frequency 
* Ad cost per rating point
Return on investment (advertising dollars per new policy)
Data Sources and Frequency	Quarterly, when advertising campaigns are active
Level of Analysis	Nationally, by media market
Cost	High
Quality/Limitations/Issues	Policy sales following a media buy must be counted carefully as policies on new homes issue immediately while those on existing homes have a 30 day waiting period
Internal Use Only/External	Primarily internal

TABLE 26. Marketing by Write-Your-Own Companies
Assessment Objectives	Track WYO company efforts to market NFIP policies
Exploratory Data Collection	
Measures	Type of media, market, frequency, reach, message
Data Sources and Frequency	WYO company reports to the NFIP, annually
Level of Analysis	Across companies by market & nationally
Cost	Moderate
Quality/Limitations/Issues	WYO companies may view this information as confidential. They also may resist a reporting requirement as burdensome.
Internal Use Only/External	Strictly internal

TABLE 27. Public and Purchaser Attitudes
Assessment Objectives	Monitor markets for non-mandatory flood insurance
Exploratory Analysis	Has not been collected
Measures	Who are the non-mandatory purchasers: basic tables plus statistical classification analysis
What is known about flood coverage and its impacts
How vulnerable do people feel to flood loss
What arguments for flood insurance are most convincing
Data Sources and Frequency	National probability sample survey, every 5 years, plus after selected major flood disasters
Level of Analysis	National, regional
Cost	High
Quality/Limitations/Issues	The impact of major flood disasters and its attenuation over time are both of interest
Internal Use Only/External	External

4 CLASSIFICATION OF THE MEASURES BY NFIP GOAL
      As noted earlier, the Evaluation of the NFIP identified four goals of the NFIP, namely:
1. [0]Decrease the risk of flood losses;
2. Reduce the costs and adverse consequences of flooding;
3. Reduce demands and expectations for federal disaster assistance after floods; and
4. Restore and preserve the natural and beneficial values of floodplains.
      
      Table 32 maps the 27 groups of performance assessment and evaluation measures identified in prior chapters into the four goals. Eleven measures relate to decreasing flood risks, 15 measures relate to reducing the costs and consequences of flooding, and 18 measures to reducing demand and expectations for flood insurance. These three goals overlap. For example, reduced flood consequences will lessen the need for disaster assistance. Similarly, decreasing flood risk is likely to reduce the consequences when floods do occur. Our mapping tries to focus on the primary thrust of the performance assessment and evaluation measures, but our choices are somewhat arbitrary. Only one exploratory measure tentatively addresses the fourth goal.
      

Table 32. Mapping of the 27 Performance Assessment and Evaluation Measures Into the Four NFIP Goals
Performance Assessment and Evaluation Measure	Decrease Flood Risk	Reduce Costs & Consequences of Flooding	Reduce Demand & Expectations for Disaster Assistance	Restore & Preserve Natural & Beneficial Value of Floodplains
1. Number of Structures at Risk of Flooding	X			
2. Environmental Impact of Designating Flood-Prone Areas as Green Space or Crop Areas				X
3. NFIP's Impact on the Costs & Consequences of Flooding		X		
4. NFIP's Impact on Annual Flood Losses		X		
5. NFIP's Impact on Federal Flood Costs			X	
6. Detailed Benefits of the NFIP 		X	X	
7. Continue & Marginally Expand the Bureau Net Data Base on Insurance-in-Force & the Associated National Actuarial Rate Report			X	
8. Insurer Compliance	X	X	X	
9. Repetitive Loss & Substantial Damage	X	X	X	
10. NFIP Penetration: the Basics			X	
11. Details of NFIP Penetration: the Possibly Unaffordable Ideal 			X	
12. Reasons for Non-Renewal			X	
13. Predictors of NFIP Penetration			X	
14. Mitigation Planning	X	X		
15. Community Information System (CIS)	X	X	X	
16. CAC/CAV Activity & Outcomes	X	X	X	
17. Floodplain Management Staff	X	X	X	
18. NFIP & CRS Compliance	X	X		
19. Evaluation of Building Standards	X	X	X	
20. Elevation Data Accuracy 	X	X	X	
21. Submit to Rate Requests & their Processing & Outcomes	X		X	
22. Efficiency in Mapping 		X		
23. Map Cost Sharing		X		
24. Map Update Need & Progress		X		
25. Nature & Outcomes of Paid NFIP Advertising			X	
26. Marketing by Write-Your-Own Companies			X	
27. Public & Purchaser Attitudes			X	


5 CONCLUSION
      Creating a sound set of performance assessment and periodic evaluation measures requires more than just thinking through what measures are useful and what additions would be useful. It must be tempered by an understanding of what information can be collected accurately without undue expense. Furthermore, one cannot be sure a measure makes sense until it is collected once and the results are analyzed to confirm that it provides informative data that are useful in monitoring or decision-making. 
      
      We recommend retaining the NFIP's current GPRA measures. The highest priority additions to current measures are: 
* Enhanced tracking of the number and nature of properties in Special Flood Hazard Areas (SFHAs) and the NFIP penetration rate among those properties
* Better tracking of unresolved problems surfaced during CACs/CAVs
* Monitoring to detect communities with concentrations of submit-to-rate properties that are non-compliant with local building codes and standards
* Annually using the HAZUS simulation model to measure the reduction in the costs and consequences of floods during the past year that resulted from NFIP mitigation efforts
* Improving information on progress with flood hazard mapping 
      
      A number of data enhancements through new data collection would support cost-effective performance measure development and provide data that improve program management. These include:
* Requiring agents selling NFIP policies to record the same information about structure value and the existence of a mortgage that they would record in writing a homeowner's policy
* Promptly entering all Community Assistance Calls and Community Assistance Visits (CACs/CAVs) into the Community Information System
* Improving the flood depth and structure elevation data recorded by claims adjusters
* Improving and accelerating the flow of claims information to local officials charged with 
making substantial damage determinations
* Tracking of NFIP marketing by Write-Your-Own (WYO) companies
      
      Ultimately, the NFIP should be able to access an inventory of floodplain structures, which will allow many critical questions to be answered. An inventory could probably be developed by overlaying digital flood maps with a national database of geo coded structure locations. An appropriate commercial geo-coded structure database should become available at some point, and Census DIME files (structure listings) that will be used to manage the 2010 decennial Census conceivably could serve this purpose. Until digital files are developed, budget constraints will require using sampling methods to answer basic questions such as market penetration and the number of post-FIRM structures that were not damaged when their SFHAs flooded or relying on the aggregate community estimates of varying quality collected in the NFIP Biennial Report. The digitized geocoding approach should be the ultimate goal of the NFIP and would allow for a range of performance measures that are beyond current technologies. 
      
      The NFIP policy and claims data are one of the most powerful tools the NFIP has to evaluate flood losses and floodplain development. The problem in the past has been lack of ground truth. FEMA did not know for sure what percentage of all structures are in the policy base and whether these insured structures have the same characteristics as those that are not insured. Assuming the market penetration information from the RAND study (Dixon et al. 2006) and some of the compliance data from the Dewberry study (Mathis and Nicholson 2006) conducted as parts of the national evaluation are correct, those studies may provide a basis for further mining the insurance data to generate information on the nation's flood risk and the NFIP's performance and impact.
      

6 REFERENCES
American Institutes for Research. (2002) "Design for the Evaluation of the National Flood Insurance Program", Washington, DC: American Institutes for Research.

Dixon L, Clancy N, Seabury S, and Overton A. (2006) "The National Flood Insurance Program's Market Penetration Rate: Estimates and Policy Implications.", Washington, DC: American Institutes for Research.

Federal Emergency Management Agency. (2006) "Flood Map Modernization: A Powerful Tool for Risk Management." Washington, DC: Federal Emergency Management Agency (FEMA).

Federal Emergency Management Agency. (2002) "NFIP Description." Washington, DC: Federal Emergency Management Agency (FEMA).

Mathis M and Nicholson S. (2006) "An Evaluation of Compliance with the National Flood Insurance Program Part B: Are Minimum Building Requirements Being Met?", Washington, DC: American Institutes for Research.

Monday J, Grill K, Esformes P, Eng M, Kinney T, and Shapiro M. (2006) "An Evaluation of Compliance with the National Flood Insurance Program Part A: Achieving Community Compliance.", Washington, DC: American Institutes for Research. 

Sarmiento C and Miller T. (2006) "Costs and Consequences of Flooding and the Impact of the National Flood Insurance Program.", Washington, DC: American Institutes for Research.


APPENDIX A. OVERALL LOGIC MODEL OF THE NATIONAL FLOOD INSURANCE PROGRAM 
Congressional Purposes	NFIP Mission per NFIP Blueprint	Simplified Goals	Objectives, FY 2000-2006; per GPRA, Blueprint	Activities	Outputs and outcomes
1. Make flood (and mudslide) insurance available on a nationwide basis through cooperative efforts of the federal government and the insurance industry 

2. Provide flexibility so that such flood insurance can be based on workable methods of pooling risks, minimizing costs, and distributing burdens equitably among those who will be protected by flood insurance and the general public

3. Subsidize flood insurance premiums of property owners who built in floodplains without the knowledge that they were doing so and to create an incentive for local governments' participation in the NFIP.

4. Encourage state and local governments to make appropriate land-use adjustments to constrict development on land that is at risk of flood damage and minimize damage caused by flood losses

NOTE: A related goal of Executive Order 11988 is to "minimize the impact of floods on human safety, health and welfare..."

5. Guide the development of proposed construction, where practicable, away from locations that are threatened by flood hazards

6. Encourage lending and credit institutions, as a matter of national policy, to assist in furthering the NFIP's objectives 

7. Assure that any federal assistance provided under the program will be related closely to all federal flood-related programs and activities

8. Encourage sound land use by minimizing exposure of property to flood losses.

NOTE: A related goal of Executive Order 11988 is "to restore and preserve the natural and beneficial values served by floodplains..."

9. Study flood hazards on a continuing basis in order to provide for an ongoing reappraisal of the flood insurance program and its effects on land-use requirements

A (1-3). Provide flood insurance to property owners at a reasonable cost in order to protect them from the financial risk of flood damage and distribute more fairly the costs associated with the risk of floods to those most at risk

B (1-3). Leverage the insurance industry's strength and expertise to provide flood insurance

C (1, 5, 6, 7). Assist in ensuring the compliance of federally regulated lenders with federal regulations and increase their participation (and that of insurance companies) through incentives and federal requirements

D (2, 3). Ensure flood insurance programs are financially responsive and actuarially sound wherever possible, given congressionally imposed rate subsidies 

E (4, 5). Encourage disaster-resistant communities

F (1, 3, 4). Encourage the participation and involvement of individuals, communities, and states

G (5, 8, 9). Reduce flood damage to buildings

H (7). Expedite the payment of claims in order to reinforce recovery efforts

I (7). Assist in maintaining credible enforcement measures to ensure compliance with floodplain management requirements

J (7). Through flood insurance, limit the costs of disaster relief and other tax write-offs 

K (8). Support environmental goals through floodplain management 

L (9). Create awareness and understanding of the risks of floods

I (1-3). Through cooperative efforts of the federal government and the insurance industry, make flood insurance widely available at actuarially sound rates or with legally mandated subsidies 

II (4-6, 9). Identify areas that are prone to flooding and inform government agencies, lenders, and the public of flood hazards (risk assessment and communication)

III (4-6, 9). In cooperation with other federal agencies, states, communities, and lenders, reduce the frequency and adverse consequences of flooding and associated NFIP claims (floodplain management and mitigation)

IV (4-7, 9). Minimize the amount of disaster assistance required for recovery from floods and reduce individual hardship through risk assessment, risk communication, floodplain management, mitigation, and insurance

V (5, 8). Support natural, beneficial uses of floodplains

a (2 AB I). Increase the number of policies  by 5 percent per year

b (4 IL II). Ensure that information, policies, and technical assistance are delivered to NFIP stakeholders promptly

c (4 CEGJ III). More clearly define the role of insurance in mitigation

d (2 CD I). Operate a financially sound insurance program

e (2 DL I). Review NFIP underwriting results and incorporate new risk information as it relates to the financial soundness of the program

f (4-6 G III,IV). Reduce potential annual flood losses by more than $1 billion 

Other Objectives
g (9 ). Conduct a comprehensive evaluation of the NFIP

1. Map floodplains to support sound floodplain management and inform flood insurance decisions 
2. Establish building and other standards to minimize damage during flood events and help standardize risk ratings 
3. Require communities to apply these standards so that their residents will be eligible to purchase flood insurance 
4. Provide funding and assistance to states and communities for mitigation programs  
5. Provide incentives, including reduced insurance premiums, for communities that exceed federal standards of floodplain management by participating in the Community Rating System (CRS) 
6. Encourage federal agencies to: a) evaluate the potential effects of the actions they may take in floodplains; and b) ensure that their planning programs and budget requests reflect consideration of flood hazards and floodplain management 
7. Provide flood insurance to property owners to encourage mitigation and decrease risk of flooding, including activities beneficial to all properties, e.g., removal of floodway obstructions 
8. Require flood insurance for any property located within a participating community, and receiving federal financial assistance, i.e., federally backed mortgages, disaster assistance or nondisaster loans and grants
9. During the term of such mortgages, require that flood insurance be maintained (no enforcement power granted to FEMA)
10. Do not provide flood insurance for property owners whose buildings are in flood hazard areas that do not comply with NFIP floodplain management requirements.
11. Review underwriting results and incorporate new risk information as it relates to the NFIP's financial soundness 
12. Conduct media campaigns and focus groups
13. Limit federal funds, including flood insurance, for new construction and substantial improvements, for areas included in the Coastal Barrier Resource System
14. Assist communities in establishing goals after flood disasters including: a) preservation of open spaces in floodplains; b) relocation of occupants; c) acquisition of land to minimize future property losses; and d) acquisition of frequently damaged structures

	Outputs
1. FIRMs that properly identify current flood-prone and other at-risk areas 
2. Building standards that reduce damage caused by flooding
3. Actions recommended by the Building Performance Assessment Team to state and local governments, construction industry, building code organizations, and property owners to reduce future damages
4. Increased number of NFIP policies in force
5. Premiums related to compliance with floodplain management requirements, resulting in increased or decreased rates depending on level of compliance within the community and the property

Intermediate Outcomes
1. A knowledgeable public that appreciates the potential impacts of floods and what can be done to mitigate their adverse impacts
2. Increased compliance with NFIP's regulations through enforcement of floodplain management requirements
3. A common agenda among federal agencies that promotes mitigation (or that does not increase the likelihood of floods) 
4. Increased amount of risk carried by property owners not paying actuarial rates, rather than the general public
5. Increased number of insured versus uninsured flood losses
6. A financially sound program of flood insurance

Ultimate Outcomes
1. Decreased risk of flood losses
2. Reduced costs and adverse consequences of flooding
3. Reduced demands and expectations for federal disaster assistance after floods
4. Natural and beneficial values of floodplains are restored and preserved