DOI: 10.5061/dryad.vhhmgqp1t
title: Data from: Extreme heat adaptation planning: A review of evaluation, monitoring, and reporting

description: Extreme heat events are increasing in intensity and duration. Although
 heat adaptation planning is increasing across the US, the effectiveness of
 adaptation strategies across contexts remains unknown. Evaluation helps
 heat adaptation planners understand the impact of investments and increase
 accountability. To understand how evaluation is or is not happening in
 extreme heat planning, we purposively sampled and analyzed 65 plans that
 would likely include extreme heat adaptation strategies. We found that
 although 55% (n=36) of plans included heat evaluation or monitoring plans
 in some form, fewer than 30% (n=19) were associated with subsequent
 reports. Of these, only six were implemented as planned, and none were
 implemented at the regional or neighborhood level. We also found that
 monitoring indicators did not match the heat impacts, vulnerabilities, and
 needs identified in the plan. We provide evaluation recommendations to
 guide and support evaluation and monitoring efforts in the heat planning
 process. This dataset is an Excel spreadsheet that includes the full plan
 list and coding, the coding instructions and the full 399 indicators list
 as well as a duplicate of the summary table included in the
 article. 
description: We conducted a qualitative content analysis of 65 plans from
 federal, state, regional, and municipal sources to understand how planners
 approached evaluation and monitoring in heat planning. Plan analysis is
 especially appropriate for heat adaptation research, as planning documents
 may be the only public-facing documentation of interagency and
 inter-jurisdiction coordination (Keith et al., 2021). Most plans are
 adopted by governing bodies and indicate a measure of government
 accountability, which is essential for climate adaptation (Mees &
 Driessen, 2019). We used purposive sampling to identify
 plans that were most likely to include heat adaptation strategies. Using
 professional networks and the Google search engine, we found nine
 standalone extreme heat adaptation plans: one state-level plan, two
 regional plans, four municipal plans, and two neighborhood-level plans.
 All but one were from the U.S. We then looked for plans from multiple
 levels of governance to identify heat adaptation strategies and evaluation
 included in broader planning efforts. We analyzed all 26 federal climate
 adaptation plans produced by Executive Order 14008 (Biden, 2021). The
 plans provided us with an overview of where heat planning and evaluation
 may or may not be happening at the highest level of government in the U.S.
 (Biden, 2021). We then pulled plans from an existing database of 175 plans
 in the 50 most populated municipalities in the U.S. that was created
 through previous research about how local governments plan for extreme
 heat (French & Hondula, 2021; Turner et al., 2022). From this
 database, we identified 20 urban plan documents that included a heat
 statement related to studying or monitoring in the Northeast and Midwest
 U.S. because we assumed these would have the highest likelihood of
 including an evaluation strategy or plan. If the plan had been updated, we
 analyzed the most recent plan. Lastly, we analyzed ten regional climate
 resilience plans in New York State to look for approaches to evaluation
 and monitoring outside of municipal geographies. We selected New York, the
 Northeast, and Midwest geographies because this research was part of a
 larger research project for the New York State Department of Environmental
 Conservation. See Table 1 for exclusion and inclusion criteria.
 In total, the 65 plans covered a variety of jurisdictions: 26
 federal, one state, ten regional, five metro-region, 21 municipal, and two
 neighborhood-level plans (See Appendix 1). The geographic bias towards
 large cities and a variety of scales reflects our purposive sampling
 designed to identify places most likely to have heat strategies
 incorporated in plans, and potential monitoring and evaluation as follow
 up (Gabbe et al., 2021). While the number of plans analyzed is smaller
 than more automated plan analyses, our interest in the varied ways
 monitoring and evaluation shows up is conducive to qualitative, “smaller
 n” analyses (Lieberson, 1991). For each plan document,
 we answered the following five questions: 1) Is extreme heat identified as
 a hazard in the plan? 2) Is there a strategy to respond to extreme heat?
 3) How was evaluation used during the planning process? 4) How are
 evaluation and monitoring planned for? And 5) What indicators are included
 in the evaluation and monitoring plans? We used qualitative analysis
 software and spreadsheets to code the plans for heat adaptation actions,
 where evaluation occurred in the planning cycle and to what extent, and
 indicators or metrics related to heat evaluation.  For
 plans that indicated an intent to evaluate or monitor, we reviewed
 websites, dashboards, reports, and evaluation activities to determine how
 governments implemented that monitoring and evaluation. Using Google
 search engine, we searched the plan lead authors’ website and the web
 search terms “plan name” AND “report* OR eval* OR monitor*.” We created a
 code book that described the coding process (See Appendix 2). Two coders
 worked together to code each plan twice, and both coders also followed the
 methodology for uncovering whether the evaluation was happening, resulting
 in an 81% agreement in results. We reviewed the evidence for each
 discrepancy to finalize the pattern coding. Biden,
 Joseph R. 2021. “Executive Order on Tackling the Climate Crisis at Home
 and Abroad.” The White House. January 27, 2021.
 https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/executive-order-on-tackling-the-climate-crisis-at-home-and-abroad/. French, M, and Emma Hondula. 2021. “How Are U.S. Cities Planning for Heat?” DesignSafe-CI. https://www.designsafe-ci.org/data/browser/public/designsafe.storage.published//PRJ-3180. Gabbe, C. J., Gregory Pierce, Emily Petermann, and Ally Marecek. 2021. “Why and How Do Cities Plan for Extreme Heat?” <em>Journal of Planning Education and Research</em>, October, 0739456X2110536. https://doi.org/10.1177/0739456X211053654. Keith, Ladd, Sara Meerow, David M. Hondula, V. Kelly Turner, and James C. Arnott. 2021. “Deploy Heat Officers, Policies and Metrics.” <em>Nature</em> 598 (7879): 29–31. https://doi.org/10.1038/d41586-021-02677-2. Lieberson, Stanley. 1991. “Small N’s and Big Conclusions: An Examination of the Reasoning in Comparative Studies Based on a Small Number of Cases.” <em>Social Forces</em> 70 (2): 307. https://doi.org/10.2307/2580241. Mees, Heleen, and Peter Driessen. 2019. “A Framework for Assessing the Accountability of Local Governance Arrangements for Adaptation to Climate Change.” <em>Journal of Environmental Planning and Management</em> 62 (4): 671–91. https://doi.org/10.1080/09640568.2018.1428184. Turner, V Kelly, Emma M French, John Dialesandro, Ariane Middel, David M. Hondula, George Ban Weiss, and Hana Abdellati. 2022. “How Are Cities Planning for Heat? Analysis of United States Municipal Plans.” <em>Environmental Research Letters</em> 17. https://iopscience-iop-org.gate.lib.buffalo.edu/article/10.1088/1748-9326/ac73a9.
description: # Data from: Extreme heat adaptation planning: A review of evaluation,
 monitoring, and reporting
 [https://doi.org/10.5061/dryad.vhhmgqp1t](https://doi.org/10.5061/dryad.vhhmgqp1t) ## Description of the data and file structure To understand how evaluation is or is not happening in extreme heat planning, we purposively sampled and analyzed 65 plans that would likely include extreme heat adaptation strategies. We used purposive sampling to identify plans that were most likely to include heat adaptation strategies. In total, the 65 plans covered a variety of jurisdictions: 26 federal, one state, ten regional, five metro-region, 21 municipal, and two neighborhood-level plans. All but one were from the United States. For each plan document, we answered the following five questions: 1) Is extreme heat identified as a hazard in the plan? 2) Is there a strategy to respond to extreme heat? 3) How was evaluation used during the planning process? 4) How are evaluation and monitoring planned for? And 5) What indicators are included in the evaluation and monitoring plans? We used qualitative analysis software and spreadsheets to code the plans for heat adaptation actions, where evaluation occurred in the planning cycle and to what extent, and indicators or metrics related to heat evaluation. For plans that indicated an intent to evaluate or monitor, we reviewed websites, dashboards, reports, and evaluation activities to determine how governments implemented that monitoring and evaluation. Using Google search engine, we searched the plan lead authors’ website and the web search terms “plan name” AND “report* OR eval* OR monitor*.” We created a code book that described the coding process (See Appendix 2). Two coders worked together to code each plan twice, and both coders also followed the methodology for uncovering whether the evaluation was happening, resulting in an 81% agreement in results. We reviewed the evidence for each discrepancy to finalize the pattern coding. ### Files and variables #### File: Heat\_plan\_eval\_and\_indicators\_Holtan\_2025.xlsx **Description:** Excel document with table of contents, list of plans, coding instructions and indicators.  ##### Variables * All variables are described and spelled out in the Excel in the coding notes tab and as summarized below: * Column A: Plan ID * Column B: Name of document * Column C: Document author/publisher * Column D: Year adopted * Column E: Planning scale * Columns F through M: Plan analysis variables: * Y=yes * N=no * R=related, but not specific enough to heat to be included in the analysis * n/a =  "not applicable" as the cells associated with that row were not applicable to the planning document as analyzed. For example, plans that did not contain a strategy related to heat, were not analyze to see if there were heat related monitoring indicators or reporting.   * Column N  Typology - the type of reporting:  * 1. Yes, Plan for process, report on something else; 2. Yes, Plan for impact evaluation, report on process; 3. Yes, Selective reporting of positive items only; 4. Yes, Data only alone; 5. Yes, Constellation of plans website with circular references 6. Yes, Evaluation through plan update/re-write; 7. No, but there is constellation of plans 8. No, but there is a new evaluation plan * Column O: Report/evaluation date (if applicable) * Column P: Report name (if applicable) * Column Q: Report link (if applicable) * Column R: Document source search category  ## Code/software No special software is needed.