Special Report: Review 

Regarding DHS OIG’s 

Retraction of Thirteen 

Reports Evaluating FEMA’s 

Initial Response to 

Disasters 

May 23, 2019 

OIG-19-41 
 

 

DHS OIG HIGHLIGHTS 

Review Regarding DHS OIG’s Retraction

of Thirteen Reports Evaluating

FEMA’s Initial Response to Disasters 

May 23, 2019 

Why We Did This 

Special Report 

After Congress raised concerns 

about the accuracy of the 

findings and conclusions in 

DHS OIG’s report on FEMA’s 

Initial Response to Catastrophic 

Flooding in Louisiana (OIG-17-

80-D), we conducted an 

internal review to determine 

the root causes of the 

deficiencies identified in that 

report and twelve similar DHS 

OIG reports purporting to 

evaluate the Federal 

Emergency Management 

Agency’s initial response to 

certain declared disasters. 

What We 

Recommend 

We recommend that DHS OIG 

take steps to improve its 

internal controls and quality 

assurance processes to 

safeguard against the 

deficiencies identified by the 

internal review. 

For Further Information: 

Contact our Office of Public Affairs at 

(202) 981-6000, or email us at 

DHS-OIG.OfficePublicAffairs@oig.dhs.gov. 

What We Found 

The thirteen reports DHS OIG retracted 

and removed from its website were not 

compliant with applicable standards. 

The deficiencies in the retracted reports 

were the result of: (a) several key 

changes to DHS OIG’s approach to 

early deployment oversight work that 

were poorly communicated and 

managed by senior leaders, (b) a flawed 

report model, and (c) internal control 

failures, including in the areas of audit 

planning, supervision, and 

independent referencing. This 

combination of factors led DHS OIG to 

publish thirteen reports concluding 

that the Federal Emergency 

Management Agency’s initial response 

to certain declared disasters was 

efficient and/or effective despite 

lacking sufficient and appropriate 

evidence to support that conclusion. 

DHS OIG’s Response 

DHS OIG concurred with the five 

recommendations in this report and 

described corrective actions it has taken, 

is taking, and plans to take, to address 

them. Appendix B provides DHS OIG’s 

management response in its entirety. 

www.oig.dhs.gov OIG-19-41 
 

 

 

OFFICE OF INSPECTOR GENERAL

 Department of Homeland Security 

Washington, DC 20528 / www.oig.dhs.gov 

May 23, 2019 

MEMORANDUM FOR: Jennifer L. Costello 

 Deputy Inspector General 

FROM: Diana R. Shaw 

 Assistant Inspector General 

 for Special Reviews and Evaluations 

SUBJECT: Special Report: Review Regarding DHS OIG’s 

Retraction of Thirteen Reports Evaluating FEMA’s 

Initial Response to Disasters 

In July 2017, the Department of Homeland Security (DHS) Office of 

Inspector General (OIG) retracted an Emergency Management Oversight 

Team (EMOT) report after the House Committee on Oversight and 

Government Reform (HOGR) raised concerns about the accuracy of the 

report’s findings and conclusions. After conducting a preliminary quality 

assurance review of similar EMOT reports, DHS OIG retracted an 

additional twelve reports in March 2018 due to similar concerns. 

At a meeting with HOGR members in March 2018, Acting Inspector 

General John V. Kelly stated that DHS OIG would conduct an internal 

review of the circumstances surrounding the retracted reports to better 

understand the factors giving rise to the deficiencies identified in them. 

Because Mr. Kelly was involved in overseeing the EMOT work at issue, he 

recused himself from the matter and delegated authority to me as the 

then-Acting Counsel to conduct the internal review and help develop a 

corrective action plan. 

The following report summarizes the results of our internal review. This 

report, along with your management response, is being distributed to the 

DHS Secretary and our congressional oversight committees, and will be 

posted on DHS OIG’s website. 

www.oig.dhs.gov 
 

 

 

                                                      

 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

BACKGROUND 

Congress established Offices of Inspector General to ensure integrity and 

efficiency in government through independent and objective oversight of 

Federal agencies. Since 2003, DHS OIG has provided oversight for the 

operational and support components comprising the Department of 

Homeland Security. DHS OIG’s former Office of Emergency Management 

Oversight (EMO) conducted audits and other reviews of DHS’ disaster 

response efforts.1 EMO’s oversight work included auditing Federal 

Emergency Management Agency (FEMA) grants awarded to state and 

local governments, conducting performance audits of FEMA’s operations, 

and sending Emergency Management Oversight Teams (EMOT) to 

evaluate FEMA’s initial response to declared disasters. 

Between fiscal years 2010 and 2017, EMO issued a series of EMOT 

reports. In July 2017, DHS OIG retracted and removed from its website 

an EMOT report regarding FEMA’s initial response to catastrophic 

flooding in Louisiana (OIG-17-80-D) after the House Committee on 

Oversight and Government Reform (HOGR) raised concerns about the 

accuracy of the report’s findings and conclusions. After conducting a 

preliminary quality assurance review of similar EMOT reports, DHS OIG 

retracted and removed from its website an additional twelve reports in 

March 2018 due to similar concerns.2 

On March 6, 2018, members of HOGR met with DHS OIG’s Acting 

Inspector General John V. Kelly to discuss the retracted reports. They 

asked whether the deficiencies that necessitated the retraction of the 

reports were the result of a “people problem,” a “process problem,” or 

1 In 2017, EMO’s staff and portfolio of work was absorbed within DHS OIG’s Office of 

Audits. 

2 FEMA’s Initial Response to Hurricane Isaac in Louisiana was Effective and Efficient 

(OIG-13-84); FEMA’s Initial Response in New Jersey to Hurricane Sandy (OIG-13-117); 

FEMA’s Initial Response in New York to Hurricane Sandy (OIG-13-124); FEMA’s Initial 

Response to the Oklahoma Severe Storms and Tornadoes (OIG-14-50-D); FEMA’s Initial 

Response to the Colorado Flood (OIG-14-111-D); FEMA Provided an Effective Response to 

the Napa, California, Earthquake (OIG-15-92-D); FEMA’s Initial Response to the 2014 

Mudslide near Oso, Washington (OIG-15-102-D); FEMA’s Initial Response to Severe 

Storms and Flooding in Michigan (OIG-15-105-D); FEMA’s Initial Response to the Severe 

Storms and Flooding in South Carolina (OIG-16-53-D); FEMA’s Initial Response to the 

2015 Texas Spring Severe Storms and Flooding (OIG-16-85-D); FEMA was Generally 

Effective in its Initial Response to the Severe Wildfires in California (OIG-16-106-D); and 

FEMA’s Initial Response to the Severe Storms and Flooding in West Virginia DR-4273 

(OIG-17-37-D). 

www.oig.dhs.gov 2 OIG-19-41 
 

 

                                                      

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

some combination of the two. Mr. Kelly assured the Committee that DHS 

OIG would undertake an objective internal review of the matter and 

report its findings to the Committee. 

An internal review team (Review Team) consisting of investigative counsel 

and analysts was established in March 2018 to investigate and report on 

the root causes giving rise to the deficiencies identified in the reports.3 

The Review Team also engaged the services of an auditing firm to get an 

outside evaluation of the reports at issue and independent guidance on 

industry standards and best practices related to auditing. The following 

sections summarize the Review Team’s findings and recommendations 

for corrective action. 

SUMMARY OF FINDINGS 

The Review Team evaluated the entire series of EMOT reports published 

by DHS OIG through 2017.4 That review revealed that DHS OIG had 

historically done early deployment work on disaster response without the 

concerns raised by the withdrawn EMOT reports. A leadership change in 

EMO in late 2011, however, led to four notable changes in how EMOT 

deployments were conducted. Those changes were poorly communicated 

to staff and not well managed by EMO’s senior leaders, including Mr. 

Kelly and several former DHS OIG employees, and resulted in a flawed 

approach to this work that lacked necessary planning and criteria. As 

that new EMOT model was cemented into place, EMO personnel began to 

think of EMOT reports as “feel good” reports — i.e., generally positive 

reports that typically concluded that FEMA’s initial response to a 

disaster was effective. Deficiencies in EMO’s internal controls system and 

quality assurance processes — including in the areas of audit planning, 

supervision, and independent referencing — resulted in a failure to 

identify and correct the issues with the EMOT product line, thus 

necessitating retraction of the thirteen reports. 

3 Because Mr. Kelly was involved in overseeing the EMOT work at issue, he recused 

himself from the matter and delegated authority to then-Acting Counsel, Diana Shaw, 

to make all decisions with respect to the review and to perform all functions necessary 

to complete the review, including developing and issuing recommendations for potential 

corrective action. 

4 DHS OIG has not published any EMOT reports since 2017. 

www.oig.dhs.gov 3 OIG-19-41 
 

 

                                                      

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

A. The Evolution of EMOTs 

OIGs have historically recognized that an important part of oversight 

work involves evaluating emergency preparedness and response 

operations. For instance, FEMA OIG deployed personnel to New York City 

in the immediate aftermath of the 9/11 terrorist attack in 2001, and 

DHS OIG deployed personnel to various sites on the Gulf Coast after 

Hurricane Katrina in 2005. 

After Katrina, and coinciding with Congress’ enactment of the PostKatrina Emergency Management Reform Act of 2006, DHS OIG started to 

formalize an early deployment model for disaster oversight. This early 

deployment model envisioned sending a skilled team of DHS OIG 

personnel to the site of a disaster to observe FEMA’s on-the-ground 

activities. This work also sought to deter fraud, waste, and abuse by 

establishing an on-site presence during the disaster response phase of 

FEMA’s work. The goal was to help FEMA get out ahead of issues before 

they became multi-million-dollar problems years later. 

The first deployments under this early deployment model were not 

treated as OIG audits, which are subject to the U.S. Government 

Accountability Office’s Government Auditing Standards (also referred to 

as “Yellow Book” standards), nor did they always result in the issuance of 

a report to FEMA. Their stated objective was to promote effectiveness and 

efficiency in FEMA’s response. The three reports DHS OIG issued under 

this early model identified concerns and contained recommendations for 

action by FEMA.5 The reports did not purport to be audit reports 

prepared to Yellow Book standards. 

B. Four Notable Changes to the EMOT Model Led to a Flawed 

Approach 

In late 2011, a leadership change in EMO brought about four notable 

changes to the EMOT model that, ultimately, set the work off course and 

resulted in a flawed product line. 

5 DHS OIG, Management Advisory Report: FEMA’s Response to Hurricane Ike (OIG-09-

78); DHS OIG, Management Advisory Report: Permanent Housing Construction on 

American Samoa (OIG-10-74); DHS OIG, American Samoa 2009 Earthquake and 

Tsunami: After-Action Report (OIG-11-03). 

www.oig.dhs.gov 4 OIG-19-41 
 

 

 

                                                      

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

First, the new Assistant Inspector General (AIG) for EMO, D. Michael 

Beard, and the new Acting Deputy AIG for EMO, Mr. Kelly,6 decided that 

EMOT deployments would always result in the issuance of a report to 

FEMA, which had not been the case previously.7 

Second, Mr. Kelly decided that reports issued following EMOT 

deployments should be issued under Yellow Book standards. However, 

the first few teams whose work was impacted by this change were not 

aware of this decision when they deployed. As a result, while performing 

their fieldwork, they did not know they were expected to conduct the 

work to the Yellow Book’s audit requirements. Further, these teams used 

EMOT-related guidance documents that were not designed to be an audit 

guide and were not designed to result in a Yellow Book-compliant audit 

report. For instance, these guidance documents directed teams simply to 

answer five questions, such as, “What were the most pressing challenges 

FEMA faced in this disaster?” Even though DHS OIG undertook thirteen 

EMOT deployments between 2012 and 2017, these documents were not 

revised to ensure they would result in Yellow Book-compliant audit work 

and audit reports. 

Third, the objective of EMOTs shifted from promoting effectiveness and 

efficiency to evaluating whether FEMA’s initial disaster response was 

effective and efficient. Notwithstanding this change in the objective, DHS 

OIG did not identify criteria that would have allowed it to adequately 

evaluate FEMA’s effectiveness and efficiency. Audit “criteria” provides a 

baseline against which a program or activity can be evaluated, and the 

identification of suitable criteria is an essential component of conducting 

a Yellow Book-compliant audit.8 

Fourth, several factors led to a formulaic approach to the drafting of 

EMOT reports that resulted in reports with a generally positive focus. As 

6 Mr. Kelly became the permanent Deputy AIG for EMO in 2012. In 2013, he became 

the AIG for EMO. 

7 DHS OIG has determined that, in light of the issues addressed in this report, the 

public interest in disclosure outweighs the personal privacy interest of certain of the 

individuals referenced herein. Accordingly, because significant public benefit would 

result from disclosure of the information contained in this report, DHS OIG has 

released the names of the senior-most decision-makers associated with this matter. 

8 U.S. Government Accountability Office, Government Auditing Standards (2018 

Revision), at pp. 113-115. An example of criteria in the disaster response context could 

be, for instance, how many individuals received assistance within the first 72 hours of a 

disaster. 

www.oig.dhs.gov 5 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

the new approach to EMOT work was being refined, there was a common 

view within EMO leadership that FEMA handled its disaster response 

duties especially well, even if it tended to fall short in other areas. 

Specifically, a former DHS OIG employee who led early EMOT work 

under this new approach and later oversaw all EMOTs after being 

promoted by Mr. Kelly (referred to herein as Senior Employee 1 (“SE1”)), 

was particularly impressed with FEMA’s dedication and work ethic in 

responding to disasters, and conveyed that view to staff. 

Mr. Kelly similarly shared this view about FEMA’s disaster response work 

and conveyed that view to staff. For instance, prior to deploying to a 

disaster, one EMOT staff member recalled Mr. Kelly telling him he would 

see “FEMA at her best,” a statement the staff member later repeated to 

other EMO staff. Statements like this risked fueling a preconceived 

notion among staff that FEMA’s response on any given disaster was likely 

to be effective and efficient, which could have impaired their objectivity 

while performing their oversight work. 

In addition to espousing a general opinion that FEMA typically handled 

the initial response to a disaster well, Mr. Kelly made a critical decision 

about what information would and would not be included in EMOT 

reports, which further fueled the notion that EMOT reports were 

intended to be positive. He decided that if EMOTs identified significant 

systemic issues while deployed, those issues would be carved out of the 

report on the initial response, and DHS OIG would issue separate “spinoff” reports on those issues. When asked by the Review Team why he 

decided to carve out such issues, he stated that believed it was preferable 

to issue a “clean report” for EMOTs because he thought a report with a 

single message would be clearer and less likely to get lost or be ignored 

by FEMA. However, it does not appear that he explained his rationale to 

EMO staff, who did not understand why they were being instructed not 

to include some of the more significant findings in their EMOT reports. 

According to Mr. Kelly, he also believed “spin-off” reports could better 

address systemic issues within FEMA that were not specific to the initial 

local disaster response effort, since DHS OIG’s EMOT reports after a 

deployment were often addressed to the FEMA region involved in the 

disaster, not FEMA Headquarters. He reasoned it would not be 

appropriate to make recommendations regarding systemic issues to 

regional personnel. However, at least some reports issued under this new 

EMOT model opined on “FEMA’s” disaster response without limiting the 

conclusion to the particular FEMA region under review. 

www.oig.dhs.gov 6 OIG-19-41 
 

 

 

                                                      

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

Ultimately, because the more serious and significant issues were “spunoff” into later reports, EMOT deployment reports were left communicating 

a largely positive message. 

C. Implementation of Those Notable Changes 

To understand how these decisions were communicated to EMOT 

personnel and impacted EMOT work, the Review Team examined the 

histories of each EMOT report produced under the new approach. This 

retrospective review revealed a pattern across EMOT projects — while the 

EMOT teams deployed to the field did not always set out to perform “feel 

good” work, when the time came to issue a report, the reports unerringly 

reached the same positive conclusion. The following examples of 

particular EMOT projects illustrate how this pattern began and 

perpetuated, ultimately resulting in a flawed product line. 

1. Hurricane Isaac EMOT, 2012 

In the fall of 2012, an EMOT deployed to Louisiana in response to 

Hurricane Isaac. The team initially planned to report on several issues 

that they believed supported recommendations for corrective action. 

However, the EMOT director, SE1, instructed the team that Mr. Kelly 

wanted to “downplay all the negatives” in the report because FEMA’s 

“overall” response to the disaster was “good.”9 The team was therefore 

instructed to “[t]alk about the things [FEMA] did right and their 

accomplishments, then maybe list the negatives in bullets — no 

recommendations unless you have evidence that these things are 

systemic.” SE1 further instructed the team that Mr. Kelly wanted the 

EMOT report “to be positive with no recommendations.” Based on these 

instructions, the team produced a report that concluded that FEMA’s 

initial response to Hurricane Isaac was overall effective and efficient, and 

contained no recommendations.10 This report served as a model for 

subsequent EMOT reports. 

9 The Review Team found no written documentation confirming that Mr. Kelly had, in 

fact, issued instructions consistent with SE1’s representations. However, when 

interviewed as part of this internal review, Mr. Kelly did not disagree with SE1’s 

representations; rather, his statements established that SE1’s representations were 

generally consistent with his thinking about how EMOT work should be reported. He 

acknowledged, however, that the statement “downplay all the negatives” could be 

problematic if taken out of context. 

10 OIG-13-84. 

www.oig.dhs.gov 7 OIG-19-41 
 

 

 

                                                      

	 

	 

	 

	 

	 


 


 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

2.	 Hurricane Sandy EMOTs, 2013 

While the Isaac EMOT report was being drafted, two other EMOTs were on 

deployments to New York and New Jersey to evaluate FEMA’s response to 

Hurricane Sandy. After the Sandy teams completed their fieldwork, Mr. 

Kelly emailed the draft Isaac report to the directors of those teams, 

stating: “[g]iven that FEMA’s initial response was generally good for both 

disasters at all locations it would be best if all three reports were similar.” 

The Review Team found that sharing the Isaac report with the Sandy 

teams and encouraging them to prepare similar reports for each of their 

deployments contributed to the development of the concept of “feel good” 

reporting on EMOTs. Indeed, three months after Mr. Kelly sent the email 

about the Isaac report, a team member on the New York Sandy EMOT 

referred to the New York report as a “feel good” report.11 

Eventually, EMOT reports came to be commonly understood throughout 

EMO as “feel good” reports. While never officially defined, the notion of a 

“feel good” report came to serve as shorthand for four key elements: 

(1)	 the overall conclusion of an EMOT report would be that 

FEMA’s disaster response was “effective and efficient”; 

(2)	 the report would not include recommendations; 

(3)	 if the report identified challenges, it would note how 

FEMA overcame them; and 

(4)	 other negative information, if systemic, would be removed 

from the report and placed in a separate report. 

Both Sandy teams produced reports along the above lines, concluding 

that FEMA provided an effective and efficient response and containing no 

recommendations.12 Consistent with Mr. Kelly’s instructions, however, 

three spin-off reports were issued as a result of the Sandy deployments 

related to housing and management issues, two of which contained 

recommendations.13 

11 This is the earliest reference to this term identified by the Review Team.
 

12 OIG-13-117; OIG-13-124.
 

13 See DHS OIG, FEMA’s Sheltering and Temporary Essential Power Pilot Program (OIG13-15); DHS OIG, The State of New York Needs To Sign Mission Assignment More 

www.oig.dhs.gov 8 OIG-19-41 
 

 

 

                                                      

 

 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

3. Oklahoma EMOT, 2013 

The next EMOT was deployed to evaluate FEMA’s response to a disaster 

in Oklahoma. Upon completing its fieldwork, the team initially drafted a 

report containing findings and recommendations. However, SE1 

instructed the team to remove systemic issues from the report so they 

could be addressed in a spin-off report.14 In an email on which Mr. Kelly 

was copied, SE1 instructed the EMOT director to “read the other EMOT 

reports – Isaac, Sandy NY, Sandy NJ. These are feel-good reports to tell 

the public how well FEMA initially responds to disasters. Yes, we will 

identify some problems, but the point is – how did FEMA overcome 

them?” The email further states that “EMOT deployments present 

wonderful opportunities to identify systemic problems, but the EMOT 

report is not the place to discuss the problems in detail. These 

opportunities allow us to write quick, meaningful reports that actually 

help FEMA, and FEMA usually loves the reports.” Mr. Kelly did not 

respond to this email to clarify or correct any of the statements in it.15 

Much like the Isaac and Sandy reports, the conclusion of the Oklahoma 

report was that FEMA’s initial disaster response was effective.16 The 

report contained no recommendations. The report stated OIG would 

issue separate reports, which it did. The spin-off reports contained 

recommendations.17 

Quickly (OIG-13-85); DHS OIG, FEMA’s Process for Selecting Joint Field Offices Needs 

Improvement (OIG-15-128-D). 

14 By this time, SE1 had been promoted by Mr. Kelly to a position overseeing all EMOT 

work. 

15 This is the only document the Review Team identified indicating that Mr. Kelly was on 

notice of the use of the term “feel good” in connection with EMOT reports. The Team did 

not uncover any evidence that he used the phrase “feel good” (or anything similar) in 

writing. The Team received conflicting testimonial evidence as to whether Mr. Kelly ever 

used the phrase in conversation, with one employee recalling that he did, and other OIG 

personnel stating that he did not. Mr. Kelly stated that he may have heard the term 

“feel good” in reference to EMOT reports, but he did not believe use of the term 

necessarily indicated that the reports were not truthful and accurate. With respect to 

this particular email, however, he stated that, upon re-reading it, it seemed to indicate 

a potential impairment of independence on the part of SE1. 

16 OIG-14-50-D. 

17 See DHS OIG, FEMA’s Dissemination of Procurement Advice Early in Disaster 

Response Periods (OIG-14-46-D); DHS OIG, Mitigation Planning Shortfalls Precluded 

FEMA Hazard Mitigation Grants to Fund Residential Safe Room Construction During the 

Disaster Recovery Phase (OIG-14-110-D); DHS OIG, FEMA Can Enhance Readiness with 

Management of Its Disaster Incident Workforce (OIG-16-127-D). 

www.oig.dhs.gov 9 OIG-19-41 
 

 

                                                      

 

 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

4. Colorado EMOT, 2013 

Around the same time as the Oklahoma EMOT, a separate team was 

working on drafting a report in connection with FEMA’s response to a 

disaster in Colorado. As with the Oklahoma EMOT, members of the team 

sought to include concerns in the report, but those efforts were 

ultimately rebuffed. One member reached out to Mr. Kelly directly to 

advocate for a different, harder-hitting approach on the Colorado EMOT 

report. He memorialized the discussion in an email, stating that he had 

explained to Mr. Kelly that he was “on track to generate an EMOT report 

full of significant (and systemic) issues and actionable recommendations 

(alongside positive aspects). I told [Mr. Kelly] that it will not resemble the 

first two EMOT reports. He said, ‘Okay,’ so I’m proceeding with working 

toward that goal.” 

A few weeks later, however, the team was instructed by SE1 that Mr. 

Kelly “really does not want serious issues discussed in the EMOT report. 

He prefers we issue a ‘feel-good’ report with no recommendations on 

FEMA’s initial response – not that you cannot mention problems, but 

focus on how FEMA dealt with problems to respond as its mission 

demands. He wants several smaller [reports] issued on immediate issues 

....” The team member who had initially wanted to prepare a report full of 

significant issues and actionable recommendations ultimately 

acquiesced, writing to SE1: “Yes – we remember and that’s our priority 

for this report. We commend FEMA on a job well done. We identify 

various problems and how FEMA overcame those challenges (which also 

establishes opportunities for future reports). There will be no 

recommendations.” 

Much like the four EMOT reports before it, the conclusion of the 

Colorado report was that FEMA’s disaster response was effective and 

efficient. As with the earlier reports, it contained no recommendations. 

This work led to separate reports on FEMA’s Disaster Assistance Helpline 

and staffing issues, which did include recommendations.18 

18 See DHS OIG, FEMA Should Take Steps To Improve the Efficiency and Effectiveness of 

the Disaster Assistance Helpline for Disaster Survivors That Do Not Speak English or 

Spanish (OIG-14-118-D); DHS OIG, FEMA Can Enhance Readiness with Management of 

Its Disaster Incident Workforce (OIG-16-127-D). 

www.oig.dhs.gov 10 OIG-19-41 
 

 

 

                                                      

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

D. Subsequent EMOTs 

Each of the five EMOT reports discussed above shared the common 

elements of a “feel good” report: (1) the conclusion that FEMA’s disaster 

response was effective and efficient; (2) no recommendations; (3) 

challenges discussed in terms of how they were overcome; and (4) 

systemic issues carved out for spin-off reports. The teams who worked on 

an additional eight EMOTs between 2015 and 2017 subscribed to this 

same approach for EMOT reporting.19 

The final report in this series, FEMA’s Initial Response to Catastrophic 

Flooding in Louisiana (OIG-17-80-D), was issued in June 2017. The team 

who prepared the report was instructed that EMOT reports typically did 

not have findings or recommendations, and that they should stick to the 

same format in their reporting. Although the team discussed producing a 

robust report, the draft the team produced fit the mold of earlier EMOT 

reports — it contained a positive conclusion and no recommendations. 

Additionally, as the draft was being revised, the EMOT director made 

several material changes to the draft without ensuring that the changes 

were supported by sufficient and appropriate evidence. For instance, he 

added superlative language like “aggressively,” “remarkable,” and 

“creatively” to describe FEMA’s disaster response efforts, and changed 

the overall conclusion of the report from “generally efficient and well 

organized” to “effective.” He also added language discounting complaints 

about FEMA’s disaster response. None of the DHS OIG senior leaders 

who reviewed and approved this report corrected these issues. 

When the report was published, HOGR staff members, who also 

happened to be in Louisiana to observe FEMA’s response to the disaster, 

could not reconcile the report’s conclusion about the effectiveness of 

FEMA’s response with their own observations. Upon learning of their 

concerns, DHS OIG conducted a quality assurance review of the EMOT 

report in question and determined that it did not comply with Yellow 

Book auditing standards. The report was retracted and withdrawn from 

DHS OIG’s website on July 19, 2017. Thereafter, DHS OIG’s Office of 

Integrity and Quality Oversight conducted a preliminary quality 

assurance review of DHS OIG’s other EMOT reports. Following that 

review, DHS OIG retracted an additional twelve EMOT reports in March 

2018 due to similar concerns. 

19 OIG-15-92-D, OIG-15-102-D, OIG-15-105-D, OIG-16-53-D, OIG-16-85-D, OIG-16-

106-D, OIG-17-37-D, OIG-17-80-D. 

www.oig.dhs.gov 11 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

E. EMO Internal Control Deficiencies 

A strong system of internal controls should have caught and corrected 

the issues described above, but failed to do so. EMO’s internal controls 

system included layers of managers and supervisors who were required 

to oversee and guide EMO’s audit work. However, the Review Team, in 

consultation with its outside auditing firm, found that EMO management 

and supervision were deficient in ensuring adequate planning that would 

have established achievable audit objectives. EMOTs generally stated 

that their objective was to evaluate the effectiveness of FEMA’s response 

to the disaster, but there were no defined criteria or steps for impartially 

conducting this assessment. 

EMO management and supervision were also deficient in ensuring teams 

adequately assessed the evidence obtained during fieldwork. For 

instance, EMOTs did not evaluate FEMA’s own internal controls in order 

to objectively determine the reliability and validity of data they received 

from FEMA, and instead almost exclusively relied on testimonial evidence 

(most of which was provided by FEMA personnel) without taking steps to 

independently confirm assertions made and to protect against bias. 

Further, EMO’s internal quality control procedures did not function as 

intended due to weaknesses in EMO’s Independent Reference Reviewer 

(IRR) process. In the IRR process, an employee who is not a member of 

the audit team is supposed to confirm that a draft report’s statements of 

fact are accurate, the report’s findings are adequately supported by 

evidence in the audit documentation, and the report’s conclusions and 

recommendations flow logically from the evidence. This process did not 

flag EMOT deficiencies. Instead, it was more focused on confirming basic 

facts and figures, rather than on ensuring that supporting conclusions 

and auditor opinions were adequately supported. 

There were also deficiencies identified in EMO’s use of DHS OIG’s 

Supervisory Review Checklist. The Checklist is the process by which 

directors, managers, and supervisors certify that they have reviewed 

audit work and determined that the work met Yellow Book standards. 

This process also failed to flag EMOT deficiencies. For instance, in 

connection with the EMOT report titled FEMA’s Initial Response to 

Catastrophic Flooding in Louisiana (OIG-17-80-D), the Checklist indicates 

that the team verified the validity and reliability of the data obtained from 

FEMA during the audit. In fact, the team only verified that the data came 

www.oig.dhs.gov 12 OIG-19-41 
 

 

 

 

                                                      

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

from a FEMA data system, not that the data contained in the system was 

valid and reliable. 

A recent U.S. Environmental Protection Agency (EPA) OIG peer review of 

DHS OIG — which included in its scope the final report in the EMOT 

series, FEMA’s Initial Response to Catastrophic Flooding in Louisiana 

(OIG-17-80-D) — reached similar conclusions about deficiencies in 

EMO’s system of quality controls. Specifically, EPA OIG concluded that 

“weaknesses in the DHS OIG’s Office of Emergency Management 

Oversight (EMO) control structure did not assure compliance with 

GAGAS [Generally Accepted Government Auditing Standards] for 

sampled audit OIG-17-80-D. We identified issues with audit planning, 

assessment of evidence, supervision, indexing and reporting disclosures 

necessary for understanding the audit scope and methodology.”20 

As a result, DHS OIG was given a “pass with deficiencies” rating by EPA 

OIG, which concluded that “except for the deficiency described above 

[regarding report OIG-17-80-D], the system of quality control in effect for 

the DHS OIG’s audit organization for the year ending September 30, 

2017, has been suitably designed and complied with to provide the DHS 

OIG with reasonable assurance of performing and reporting in conformity 

with applicable professional standards in all material respects.” DHS OIG 

concurred with all four recommendations made by EPA OIG.21 

CONCLUSION 

While the Review Team did not find that EMOT members always set out 

to perform “feel good” work when deployed to the field, when the time 

came to issue a report, the EMOT reports unerringly took the same 

shape and reached the same conclusion. The Review Team determined 

that the issues with the thirteen retracted EMOT reports were the result 

of: (1) several key changes made to the EMOT model that were poorly 

communicated and managed; (2) a flawed report model; and (3) 

deficiencies in EMO’s internal controls. 

Had the EMOT work been performed as originally intended — as a tool 

for collecting and analyzing information to identify issues for future audit 

work — it could have provided valuable insight on FEMA’s disaster 

20 EPA OIG, System Review Report of DHS OIG (Aug. 27, 2018), 

https://www.oig.dhs.gov/sites/default/files/OIG_SystemReviewReport_082718.pdf. 

21 Id. 

www.oig.dhs.gov 13 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

response efforts and presented opportunities to prevent issues from 

developing into more serious, costly problems. However, the decision to 

move away from “promoting” efficiency and effectiveness in FEMA’s initial 

disaster response to “evaluating” that response, coupled with DHS OIG’s 

failure to develop appropriate criteria to measure efficiency and 

effectiveness, flawed the work from the start. 

RECOMMENDATIONS 

We recommend that the Deputy Inspector General: 

1. Confirm that all DHS OIG reports bearing the four signature traits 

of a “feel good” report have been retracted and removed from DHS 

OIG’s website. 

2. Ensure that all future early deployment work conducted by DHS 

OIG, if any, is performed in accordance with applicable standards. 

3. Design and implement improvements to DHS OIG’s quality 

assurance processes to resolve the internal controls deficiencies 

identified by the internal review. 

4. Close all open recommendations outlined in the EPA OIG peer 

review. 

5. Refer this report to the Council of Inspectors General on Integrity 

and Efficiency for whatever action it deems appropriate. 

CORRECTIVE ACTIONS 

Upon completing fact development and consulting with its outside 

auditors, members of the Review Team briefed HOGR on the findings of 

the internal review on October 19, 2018. On October 25, 2018, members 

of the Review Team provided the same briefing to the Senate Homeland 

Security and Governmental Affairs Committee. Immediately following 

these congressional briefings, Mr. Kelly was briefed on the findings of 

the review. Over the next few months, DHS OIG’s senior leaders — 

including Deputy Inspector General, Jennifer Costello; Assistant 

Inspector General for Audits, Sondra McCauley; and Assistant Inspector 

General for Integrity and Quality Oversight, Thomas Salmon — began 

www.oig.dhs.gov 14 OIG-19-41 
 

 

 

                                                      

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

developing and implementing a corrective action plan to address the 

deficiencies identified in the review.22 That plan built on corrective 

actions DHS OIG had already begun implementing after retracting the 

thirteen EMOT reports. 

As reflected in the management response produced in full in Appendix B, 

the Deputy Inspector General concurred with our recommendations and 

outlined proposed corrective actions for all recommendations. We agree 

with the proposed actions and consider the recommendations open with 

corrective actions ongoing. 

22 Because of Mr. Kelly’s recusal, he delegated authority for the development and 

implementation of corrective actions to the above-referenced DHS OIG senior leaders. 

www.oig.dhs.gov 15 OIG-19-41 
 

 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

APPENDIX A 

Objective, Scope, and Methodology 

DHS OIG was established by the Homeland Security Act of 2002 (Public 

Law 107-296) by amendment to the Inspector General Act of 1978. 

The objective of this review was to determine the factors that led to DHS 

OIG retracting thirteen EMOT reports regarding FEMA’s initial response 

to specific disasters. DHS OIG retracted one of these reports, FEMA’s 

Initial Response to Catastrophic Flooding in Louisiana (OIG-17-80-D), in 

July 2017 after the House Committee on Oversight and Government 

Reform raised concerns about the accuracy of the report’s findings and 

conclusions, and retracted an additional twelve reports in March 2018, 

after a preliminary quality assurance review identified similar concerns. 

To conduct this review, DHS OIG assembled an internal review team 

(Review Team) in March 2018 consisting of investigative counsel and 

analysts. Because Acting Inspector General John V. Kelly was involved in 

overseeing the EMOT work at issue, he recused himself from the matter 

and delegated authority to then-Acting Counsel Diana R. Shaw to 

conduct the internal review and help develop a corrective action plan. 

The Review Team collected and analyzed thousands of pages of emails, 

draft and final audit reports, audit work papers, and other documents 

related to the thirteen retracted reports. We also reviewed historical DHS 

OIG guidance and other documents concerning the creation and 

development of the EMOT concept. Additionally, the Review Team 

interviewed 26 current and former DHS OIG employees, including DHS 

OIG personnel involved in each of the thirteen retracted EMOT reports. 

Those interviewed included line and supervisory auditors and audit 

directors, and other current and former management in DHS OIG’s Office 

of Audits and its former Office of Emergency Management Oversight. The 

Review Team also engaged the services of an auditing firm to get an 

outside evaluation of the EMOT work at issue and independent guidance 

on industry standards and best practices related to auditing. 

This special report was prepared according to the Quality Standards for 

Federal Offices of Inspector General issued by the Council of the 

Inspectors General on Integrity and Efficiency, and reflects work 

www.oig.dhs.gov 16 OIG-19-41 
 

 

 

                                                      

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

performed between March 2018 and September 2018 pursuant to 

Sections 2-2 and 2-3 of the Inspector General Act of 1978, as amended.23 

This special report summarizes the Review Team’s findings regarding the 

factors that led to a series of deficient DHS OIG audit reports for the 

purpose of keeping the Secretary of DHS and the Congress fully and 

currently informed about problems and deficiencies relating to the 

administration of DHS programs and operations and the necessity for 

corrective action. This report is designed to promote the efficient and 

effective administration of, and to prevent and detect fraud, waste, and 

abuse in, the programs and operations of DHS. 

23 Under Section 2-2 of the Act, DHS OIG is charged with providing leadership and 

coordination and recommending policies for activities designed (A) to promote economy, 

efficiency, and effectiveness in the administration of, and (B) to prevent and detect fraud 

and abuse in DHS programs and operations. Under Section 2-3 of the Act, DHS OIG is 

charged with keeping the DHS Secretary and the Congress fully and currently informed 

about problems and deficiencies relating to the administration of DHS programs and 

operations and the necessity for and progress of corrective action. 

www.oig.dhs.gov 17 OIG-19-41 
 

 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

APPENDIX B 

DHS OIG’S MANAGEMENT RESPONSE 

www.oig.dhs.gov 18 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

www.oig.dhs.gov 19 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

www.oig.dhs.gov 20 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

www.oig.dhs.gov 21 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

www.oig.dhs.gov 22 OIG-19-41 
 

 

 OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

APPENDIX C 

Report Distribution 

Department of Homeland Security 

Acting Secretary 

Acting Deputy Secretary 

Federal Emergency Management Agency 

Acting Administrator 

Acting Deputy Administrator 

Congress 

Congressional Oversight and Appropriations Committees 

www.oig.dhs.gov 23 OIG-19-41 
Additional Information and Copies 

To view this and any of our other reports, please visit our website at: 

www.oig.dhs.gov. 

For further information or questions, please contact Office of Inspector General 

Public Affairs at: DHS-OIG.OfficePublicAffairs@oig.dhs.gov. 

Follow us on Twitter at: @dhsoig. 

OIG Hotline 

To report fraud, waste, or abuse, visit our website at www.oig.dhs.gov and click 

on the red "Hotline" tab. If you cannot access our website, call our hotline at 

(800) 323-8603, fax our hotline at (202) 254-4297, or write to us at: 

Department of Homeland Security 

Office of Inspector General, Mail Stop 0305 

Attention: Hotline 

245 Murray Drive, SW 

Washington, DC 20528-0305