U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. XXX ii DECEMBER XX, 2015

Dr Discussion aft

Audit of the Office of Compliance Inspections and 

Examinations’ Investment Adviser Examination 

Completion Process

July 21, 2017

Report No. 541
 UNITED STATES

SECURITIES AND EXCHANGE COMMISSION

WASHINGTON, D.C. 20549

 OFFICE OF

INSPECTOR GENERAL

M E M O R A N D U M

July 21, 2017

TO: Peter Driscoll, Acting Director, Office of Compliance Inspections and 

Examinations 

FROM: Carl W. Hoecker, Inspector General

SUBJECT: Audit of the Office of Compliance Inspections and Examinations’ Investment 

Adviser Examination Completion Process, Report No. 541

Attached is the Office of the Inspector General (OIG) final report detailing the results of our 

audit of the Office of Compliance Inspections and Examinations’ (OCIE) investment adviser 

examination completion process. The report contains three recommendations for corrective 

action that, if fully implemented, should improve OCIE’s internal controls over the examination 

completion process and enhance oversight of examination outcomes generally. 

On June 28, 2017, we provided management with a draft of our report for review and 

comment. In its July 14, 2017, response, management concurred with our recommendations. 

We have included management’s response as Appendix II in the final report.

Within the next 45 days, please provide the OIG with a written corrective action plan that 

addresses the recommendations. The corrective action plan should include information such 

as the responsible official/point of contact, timeframe for completing required actions, and 

milestones identifying how the agency will address the recommendations.

We appreciate the courtesies and cooperation extended to us during the audit. If you have 

questions, please contact me or Rebecca L. Sharek, Deputy Inspector General for Audits, 

Evaluations, and Special Projects.

Attachment

cc: Jay Clayton, Chairman

Lucas Moskowitz, Chief of Staff, Office of Chairman Clayton 

Sean Memon, Deputy Chief of Staff, Office of Chairman Clayton 

 Peter Uhlmann, Managing Executive, Office of Chairman Clayton

Michael S. Piwowar, Commissioner 

Richard Grant, Counsel, Office of Commissioner Piwowar 

Kara M. Stein, Commissioner

Robert Peak, Advisor to the Commissioner, Office of Commissioner Stein 

Robert B. Stebbins, General Counsel 

Bryan Wood, Director, Office of Legislative and Intergovernmental Affairs 
Mr. Driscoll

July 21, 2017

Page 2 

John J. Nester, Director, Office of Public Affairs

Rick A. Fleming, Investor Advocate

Daniel Kahl, Chief Counsel, Office of Compliance Inspections and Examinations

Robert Fisher, Managing Executive, Office of Compliance Inspections and 

Examinations 

Darlene L. Pryor, Management and Program Analyst, Office of the Chief Operating 

Officer
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

i 

d

Executive Summary Audit of the Office of Compliance Inspections 

and Examinations’ Investment Adviser 

Examination Completion Process

Report No. 541

July 21, 2017

Why We Did This Audit 

The U.S. Securities and Exchange 

Commission’s (SEC) National 

Examination Program (NEP), 

conducted by the Office of 

Compliance Inspections and 

Examinations (OCIE), is risk-based 

and data-driven. As part of the NEP,

OCIE examines SEC-registered 

entities, including about 

12,000 investment advisers (IAs). 

According to the SEC’s Fiscal Year 

(FY) 2016 Agency Financial Report, 

“OCIE uses the findings from these 

examinations to improve industry 

compliance, detect and prevent 

fraud, inform policy, and identify 

risks.” We initiated this audit to 

assess the controls over OCIE’s IA 

examination completion process, an

to follow-up on prior Office of 

Inspector General (OIG) 

recommendations. 

What We Recommended

We recommended that OCIE 

(1) design control activities related to 

the review and approval of 

examination work products to require

adequate segregation of duties, 

(2) update NEP policies and 

procedures to more clearly define the 

requirements for documenting in

TRENDS examination meetings and 

interviews, and (3) develop and 

disseminate to OCIE staff guidance 

for assigning final examination risk 

ratings before closing examinations. 

Management concurred with the 

recommendations, which will be 

closed upon completion and 

verification of corrective action. 

What We Found 

We found that controls over OCIE’s IA examination completion 

process are generally effective but improvements are needed. We 

reviewed documentation from all IA Corrective Action Reviews 

(CARs) OCIE approved between FYs 2015 and 2016 and closed in 

the Tracking and Reporting Examination National Documentation 

System (TRENDS) as of November 22, 2016. We also reviewed 

documentation from a statistical sample of 240 of the 2,443 IA 

examinations OCIE approved and closed in TRENDS during the 

same period. We did not find any deficiencies related to the IA 

CARs we reviewed. Moreover, we determined that OCIE has 

addressed prior OIG recommendations. However, we also 

identified deficiencies in OCIE’s IA examination completion controls 

that warrant management’s attention. Specifically, we found that:

• two IA examination completion controls regarding control sheets 

and post-exam fieldwork lacked adequate segregation of duties; 

• examiners did not always document preliminary exit interviews 

with examined IAs; and

• examiners either did not assign final risk ratings, or may have 

assigned final risk ratings inconsistently.

These deficiencies occurred because sufficiently robust policies and 

controls were not in place to prevent their occurrence. If OCIE does 

not appropriately review and consistently document IA examination 

results and risk assessments (1) examination work products may be 

more susceptible to error, (2) OCIE examiners’ ability to sufficiently 

review prior examination findings and perform comprehensive risk 

assessments may be reduced, and (3) OCIE may not effectively 

consider the results of examinations during its evaluation of risk for 

future examinations. OCIE can improve its IA examination 

completion process and internal controls by updating or 

documenting policies and procedures consistent with the Standards 

for Internal Control in the Federal Government. 

During the audit, we also inquired about the status of 

(1) recommendations OCIE received in November 2016 from a 

consultant’s efficiency study, and from an internal steering 

committee; and (2) plans to apply to the NEP the Government 

Accountability Office’s Risk-Management Framework. We 

discussed with OCIE management, including the Acting Director, 

these other matters of interest, which did not warrant 

recommendations. We will continue to monitor these matters, as 

needed.

For additional information, contact the Office of Inspector General at 

(202) 551-6061 or http://www.sec.gov/oig. 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 ii JULY 21, 2017 

TABLE OF CONTENTS

Executive Summary ..................................................................................................... i 

Background and Objectives....................................................................................... 1 

Background .................................................................................................................. 1 

Objectives .................................................................................................................... 1 

Results ......................................................................................................................... 3 

Controls Over OCIE’s IA Examination Completion Process Are Generally 

Effective But Improvements Are Needed...................................................................... 3 

Recommendations, Management’s Response, and Evaluation of Management’s 

Response ............................................................................................................... 7 

Other Matters of Interest............................................................................................. 9 

Appendices

Appendix I. Scope and Methodology ......................................................................... 11 

Appendix II. Management Comments........................................................................ 15 

ABBREVIATIONS

CAR Corrective Action Review

DAS Data and Analytic Solutions, Inc.

ERP Enterprise Resource Performance, Inc.

FY fiscal year

GAO U.S. Government Accountability Office 

IA investment adviser

NEP National Examination Program

OCIE Office of Compliance Inspections and Examinations

OIG Office of Inspector General

SEC or agency U.S. Securities and Exchange Commission

TRENDS Tracking and Reporting Examination National Documentation 

System
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 1 JULY 21, 2017

Background and Objectives 

Background 

The Office of Compliance Inspections and Examinations’ (OCIE) conducts the U.S. 

Securities and Exchange Commission’s (SEC or agency) National Exam Program 

(NEP). The NEP’s mission is to protect investors, ensure market integrity, and support 

responsible capital formation through risk-focused strategies that (1) improve 

compliance with Federal securities laws, (2) prevent fraud, (3) monitor risk, and 

(4) inform the SEC’s regulatory policy. Under the NEP, OCIE uses a risk-based, datadriven process to select entities for examination. Examiners in Washington, DC, and 

the SEC's 11 regional offices (examination staff) are charged with overseeing about 

26,000 market participants. These include about 12,000 investment advisers (IAs), 

about 10,000 mutual funds and exchange-traded funds, more than 4,000 brokerdealers, 400 transfer agents, and self-regulatory organizations including 21 securities 

exchanges, 7 active registered clearing agencies, and the Financial Industry Regulatory 

Authority. In addition, OCIE conducts a limited number of Corrective Action Reviews 

(CARs) to verify whether examined entities took corrective actions in response to 

deficiencies identified during previous OCIE examinations. Because IA’s represent the 

largest segment of the population examined by OCIE, OCIE allocates the majority of its 

examination resources to its IA program.

To standardize examination processes and documentation and to provide guidance and 

requirements for examination staff, OCIE management established NEP policies and 

procedures. In addition, OCIE management implemented the Tracking and Reporting 

Examination National Documentation System (TRENDS), which examination staff use 

to document work performed and deficiencies identified during examinations. NEP 

policies and procedures and TRENDS contribute to OCIE’s control environment and 

provide a foundation of required processes and automated controls that facilitate 

examination completion.

Objectives 

Our objective was to determine whether OCIE established effective controls over its IA 

examination completion process, including but not limited to the issuance and resolution 

of deficiency letters and the performance of CARs, to improve compliance with Federal 

securities laws, prevent fraud, inform policy, and monitor risk. Specifically, we sought to 

determine whether OCIE: 

1. established effective controls (including systems, policies, and procedures) to 

complete examinations and resolve identified deficiencies;

2. established effective controls (including systems, policies, and procedures) to 

perform and document CARs; and
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 2 JULY 21, 2017

3. effectively used findings from examinations and CARs as part of its risk-based, 

data-driven examination selection process.

We also reviewed OCIE’s planned and completed corrective actions resulting from a 

prior Office of Inspector General (OIG) evaluation.

1

Our audit covered fiscal years (FYs) 2015 and 2016 (the period between October 1, 

2014, and September 30, 2016). Because the majority of OCIE’s examination 

resources are allocated to its IA program, we focused on IA CARs and examinations. 

Specifically, our universe consisted of IA CARs and IA examinations approved in FYs 

2015 and 2016 that were closed in TRENDS as of November 22, 2016. To address our 

objectives, we reviewed: 

• NEP policies and procedures, OCIE’s Risk and Control Matrix, and TRENDS 

guidance used to complete examinations and resolve deficiency letters; 

• TRENDS documentation from all IA CARS and from a statistical sample of 240 of 

the 2,443 IA examinations approved and closed in TRENDS during the period 

reviewed to determine compliance with requirements, including requirements for 

issuing and resolving deficiency letters; and

• OCIE’s use of examination data in its risk-based, data-driven examination 

selection process and relevant examination planning documents and guidance. 

Appendix I includes additional information on our scope and methodology, our review of 

internal controls, and prior coverage.

 

1 U.S. Securities and Exchange Commission, Office of Inspector General, Office of Compliance 

Inspections and Examinations’ Management of Investment Adviser Examination Coverage Goals (Report 

No. 533; March 10, 2016). 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 3 JULY 21, 2017

Results

Controls Over OCIE’s IA Examination Completion Process Are 

Generally Effective But Improvements Are Needed

We did not find any deficiencies related to the IA CARs approved in FYs 

2015 and 2016 and closed in TRENDS as of November 22, 2016. 

Moreover, we determined that OCIE has addressed recommendations

made in OIG Report No. 533. However, we tested each of the 240 IA 

examinations we reviewed for compliance with 16 IA examination 

completion requirements established in OCIE policies and procedures2

and identified deficiencies that warrant management’s attention. 

Specifically, although 12 of the 16 controls we tested (or 75 percent) were 

sufficiently designed and operating effectively, we found that: 

• 2 IA examination completion controls lacked adequate segregation 

of duties; 

• examiners did not always document preliminary exit interviews with 

examined IAs; and

• examiners either did not assign final risk ratings, or may have 

assigned final risk ratings inconsistently. 

As described further below, these deficiencies occurred because 

sufficiently robust policies and controls were not in place to prevent their 

occurrence. If OCIE does not appropriately review and consistently

document IA examination results and risk assessments (1) examination 

work products may be more susceptible to error, (2) OCIE examiners’ 

ability to sufficiently review prior examination findings and perform 

comprehensive risk assessments may be reduced, and (3) OCIE may not 

effectively consider the results of examinations during its evaluation of 

risk for future examinations. OCIE can improve its IA examination 

completion process and internal controls by updating or documenting 

policies and procedures consistent with the Standards for Internal Control 

in the Federal Government. 

Two Controls Lacked Adequate Segregation of Duties. NEP policies and 

procedures require that examination staff document their actions, analysis, 

observations, and conclusions in TRENDS control sheets. Control sheets are 

organized by risk areas and contain fields where OCIE staff can enter examination 

 

2 We determined the 16 IA examination completion requirements by reviewing and selecting testing 

attributes from OCIE’s Risk and Control Matrix, NEP policies and procedures, and TRENDS guidance. 

For OCIE’s Risk and Control Matrix, we included testing attributes from 6 of 97 control activities. 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 4 JULY 21, 2017

findings, sampling logic, key procedures performed, general comments, and control 

sheet approval information. NEP policies and procedures further require that 

examination staff submit control sheets to an Exam Manager or Assistant Director for 

review. Exam Manager or Assistant Director approval is also required for postfieldwork completion steps documented in TRENDS, including control sheet review, 

final risk rating selection, and finalization of examination findings. Nonetheless, we 

found a lack of segregation of duties for the submission and review of TRENDS control 

sheets and post-fieldwork completion information. 

Specifically, for 58 of the 240 IA examinations we reviewed (or about 24 percent), an 

OCIE supervisor3 both submitted and reviewed in TRENDS at least one examination 

control sheet. In addition, OCIE supervisors both submitted and approved postfieldwork completion information in TRENDS for 56 of the 240 IA examinations we 

reviewed (or about 23 percent). Therefore, we estimate that about 600 of the 2,443 IA 

examinations approved in FYs 2015 and 2016 (or about 25 percent) lacked adequate 

segregation of duties in at least one of these two areas.4

 In October 2016, the U.S. 

Government Accountability Office (GAO) also reported OCIE’s lack of segregation of 

duties related to TRENDS workbook approval.5 GAO found that, in 11 of the 92 exams 

it tested, the same Exam Manager both submitted and approved examination

workbooks. 

According to GAO’s Standards for Internal Control in the Federal Government,

6

management considers segregation of duties in designing control activity 

responsibilities so that incompatible duties are segregated and, where such 

segregation is not practical, management designs alternative control activities to 

address the risk. Moreover, management considers the need to separate control 

activities related to authority, custody, and accounting of operations to achieve 

adequate segregation of duties. In accordance with these principles, mitigating control 

activities should be used when segregation cannot be obtained or is not practical. 

Although NEP policies and procedures require that examination staff use TRENDS 

control sheets to document their actions, analysis, observations, and conclusions, 

neither NEP policies and procedures nor TRENDS prohibits OCIE supervisors from 

reviewing and approving control sheets and post-fieldwork completion information that 

they created. According to OCIE management, the lack of segregation of duties is 

 

3 OCIE supervisors include both Exam Managers and Assistant Directors.

4 We estimate that supervisors both submitted and reviewed at least one control sheet for as many as 

619 IA examinations. We also estimate that supervisors both submitted and approved post-fieldwork 

completion information in TRENDS for as many as 598 IA examinations.

5 U.S. Government Accountability Office, Management Has Enhanced Supervisory Controls and Could 

Further Improve Efficiency (GAO-17-16, October 2016).

6 U.S. Government Accountability Office, Standards for Internal Control in the Federal Government (GAO14-704G, September 2014).
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 5 JULY 21, 2017

mitigated by a requirement that an Assistant Director sign each examination disposition 

letter, which indicates overall approval of the entire examination. Therefore, OCIE 

management supports the current practice and values the flexibility it provides. 

Because OCIE supervisors submitted examination control sheets and post-fieldwork 

status information that they also reviewed and approved,

7 control activities in these 

areas are not operating as effectively as they should. Without adequate segregation of 

duties in examination review and approval processes, IA examination work products 

may be more susceptible to error. Undue risk is created when workpapers that support 

examination conclusions are not directly reviewed by an OCIE supervisor, as intended 

by the design of the control activities and NEP policies and procedures. We believe 

that segregation of the review and approval of control sheets and post-fieldwork 

completion information is practical within OCIE’s NEP operations.

Examiners Did Not Always Document Preliminary Exit Interviews. OCIE 

examiners conduct preliminary exit interviews to inform the examined entity of the next 

steps in the examination process, as well as to discuss preliminary findings or concerns 

that require immediate attention. The preliminary exit interview is generally held on the 

last day of the on-site review. We found that examiners who conducted 133 of the 

240 IA examinations we reviewed (or about 55 percent) did not document preliminary 

exit interviews. Therefore, we estimate that as many as 1,327 of the 2,443 IA 

examinations approved in FYs 2015 and 2016 (or about 54 percent) lacked 

documentation of preliminary exit interviews. 

According to NEP policies and procedures, notes from interviews are considered 

examination workpapers, and examination staff should store all workpapers in TRENDS 

or in an approved off-TRENDS storage site. NEP policies and procedures also state 

that the examination team, in consultation with their supervisors, should determine the 

appropriate note-taking protocol for the examination. Generally, at least one member of 

the examination team should take notes of interviews, and all notes taken should be 

maintained with the workpapers and should include the date of the interview, the name 

of participants, and the matters discussed. 

Although NEP policies and procedures require examiners to retain interview notes in the 

examination workpapers, the policies and procedures do not specifically address 

documentation requirements for preliminary exit interviews. Also, TRENDS guidance

does not specify expectations for the “Exit Interview” fields in TRENDS, and TRENDS 

allows examination staff to close examinations without including preliminary exit

information in these fields.8

 We determined that examiners primarily use the TRENDS 

“Exit Interview” fields to document the exit conference call, which is a separate meeting 

 

7 In TRENDS, the control sheet approval history indicates a reviewed status when complete, while the 

post-fieldwork completion history indicates an approved status when complete.

8 The TRENDS “Exit Interview” fields document whether an exit interview was conducted, the exit 

interview date, and a narrative of the examiners’ “Interview Comments.” 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 6 JULY 21, 2017

typically held after the examination findings have been determined and approved by an 

OCIE supervisor. According to OCIE management, regional offices have varying 

practices with respect to whether or not examiners conduct preliminary exit interviews 

and how, if at all, to document such interviews. 

However, unclear guidance and expectations can lead to inconsistent examination 

documentation among OCIE examination staff. Key interviews that support examination 

conclusions, including identification of deficiencies, may not be documented in 

TRENDS, which could reduce OCIE examiners’ ability to sufficiently review prior 

examination findings and perform comprehensive risk assessments. Additionally, 

examiners may be unable to recall certain details or conclusions reached from the 

preliminary exit interview, which may inhibit their ability to support discussions held with 

the examined entity in the future, if needed. 

Examiners Did Not Always Assign Final Examination Risk Ratings or May Have 

Assigned Them Inconsistently. Final examination risk ratings are one of many 

components OCIE uses in its risk-based, data-driven examination selection process. 

However, we found that examiners did not assign final risk ratings for 23 of the 240 IA 

examinations we reviewed (or about 10 percent). Therefore, we estimate that 

examiners did not assign final risk ratings for as many as 226 of the 2,443 IA 

examinations approved in FYs 2015 and 2016 (or about 9 percent). Moreover, those 

examiners who assigned final risk ratings in FYs 2015 and 2016 may have done so 

inconsistently. 

According to OCIE management, before May 2016, OCIE management and TRENDS 

system controls did not require examiners to assign a final risk rating before closing an 

examination. We confirmed this assertion through our testing. Specifically, the 23 IA 

examinations we reviewed that were missing final risk ratings were closed before May 

2016. All IA examinations we reviewed that were closed after May 2016 included final 

risk ratings. Additionally, we confirmed that the current TRENDS guidance states that 

final risk rating information must be included in TRENDS before closing an examination. 

Although the current version of TRENDS ensures examiners must assign a final risk 

rating before closing each examination, NEP policies and procedures do not provide 

examiners with guidance for assigning final risk ratings. Likewise, TRENDS guidance

does not direct examiners on how to complete this step, other than to “select a final risk 

rating for the registrant from the drop down [menu].” According to OCIE management, 

examination staff assign final risk ratings based on their judgment, experience, and the 

findings of the examination. 

However, prior examination results can influence OCIE’s risk-based, data-driven 

examination selection process, and the lack of guidance and instruction provided to 

examiners regarding final risk rating selection may have contributed to incomplete or 

inconsistent final IA examination risk ratings in FYs 2015 and 2016. Specifically, 

according to personnel from OCIE’s Office of Risk Analysis and Surveillance, OCIE’s 

risk assessment process is based, in part, on prior OCIE examinations. Personnel from 

the Office of Risk Analysis and Surveillance analyze prior examination results along with 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 7 JULY 21, 2017

other factors, including annual NEP priorities and data from regulatory filings, to develop 

an initial risk profile which constitutes several risk-based outputs for each SECregistered IA (for example, inherent risk, workforce risk, and controls-based risk). The 

Office of Risk Analysis and Surveillance then prepares annual IA examination planning 

information by IA, which is accessible to all OCIE examiners through OCIE’s intranet. 

Therefore, a continued lack of guidance regarding assigning final examination risk 

ratings could lead to inconsistent final risk ratings in the future, which could negatively 

impact OCIE’s ability to assess and select IAs for future examination. 

Recommendations, Management’s Response, and Evaluation of 

Management’s Response

To improve OCIE’s internal controls over the IA examination completion process, we 

recommend that OCIE:

Recommendation 1: Design control activities related to the review and approval of 

examination work products to require segregation of duties, and update NEP policies 

and procedures and TRENDS system controls and guidance, as needed, to reflect this 

requirement.

Management’s Response. OCIE’s Acting Director concurred with the 

recommendation and stated that OCIE has controls in place to ensure Assistant 

Directors review important aspects of the examination from pre-examination 

fieldwork to citing a regulated entity for deficiencies. The Acting Director further 

stated that OCIE will strengthen other controls, as well as update its policies and 

procedures and TRENDS, to ensure adequate segregation of duties at intermediate 

phases of the examination process. Management’s complete response is reprinted 

in Appendix II. 

OIG’s Evaluation of Management’s Response. Management’s proposed actions 

are responsive; therefore, the recommendation is resolved and will be closed upon 

verification of the action taken. 

Recommendation 2: Update NEP policies and procedures to more clearly define the 

requirements for documenting in TRENDS examination meetings and interviews, 

including preliminary exit interviews, and make corresponding revisions to the TRENDS 

system controls and guidance, as needed. 

Management’s Response. OCIE’s Acting Director concurred with the 

recommendation and stated that OCIE senior management should more clearly 

define expectations for examination staff when it comes to conducting and 

documenting preliminary exit interviews. The Acting Director also acknowledged

that varying practices exist among regional offices due to a lack of clearly articulated 

policies and procedures. Management’s complete response is reprinted in 

Appendix II.
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 8 JULY 21, 2017

OIG’s Evaluation of Management’s Response. Management’s proposed actions 

are responsive; therefore, the recommendation is resolved and will be closed upon 

verification of the action taken. 

Recommendation 3: Develop and disseminate guidance for assigning final 

examination risk ratings, and notify all OCIE examination staff of the requirement and 

importance of selecting final examination risk ratings before closing examinations. 

Management’s Response. OCIE’s Acting Director concurred with the

recommendation and stated that OCIE’s Exam Process Steering Committee 

reviewed the use of final examination risk ratings and, in consultation with OCIE 

senior management, will recommend guidance for assigning final examination risk 

ratings. Management’s complete response is reprinted in Appendix II. 

OIG’s Evaluation of Management’s Response. Management’s proposed actions 

are responsive; therefore, the recommendation is resolved and will be closed upon 

verification of the action taken. 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 9 JULY 21, 2017

Other Matters of Interest

During the audit, we inquired about the status of (1) recommendations OCIE received in 

November 2016 from a consultant’s efficiency study, and from an internal OCIE Exam 

Process Steering Committee; and (2) plans to apply to the IA program GAO’s RiskManagement Framework. We discussed with OCIE management, including the Acting 

Director, these other matters of interest, which did not warrant recommendations. 

Consultant’s Study on Efficiency in the NEP. Under contract with the SEC, 

Enterprise Resource Performance, Inc. (ERP) conducted a 12-month assessment of 

the NEP and identified opportunities to increase the NEP’s effectiveness and 

efficiency. ERP’s final report, dated November 16, 2016, included recommendations 

for potential program enhancements related to OCIE’s organizational structure, 

workload management, examination staffing, and other NEP processes.

SEC Chairman, Jay Clayton, was sworn in on May 4, 2017. OCIE management has 

provided to Chairman Clayton’s staff ERP’s report, and had previously briefed the 

Acting Chairman on the report’s contents. As of the date of this audit report, OCIE 

management’s review and consideration of the ERP report recommendations was 

ongoing. 

Exam Process Steering Committee Recommendations. In February 2016, OCIE’s 

Exam Process Steering Committee9 surveyed OCIE regional offices to study registrant 

risk analysis and exam selection practices. After reviewing the survey information, the 

Committee conducted follow-up interviews with each regional office in April 2016. On 

November 17, 2016, the Committee provided OCIE management a final report, which 

included five recommendations.

OCIE accepted the Committee’s recommendations and plans to complete necessary 

updates to NEP policies and procedures by the end of FY 2017. Implementation of the 

recommendations should strengthen OCIE’s candidate selection process and ensure 

regional offices follow minimum common procedures and documentation requirements, 

which should improve consistency and enhance OCIE’s overall NEP control 

environment.

Implementation of GAO’s Risk-Management Framework. In response to a 

recommendation from OIG Report No. 533, OCIE engaged a consultant (IBM 

Corporation) to better understand how each of the five components of GAO’s Risk- 

 

9 The Exam Process Steering Committee was formerly known as the Risk and Exam Process Steering 

Committee.
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 10 JULY 21, 2017

Management Framework10 could be applied to the NEP. OCIE awarded the contract to 

IBM Corporation in September 2016, and expects the project to be completed in early 

2018. 

As needed, we will continue to monitor and report on how OCIE prioritizes and takes 

positive steps to (1) assess and implement recommendations from ERP’s efficiency 

study and from the internal Exam Process Steering Committee, and (2) consider the 

application of GAO’s Risk-Management Framework to the NEP. 

 

10 U.S. Government Accountability Office, Further Refinements Needed to Assess Risks and Prioritize 

Protective Measures at Ports and Other Critical Infrastructure (GAO-06-91, December 2005).
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 11 JULY 21, 2017

Appendix I. Scope and Methodology

We conducted this performance audit from October 2016 through July 2017 in 

accordance with generally accepted government auditing standards. Those standards 

require that we plan and perform the audit to obtain sufficient, appropriate evidence to 

provide a reasonable basis for our findings and conclusions based on our audit 

objectives. We believe that the evidence obtained provides a reasonable basis for our 

findings and conclusions based on our audit objectives.

Scope. Our objective was to determine whether OCIE established effective controls 

over its IA examination completion process, including but not limited to the issuance and 

resolution of deficiency letters and the performance of CARs, to improve compliance 

with Federal securities laws, prevent fraud, inform policy, and monitor risk. Specifically, 

we sought to determine whether OCIE: 

1. established effective controls (including systems, policies, and procedures) to 

complete examinations and resolve identified deficiencies;

2. established effective controls (including systems, policies, and procedures) to 

perform and document CARs; and

3. effectively used findings from examinations and CARs as part of its risk-based, 

data-driven examination selection process.

We also reviewed the implementation of corrective actions from a prior OIG evaluation 

titled, Office of Compliance Inspections and Examinations’ Management of Investment 

Adviser Examination Coverage Goals (Report No. 533; March 10, 2016).

The audit covered FYs 2015 and 2016 (the period between October 1, 2014, and 

September 30, 2016). Because the majority of OCIE’s examination resources are 

allocated to its IA program, we focused on IA CARs and examinations. Specifically, as 

discussed further below, our universe consisted of IA CARs and IA examinations 

approved in FYs 2015 and 2016 and closed in TRENDS as of November 22, 2016.11 

Methodology. We performed fieldwork at the SEC’s Headquarters in Washington, DC. 

To address our audit objectives, we reviewed: 

• NEP policies and procedures, OCIE’s Risk and Control Matrix, and TRENDS 

guidance used to complete examinations and resolve deficiency letters; 

 

11 OCIE considers an examination approved once the examination fieldwork is reviewed and approved in 

TRENDS and the disposition letter is signed and issued to the examined entity. An approved 

examination may be in “pending response” status after the disposition letter is issued. To ensure we 

tested examinations that were complete in TRENDS, we included only IA examinations approved in FYs 

2015 and 2016 with a “closed” status in TRENDS as of November 22, 2016. 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 12 JULY 21, 2017

• Federal securities laws related to the SEC’s authority to conduct inspections and 

examinations; 

• TRENDS documentation from all IA CARS and from a statistical sample of 240 of 

the 2,443 IA examinations approved and closed in TRENDS during the period 

reviewed to determine compliance with requirements, including requirements for 

issuing and resolving deficiency letters; and

• the corrective actions taken and planned to address the recommendations made 

in OIG Report No. 533. 

We also:

• interviewed officials from OCIE’s NEP and Office of Chief Counsel to gain an 

understanding of the NEP and to discuss the IA examination and CAR test 

attributes we identified; 

• interviewed officials from OCIE’s Office of Risk Analysis and Surveillance to gain 

an understanding of how OCIE uses examination data in its risk-based, datadriven examination selection process; 

• confirmed OCIE’s use of examination data in its risk-based, data-driven 

examination selection process by reviewing relevant examination planning 

documents and guidance;

• conducted a walkthrough of TRENDS; and

• assessed the reliability of relevant TRENDS data.

We engaged a contractor—Data and Analytic Solutions, Inc. (DAS)—to develop our 

sampling methodology. The sample size was determined using a desired confidence 

level of 95 percent, a tolerance rate of 50 percent, and a presumed error rate of 

±6 percent. As of November 22, 2016, OCIE approved and closed in TRENDS 

2,443 IA examinations for FYs 2015 and 2016. Using the sampling methodology 

developed by DAS, we selected a statistical sample of 240 of these 2,443 IA 

examinations, stratified by year, SEC regional office, and examination disposition 

type.12 Therefore, the sample included randomly selected IA examinations of different 

examination disposition types from each regional office population in each year. We 

tested each sampled IA examination for compliance with 16 IA examination completion 

requirements. We determined these 16 IA examination completion requirements by 

reviewing and selecting testing attributes from OCIE’s Risk and Control Matrix, NEP 

 

12 The statistical sample ensured that all SEC regional offices were represented for both FYs under 

review, and that each examination disposition type (Deficiency Letter, Close-Out Letter, and No-FurtherAction Letter) was represented in each regional office sample. 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 13 JULY 21, 2017

policies and procedures, and TRENDS guidance. For OCIE’s Risk and Control Matrix, 

we included testing attributes from 6 of 97 control activities. 

We tested all IA CAR examinations for FYs 2015 and 2016, which were approved and 

closed in TRENDS as of November 22, 2016, for compliance with the five examination 

requirements established in NEP policies and procedures that are unique to CAR 

examinations. 

Internal Controls. To assess internal controls relative to our objectives, we reviewed 

OCIE’s management assurance statements and risk assessments for FYs 2015 and 

2016. In its FY 2015 management assurance statement, OCIE management reported 

that it tested 26 control activities to evaluate the design and effectiveness of OCIE’s

internal controls. OCIE management identified areas requiring improvement but 

reported that the areas did not create the risk of material weakness. OCIE 

management concluded that its controls and processes in place provide reasonable 

assurance that the NEP’s operations and programs are effective and efficient. 

Additionally, in its FY 2016 management assurance statement, OCIE management 

reported testing 21 control activities to evaluate the design and effectiveness of OCIE’s 

internal controls. OCIE management reported no material weaknesses in controls and 

concluded that the controls and processes in place provided reasonable assurance that 

the NEP’s operations and programs were effective and efficient. 

We assessed OCIE’s control environment in accordance with GAO’s Standards for 

Internal Control in the Federal Government. We tested key internal controls associated

with OCIE’s IA examination completion process and requirements specific to IA CARs. 

Specifically, as part of our review of select IA examinations and CARs, we assessed 

OCIE's controls related to (1) management review of examination work products, 

(2) documentation of examination interviews and meetings, (3) TRENDS review and 

approval processes, (4) documentation of examination findings and disposition letters, 

and (5) selection of final examination risk ratings. We found that controls over the IA 

examination completion process were generally effective in supporting OCIE’s mission, 

but noted that improvements should be made. As stated in this report, we identified 

opportunities for OCIE to improve how management and staff review examination 

fieldwork, document preliminary exit interviews, and select and assign final examination 

risk ratings. Our recommendations, if implemented, should strengthen OCIE’s control 

environment and improve the IA examination completion process. 

Computer-processed Data. GAO’s Assessing the Reliability of Computer-Processed 

Data (GAO-09-680G, July 2009) states that “data reliability refers to the accuracy and 

completeness of computer-processed data, given the uses they are intended for. 

Computer-processed data may be data (1) entered into a computer system or 

(2) resulting from computer processing.” Furthermore, GAO-09-680G defines 

“reliability,” “completeness,” and “accuracy” as follows: 

• “Reliability” means that data are reasonably complete and accurate, meet your 

intended purposes, and are not subject to inappropriate alteration. 
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 14 JULY 21, 2017

• “Completeness” refers to the extent that relevant records are present and the 

fields in each record are appropriately populated.

• “Accuracy” refers to the extent that recorded data reflect the actual underlying 

information. 

To address our objectives, we relied on examination data from OCIE’s TRENDS. To 

assess the reliability, completeness, and accuracy of the data from TRENDS, we 

interviewed OCIE personnel, obtained access to TRENDS, and conducted walkthroughs to determine the system’s capabilities. We used TRENDS reports to perform 

limited reliability testing and reviewed TRENDS information system controls, as well as

security assessment and contingency planning documents. We determined that the 

TRENDS report data was sufficiently reliable to address our audit objectives. 

Prior Coverage. During 2016, the SEC OIG and GAO issued the following two reports 

of particular relevance to this audit: 

SEC OIG: 

• Office of Compliance Inspections and Examinations’ Management of Investment 

Adviser Examination Coverage Goals (Report No. 533, March 10, 2016) 

GAO:

• Management Has Enhanced Supervisory Controls and Could Further Improve 

Efficiency (GAO-17-16, October 2016) 

These reports can be accessed at: 

https://www.sec.gov/about/offices/oig/inspector_general_audits_reports.shtml (SEC 

OIG) and https://www.gao.gov (GAO).
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 15 JULY 21, 2017

Appendix II. Management Comments
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 16 JULY 21, 2017
U.S. SECURITIES AND EXCHANGE COMMISSION OFFICE OF INSPECTOR GENERAL

REPORT NO. 541 17 JULY 21, 2017

Major Contributors to the Report

Colin Heffernan, Audit Manager

John Gauthier, Lead Auditor

Michael Gainous, Auditor

To Report Fraud, Waste, or Abuse, Please Contact:

Web: www.reportlineweb.com/sec_oig 

Telephone: (877) 442-0854 

Fax: (202) 772-9265

Address: U.S. Securities and Exchange Commission

Office of Inspector General

100 F Street, N.E.

Washington, DC 20549

Comments and Suggestions 

If you wish to comment on the quality or usefulness of this report or suggest ideas for 

future audits, evaluations, or reviews, please send an e-mail to OIG Audit Planning at 

AUDplanning@sec.gov. Comments and requests can also be mailed to the attention of 

the Deputy Inspector General for Audits, Evaluations, and Special Projects at the 

address listed above.