A Framework for the Regulatory use of Penetration Testing in
the Financial Services Industry

March 2018

1 Table of Contents
Disclaimer ..................................................................................................................................................... 2
Executive Summary ...................................................................................................................................... 3
Contributing Organizations .......................................................................................................................... 6
Introduction .................................................................................................................................................. 7
Background .................................................................................................................................. 7
Purpose of this Framework ......................................................................................................... 8
Testing Options ............................................................................................................................ 8
Testing Lifecycle........................................................................................................................... 9
1.4.1

Threat Intelligence Phase ...................................................................................................... 9

1.4.2

Planning Phase .................................................................................................................... 10

1.4.3

Testing Phase ...................................................................................................................... 10

1.4.4

Analysis and Response Phase.............................................................................................. 10
Regulatory Role ......................................................................................................................... 10

The Testing Lifecycle .................................................................................................................................. 112

Threat Intelligence ............................................................................................................................. 11
Scenario Development .............................................................................................................. 11
Select and Prioritize Testing Scenarios ...................................................................................... 11
Validation................................................................................................................................... 12
Maintenance.............................................................................................................................. 12

3

Planning Phase ................................................................................................................................... 13
Project Management ................................................................................................................. 13
Risk Management ...................................................................................................................... 14
Scoping ...................................................................................................................................... 14
Testing Options .......................................................................................................................... 15
Timing of Tests........................................................................................................................... 15
Rules of Engagement (ROE) ....................................................................................................... 15
Resourcing / Qualifications........................................................................................................ 16

4

Testing Phase ...................................................................................................................................... 18
Operational Planning ................................................................................................................. 18
Execution ................................................................................................................................... 20
Review ....................................................................................................................................... 21

5

Analysis and Response Phase ............................................................................................................ 221

Analysis ...................................................................................................................................... 22
Response.................................................................................................................................... 23
Notification ................................................................................................................................ 23
Reporting ................................................................................................................................... 23
Data Protection.......................................................................................................................... 24
Distribution ................................................................................................................................ 24
6

Conclusion .......................................................................................................................................... 25

7

Glossary .............................................................................................................................................. 26

Disclaimer
These materials are for general informational purposes only, and are not intended to provide, and do
not constitute, investment, tax, business or legal advice to any individual or entity. The views andopinions expressed in these materials are solely those of the authors and do not necessarily reflect theofficial policy or position of GFMA, SIFMA, AFME, ASIFMA, or their employees, or members. We makeno representations warranties or guarantees, expressed or implied, that the information contained
herein is up-to-date, accurate, or complete, and we have no obligation to update, correct, or
supplement this information, or to otherwise notify you, in the event that any such information is or
becomes outdated, inaccurate, or incomplete. To the fullest extent permitted by law, we expressly
disclaim all warranties of any kind, whether expressed or implied, including but not limited to implied
warranties of merchantability, fitness for a particular purpose, title, and non-infringement. Reference
herein to any specific commercial product, process, or service by trade name, trademark, manufacturer,
or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by
GFMA, SIFMA, AFME, ASIFMA, or their employees, or members.
2

Executive Summary
Cybersecurity is a major priority for the financial services industry. Penetration testing and red teaming(hereafter “testing”) serves as one of the foremost tools in enabling a robust security program within afinancial institution. Such testing allows firms to evaluate their systems and the controls that protect
them in order to identify and remediate vulnerabilities, thereby strengthening their infrastructure andorganization against cyber threats.
Increased interest by global regulators has led to the proliferation of regulatory-mandated testing
initiatives. While these tests are understandably important pieces of regulatory oversight, they canpresent risks to firms and the firms’ clients if the test results become public or are inadvertently
disclosed or stolen. This is especially true if penetration testing is not approached in a collaborative andcoordinated manner.
Towards that end, the Industry recently published a set of principles to harmonize the growing
regulatory demand for penetration testing and red-teaming. The shared goals are to:
•

•

•

•

Provide regulators the ability to guide penetration testing and red teaming programs to meetsupervisory objectives through use of scenarios based on current risks that drive scheduling andscoping of testing activities
Provide regulators with a high degree of confidence that testing is conducted by trained,
certified and qualified personnel with sophisticated tools that can accurately emulate
adversaries, as required
Provide regulators transparency into the testing process and results for both regulator-driven
and firm-driven testing as well as assurance that firm governance identifies and properly
addresses weaknesses
Ensure testing activities are conducted in a manner that minimizes operational risks and ensuresdata security by including strict protocols for distributing test data and results

Building off of these principles, this Framework is designed to create an agreed upon approach forregulators and financial services firms to conduct effective testing to satisfy both supervisory and firmoriginated requirements. The Framework’s objectives are to:▪

Engage regulators globally with a common framework to facilitate open dialogue;▪

Ensure regulatory concerns and recommendations are considered; and,▪

Establish an industry-wide process where emerging technologies, threats, industry-leading
practices and regulatory requirements drive continued iteration of the Framework.

The target audience for this Framework includes those in the financial services industry who conduct,
rely or call for the execution of penetration testing and red teaming, including (but not limited to):••••Financial Industry Regulators
Firm Executives
Firm Information Security Professionals
Firm Information Owners / Technology Specialists
3

•
•
•

Testers
Third-party Stakeholders (e.g., in the case of managed systems testing, cloud computing
providers, etc.)
Other Industries like Fintech, Telecommunication, Media, etc.

The Framework documented below outlines a four-phased Testing Lifecycle (see diagram) to ensurefirms are following industry best practices while simultaneously meeting regulatory demands. The fourphases of firm-led red teaming or penetration testing are the following:
•

•
•
•

Threat Intelligence Phase – A firm’s internal intelligence should be augmented by government
agencies and sector level financial industry resources. Final threat intelligence scenarios should
be approved by regulators, where applicable.
Planning Phase – Test activities should be prioritized and scheduled according to threat
intelligence and regulator input in planning the scope of the exercise.
Testing Phase – Testing should begin after operational planning and attack methodologies areagreed upon.
Analysis and Response Phase – This phase includes the development of executive / technicalreports and associated firm responses. Summary versions of final reports may be distributed
internally within the firm and to regulators and would include a sign-off from the organization’s
Board on the identified vulnerabilities and associated remediation plan.

Testing Lifecycle

While the Framework provides an approach for penetration testing and red teaming, it is not intended
to serve as a detailed industry playbook for conducting testing. It is primarily focused on the interaction
between regulators and firms when conducting tests and is not intended to provide granular technicaldetails of the testing process.

4This document, while directional in nature, is designed to guide both regulators and the financial
services industry to develop a safe, secure and scalable testing program to manage inherent
shareholder, investor, market and firm reputation/financial risks arising out of a potential cyber-attack.

5

Contributing Organizations
The following financial firms, trade associations and consulting organizations provided substantial
support and input towards developing this document and its contents.

6

Introduction
BackgroundThe growing regulatory interest in penetration testing and intelligence-driven red teaming and theproliferation of various frameworks and approaches initiated the development of this Framework toensure consistent, safe, secure and scalable testing regimes. Penetration tests are powerful tools toassess a firm’s cyber security program but due to their invasive nature, testing data and results are
particularly sensitive. Testing results with vulnerability data can provide a clear roadmap to attack afirm, therefore it is imperative that the distribution of detailed test results be tightly controlled. With
more jurisdictions interested in these invasive tests, firms face increasing operational risk without anagreed framework for performing tests that can safely fulfill supervisory requirements.
To manage that perceived operational risk, the Global Financial Markets Association (GFMA) together
with the Securities Industry and Financial Markets Association (SIFMA), Association for Financial Markets
in Europe (AFME), Asia Financial Markets Association (ASIFMA), in partnership with the financial
industry, issued a joint comment letter1 during July 2016 outlining issues associated with regulatorydriven testing followed by a set of principles2 issued December 2017 intended to harmonize the growing
regulatory demand for penetration testing.
The principles advocate for firms with robust in-house penetration testing or red teaming capabilities tocontinue to utilize their existing programs, while giving firms the option to enhance those programs
through alignment with an agreed-upon harmonized penetration testing approach.
GFMA then brought together a team of subject matter experts from firms across the financial industry,including the Financial Services Roundtable, with the goal of developing a framework for joint
agreement between regulators and firms for conducting firm-led penetration testing and red teaming.
To develop such a framework, the industry divided their efforts between three working groups
composed of subject matter experts that met on a weekly basis over a 6-month period from July to
December 2016, to discuss and collaborate in the following areas:•••

Scoping and scheduling tests
Executing the tests
Communicating test results to management and regulators

The framework includes processes and procedures to provide regulators with a high level of confidence
that financial institutions utilize industry best practices and perform effective security assessments.

1
2

http://www.gfma.org/correspondence/item.aspx?id=827
http://gfma.org/uploadedFiles/News/GFMA_in_the_News/2017/GFMA-Penetration-Testing-Principles.pdf

7

Purpose of this Framework
This framework provides a guide for the development of a safe, secure and scalable testing program andproposes a basis for joint agreement between financial service firms and regulators for conducting
effective testing while managing operational risk. This Framework offers guidance and best practices onbuilding penetration testing and red teaming programs aligned with firm needs and regulatory
expectations while providing regulators confidence that firms are conducting quality tests thatappropriately assess security programs. With quality testing programs in place, firms should be able tosatisfy different testing requests on a single system with a single test of that system eliminating wasted
resources and at the same time keeping sensitive data tightly controlled.
The Framework design was guided by the following core principles which aims to:
•

•
•

•
•

Provide regulators the ability to guide penetration testing and red teaming programs to meetsupervisory objectives through use of scenarios based on current risks that drive scheduling andscoping of testing activities
Provide regulators with a high degree of confidence that testing is conducted by trained,
certified personnel with sophisticated tools that can accurately emulate adversaries, as required
Provide regulators transparency into the testing process and results for both regulator-driven
and firm-driven testing as well as assurance that firm governance identifies and properly
addresses weaknesses
Ensure testing activities are conducted in a manner that minimizes operational risks andenhances data security by including strict protocols for distributing test data and results
Provide firms with a capability to test their cybersecurity framework based on global bestpractices

Additionally, testing conducted according to this Framework should provide confidence in the testingprocess regardless of whether the firm or regulator initiated the testing. Firms should be able to test
their systems periodically and leverage test results to satisfy regulatory requirements in different
jurisdictions around the world.

Testing Options
The Framework acknowledges that there are many testing types available for firms to assess theeffectiveness of their security programs ranging from penetration testing, application vulnerability
scanning to adversary emulation testing or red teaming. This Framework focuses on penetration testing
and red teaming (aka “adversary emulation testing”) due to the high risks to operational disruption thatsuch testing could cause if undertaken without appropriate planning and involvement from across anorganization by a qualified team of practitioners.
Penetration testing options may range from external / internal infrastructure, web / server-client based
applications, third-parties, or software / code analysis, whereas adversary emulation testing seeks toreplicate the actions of sophisticated threat actors. Testing typically begins outside the firm and ends
with agreed-upon actions and objectives for attacking target systems and processes inside the firm.
Determining which business processes to be tested can be based on a number of factors but should startwith those which are relevant to the target organization. For example, a business-based risk assessment
8

can be used to determine the relevance of a business function or business process to the particularorganization. A business-based risk assessment may include: (1) the importance of the firm andbusiness process to the stability of financial system; (2) potential business impact of a proposed
assessment; (3) the effort required for testing; (4) the size and complexity of what is being tested; and(5) the length of time since an assessment was last performed on that particular business process.
Due to the complicated and potentially disruptive nature of testing, this framework focuses on thenecessary steps to successfully leverage this powerful tool to evaluate financial firms’ cybersecurity
programs.

Testing Lifecycle
Regardless of the type of testing firms choose to perform they are encouraged to follow a four-phased
Testing Lifecycle (see diagram) to ensure they are adhering to industry best practices while
simultaneously meeting regulatory demands.
It is important to note that different types of testing may emphasize different aspects of the lifecycle,
e.g., red teaming will require a more rigorous threat intelligence phase than application penetration
testing. Firms should select their type of testing and include an evaluation of how much time and effort
they will spend on each phase to ensure their testing results are the product of a thorough testing
lifecycle process.

Testing Lifecycle

1.4.1

Threat Intelligence Phase

The threat intelligence phase identifies key threats facing the financial sector. Firms, regulators andgovernment agencies responsible for cybersecurity should identify and validate key threats which will
act as the basis for threat scenarios. These key threat scenarios should be used by individual firms when
scoping and executing penetration tests and adversary emulation testing.
91.4.2

Planning Phase

The planning phase covers test preparation. Firms will collaborate with regulators to determine testing
expectations, and review business and technical processes to ensure assessment of appropriate
systems. Firms will articulate the scope of the test based on their own business processes and thebusiness functions associated with the testing expectations. A business level description of scope will
enable the firm to test systems based upon on their own evaluation of that business function.

1.4.3

Testing Phase

The testing phase covers test execution within each firm. Firms are responsible for executing tests
based upon the scope and the procedures identified in the Planning Phase. Tests performed inaccordance with the Framework should give regulators confidence that the executed tests provide
quality results and guide firms on how to improve their cybersecurity programs.

1.4.4

Analysis and Response Phase

The Analysis and Response Phase covers evaluation of the test results, confirms identified risks, andprepares and tracks firm responses to said risks. As part of this phase, regulators will be provided theability to review the test results and responses prepared by the firm.

Regulatory Role
Regulatory coordination and input play a key role in regulator-sponsored testing activities. Firms andregulators should engage in collaborative discussions on testing scheduling and goals. When regulators
request that financial institutions complete testing activities, both they and the industry should be clear
on the scoping of the testing activity, the timing of the testing, the testing requirements and expected
test outcomes. Firms should collaborate with regulators and allow them to review their businessprocesses to determine the appropriate systems within scope for testing. Regulators should identify therequirements for the test while allowing firms flexibility on how to address them.
Firms that follow the Framework are likely to produce high quality testing and have appropriate
procedures in place to improve their security programs based upon previous test results.

10

The Testing Lifecycle
The following sections provide detailed information on each phase of the Testing Lifecycle.

2 Threat Intelligence
Threat intelligence-based scenarios
mimicking real-life cyber adversaries are
essential to the success of testing
activities. Threat intelligence scenarios
should reflect the most significant risks
faced by the financial sector. It is
recommended that industry and regulators
jointly identify and prioritize the threats inorder to develop test scenarios at thefinancial sector level. This will ensure thethreat profile does not vary significantly
from firm to firm. Developing sector-level
scenarios benefits the industry by bringing
together key stakeholders and subject
matter experts to provide a high quality,
consistent view of the inherent threats
industrywide.

Testing Lifecycle

Threat intelligence-based scenarios should consider key financial market participants, including banks,
broker-dealers, financial market infrastructures, financial market utilities, critical third-parties and thegeographies in which they operate.

Scenario Development
To provide a broader view to the sector, threat intelligence scenarios should be prepared by a financial
sector representative group (“Sector Representatives” or “Sector Group”) with appropriate financial
sector threat intelligence expertise. Appropriate threat intelligence expertise is sourced from financial
firms, regulators, Subject Matter Experts (SMEs), Information Sharing and Analysis Centers (ISACs) andgovernment agencies. Output from the Sector Group will include threat intelligence scenarios
prioritized by risk to the financial sector that reflect the current threat landscape. For example, if thecurrent threat trends involve payment systems, the financial sector group could create a testing scenario
based around payment systems. Government agencies with a responsibility for cybersecurity will havean opportunity to validate the threat scenarios (e.g., National Cyber Security Center in the UK or the
Department of the Treasury Office of Critical Infrastructure Protection in the U.S.).

Select and Prioritize Testing Scenarios
A Sector Group should select and prioritize scenarios in accordance with their evaluation of therespective risk to the financial sector. Example sector groups are the Financial Services Information
Sharing and Analysis Center (FS-ISAC), the Critical Infrastructure Partnership Advisory Council (CIPAC US), the Association of Banks in Singapore (ABS-SG) and the Cross Market Operational Resilience Group(CMORG - UK).
11

The Sector Group would identify a set of relevant threat scenarios based on the information sources.
The Sector Group could also identify and define sub-scenarios, extending one or more of these variables,
to provide more detailed threat scenarios. The FS-ISAC is currently developing a bi-annual financial
sector specific threat intelligence scenario which meets the suggested requirements of this Framework.
Firms and regulators may be able to use the FS-ISAC sector-wide intelligence to inform the scenarios
used for their testing purposes and develop firm or country-specific sector threat profiles.
Firms are encouraged to develop multiple scenarios for their testing purposes. Scenarios shouldincorporate various tools, techniques and procedures reported to be in use by real world adversaries.
Testers should be given flexibility, within the bounds of the planned scope of the test, to pursue
different attack methods to ensure a realistic testing environment while minimizing operational risk.

Validation
A financial sector group with expertise in threat intelligence, led by the FS-ISAC, should validate any
identified threat intelligence scenarios. Once the Sector Group completes their review, they should
create and distribute a sector-wide assessment to financial institutions, government agencies andregulators. We encourage regulators to engage in a broader discussion with the sector after they
complete their review of the assessment and propose modifications to the sector testing scenarios.
The Framework highlights the need for industry and regulators to have a formal approach for the review
and validation process. This approach would help build a consensus around the validated testing
scenarios that can then be used at a national level. A successful validation process should define
industry participation and engagement.

Maintenance
The threat landscapes change rapidly. For the purposes of regulatory sponsored security assessment,
the financial Sector Group should re-assess and re-prioritize threat intelligence scenarios at least biannually.

12

3 Planning Phase
The Planning Phase covers test preparation
activities. Rigorous test preparation isrequired to ensure operational risks are
effectively managed and test objectives areachieved.

Testing Lifecycle

The Planning Phase includes project
management of testing activities, risk
management considerations, and the scope
and timing of tests (e.g., threat scenarios,
testing options, rules of engagement), andnecessary qualifications of testers. This
phase is designed to establish thegovernance structure, prioritization of tests,
the scope and related entity/assessment
types, and testing scenarios. Targets fortesting are identified based on input frombusiness owners, as well as threat
information and data security criteria.
Selection may be based on risk type, priortest results, or other criteria as appropriate.
Organizations should define the purpose and associated limits of the tests and document them in a
detailed plan. Technology owners should be involved in selecting the test purpose and limitations andmay provide technical information pertaining to the target system such as architecture diagrams,
process documentation and technological configurations.
Organizations should establish agreements regarding scope, and assignment of teams and resources
during the early stages of the overall process. This includes designating appropriate oversight andworking groups (these are both identified as control groups for the test activity). The working groupshould provision the scoping documents and background information relating to the functions andsystems in-scope.
It is recommended that the control group numbers be kept to a minimum. Members of either theoversight or working group are reminded to re-affirm their contractual confidentiality obligations. This
is to ensure that testing details remain confidential. Minimizing the numbers in the control group andadhering to confidentiality maintains the integrity of the test and provides an excellent opportunity forthe organization to test the preparedness of their controls and response teams. Firms may consider
having the working group sign a non-disclosure agreement to protect the integrity of the test.

Project Management
Organizations should use project management best practices during a firm-led penetration test,
particularly during an adversary emulation test. A project plan should outline the schedule for different
testing phases. The plan should be communicated to stakeholder groups, be closely adhered to andshould have protocols in place if deviation from the plan becomes necessary. Since testing may occur in13

a production or non-production environment, project management planning is essential to avoid andmitigate operational risks.

Risk Management
Risks are inherent during any type of test but are particularly evident in an adversary emulation test.
The possibility of causing a denial-of-service incident, unexpected system crash, damage to live criticalsystems, or the loss, modification, or disclosure of production data highlight the need for active androbust risk management.
To reduce the risks associated with testing, sufficient planning and coordination must take place
beforehand. This planning should include agreements between the test subjects and testers on theRules of Engagement. This would also include scope and, where required, a contractual arrangement
covering the testing – including indemnification and liability provisions. Firms should develop an
oversight group during the planning stages to manage the overall risk of the firm. Firms should conduct
thorough due-diligence of in-scope systems prior to any testing to ensure that backups systems are inplace, and recovery procedures are up to date and have been recently tested. A rules of engagement(ROE) document should be created that clearly defines the testing parameters, approvals, andescalations (see Section 3.6 for more detail relating to ROE).
Risks should be carefully managed throughout the Planning Phase. Higher risk activities can be managed
by conducting tests off-hours, or against non-production systems (however, testing parties should
ensure these systems are operating with the same parameters of the production systems to ensure avalid test).
The standard language and structure requirements for contracts with penetration testing and adversary
emulation testing vendors should include a requirement for third-party testers to meet security andconfidentiality requirements at least as stringent as those followed by the underlying institution
regarding confidential information handling, including PII (i.e., personally identifiable information). The
contracts should also include a clause related to data destruction requirements and breach notification
provisions.

Scoping
Testing scope should focus on a firm’s business functions and designed in a way that ensures the testing
activity is completed within an agreed-upon timeframe. Communication between a firm and its
regulator should be focused on the business process. Firms should determine the test scope and share
the test parameters with regulators.
Firms may apply risk and firm-based criticality ratings to in-scope systems, infrastructure, applications,functions, and data. Firms should examine these assets to effectively select the final target set.
Financial institutions should create a scoping document that allows for a consensus between firms andregulators.
The scope of the testing should be based on the threat intelligence scenarios and key business processes
and could range beyond traditional endpoint and network level testing to include physical threats,
threats from third-parties / suppliers and mobile devices, including BYOD environments. Threat
intelligence is essential to establish and maintain a library of testing scenarios. The agreed-upon key
14

business processes tested should be matched to one or more of the sector-level identified threat
scenarios.
Testing Phase (Section 4) objectives should be defined and linked to the threat scenarios previously
identified. Mapping objectives to threat intelligence scenarios will allow firms to identify successful
outcomes and calculate actual impact to firms. This in turn will help drive prioritization of remediation
activities.

Testing Options
As mentioned previously, this Framework acknowledges that there are many testing types ranging fromapplication and/or network penetration testing to adversary emulation testing. Different types oftesting present different risks to different systems, such as in production systems and those emulated in
testing / development environment. Firms should understand the benefits of and risks associated withthe different types of testing and choose the method that best suits their needs in evaluating theircybersecurity programs.

Timing of Tests
Firms and regulators should determine the appropriate timing of test execution based upon the type of
test, the criticality of the system or process to be tested, the potential for operational disruption causedby the testing and how recently the systems or processes have been tested. Frequency of testing for
operationally sensitive systems should be considered carefully to weigh value of testing against risk ofany potential disruption.
To maximize the utility and scalability of the regulatory use of penetration testing, a standard schedule
is recommended that allows both regulators and firms to adequately plan and execute the necessary
type and scope of testing. Firms are often planning their testing activities late in the year for execution
the following year. One proposed schedule would be as follows:
Month 1: Regulators and Firms, through bi-annual reports provided by the FS-ISAC, engage and agree
on the most critical identified threats, vulnerabilities and adversary tactics.
Month 2-Month 3: Regulators and Firms meet to discuss Firm planned testing for the following year inlight of institution needs and current threat intelligence.
Month 4: Regulators, based upon threat intelligence received and input from Firms, provide guidelines
for systems to be tested and scope of testing.
Month 5: Regulators provide schedule for testing to be performed in order for results of previous years
testing and current threat intelligence to guide the following testing cycles.
Month 5- Month 11: Firms perform testing during this time period.

Rules of Engagement (ROE)The ROE is an agreement between the firm and testers to provide assurance that tests and associatedrisks are being managed in a controlled manner. This planning document is essential in testing andshould be agreed upon between all parties involved. An effective ROE outlines at a minimum thefollowing:
15

•

•

Technical Scope and Span to be Tested:o Functionso Peopleo Firm assetso Specific out-of-scope areaso Specific out-of-scope periods of timeo Activities to be conducted during testingo Locationso Specific Technologies
o Third-parties / suppliers
Administrative Details:o Points of contact – trusted agent(s), testers, and leadershipo Testing guidelines
o Projected timelines
o Communication Plan – guidelines between trusted agents and testerso Reporting mechanism / frequency
o Acceptance of liabilities, responsibilities, and risks

The ROE is typically a project document that may be altered during the course of the test based on
approval from control group(s) and the tester(s). The ROE’s fluidity is essential to maintain testingflexibility and allows the tester to remain within the scope and boundaries of the ROE yet practice best
judgment when examining the environment. The scope and boundaries of the ROE should only bealtered with the permission of the firm.

Resourcing / Qualifications
Testers, both individuals and teams, must: (1) possess a minimum level of expertise, measured via
numerous certification criteria established by international standard-setting bodies, such as CREST,OSCP and GIAC; (2) gain qualification through vetted industry examination processes; and (3) be guided
by a strict code of conduct. The aforementioned accreditation standards are globally recognized andshould be acceptable in determining the competency of testing personnel. However, other recognizedaccreditation options with similarly rigorous standards may be considered as well. Firms should
demonstrate and verify to regulators that their testers and teams meet these baseline standards.
The Framework suggests the use of internal resources for security assessments, and particularly foradversary emulation testing if the accreditation requirements are met. Furthermore, organizations
should grant internal resources the appropriate levels of independence and oversight.
The Framework suggests the creation of a commonly accepted accreditation standard. Adopting an
industry-wide accreditation standard, enables firms and regulators to:•••

Establish a set of practices that defines and prescribes strict adherence to levels of competence,skill, experience, and knowledge requirements for practitioners at different levels (e.g.,foundational, moderate, expert)
Accredit a testing organization for their ethics, reputation, credibility and delivery oversight,
covering the methods for assessment, risk management, quality assurance, provision of ongoingtraining and development, innovation, and research
Ensure teams can provide adequate information protection, operate within appropriate testing
facilities, use an approved suite of tools, and have adequate de-confliction procedures
16

•
•
•

Establish clear criteria for trustworthiness of results through meeting professional standards,
obtaining relevant qualifications, and adopting a Code of Conduct committing to professionaland ethical responsibilities
Establish certification criteria specifically as it relates to third-party vendors
Ensure that legal and regulatory requirements of the firm are met when testers receive accessto client data during testing (e.g., cross-border requirements)

The application of an internationally agreed upon accreditation standard to both internal and third-party
teams would allow for multiple regulators to benchmark internal teams against peers with consideration
for firm size, scale, business, and risk profile, and against third-party security assessment organizations.
A key goal of the Framework is to establish confidence in a testing team’s (internal or third party) ability
to deliver the levels of assurance in the execution of regulator-driven security assessments andparticularly adversary emulation testing activities.

17

4 Testing Phase
The Testing Phase brings together Industry
Threat Intelligence, testing scenarios andfinancial institutions’ scope to deliver apractical assessment of defensive security
controls and detection and response
capabilities. In adversary emulation testing
the Tactics, Techniques and Procedures(TTPs) are representative of the threat
actors within a threat landscape and will be
used to deliver a realistic simulation.

Testing Lifecycle

The activities within the Testing Phase are:• Operational Planning• Execution• Review

The time allocated for testing is determined by the scope and resources of the financial institution, any
external requirements for a given engagement, and availability of supporting information supplied by
the financial institution (e.g., regular vulnerability reports or previous assessment data). A test could
have limitations that would not apply to genuine threat actors and therefore limitations influencing
adversary emulation testing should be considered.
The Testing Phase will involve specialist security testers experienced in executing adversary emulation
exercises, which may exist either internally within the financial institution or as part of a third-party
engagement with a specialist security consultancy firm. Specialist accreditations are available for bothinternal and external professionals as discussed in Section 3.7 of this document.

Operational Planning
Operational Planning includes the process of ensuring the outputs from the threat intelligence andwider Planning Phase drive the test execution. Furthermore, the Framework suggests that theownership of Operational Planning should reside within the firm. Appropriate oversight should be
provided by the firms’ governance and working groups.
Inputs, activities, and outputs of the Operational Planning phase for adversary emulation testing aredetailed below:••

Inputs: Industry Threat Intelligence sources, vetted scope, and developed scenarios
Activities: Test preparation and firm specific intelligence gathering

18•

Outputs: Targeting Report3, Penetration Testing Plan, and possible modifications to Rules ofEngagement

The following testing elements are an example of what may be examined during the Operational
Planning phase:
Information Gathering: Reconnaissance against a firm to discover publicly available informationsuch as company email addresses, document metadata, IP addresses, and domain name servers.
Enumeration: Collection of data related to hosts/servers on the network. Some of the datathat may be discovered includes users, ports, running services, and operating systems.
Vulnerability Identification: Enumeration findings are examined to discover vulnerabilities that
may be exploited by the tester. Manual analysis as well as automated scanning tools may be
used to identify network vulnerabilities.
Exploit: The act or attempt to validate that a vulnerability is susceptible to an exploit.
Compromise: Successful exploitation of an identified vulnerability. The outcome from the
exploitation of the target from the threat actor’s perspective, which allows for information gain,or acquisition of unauthorized access onto in-scope system.
Post-Exploitation: After gaining unauthorized access to a host/server, the tester aims toescalate privileges (if the tester is using a low privileged account) and pivot to otherhosts/servers to steal more information.
De-chain: A point in the test execution where a scenario is artificially progressed to compensate
for time limitations or replicate control failures (e.g., if a phishing exercise has not resulted incompromise of a system within a given time frame, the testers may be given access to allow thetesting to progress).
As a verification of the testing planning phase it should be possible to map all of the testing activities tothe threat scenarios developed during the Threat Intelligence Phase (Scenario Development Section
2.1).

3

Targeting consists of industry level threat intelligence developed scenarios, scope, and reconnaissance [e.g.,
open-source intelligence (OSINT), social-media intelligence (SOCINT), passive and/or active information gatheringor enumeration]. The goal of targeting is to utilize this information to formulate a targeting report that adheres tothe validated scenarios and focuses on systems that may provide a cyber advantage (e.g., a list of potential targets
for phishing attacks, or previously publicized vulnerabilities of in-scope assets). Targeting will be undertaken by
the testing team prior to executing the penetration test.

19

Execution
Penetration testing and, in particular, adversary emulation testing utilizes numerous techniques andapproaches. This document will not mandate or describe a specific methodology. However,experienced testers are expected to replicate the modus-operandi of threat actors and their TTP’s
according to the scenarios selected for testing.
The key inputs, activities and outputs of the Execution Phase are outlined below:•••

Inputs: Targeting Report, Test Plan, Rules of Engagement
Activity: Scenario driven penetration testing as detailed in the test plan
Output: Initial testing results

In this phase, testers are positioned to test against a targeted business process within the agreed-upontimeframe. The Targeting Report and results of the testing will drive the next test activity.
Regular updates on progress will be provided to trusted agents as detailed in the Communications Plan,and in accordance with the ROE.
Test execution can be enhanced using results from similar activities, which may be undertaken as part ofa business-as-usual process within the financial institution; for example, if monthly vulnerability
assessments are undertaken, this data can feed into the test execution process without needing torepeat previous test elements.
In testing the agreed scenarios, there may be common stages throughout that do not need to be
replicated each time. For example, if more than one scenario suggests phishing as an attack vector, this
only needs to be undertaken once. There may be common themes (see table below) across the
scenarios, and utilizing previous results ensures the most effective use of available time and resources
within a testing cycle.
Infiltration: Gaining access to and persistence on systems within the scope of the test; this could occur
via social engineering, exploiting vulnerabilities, or by taking advantage of misconfigurations.
Lateral Movement: Moving around the network to identify key assets within the scope of the test,
building up information of the financial institution’s systems and processes.
Action on Objective: The successful execution of an attack which could result in user / privileged-level
access, sensitive information leakage, or other follow-on malicious activity.
Upon completion of execution, initial details such as host information, vulnerability, exploitability,
impact, recommended actions for mitigation / prevention, and follow-up actions should be recorded as
inputs into the reporting cycle.

20

Review
During the Review portion of the Testing Phase, the initial results from the penetration testing execution
should be reviewed jointly by the testing team and the financial institution control groups articulated in
the Planning Phase (Section 3).
The inputs, outputs and core tasks are summarized below:

Firms should provide additional context to properly apply a risk rating for each finding (technicallyverified vulnerability associated with an in-scope target). A “significant finding” is a finding that
potentially possesses a substantial damaging effect of the firm’s security, financial, reputational or legal
posture. A draft test report of initial findings should be provided to each appropriate audience (e.g.,technical findings are provided to the team that will take appropriate action to resolve the findings).
Distribution and discussions of initial findings will be detailed as part of the engagement. If a third-party
conducts the test, it may provide out-brief workshops or walk-throughs of the test findings / results forvalidation. The oversight control group will validate and accept these findings. Many organizations havefound value in not only reviewing the initial findings via a replay of the test engaging both the red andblue teams to walk through the testing procedures and articulate how the testing progressed to identify
the strengths and weaknesses of the firm’s defense.
Any significant finding must be communicated as described within the ROE to ensure risk exposure canbe managed at or close to the time of the finding being discovered. Firms have the responsibility toshare any significant findings with other firms which might be impacted with the same vulnerability thatmight have been discovered during testing. At the time of review, these findings may have already been
remediated. All findings revealed at the time of the test should be captured in the draft Test Report,
including any remediation efforts or actions taken by the firm related to the finding.
Tests may identify areas outside of scope that cannot be explored within the agreed-upon timeframe.
These potential targets of opportunity should be recommended for review as part of an additional workstream or follow-on testing.
The draft results from the review phase will form the core elements of the Analysis and Response Phase(Section 5).
21

5 Analysis and Response Phase
The Analysis and Response Phase covers
analysis of current test results, theappropriate firm response actions to thetest, and protocols for reporting testing
results. IT firm management and senior
manager involvement is crucial, whereappropriate, at different phases,
particularly throughout the Analysis andResponse portions. Relevant
stakeholders should be involved during
the acceptance, mitigation or transfer ofthe risks uncovered during the Testing
Phase.

Testing Lifecycle

IT and risk management personnel should
be involved in developing the appropriate
responses, which may involve allocating
firm resources and prioritization ofremediation efforts.
The graphic below represents a typical process flow of the Analysis and Response Phase. Core tasks, aswell as common input and output documents are defined for each process. “Governance” represents
oversight provided by firm leadership up to and including the Board when appropriate throughout theentire phase.

Analysis
The test findings represent potential weaknesses or vulnerabilities which will be evaluated by firms toestablish the actual level of risk they pose. In the Analysis section, firms should utilize existing controls
to determine the actual level risk to the firm. This risk should be described using firms’ internal risk
22

definitions and framework. Based on the level of actual risk, firms may determine that additional
controls are necessary.
After analysis is complete, a Test Summary should be prepared, which includes a summary view of thetest activity, categorized test findings and risks aligned to existing mitigating controls. The Frameworksuggests the usage of the Financial Services Sector Profile4 of the NIST Cyber Security Framework forsynchronizing the language used by firms when categorizing test findings.
Often an evaluation of the test by all stakeholders: testers, oversight team and defenders, will provide
valuable insight on the efficacy of the testing and how to improve testing procedures in the future. Thisevaluation should be documented by the firm and used for any future testing.

Response
During the Response process, firms should utilize the Summary Report from the Analysis process toconsider actionable responses to the identified risks. These responses may include the enhancement ofexisting controls, the development of new controls, or the update of application code to fix theidentified issue. The control enhancements for the identified risks should be reviewed and approved byappropriate senior stakeholders in the firm to ensure organizational accountability.
Risks identified by tests and the firm’s response should be coordinated with senior technology leaders,
operational leaders and risk committees. They should provide oversight for the identified control
enhancements and ensure that they are fulfilled. Firm’s operational risk oversight and tracking
mechanisms should be used.
Firms should follow accepted protection guidelines when managing data that contains vulnerability
information. Priorities may be formulated during this phase based on the risk level of the findings. For
instance, priorities may be defined based on the risk levels (e.g., critical, high, medium, and low).
A Response Plan should be prepared and encompass the identified risks from the Summary Report andactions to be taken by the firm, including the evaluation of existing controls and planned control
enhancements with target dates.

Notification
Determinations regarding what findings are presented to executive management should be madeconsistent with documented policies within the firm for alerting management to risks. It is importantthat findings be brought to the attention of firm management consistent with the firm’s overall risk
reporting procedures.

Reporting
Final reports will be produced from a combination of test findings, summary reports, and responseplans. The key final reports should be at the following levels:
4

The Financial Services Sector Profile of the NIST Cybersecurity Framework was developed by the U.S. Financial
Services Sector Coordinating Council to act as a harmonizing element for the financial services sector members aswell as financial services regulators. The Profile provides a single vocabulary, taxonomy and approach for
managing cyber risk within the financial services sector. It can act as a common link for organizations to articulate
their cybersecurity risk management program with other firms and their regulators and if used as a basis forfinancial supervision can coordinate regulatory activity across jurisdictions leading to sector wide improvement inrisk management as well as broad efficiencies in supervision.

23

•

Executive Level - Senior stakeholders in firms should be provided with Test Summaries andResponse Plans. Executive management should receive a high-level report that outlines thethreatened assets, impact to the organization, and recommended remediation actions.

•

Regulatory Level - Regulators will be provided access to a summarized version of the Test
Response Plan and a summary of the test results, which will include key risks identified and firm
responses.

•

Technical Level - The most detailed / technical versions of the report should be reviewed withthe firm’s IT security team on firm premises and not leave control of the firm.

Data Protection
Testing information and reports must be protected with appropriate technical, physical andadministrative safeguards during transit and at rest. Test reports should be sent and stored in a mannerthat protects their confidentiality and integrity (e.g., using encrypted email, secure files transfer
mechanisms, use of out-of-band authentication)

Distribution
Due to the sensitive nature of the data, these reports should be distributed on a need-to-know basis,strictly controlled and appropriately protected. Distribution of reports outside of the firm should
happen infrequently and highly sensitive data should not leave the firm at all. Regulators should be
provided reasonable access to review such sensitive data within the firms’ control.
Specific details about vulnerabilities and systems affected in the test report could cause significant
damage and follow-on information security concerns and/or reputational harm. Detailed results are
particularly sensitive and should be very tightly controlled to internal review.
General findings and especially best practices the testing has identified may be shared as appropriate forthe benefit of the sector.

24

6 Conclusion
This Framework is intended to serve as a high-level directional guide for both firms and regulators
during the firm-led / regulator-driven penetration testing or red teaming process. The four-phased
Testing Lifecycle provides assurance of quality by firm designated testers and encourages clear
communication amongst the regulatory community and financial firms.
While the goal of this Framework is to promote a better understanding of the requirements andexpectations of both firms and regulators, it also assists the industry in reducing risk by defining
foundational processes globally. The reduction in risk extends beyond the firms; as clients, shareholders
and investors will have an understanding that the industry and regulators work together via a common
framework.
From a regulatory viewpoint, it suggests timely collaboration from regulators during each phase of thepenetration testing process. This Framework encourages regulators involvement during the threat
scenarios development phase; as well as Test Plan and Response Plan reviews. Involvement during key
milestones in the process ensures that firm-led Penetration Tests have met industry and global
standards.
As adversarial cyber testing continues to evolve, there may be a need for a more in-depth playbook thatexamines the details surrounding the testing process. Such details could provide more uniformity
among the firms and standardize how the testing process is conducted throughout financial industry.

25

7 GlossaryTerm

Abbreviation

Description

Adversary Emulation or Red
Teaming

N/A

A type of cyber security testing in which thetester assumes the role of an adversary to exploit
vulnerabilities found in IT systems.

Threat Intelligence

TI

Intelligence centered on emerging cyber threatsthat is typically used to understand the threat
trends and make informed decisions.

NIST

An organization of the United States Commerce
Department that develops and facilitates thedevelopment of standards for the industry.

National Institute of Standards
and Technology

Trusted AgentTA

Open-Source Intelligence

OSINT

Social-Media Intelligence

SOCINT

Test Findings

N/A

Vulnerability

N/A

A person in the firm responsible forcommunicating with the tester while maintaining
confidentiality of the ongoing test. Topics fordiscussion between the trusted agent and tester
include changing the attack methodology,
evidence of past or ongoing breach, and status
updates.
A type of intelligence wherein information isgathered from publicly available sources.
A type of intelligence wherein information isgathered from popular social media sites.
Confirmed vulnerabilities to be presented in areport that outlines their severity based on risk
and impact to the firm.
A flaw in a system that can be exploited by anadversary.

26