Model use and misuse
Speech given by
David Rule, Executive Director of Insurance Supervision
Association of British Insurers Prudential Regulation Seminar
14 May 2019
I am grateful to the following for contributions to this speech: Liam Clarke, Chintan Patel, Paul Collins,
Amanda Istari, Brian Howell, Sarah O'Sullivan, Anu
Ralhan, Peter McIntyre and David Phillips.
1
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
I would like to start by thanking the ABI for the invitation to make this keynote address. Today I want to talk
about insurance models and the growing importance of model risk management. In doing so, I will describe
some recent findings from the PRA's work to
guard against weakening over time of capital requirements
calculated from internal models or 'model drift'.
A model applies theories, techniques and assumptions to process input data into quantitative estimates for a
particular purpose.1 Make that purpose providing insurance and it is close to a definition of what insurers do.
No surprise therefore that insurers are big users of models. But models can go wrong. They might have
fundamental errors. Probably more commonly, they can be misused: for example, because management do
not understand the limitations of a model. In insurance, a model going wrong might mean under-pricing of
products, under-reserving against risks or treating certain customer groups unfairly. In the particular case of
internal models used to calculate regulatory capital requirements, it could result in insufficient capital to
protect policyholders.
Insurance Capital Models
Let us start with capital models. Solvency II allows insurers to apply to the PRA for permission to use a
model to calculate their capital requirements. The purpose of these models is to estimate the amount of
capital needed to protect the insurer against losses in a '1 in 200' scenario over one year. They are highly
complex. Taking the example of a life insurer, a typical model might involve:

Producing a base balance sheet. Many assets and liabilities can be valued without models: for
example using market prices. But other valuations are themselves based on models;
 Estimating a distribution of outcomes over one year for the material risks to which the insurer is
exposed: such as, changes in interest rates, credit spreads, inflation, property prices, longevity, etc;

Assessing to what extent these risks are inter-dependent: for example, asset prices are related to
interest rates;

Estimating the impact of the combination of risks to which the insurer is exposed on its balance
sheet across many different scenarios;
 - -
Taking these outputs and converting them into a simpler set of equations or proxy model that
relates the insurer's net assets to a series of risk factor inputs. The purpose of estimating a proxy
model is to make it tractable to run many more scenarios. The complexity of the underlying valuation
models would make this impractical computationally.

Running hundreds of thousands of scenarios through the proxy model in order to generate a
distribution of probable balance sheet outcomes over one year; and
1
This definition is based on the one in Supervisory Guidance on Model Risk Management, SR Letter 11-7, Federal Reserve Board and
Office of Comptroller of the Currency, April 4 2011
2
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx

Focusing on the tail of losses and estimating the 99.5th percentile of the distribution to identify the
minimum amount of capital necessary to protect the insurer against a '1 in 200' scenario.
(see Chart 1)
The use of models in Solvency II is more ambitious than in banking regulation. Comparing use of models in
banking and insurance regulation:

The insurance capital model framework encompasses the whole balance sheet whereas banking
models focus on particular risks (market, credit, counterparty). In the case of credit risk, models are
built for particular portfolios;

Insurance models estimate overall capital requirements taking account of assumed diversification
benefits across different risks whereas bank capital requirements are determined risk by risk and
added together, with no diversification benefits across risks, although diversification is taken into
account within risks;

Regulation determines or constrains more elements of bank than insurance models;

In banking regulation, the standardised approach is calibrated to give higher capital requirements
than internal models for a typical bank whereas in insurance the two approaches are calibrated to
the same confidence level; and

It is more straightforward for banking supervisors to top up capital requirements using so-called
Pillar 2 of the bank capital framework. Once an internal model has been approved under Solvency II,
supervisors have more limited scope to apply capital add-ons, which can only be temporary.
I make no apology therefore for the exacting nature of our internal model approval and change processes for
-
UK insurers. Models must meet the tests and standards set out in Solvency II. These are both quantitative
- -
around calibration, statistical quality and profit and loss attribution and qualitative around independent
validation, documentation, model governance and the use test. The PRA has approved 24 UK insurers to
calculate capital requirements using a model, which is by some way the highest number by EU country. That
gives us the advantage of peer comparison. We challenge hard where we think model calibrations are weak
based on either comparison with other insurers' models or our own view of a risk. At the same time, though,
we recognise that every insurer's risk profile is different.
Insurers are many and varied. The standard formula
cannot be flexible enough to work well for all of them. For this reason, internal models are an important
element of the Solvency II framework.
Part of the reason for our intensive review of insurer models is that the interests of management and
shareholders do not necessarily align with our own. Financial performance is measured by return on equity.
Especially in a highly competitive trading environment, lowering regulatory capital requirements may seem a
more achievable way to boost return on equity than increasing underlying profits. One legitimate approach is
to reduce underlying risks. If an insurer's internal model is well designed,
lower risks should lead to a fall in
3
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
capital requirements. We become concerned when genuine risk reduction tips over into weakening of
assumptions or more aggressive modelling of risks. It is one thing we look for when reviewing material model
changes proposed by insurers. Another is selectivity. Insurers might seek to make changes to their models in
areas where risks are reducing or they have overestimated them but neglect areas where risks are
increasing or they have underestimated them.
Model Drift
Model drift is the risk that the capital requirements calculated using an internal model may gradually weaken
over time such that they no longer reflect the risks to which the firm is exposed. No measure of model drift is
perfect. The PRA tracks movements in internal model capital compared to the standard formula and the net
The first benchmarks the tailored risk calibrations in an insurer's internal model
best estimate of liabilities.2
against the 'off the peg' measures of risk specified in Solvency II. The second
compares them to the size of
the insurer's balance sheet - a sort of insurance 'leverage ratio'. Supervisors supplement these aggregate
measures with 'bottom up' analysis to compare the outputs of internal models, looking at calibrations risk by
risk, over time and across insurers.3
(see Chart 2)
In 2016 and 2017, the aggregate capital requirements of UK general insurers with internal models increased
more or less in line with the standard formula and best estimate of liabilities. That is the outcome I would
expect, other things being equal. No generalised model drift was apparent. For life insurers in aggregate, by
contrast, both standard formula capital and best estimates of liabilities rose considerably more than internal
model capital. At face value, this looks like considerable model drift.
For a couple of reasons, though, we are not yet ringing the alarm bells. First, it is only two years of data. We
will need a longer period to draw strong conclusions about trends. Second, some life insurers with internal
models have been growing in areas where the standard formula over-states the risks: in particular, illiquid
assets within Matching Adjustment portfolios. Third, life insurers have been expanding unit-linked business
and transferring an increasing proportion of longevity risk on new annuities business through reinsurance,
both of which would tend to lead to faster growth in the best estimate of liabilities than capital requirements.
In summary, the two benchmarks against which we compare internal model capital might not have been fully
reliable measures of changing risks. Our 'bottom up' analysis of internal model outputs has also not
suggested any generalised dilution of standards. Nonetheless, the significant reduction in internal model
capital compared to the standard formula is not a trend we would expect to continue over time. We will be
watching it carefully.
2
See Monitoring Model Drift and Standard Formula SCR Reporting for Firms with an Approved Internal Model, Prudential Regulation
Authority, Supervisory Statement 15/16.
3
See Solvency II Regulatory Reporting: Internal Model Outputs, Prudential Regulation Authority, Supervisory Statement 25/15.
4
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
to any selectivity in insurers' model development is the PRA having its own model
One counter-balance
change agenda. For example, over the past couple of years, we have asked life insurers to adapt their
interest rate modelling to include the possibility of negative rates and we have pushed them to improve their
modelling of default and downgrade risks in Matching Adjustment portfolios.
Proxy Modelling and Use of Management Actions
Two areas we have reviewed recently are proxy modelling and use of management actions in internal
models. I mentioned earlier that life insurers build proxy models as an approximation of more complex
valuation models. They use them to calculate and rank losses under a very large number of one-year
scenarios, to create a distribution of possible losses, to identify the 99.5th percentile loss and thus to
calculate the regulatory capital requirement. If an insurer calibrates a proxy model poorly, regulatory capital
will be wrong, even if the underlying valuation models are broadly correct. Weaknesses could also lead to
bad decisions in other contexts where insurers use proxy models, including capital allocation, pricing and risk
management. For example, insurers often employ proxy models for stress and scenario testing.
In May 2018, the PRA issued a survey to a number of life insurers with a proxy model. The responses
revealed that over the previous couple of years, some insurers had improved the quality of their proxy
modelling considerably. Others had not made the same investment. Standards of validation also varied. The
best firms had increased the number of validation tests by improving the speed of valuation models, were
placing the validation tests at points carefully selected to challenge the proxy model calibration and were
conducting more validation after the reporting date.
Another area that we have reviewed recently is what future management actions insurers assume when
calculating their best estimate of reserves and internal model capital. Typical management actions might
include, for example, an insurer assuming that it can replace reinsurance following the default of a reinsurer
or at policy renewal, or a life insurer assuming that it could reduce bonus rates on a with-profits fund if it was
in financial distress. Solvency II allows insurers to assume management actions provided it would be
reasonable to implement them in stressed market conditions. We found that some insurers, however, were
inconsistent and insufficiently rigorous in their approach. Good practice is to bring all future management
actions together into a single document that, at least at a high level, the board can review. This plan should
quantify each management action and show how it is used in the calculation of reserves, capital
requirements or otherwise (for example, in stress testing).
5
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
Model Risk Management
I have focused so far on models used to help calculate regulatory capital and reserves. Insurers use many
other models though, notably for pricing. For many of these models, new technology is bringing rapid
change. Processing power continues to increase, including from use of the Cloud. This is enabling
development of models that employ artificial intelligence techniques, such as machine learning, to analyse
-
so-called Big Data larger, more complex information from new sources, some of it structured but a lot of it
unstructured. One example is real time information about customer behaviour taken from computing devices
-
embedded in everyday objects, such as mobile phones, home appliances and cars the so-called internet of
things. Together with digital distribution of insurance, these changes bring huge opportunities. For example,
better information may bring improved pricing, wider availability of insurance, more tailored and relevant
products, enhanced fraud detection and better customer service.
At the same time, they are raising new questions and risks. Many are ethical and conduct related. For
example, when designing a pricing algorithm what characteristics and behaviours of people is it acceptable
to include? How transparent do insurers need to be to their customers about these algorithms? Where is the
boundary between collecting data on behaviour to improve pricing and unacceptable interference in privacy?
Is it acceptable to charge a customer more because their behaviour suggests they might be willing to pay
more?
All these questions highlight the growing importance of managing model risk, including these ethical issues.
The US Federal Reserve System and Office of the Comptroller of the Currency published the seminal
guidance on model risk management in 2011.4 Although targeted at banks, most of it is also relevant to any
organisation making extensive use of models. Its recommendations cover model development,
implementation and use; independent model validation, comprising evaluation of conceptual soundness,
ongoing monitoring and backtesting; and effective governance, policies and controls, including the
importance of a comprehensive model inventory and good documentation.
Model risk can arise in two ways: from fundamental flaws in models and from misuse of models. Turning first
to model flaws, the risk is not that the model is imperfect - that is inevitable. Rather it is that errors are so
fundamental that the model produces inaccurate or undesirable outputs when viewed against design
internal model might produce a capital
objectives and intended business uses. For example, an insurer's
requirement that is significantly too low; or a pricing model might include analytical methods that lead to
outcomes that are outside the insurer's risk appetite or, in the extreme, in breach of regulations or illegal.
The Model Risk Working Party of the Institute and Faculty of Actuaries have published two good papers on
model risk.5 The second one6 relates the case of the loss of the Mars climate orbiter in 1998 as an example
4
Supervisory Guidance on Model Risk Management, SR Letter 11-7, Federal Reserve Board and Office of Comptroller of the Currency,
April 4 2011
5
https://www.actuaries.org.uk/practice-areas/risk-management/disbanded-research-working-parties/model-risk
6
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
of flawed model design. The trajectory of the spacecraft had been incorrectly calculated, which meant that
the spacecraft had actually been orbiting much closer to Mars than had been targeted causing the space
probe to disintegrate in the planet's atmosphere. The problem was as basic as
the use of incorrect units in
part of the navigation software. An external contractor had produced software to provide Thruster
The navigation team at NASA's Jet Propulsion
performance data in English units of pound-seconds.
Laboratory mistook this data as being in the required metric units of Newton-seconds. This led to errors in
the spacecraft's trajectory calculations. One of the root causes was NASA's adoption of a 'faster, better
cheaper' philosophy that had not given sufficient prominence to risk
management. It had also not carried out
an independent, end-to-end validation. Finally, developers had had an insufficient sense of ownership of
the whole project as opposed to their bit of it.
The second source of model risk is misuse of models. This can arise when management take decisions
based on the outputs of a model without sufficiently understanding the uncertainty around those outputs.
Complicated, dynamic problems are more difficult to model than simple, stable problems. Model outputs are
therefore more uncertain for problems that are more complex and where the underlying process in the real
world is changing. Calculating the capital requirements of an insurer is surely a prime example.
The IFoA Model Risk Working Party has an interesting typology of corporate cultures according to their
concern for model uncertainty and the legitimacy of modelling. Although they explain that all are valid
viewpoints, I would feel more comfortable if insurers fell into the category of conscientious modellers, with
modelling having high legitimacy but management also having a high concern for model uncertainty.
(see Chart 3)
The Solvency II use test requires models to play an important role in risk management, decision-making and
capital allocation. Management should therefore have confidence in their models. However, that confidence
must be tempered by a proper understanding of model limitations and assumptions, which the use test
indeed also requires. Insurers should try to de-mystify models as much as possible and avoid any sense that
they are "black boxes". They need to understand for what purpose a model has been developed and the
boundaries of its legitimate application. Criticism of the model ought to be welcomed and followed up: for
example, arising from independent validation. Stress testing should be used to explore and improve
understanding of model weaknesses. Management should challenge the outputs of complex models using
7
other simpler models, rules of thumb and expert judgement. For example, internal model estimates of
capital can be compared to simpler standard formula or leverage measures. We encourage insurers to
develop their own model drift measures and not limit themselves to the PRA's metrics. To take another
example, in a rapidly changing world, it is good practice for insurers to pay close attention to claims
6
Model risk: illuminating the black box, British Actuarial Journal, 22 August 2017, R. Black, A. Tsanakas, A. D. Smith, M. B. Beck, I. D.
Maclugash, J. Grewal, L. Witts, N. Morjaria, R. J. Green and Z. Lim.
7
In the Dog and the Frisbee, 2012, Andy Haldane and Vas Madouros explained how simple rules of thumb often outperform
complicated models in complex, uncertain environments. https://www.bankofengland.co.uk/paper/2012/the-dog-and-the-frisbee
7
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
experience as a risk indicator. If recent claims experience is suggesting increasing risk when models are not,
that should raise questions for management.
The risk of misuse of models might be particularly acute when an insurer buys in a model from a third party.
Its people have not developed the model and become familiar with its limitations. Management might be
tempted to put too much trust in the reputation of the supplier and assume that if many other insurers are
using the model, 'it must be fine'.
The Senior Managers and Certification Regime requires insurers to allocate certain prescribed
responsibilities to individual senior managers. Responsibilities for managing capital, funding and liquidity and
good understanding of an insurer's internal
for the Own Risk and Solvency Assessment, for example, entail a
capital model. In practice however, it is unrealistic to expect that everyone using a model will fully
comprehend how it works. The crux of good model risk management is to ensure that decision makers know
enough. My former boss, Andrew Bailey summarised this when discussing our expectations of boards in
2015, "the challenge is to reduce complexity to simplicity, so
that Board members feel that they understand:

where is the model expected to work well;
 in what circumstances is it likely to break down;

is the overall model output credible;
 what "moves the dial" in terms of key assumptions or judgements; and
 are those assumptions and judgements reasonable?"8
LIBOR Transition
Before I conclude, I would like to spend a moment on a different subject, LIBOR transition. As many of you
will be aware, the FCA's agreement with panel banks to provide LIBOR quotes
will end in December 2021,
now just over 30 months away. Across financial markets, SONIA is the Working Group on Sterling Risk Free
Reference Rates' preferred benchmark for the transition to sterling
risk-free rates from LIBOR. Liquidity is
-
now building in SONIA markets both derivatives and cash.
Our Dear CEO letter in September 2018 has had the desired impact of raising the profile of LIBOR
discontinuance, with a requirement on the largest firms to provide transition plans, and giving greater visibility
of the issue to other firms. We, and the FCA, have been clear that this is a significant change and achieving
an orderly transition is a high priority for us. We will shortly publish the main points arising from responses to
that Dear CEO letter.
For insurers, the Solvency II discount curves for major currencies are currently LIBOR-based. We are aware
that insurers need clarity about when and how these discount curves will transition to replacement risk-free
8
Governance and the role of boards, Andrew Bailey, 2015,
https://www.bankofengland.co.uk/speech/2015/governance-and-the-role-ofboards.
8
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
rates. We understand the challenges this poses to insurers, and we are working constructively with EIOPA
and others to address these issues. We encourage insurers to continue to focus on the actions within your
control such as identifying where LIBOR exposure is on your balance sheets, engaging with counterparties,
and preparing for operational changes.
Conclusion
To conclude, as insurers deploy increasing numbers of models, drawing on massive amounts of data and
with new analytical techniques, it is vital that they give sufficient attention to effective model risk
management. Insurers need to be sufficiently confident to use their models but not so over-confident that
they misuse them.
9
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
Chart 1: Typical method used by large life insurers to calculate their Solvency Capital Requirement (SCR) using an Internal Model (IM)
10
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
PRA's analysis of selected
Chart 2: internal model drift metrics from 1 Jan 2016 to 31 Dec 2017
40%
Life insurance firms
30%
General insurance
20%
firms
10%
0%
-10%
-20%
Internal Model Standard Net Best IM SCR / SF SCR IM SCR / Net
(IM) SCR Formula (SF) Estimate BEL
SCR Liabilities (BEL)
Source: Solvency II regulatory reporting and Bank calculations
11
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx
Chart 3: Alternative perceptions of modelling and its uses
Source: Adapted from Model risk - illuminating the black box (https://www.actuaries.org.uk/practice-areas/risk-management/disbanded-
research-working-parties/model-risk)
12
All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx