Title: Policy Research Unit in Policy Innovation Research
Funder: Department of Health (England)

Abstract: Vision for the proposed Unit The Unit will play a leading role in the ‘co-production’ of rigorous research evidence to inform the early stages of policy innovation through a joint enterprise between a coherent team of researchers and a wide range of policy decision makers in the Department of Health (DH) and the NHS. Our understanding of the brief for the new Unit is that it will focus on evaluation methods and evaluative studies (primarily but not necessarily exclusively in relation to pilots and other forms of structured innovation), and will relate them directly to the policy initiation, development and implementation process, potentially across the entire portfolio of the DH. The most recent thinking on how to improve the take up and use of research evidence for government and managerial decision making emphasises the importance of the ‘co-production’ of useable, rigorous evidence. This vision of ‘policy-based evidence-making’ will orientate the work of the proposed Unit. It means working so that the use of existing research evidence and the requirements for new evaluations are built into the policy initiation and development process from the outset, thereby breaking down the conventional sequence in which considerations of research and evaluation tend to follow much policy development work and intrude at a relatively late stage in the policy process, sometimes disruptively. The proposed Unit is unique in that it will be a generic resource capable of contributing to the initiation of policy innovation and pilots across the Department’s business. To do this, we will need to be informed of policy thinking at the earliest possible stage and to be trusted with confidential information. One way of enabling this to happen is to build stronger formal and informal relationships between the Unit and the DH than has been customary to date. This ‘policy-based evidence-making’ is a challenging aspiration while maintaining rigour and independence. To make it a reality, we propose a broad multi-disciplinary resource for policy innovation analysis with involvement in all steps on the pathway from horizon scanning, through innovation design and development, to implementation and evaluation, and including both policy advice and research. The ‘co-production’ process will need to vary from project to project and in accordance with DH needs and timescales. It may therefore be helpful to give an example from the early stages of a theoretical new research project– to illustrate how we may go about defining the research question, and identifying the evaluation options. First of all, this could involve early briefings where the Director presents the results of early horizon scanning and intelligence gathering undertaken by the PIRU. This would set the background; in the advance of a specific new policy initiative he could present the results of a rapid evidence synthesis to policy colleagues and use this to jointly discuss and identify possible evidence gaps, and to outline the range of possible outcomes of any new initiative; such discussion would also be used to agree on a preliminary set of evaluation questions. These would be further refined in discussion with DH and appropriate PIRU colleagues. This initial discussion would then help PIRU to understand the specific evaluation needs of DH, and this, in turn, could lead to further discussion of what evaluation options are likely to be possible/useful from the perspective of both PIRU and DH. On past experience, this can be done successfully in one-to-one meetings, in more formal seminars and in open discussions with larger groups, as well as by more informal methods, including electronically. Such meetings and interactions can help both researchers and policymakers clarify the objectives and logic models underlying initiatives, and identify the resources required for evaluation. They can also ensure that any new research draws on the most relevant sources of data and expertise from within both DH and PIRU, and at later stages the joint development of new research like this can ensure that the findings of relevant research can be disseminated rapidly and effectively. Aims of the proposed Unit Consistent with the vision of ‘policy-based evidence-making’, we aim to: • Develop a way of working with the DH based on close relationships so that Unit staff are fully integrated at an early stage into the processes leading to important policy innovations and that DH staff are fully involved in the design of evaluations so that innovations are both ‘evaluable’ and generate learning for policy development; • Increase and strengthen the ability of the Department to use evidence from previous research and future policy innovations. Strategic objectives • To develop early awareness of emerging and new challenges facing the health and social care system in England in the Unit and among DH officials with a view to providing advice on where pilots and innovation are most likely to be needed and beneficial in maintaining improvements in health and well-being; • To participate in the processes by which policy innovations (primarily pilots and demonstration programmes) are designed and developed so that the requirements of evaluation for policy learning are integral to the process and innovations are ‘evaluable’; • To advise on the design and methods of evaluation of policy innovations so that evaluations are robust and capture impacts on different population groups, across different parts of government, but are tailored to the realities of practical policy making; • To undertake evaluation of high priority policy pilots or demonstration programmes using formative and/or summative approaches and ensure that the findings are used for future policy development. Operational objectives • To develop and advise on methods to identify the need for evaluation, and to improve the quality, cost and utility of evaluations, paying particular attention to the ability to capture the direct and indirect effects of policy innovations on different population sub-groups, in different contexts and over time; • To produce rigorous, policy-shaping and policy-focused evaluation advice and analysis over differing timescales rooted in and informed by the contingencies of the particular policy process and the realities of regional and local policy implementation; • To advise policy decision makers, principally but not exclusively in the DH, on the interpretation of the findings of evaluations of policy innovations; • To actively transfer knowledge and promote the use of the Unit’s work in collaboration with interested parties within the DH, local government and the NHS. • To build new evaluation capacity, through hosting doctoral and masters students to work on projects in the unit and by offering short courses to other researchers, policy makers and others. Key research themes and activities In principle, ‘pilots' are attractive ways of making policy development more evidence-based. However, it is frequently not possible to mount straightforward trials of policy pilots. For instance, randomised controlled trials may be inappropriate in the case of some complex or hard-to-define interventions. In other cases, it is hard to find willing sites to act as ‘controls’ because of the enthusiasm generated by a call for pilots. As a result, evaluations can be unreliable, particularly to serve as indicators of mainstream implementation due to selection effects, lack of valid comparison groups, lack of baseline, pre-pilot data, inadequate timescales, and an inability to collect appropriate or any outcome data. A recent House of Commons Health Committee report concluded that insufficient time and thought is given by policy makers in the health field in the UK to the design, introduction and sustaining of policy innovations and pilots for meaningful evaluation to be possible in many cases, thereby wasting opportunities to learn what ‘works’ (1). One of the biggest problems is the tendency for innovations to be introduced very widely thereby severely limiting the scope for learning from comparative evaluation (e.g. Sure Start and NHS London polyclinic programme). However, the perceived imperatives of the policy process and everyday operational considerations mean that useful evaluation presents significant implementation challenges distinct from the more familiar ones of evaluation design, analysis, interpretation and application. We will provide responsive, policy-focused analysis within the limits of our resources so that the DH can manage these challenges better in future, through: • horizon scanning and intelligence gathering for the early identification of new and emerging issues likely to call for policy innovation and piloting; • assessing the need for evaluation and its feasibility in relation to early stage policy innovations; • rigorous syntheses of existing evidence ahead of the initiation of pilots, including evidence and experience from other countries and exploitation of high quality individual-level clinical databases and other routine information sources, to inform pilot development (e.g. to help identify which problem the innovation aims to address, to clarify the theory about how the intervention is expected to work, to clarify the nature and extent of uncertainty surrounding the intervention (e.g. to help determine whether evaluation is worthwhile), and to estimate the scale of potential intended and unintended effects from previous studies of the same or conceptually similar interventions (e.g. to help with evaluation design); • advice on the design of policy initiatives so that evaluation is designed in from the outset, helping policy makers clarify the objectives and logic models underlying initiatives, and identify the resources required for evaluation, building the needs of evaluation into policy development (e.g. so that an evaluation team is in place before pilots begin, and intervention and comparison sites are chosen rigorously) and ensuring that innovations and their related evaluations are pursued for sufficiently long and in a consistent enough manner for their effects to be reliably identified; • advice on the overall design of evaluations (e.g. feasible experimental studies, including increasing the use, where appropriate, of a range of pragmatic RCT designs such as cluster RCTs and step wedge designs, and ensuring comparability of intervention and control groups in non-experimental studies using techniques such as risk profiling, propensity score matching, predictive modelling, etc) and on specific methods to be used (e.g. specification of datasets to be collected by pilots); • supporting the subsequent process of pilot development and implementation (e.g. to encourage consistency in key aspects of the delivery of an intervention if required for evaluation to yield policy learning); • advising on the interpretation of findings (e.g. assessing the extent to which the effects seen have been caused by the pilot and are likely to be replicable more widely, and relating the findings of a single DH study to the wider evidence from the UK and overseas; • supporting learning from pilot evaluations (e.g. presentation of findings in an accessible and relevant manner to guide the roll out of pilots, but also deriving lessons for future policy development processes from the experience of mounting and evaluating pilots); • modelling and simulations to estimate the likely effectiveness and cost-effectiveness of pilots ex ante and to estimate their potential effects if ‘rolled out’ more generally ex post, including equity consequences; • data linkage and warehousing of ‘routine’ data to provide a flexible resource for relatively low marginal cost ex ante assessment and long-term monitoring and evaluation of policies and pilots (e.g. including studies of ‘universal’ policies as well as post-evaluation monitoring of demonstration programmes); • becoming the Unit of choice for the Department in undertaking major evaluations of policy innovations; • capacity building through providing DH and NHS staff with opportunities to participate in policy evaluation (e.g. hosting NIHR SDO management fellows) and offering more formal training for DH staff and others (e.g. PhD students) in evidence-informed policy initiation and development processes as well as methods of policy evaluation. Thus the Unit will provide responsive advice, but also undertake primary research evaluating specific pilots. The research work will grow directly from the responsive advice, and, in turn, lead to subsequent advice. Some evaluation work will be undertaken within the Unit’s core funding, depending on the volume of more rapid, responsive work, but larger projects will require additional funding. As well as policy-focused evaluations, this research will include methodological studies embedded within its evaluations (see Methodological Development, below). Given the remit of the Unit and its skills across health services, social care and public health, we will work across the three policy sectors as far as possible. While it is easy to think about policy innovations that incorporate concerted action simultaneously in the NHS and in social care, it is much harder to design and undertake evaluations that can track users of services between the two sectors reliably and that can capture health and social outcomes over time. One strand of methods development work in the unit will focus on developing ways of implementing such studies (see below). For example, interventions designed to improve the functioning and quality of life of people with long term health problems are likely to have unpredictable effects on the costs of their long term care and support. It will be important for the Unit to be in a position to study such relationships. Engagement with policy and policy makers’ needs The Unit will need to be in regular confidential contact with DH policy teams to discuss early stage policy ideas. This will be especially important in the case of major policy reviews. The aim is to ensure that evaluations produce evidence likely to assist Ministers make informed decisions, and to provide input to processes such as Spending Reviews and Impact Assessments. In addition to regular discussion with policy teams, the Unit will develop a range of secondments and exchanges initially with the DH, but possibly, over time with other relevant agencies. For example, it is envisaged that all research staff will spend some time attached to the DH (e.g. in the Strategy Group of the Policy and Strategy Directorate) or a related agency, a model with some similarities to the clinical advisors in the DH currently) and that the Unit will host DH staff regularly so as to ensure that the Unit is aware of the latest thinking and can keep up with the pace of policy developments. The Personal Social Services Research Unit (PSSRU) has considerable experience of having staff seconded from the DH to work on particular projects and of academic staff who divide their time between the two organisations (and also secondments to the Treasury). However, hitherto, these arrangements have tended to be restricted to DH economists and have been somewhat ad hoc. The proposed Unit aims to widen the scope of these exchanges and secondments, and regularise them. The work of the Unit is expected to generate insights as to how the norms and processes of policy development and commissioning of evaluation within the DH facilitate or hinder its use of evidence for policy. The success of the Unit will thus depend not only on how it does its work, but also on the openness of the DH to exploring new ways of carrying out the initial stages of policy making in light of experience of working with the Unit. During the establishment phase between acceptance of the bid and the formal start of the new Unit in January2011, we envisage discussions about the implications of the new Unit for the Department’s ways of working with external advisers and contractors. The Unit will seek additional funds for an external evaluation of its work and periodically reviewing its experience of working with DH and, if necessary, suggest changes that both organisations can work on. We also plan to include in our ‘first three projects’ a piece of retrospective research on recent pilots, how they were developed and implemented, and how evaluation either did or did not get used and by whom subsequently. This would bear directly on this concern to inform the evolution of the way the DH does its business and to understand better at the outset the environment we are likely to be working in. [1] House of Commons Health Committee. Health inequalities. Third report of session 2008-09. Volume 1, report. HC-286-1 London: The Stationery Office, 2009


Research activity codes: 8.4 RESEARCH DESIGN AND METHODOLOGIES (HEALTH SERVICES);8.3 POLICY, ETHICS AND RESEARCH GOVERNANCE
Health categories: GENERIC HEALTH RELEVANCE