Appraisal policy

© Crown copyright 2012

You may re-use this information (excluding logos) free of charge in any format or medium, under the terms of the Open Government Licence. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence or email psi@nationalarchives.gsi.gov.uk.

Where we have identified any third-party copyright information, you will need to obtain permission from the copyright holders concerned.

This publication is available for download at nationalarchives.gov.uk. 1 Introduction

1.1 The decision to develop a new policy

This policy has been developed in response to the changed conditions for records management within UK government, especially the development of digital records. The National Archives established an Appraisal Policy Project, overseen by a project board, which studied ways to streamline ‘Grigg’ methods of review, the impact of digital records on appraisal methods and the theory and practice of alternative appraisal methods used in archives elsewhere. The draft Appraisal Policy was the subject of a public consultation exercise between 5 April 2004 and 30 June 2004.

1.2 Scope

The policy applies to all public records as defined in the Public Records Act of 1958: ‘administrative and departmental records belonging to Her Majesty’, including ‘not only written records but records conveying information by any other means whatsoever’.¹

Although primarily developed for the records of central government, this policy may be applied to the appraisal of all records created by public authorities as defined in the

¹ See nationalarchives.gov.uk/information-management/legislation/public-records-act.htm Freedom of Information Act 2000 as a means to fulfil their obligations under the Lord Chancellor’s Code of Practice on the Management of Records under section 46 of the Act.

The policy should be read and understood in conjunction with the Acquisition Policy (2000) which sets out The National Archives’ acquisition objectives in the form of eight collection themes which provide ‘the basic definition of the archive which will be acquired in the future’.

1.3 The ‘Grigg system’

The report of the Royal Commission on Departmental Records chaired by Sir James Grigg was the basis of the Public Records Act of 1958 which ended the haphazard legal position as regards duties of custody and retention of public records. The Act defined public records, it assigned duties between The National Archives and departments and within departments regarding custody and control, and it determined timings of transfer of historical records to The Archives (at 30 years) and rights of public access (50 years amended to 30 years in the 1967 Act). In addition to proposing new legislation to define these statutory responsibilities, the Grigg report proposed a system for the review of public records which still obtains today. The key elements of that system were endorsed, with reservations, by a subsequent Committee of Inquiry chaired by Sir Duncan Wilson in a report published in 1981.

The Appraisal Policy recognises that the Grigg system has been an effective means of enabling departments to manage their records and has provided an ordered mechanism for the transfer of records to the archive. Close contacts have been developed between The National Archives and departmental records staff, allowing each to voice and respond to the concerns of the other. The National Archives’ holdings are renowned, records can be readily located and their context is well-explained in the catalogue. This represents a significant achievement for an archive which spans nearly 1,000 years of continuous record-holdings by governments which exercised extensive overseas influence and control.

______________________________________________________________________

2 nationalarchives.gov.uk/information-management/projects-and-work/acquisition-disposition-strategy.htm 3 Committee on Departmental Records. Report. (1954) Cmd. 9163. A further account of the history of this system may be found at nationalarchives.gov.uk/policy/act/history.htm 4 Modern Public Records Selection and Access. Report of a Committee appointed by the Lord Chancellor (1981) Cmd. 8204 1.4 The shape of the future

While recognising past achievements the research conducted in the course of the Appraisal Policy Project indicated that significant changes were needed to accommodate digital records, to streamline selection techniques for paper records, to provide methods which facilitate the application of consistent selection decisions for records held in a variety of media and across government departments, and to ensure records are managed in ways compliant with recent information legislation.

The policy statement which follows this introduction sets out an agenda for change in methods used to select records for permanent preservation. The Appraisal Policy will be supported by standards on procedures for appraisal for both paper and digital records, procedures for the migration of digital records with archival value and by Operational Selection Policies (OSPs)(^5) based on policy principles.

1.5 Summary of the papers

The Appraisal Policy outlines The National Archives’ strategic aims Appendix 1 summarises the long-term shift in appraisal methods envisaged Appendix 2 provides definitions of business and archival values Appendix 3 summarises the ‘Grigg system’ A Background Paper developed for the consultation exercise supports and explains the statements in this policy(^6) The survey of procedures which accompanied the draft Appraisal Policy will be refined and made available on The National Archives’ website

2 Appraisal policy statement

2.1 Appraisal

2.1.1 Appraisal is the process of distinguishing records of continuing value from those of no further value so that the latter may be eliminated.

2.1.2 Our policy is to develop and implement strategies which enable robust and coherent appraisal decisions to be made.

(^5) See Appendix 3 for a description of OSPs. (^6) nationalarchives.gov.uk/documents/information-management/background_appraisal.pdf 2.1.3 Appraisal supports records management because, when carried out successfully it:

- maintains the efficiency of departmental records management systems by limiting their contents to those records which are of continuing value, for business and archival purposes
- ensures that records of long-term or archival value are identified so that provision can be made for their preservation

2.2 The need for an appraisal policy

2.2.1 The development of digital records has prompted the need for The National Archives and UK government departments to appraise, or provide appraisal criteria for, records already created digitally and for digital records being created now.

2.2.2 For the next 20 years The National Archives will accession records predominantly in paper form, and departments will need to provide for the appraisal and retention of their paper records.

2.2.3 There are limited resources within the UK civil service available for dealing with records management and their effectiveness needs to be maximised and the case made for increasing resources to fulfil statutory requirements.

2.2.4 The public availability environment created by Freedom of Information legislation requires departments to document their appraisal work and provide justifiable reasons for keeping or destroying records.

2.2.5 Agencies, non-departmental public bodies (NDPBs) and other bodies have been created over the past 15 years ‘to carry out the executive functions of government within a policy and resources framework set by departments’(^7). As a result the machinery of government has radically altered since the Grigg Committee considered departmental records management requirements.

2.3 Aims

2.3.1 The National Archives will engineer a shift to a system of appraisal that will be applicable to the new environment created by digital records.

______________________________________________________________________

(^7) The Prime Minister's Office of Public Service Reform Better government services. Executive agencies in the 21(^{st}) century (2002) 2.3.2 The National Archives will ensure the continued and timely transfer of paper records to The Archives, as these will continue to be the main form in which we will accession records for at least the next 20 years.

2.3.3 The National Archives will ensure that appraisal for archival purposes selects records of the highest archival value, avoiding duplication (see Appendix 2).

2.3.4 The National Archives will provide appraisal methods which will integrate the appraisal of digital and paper records as well as records produced in any other medium.

2.4 Principles of appraisal

2.4.1 The future of the Grigg system

2.4.1.1 Elements of the Grigg system – the specific timings for review and extensive file by file review - are not practical for assessing the archival value of digital records. (See 2.4.3 for alternative approaches.)

2.4.1.2 The Grigg system will be retained as the means to safeguard the continuing transfer of records in all other media to The National Archives as long as departments need it to ensure compliance with section 3(4) of the Public Records Act 1958. To this end The National Archives will strengthen elements of the Grigg system by:

- reasserting the importance of disposal schedules, and of first review where it persists, and ensure consistent criteria are applied in departments
- reasserting the importance of clear selection criteria and The National Archives monitoring at second review, while ensuring that second review is targeted on records with high potential archival value

______________________________________________________________________

8 ‘Public records selected for permanent preservation under this section shall be transferred not later than 30 years after their creation either to the Public Record Office or to such other place of deposit appointed by the Lord Chancellor under this Act as the Lord Chancellor may direct’

9 A disposal schedule is ‘a set of instructions allocated to a folder to determine the length of time for which the folder should be retained by the organization for business purposes and the eventual fate of the folder on completion of this period of time’. See nationalarchives.gov.uk/information-management/projects-and-work/information-records-management.htm. Accommodating the need to retain records for archival purposes is an element in the development of disposal schedules implementing Grigg recommendations for a government-wide approach to case files and datasets\\textsuperscript{10}, to ensure rational archival selection and to aid departmental management of such records

2.4.2 Record creation and disposition

2.4.2.1 Effective appraisal, especially in the digital environment, is dependent on good systems of records creation and business scheduling of records.

2.4.2.2 The National Archives will assist in this process and will:

- influence the current implementation of Electronic Records Management (ERM) Systems\\textsuperscript{11} so that file plans are created which enable the appraisal of records for business and archival value
- provide procedures for the appraisal, migration and transfer of digital records
- reassert the importance of other elements of robust systems for the creation of records, including clear guidelines for naming, filing and storing documents, in all formats
- encourage and assist in the development of disposal schedules across all records in all formats

2.4.3 Macro-appraisal

2.4.3.1 Macro-appraisal can be defined as assessing the value of records at a government, departmental or unit level rather than at an individual document or file level.

\\textsuperscript{10} Case files, sometimes referred to as ‘particular instance papers’ are records series whose component files cover the same subject matter though each relating to a different person, body or place. As the Grigg Committee commented: ‘While each individual document may be of little importance by itself, taken together or by way of sample these papers enable certain broad conclusions as to historical, economic, or social trends to be drawn’. (para 62). Datasets may be the approximate digital equivalent of case files

\\textsuperscript{11} An Electronic Records Management System may be defined as: ‘An Environment applied to digital records using application level software to control the behaviour of digital records throughout their life [within the system] until their disposal is determined and in particular the accrual of metadata 2.4.3.2 Macro-appraisal encourages government-wide or organisation-wide analysis of functions as a guide to identifying records of value for business and archival purposes. It may be appropriate to digital records because, by identifying records produced by the most significant functions, it provides the means to make appraisal decisions without the need for file by file scrutiny or the ‘historical perspective’ provided by the passage of time.

2.4.3.3 The National Archives traditionally assessed the archival value of records with reference both to their provenance (the organisational and functional context in which they were created) and to their content. The National Archives will in future make an understanding of the functions which produced public records an overt, preliminary stage in appraisal work.

2.4.3.4 The National Archives will make the final assessment of the archival value of public records according to the selection criteria laid out in Appendix 3 of the Policy and in the Acquisition and Disposition Policies.

2.4.3.5 Macro-appraisal will be used as an aid to:

- inform the way paper review is conducted
- identify areas of file plans created in an ERM system of potential archival value
- provide appraisal guidelines for ‘hybrid’ records (records stored in more than one format but especially in a combination of digital and paper form) produced in the period between the introduction of the computer and the introduction of full ERM systems
- identify datasets and case files of potential archival value
- identify areas of overlap between departments, so avoiding duplication in material selected for the archive

2.4.3.6 The National Archives will develop generic archival appraisal guidance for categories of records such as those produced by similar types of departments (agencies, regulatory bodies) or those produced by activities common to many departments (research papers, inspection reports). 2.5 Implementation

2.5.1 Methods and procedures have been developed to fulfil the aims in this policy and to allow for the controlled transition to modified systems of review. These were published as Appendix 3 of the consultation paper and are available on The National Archives’ website.

2.5.2 Further implementation will be developed in consultation with government departments.

2.5.3 The National Archives will continue to:

- provide guidance on the maintenance of digital records over time and developing disposal schedules for common administrative records
- provide selection criteria for archival records and monitor departmental selection of public records in whatever medium for permanent preservation and transfer to the archive

2.6 Responsibilities under the Public Records Act, 1958

Departments are responsible for making ‘arrangements for the selection of those records which ought to be permanently preserved and for their safe-keeping’.

The National Archives is empowered to give guidance, co-ordinate and supervise all action taken in pursuance of the above(^\\text{12}).

3 Your questions answered

The following questions arose in the course of the public consultation exercise and are answered here. Further common questions about the public records system are answered

(^{12}) Sections 3(1) and 3(2) of the Acts state:

It shall be the duty of every person responsible for public records of any description which are not in the Public Record Office or a place of deposit appointed by the Lord Chancellor under this Act to make arrangements for the selection of those records which ought to be permanently preserved and for their safe-keeping. Every person shall perform his duties under this section under the guidance of the Keeper of Public Records and the said Keeper shall be responsible for co-ordinating and supervising all action taken under this section. in the frequently asked questions section attached to The National Archives’ Acquisition Policy.

3.1 What is the expected time frame within which the shift from Grigg will occur?

There is no specific time frame common to all departments. The National Archives will prioritise areas where guidance is needed and departments and types of records where generic archival appraisal policies can be devised, and will respond to specific departmental requests for guidance and directions relating to paper or digital records. Departments are encouraged to continue with Grigg methods for paper (other than those produced in a hybrid paper/digital environment) and these may well persist for another 30 years.

3.2 Have appraisal values changed?

The Policy proposes that an assessment of the value of the functions carried out by an organisation will be the main element in the appraisal of digital records produced by it. Even so for datasets, where they are the digital equivalent of case files, procedures have been established in which the informational content and research value of the records remains an important element in appraisal. The statements in para 2.4.3.4 and in Appendix 2 are intended to demonstrate that The National Archives will continue to make decisions about archival value on the basis both of an understanding of a record’s provenance and of an assessment of the record’s information.

3.3 Will there be a reduction in the numbers of review staff?

The National Archives would encourage a streamlining and sharing of knowledge among staff dealing with appraisal for archival purposes and to implement FOI and ERM systems. There will need to be a process of retraining and reorientation of working practices. The National Archives believes that the expertise of existing review staff can and should be utilised to guide a more targeted approach to second review and to assist in the development of disposal schedules for paper and for file plans under ERM systems.

3.3 Will there be training provided in the meaning of macro-appraisal?

The Appraisal Policy will necessitate a reconsideration of the training needs of all staff involved in the process of appraisal at The National Archives and in departments. 3.4 Will records be transferred to The National Archives earlier if Grigg review timings no longer apply across the board?

Section 3(4) of the Public Records Act (1958) still applies(^\\text{13}), so records must be transferred to The National Archives 30 years after their creation. The need to identify digital records for permanent preservation at or near creation creates the possibility of early migration of such records to the archive. Paper records may also be reviewed earlier, for instance under procedures whereby first and second review are merged. We shall be examining the options for this in the light of continuing departmental business needs and Freedom of Information legislation.

3.5 Where can I find the proposed procedures for appraisal and review?

In the course of the Appraisal Policy Project some pilots were conducted into new ways of carrying out second review. The draft Appraisal Policy which was circulated for public consultation contained an Appendix 3 which outlined procedures for: appraising the records of agencies using macro-appraisal, cross-departmental methods; appraising case files and datasets produced in the course of carrying out a function across departments; streamlining full file by file second review methods or merging first and second review. That Appendix 3 has not been published as part of the finalised Appraisal Policy but will be revised in the light of comments received during the consultation exercise and published on The National Archives’ website as guidance. It will be regularly updated in the light of experience. It will form part of discussions between The National Archives and departments on their appraisal needs in the implementation phase.

(^{13}) ‘Public records selected for permanent preservation under this section shall be transferred not later than 30 years after their creation either to the Public Record Office or to such other place of deposit appointed by the Lord Chancellor under this Act as the Lord Chancellor may direct’ APPENDIX 1: Summary of the proposed shift in the timing, purpose, staffing and scope of appraisal

| Elements in the appraisal process | The present: Grigg review | The future: | |-----------------------------------|---------------------------|-------------| | | | Appraisal (digital and hybrid records) | | | | Appraisal + review (paper and other records) |

| When are appraisal criteria applied? | 1<sup>st</sup> review (5 years from closure) | At any point, but | |-------------------------------------|---------------------------------------------|------------------| | | 2<sup>nd</sup> review (25 years from the initial date of the record) | (a) for electronic records: | | | | ● before creation through file plans | | | | ● at creation through filing on to folders with disposal already attached | | | | ● at moment of migration for preservation of semi-current records | | | | ● at moment of migration to more permanent storage | | | | (b) for paper records: | | | | ● maintain current timings but earlier 2<sup>nd</sup> review may be appropriate | | | | ● review with electronic records where they are linked |

| Why are appraisal criteria applied? | 1<sup>st</sup> review to assess the future business use of the record (+ an awareness of historical value) | Business and archival appraisal values are defined in Appendix 2 | |-------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------| | | 2<sup>nd</sup> review to assess the historical value of the record | |

| Who applies the appraisal criteria? | 1<sup>st</sup> review by business units in departments or by a discrete group of 1<sup>st</sup> reviewers in departmental records centres: some The National Archives spot checks | These units are still involved but in different ways. | |-------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------| | | 2<sup>nd</sup> review by discrete group of 2<sup>nd</sup> reviewers in departments with direct The National Archives monitoring | Decisions about specific records or sets of records are preceded by analysis of all or a wide range of the records related to the organisation or function. This initial process will involve more collective effort for both business and archival purposes, enlisting the expertise of The National Archives client managers, reviewers, and past and present business users. |

| What is the range of records to which archival appraisal criteria are applied? | Each group of records is reviewed separately, with limited cognisance of decisions made for similar or related records in other departments or divisions | The new method will encourage: | |---------------------------------------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------| | | The new method will encourage: | (a) the consideration of all the records produced by an organisation, or by several organisations in the course of fulfilling the same function | | | | (b) awareness of where similar or related records reside in government | | | | (c) development of common/generic selection criteria for types of records and types of record-creating bodies | APPENDIX 2: Appraisal values

1. The value of records has two elements: their ‘primary’ value to the organisation (business value) and their ‘secondary’ value to society, providing a resource for historical research to a wide range of future users (archival value).

2. Appraisal of the business value of records should be carried out in such a way as to:

   - assist efficient and effective administration through reducing the time taken to retrieve information
   - ensure pertinent material is retained for as long as it is necessary to enable informed policy development
   - allow departments to account for the management of resources and policy-making to the public (UK citizens and organisations, and overseas bodies), and in the course of legal and financial scrutiny

3. The aims of archival appraisal are to:

   - determine which records have the highest archival value. All records can be deemed of potential archival value but an archive must be able to preserve and conserve records permanently and must therefore limit its acquisitions to records of the highest archival value
   - avoid duplication in the selection of archival material
   - develop consistent archival appraisal decisions within and across departments
   - support the strategic objectives and collection themes of The National Archives’ Acquisition Policy

4. Records should be selected for the archive which show the significance of the functions and activities of departments and which provide information relevant to The National Archives’ Acquisition Policy.

5. Records of the highest archival value are those which:

   - are most capable of documenting change, continuity and development over time and of assisting historical interpretation of such changes
   - document the functions of departments and of government as a whole

6. Appraisal should be carried out in such a way as to select records of the highest archival value in respect of:

   - events
   - institutions
   - policies • processes and procedures • social development, including demographic, cultural and economic change • changes to the physical environment

APPENDIX 3: Summary of the ‘Grigg System’

The ‘Grigg system’ has two main elements: a system of timing and procedures recommended by Grigg and then implemented by The National Archives and departments; and the advice and guidance given by The Archives on how reviewers assess the value of records.

The Grigg system

a) All registered paper files, except case files, are closed after a maximum of five years b) Five years after a file has passed out of active use departments (either the business users or the departments’ records centres) carry out ‘first review’ at which they decide whether the file has any continuing administrative value to the organisation or could have administrative value in the future. Files may be destroyed immediately or earmarked for destruction without further review after a stated period c) Twenty five years from the initial date of a file (supposing it survived first review) reviewers in departmental records centres, under The National Archives’ supervision, carry out “second review” at which they decide whether the file has ‘historical’ value. Those that have are transferred to The National Archives, or a recognised place of deposit d) ‘Case files’ (today a term which includes datasets) are dealt with outside the system of review. Grigg proposed that all case files created across government be appraised as a whole, enlisting historical advice e) Specific guidance applies to unregistered files, such as private office papers, films, sound recordings f) The National Archives advice on how to decide what records are of historical value has been given in various manuals for departments and, more recently, through the Acquisition and Disposition Policies and the accompanying Operational Selection Policies (OSPs) g) The Grigg Committee, the subsequent Wilson Committee, and ensuing (National Archives) advice all emphasise the need for good systems of records creation and the extensive use of disposal schedules h) The timing of disposal for common administrative records is closely regulated in many departments by disposal schedules, supported by general guidelines provided by The National Archives for classes of administrative records, such as accounting records, legal records, personnel records, estate records and so on. Determining the value of records

The National Archives gives guidance to enable reviewers to judge the value of records, especially in terms of ‘historical value’:

a) At first review the main criterion is administrative need for departmental purposes supplemented by two caveats made in the Grigg report: that first review should take place as soon after closure of the file as possible thereby ensuring more records are retained; and that at first review consideration be given as to whether the function being carried out might arise again in the future, for example, rationing. In addition, The National Archives advises that in a few cases the records’ possible historical value be taken into consideration at this stage.14

b) For second review The National Archives’ orange Manual of Records Administration defined two types of selection criteria for records: selection to show the significance of the functions and activities of the department or selection on the basis of the informational content of the records. The Manual further recommended that reviewers should provide material for research into:

- the history of the department, its organisation and procedures
- the formulation of policy and legislation and, more selectively, its implementation and interpretation
- notable events or persons when the records add significantly to what is already known
- major events, developments or trends in political, social, legal or economic history
- scientific, technological or medical research and development
- regional or local conditions when it is unreasonable to expect information to be available locally, when it is convenient to hold it centrally, or when it is known that significant local information does not survive locally
- demographic, medical, social, cultural and economic history and historical geography, by means of statistical and quantitative research.15

______________________________________________________________________

14 PRO Government Services Division Manual of Records Administration August 1993, para 3.3.6. Grigg states that the terms ‘for its own Departmental purposes’ be interpreted to cover ‘the likelihood of the paper being required as a precedent or as a guide to possible action should a similar set of circumstances arise in the future.’ Such a proviso would catch records showing a Department was organised in order to carry out a set of policies or procedures or it might stop the destruction of papers because the activities they dealt with had ceased, such as rationing. The National Archives, in common with many archives around the world generally adopted the taxonomy of value developed by an American archivist, TR Schellenberg. He argued that a record has two different layers of value - a primary value or the value to the organisation that created them and a secondary value to historians. ‘Primary value’ consisted in the value to the organisation for administrative, legal and fiscal purposes of records and ‘secondary value’ consisted in value to other users, to historians, value that was never intended by the creator. And within secondary value, there was ‘evidential’ value - a value derived from the way the record documented the history structure and functions of an organisation and ‘informational’ value or value in providing research material on persons, places and subjects. See TR Schellenberg, Modern Archives. Principles and Techniques (Chicago: University of Chicago Press, 1956); TR Schellenberg, The Management of Archives (New York: Columbia University Press, 1965)

15 Manual para 3.6.11 c) In 2000 The National Archives published its Acquisition and Disposition Policies (see 1.2). The Acquisition Policy defines the limits of the archive. It works in conjunction with the Disposition Policy which is a framework of principles on which to base decisions to offer public records to an archival institution other than The National Archives.

d) Since 2000 these two policies have been developed using Operational Selection Policies (OSPs). 28 of these exist which probably now cover 20% of the records accessioned into The National Archives each year. The OSPs cover specific sets of case files (coroners’ records), or disposal of all of the records of a department (Royal Mint) or may cover a theme such as recreational use of the countryside.

e) The Grigg Report recommended that a Committee be set up under the auspices of the PRO to conduct a census of what case files are in departments, and ‘to determine what papers, if retained, would give the greatest amount of information in the smallest amount of space’ and so ‘decide which papers, and in what quantities, should be preserved’\\textsuperscript{16}. A Particular Instance Papers Committee (PIPC) met in the 1960’s, but the census was never conducted. Since 200 the role of the PIPC has, to an extent, been filled by the Records Review Panel, an internal committee based in the Records Management Department of The National Archives.

\\textsuperscript{16} Departmental Records para 109