Data Quality 2017-18 City of York Council Internal Audit Report

Business Unit: Customer and Corporate Services, Health, Housing and Adult Social Care Responsible Officer: Assistant Director Customer Services & Digital, Assistant Director Adults & Social Care Service Manager: Shared Intelligence Bureau Group Manager Date Issued: 15/05/2018 Status: Final Reference: 11040/009

| Actions | P1 | P2 | P3 | |---------|----|----|----| | | 0 | 0 | 0 |

Overall Audit Opinion: High Assurance Summary and Overall Conclusions

Introduction Public services require information that is accurate, reliable, complete, relevant and timely in order to effectively manage service delivery, to make decisions and to account for performance. A balance must be achieved between the need for information and the cost of collecting the supporting data.

The council’s performance indicators are reported on an application called the KPI Machine which acts as a central repository enabling management to access this information at any time. Council data is made available to the public through the York Open Data Platform – a separate web-based system.

Corporate arrangements for data management are performed by the Strategic Business Intelligence Hub which is responsible for the collation, processing, integrity checking and reporting of data held by the council.

There is an ever-rising demand for Health and Adult Social Services and the council needs reliable and insightful performance data to provide these vital services. Performance information for Adults and Children’s Social Care is recorded on the MOSAIC system. The MOSAIC system was implemented in 2015 to replace RAISE and frameworki.

This audit provided a detailed review and re-performance of a number of indicators from the MOSAIC System.

Objectives and Scope of the Audit The purpose of this audit was to provide assurance to management that procedures and controls within the system will ensure that:

- Datasets provided are complete, accurate and relevant for the calculation of the indicator;
- Data is correctly processed in order to calculate the indicator;
- The data gathering process for each indicator is understood and reviews are undertaken of the final output figures to confirm their accuracy.

This audit did not review the performance management system itself but observed the accuracy of calculation of indicators from the processing of the base data to the final representation of the figures in performance reports. Key Findings

This audit found that there is an effective system of controls in place to calculate and record the key performance indicators under review. In order to assess the efficacy of the control environment, a review of the calculation process, including any written guidance available, was conducted. Walkthrough testing with the latest published datasets was also performed for the sample of chosen indicators:

- Delayed Transfers of Care (Delayed Discharges);
- The % of adults receiving secondary mental health services in paid employment;
- The % of adults receiving secondary mental health services living independently.

Before the raw datasets are accessed by the BIH, they have already been collated and reviewed by NHS Digital. These checks are supplemented by a series of controls to ensure the accuracy of the final indicators by the BIH. At the heart of these controls is the creation of detailed process maps for each indicator by the Senior Business Intelligence Officer. These explain the procedure for collecting and processing the raw dataset into the final figures. A review of the process maps found that they provide clearly structured, accurate guidelines for the necessary calculations and a framework to ensure consistency in the implementation of controls when producing the indicator. The inclusion of process maps for each indicator prevents the procedure from becoming reliant on the knowledge and experience of the officer producing the indicator. This ensures that any officer could understand how to calculate the indicators accurately. Although there are a significant number of manual processes involved in calculating and recording the performance indicators, these elements are clearly guided by the process maps and an extra layer of control is included within the design of the relevant Excel worksheets. During testing, it was found that the automation and design of the graphical output ensures even novice officers will be made aware of any deviations from past indicator values, signposting areas for further investigation.

Throughout the process of calculating the indicators, these formal controls are supplemented by frequent spot-checks against previous years’ results to confirm the data is being correctly processed and the formal controls are functioning correctly. Review of the final indicator involves a reasonableness check against the existing database of previous year’s values and a re-performance, where necessary, to directly verify the accuracy of the result. In practise, this means any errors are likely to be identified quickly and again, there is less reliance on an individual’s ability to identify what a ‘typical value’ would constitute.

Therefore a number of checks are in place to ensure not only that the data is being correctly processed in order to calculate the indicators, but that staff understand and appreciate the importance of continued re-evaluation to guarantee the accuracy of the procedure. The absence of error in the annual published figures is further indication of the strong controls that have been implemented.

Overall Conclusions

It was found that the arrangements for managing risk were very good. An effective control environment appears to be in operation. Our overall opinion of the controls within the system at the time of the audit was that they provided High Assurance. Audit Opinions and Priorities for Actions

### Audit Opinions

Audit work is based on sampling transactions to test the operation of systems. It cannot guarantee the elimination of fraud or error. Our opinion is based on the risks we identify at the time of the audit.

Our overall audit opinion is based on 5 grades of opinion, as set out below.

| Opinion | Assessment of internal control | |-----------------------|-------------------------------------------------------------------------------------------------| | High Assurance | Overall, very good management of risk. An effective control environment appears to be in operation. | | Substantial Assurance | Overall, good management of risk with few weaknesses identified. An effective control environment is in operation but there is scope for further improvement in the areas identified. | | Reasonable Assurance | Overall, satisfactory management of risk with a number of weaknesses identified. An acceptable control environment is in operation but there are a number of improvements that could be made. | | Limited Assurance | Overall, poor management of risk with significant control weaknesses in key areas and major improvements required before an effective control environment will be in operation. | | No Assurance | Overall, there is a fundamental failure in control and risks are not being effectively managed. A number of key areas require substantial improvement to protect the system from error and abuse. |

### Priorities for Actions

| Priority | Description | |----------|-----------------------------------------------------------------------------| | Priority 1 | A fundamental system weakness, which presents unacceptable risk to the system objectives and requires urgent attention by management. | | Priority 2 | A significant system weakness, whose impact or frequency presents risks to the system objectives, which needs to be addressed by management. | | Priority 3 | The system objectives are not exposed to significant risk, but the issue merits attention by management. | Where information resulting from audit work is made public or is provided to a third party by the client or by Veritau then this must be done on the understanding that any third party will rely on the information at its own risk. Veritau will not owe a duty of care or assume any responsibility towards anyone other than the client in relation to the information supplied. Equally, no third party may assert any rights or bring any claims against Veritau in connection with the information. Where information is provided to a named third party, the third party will keep the information confidential.