Data Quality City of York Council Internal Audit Report 2015/16

Business Unit: Customer & Business Support Services Responsible Officer: Assistant Director – Customers and Employees Service Manager: Group Manager – Shared Intelligence Bureau Date Issued: 20 June 2016 Status: Final Reference: 11040/007

| Actions | P1 | P2 | P3 | |---------|----|----|----| | Overall Audit Opinion | Substantial Assurance | Summary and Overall Conclusions

Introduction

Public services require information that is accurate, reliable, complete, relevant and timely in order to effectively manage service delivery, to make decisions and to account for performance.

Users of performance information must be able to place reliance on the data supplied but a balance must also be achieved between the need for information and the cost of collecting the supporting data with the necessary accuracy, detail and timeliness.

The council’s performance indicators are reported on an application called the KPI Machine which acts as a central repository enabling management to access this information at any time. The KPI Machine is also used to update Members on performance. Council data (as well as that of other organisations) is made available to the public through the York Open Data Platform – a separate web-based system. Corporate arrangements for data management have been centralised with the establishment of the Strategic Business Intelligence Hub which is responsible for the collation, processing, integrity checking and reporting of data held by the council.

A total of 10 council indicators were selected for detailed review. Five of these were statutory indicators chosen from the council’s BI schema, with the remaining five being selected from the KPI machine.

Objectives and Scope of the Audit

The purpose of this audit was to provide assurance to management that procedures and controls within the system ensure that:

- Datasets provided are complete, accurate and relevant for the calculation of the indicator
- Data is correctly processed in order to calculate the indicator
- The data gathering process for each indicator is understood and reviews are undertaken of the final output figures to confirm their accuracy

This audit included a review of arrangements to ensure data quality to the extent possible within the control framework of the Business Intelligence Hub (BIH) and did not include an in-depth review of the quality of data and collection methods at source. Key Findings

Overall, a sound control environment was found to be in place within the Business Intelligence Hub (BIH) to ensure correct processing of data and calculation of final indicator figures. However, for all indicators tested, reliance is placed on the accuracy and completeness of pre-existing system reports and these reports are not regularly reviewed to ensure they are set up correctly and continue to work as intended.

Audit testing performed on the indicators reviewed did not uncover any issues with processing that directly affected the accuracy of reported performance. Review of final indicator figures is, however, limited to basic reasonableness checks. Controls such as re-performance, reconciliation and second officer review which would directly verify the accuracy of the figure are not in place. It is recognised that these controls are resource intensive and the absence of error in reported performance suggests that the risks do not justify such resource intensive controls. That said, it would not be unreasonable for the BIH to periodically spot-check back to source data and confirm the accuracy and completeness of the source system to ensure that the data is correct on reaching the Hub.

There are no set rules for direction of travel and this was found to be a subjective judgement made by the Business Intelligence Officers responsible for each indicator. While knowledge of the service areas was generally found to be very good, interpretation of performance may differ from that of the services.

Substantial improvement was found to have been made in the standard of procedure notes and these are now available for the majority of indicators, although inclusion of a formal definition of the indicator within these documents would represent a further improvement.

Overall Conclusions

It was found that the arrangements for managing risk were good with few weaknesses identified. An effective control environment is in operation, but there is scope for further improvement in the areas identified. Our overall opinion of the controls within the system at the time of the audit was that they provided Substantial Assurance. 1 Reliance on system reports

| Issue/Control Weakness | Risk | |--------------------------------------------------------------------------------------|----------------------------------------------------------------------| | For a high proportion of indicators reliance is placed on the completeness, accuracy and relevance of a system report. | Incorrect performance is calculated and, as a result, decisions are made on the basis of erroneous information. |

Findings

Audit testing found that a high proportion of indicators are reliant on the completeness, accuracy and relevance of pre-existing system reports for the calculation of the indicator.

While Business Intelligence Officers were able to demonstrate good understanding of data collection at source, it is important for the Business Intelligence Hub to have assurance that these reports remain appropriate for the calculation of the indicator to ensure their continued accuracy.

For the indicators tested, there had not been any review of the reports themselves to check that they are set up with the correct parameters and are extracting complete, accurate and reliable data from the source systems.

Agreed Action 1.1

An internal reporting information architecture working group will be established within the Business Intelligence Hub in order to provide a meeting structure to co-ordinate standards on report development and sign off and to approve any changed or new reports.

The Business Intelligence Hub will also look to establish, alongside IT and business areas, governance boards when there are newly implemented major case management systems. These boards will include reporting as an agenda item which will provide a mechanism for maintaining assurance that all reports remain appropriate.

The Business Intelligence Hub is centralising all reporting through the KPI machine. A piece of work to capture metadata on when reports were last run, tested and viewed by the relevant business area and by a member of the management team will be undertaken. It is the intention to put this architecture in place for 80% of reports that the Hub is responsible for by the end of 2016/17.

| Priority | Responsible Officer | Timescale | |----------|---------------------|-----------| | 2 | Group Manager – Shared Intelligence Bureau | 31 March 2017 | 2 Reasonableness checks

| Issue/Control Weakness | Risk | |---------------------------------------------------------------------------------------|----------------------------------------------------------------------| | Checks of the accuracy of the final indicator figure are predominantly limited to | Erroneous final indicator figures are not identified, incorrect | | reasonableness checks rather than direct verification of accuracy. | performance is reported and decisions are made on the basis of | | | inaccurate information. |

Findings

For the indicators tested, review of the final output figure was found to be limited to basic reasonableness checking against previous months’ figures, with no direct verification of accuracy being undertaken. Accordingly, identification of processing errors relies on the reported figure being significantly different to expected performance. Although bi-weekly background validation routines are also run on the indicators uploaded onto the KPI machine, these are limited in the same way as reasonableness checking in that only unexpected variances are detected.

Whilst it would be too resource intensive to check indicator figures back to source data each time they are produced, there would be value in doing periodic spot-checks to source data in order to gain assurance that the indicator is being correctly calculated and reported. This would compliment the reviewing of the reports themselves (as noted in finding 1) and together would provide more assurance for the BIH that performance is being accurately reported.

Agreed Action 2.1

The Business Intelligence Hub is currently testing its ability to have systematic and programmatic solution towards flagging-up all data abnormalities (i.e. an automated Quality Assurance process that is designed to bring to attention any potentially erroneous values) which would help pinpoint and target any issues.

The Business Intelligence Hub will also introduce a ‘date of review’ field in the KPI database and programme the Machine to randomly select KPIs for source data review every six months.

| Priority | Responsible Officer | Timescale | |----------|---------------------|-----------------| | 2 | Group Manager – | 31 March 2017 | | | Shared Intelligence Bureau | | 3 Direction of travel

| Issue/Control Weakness | Risk | |---------------------------------------------------------------------------------------|----------------------------------------------------------------------| | Subjectivity in the determination of 'good', 'bad' or 'neutral' performance when | Level of reported performance differs from that of service | | reporting on direction of travel. | interpretation of performance. | | | Good and poor direction of travel scorecards reported to the | | | Council Management Team are not a true reflection of | | | performance as per service interpretations. |

Findings

Determining the direction of travel is a manual process that requires an assessment of the change in the indicator figure in relation to performance tolerances and sensitivity of the indicator to change.

This assessment is undertaken on the basis of Business Intelligence Officer knowledge and judgement. There are no set rules for determining what degree of change in performance represents good or bad direction of travel. Whilst it is appreciated that it is difficult to have set rules, judgements on direction of travel have not been determined in consultation with the services.

Agreed Action 3.1

The Business Intelligence Hub intends to develop and implement KPI Machine v2 throughout 2016/17. This version will include an automated Direction of Travel. The Hub has already built the functionality within a development environment and it is currently undergoing testing.

Once a new indicator is registered within the database, a value for 'Up is good' true/false will assigned together with the '% degree of change for neutral score'. This will then provide a basis for a report to be written that will compare the latest data with previous data-points and then check this against the direction of travel.

| Priority | 3 | |----------|---| | Responsible Officer | Group Manager – Shared Intelligence Bureau | | Timescale | 31 March 2017 | 4 Indicator definitions

| Issue/Control Weakness | Risk | |------------------------|------| | There is no formal definition for indicators. | In the event that key members of staff leave the council, important finer details in the production of the indicator are lost and incorrect performance is reported. |

Findings

Although procedure notes were found to be in place for almost all indicators and generally provided a sufficient level of detail to enable performance of the processing task, none of these documents included a formal definition of the indicator.

Inclusion of a formally accepted definition is fundamental in determining exactly what is and what is not included in the calculation of the indicator and, therefore, for the accuracy of the final output. Furthermore, a definition would allow the BIH and services to be assured that they are working with the same definition, minimising the chances of confusion, misunderstanding or misinterpretation of what the indicator shows.

Agreed Action 4.1

A KPI Dictionary will be developed. However, at present there are over 12,000 items of management information. This will require a prioritised approached starting with those KPIs that are published on the York Open Data Platform.

| Priority | Responsible Officer | Timescale | |----------|---------------------|-----------| | 3 | Group Manager – Shared Intelligence Bureau | 31 March 2017 | Audit Opinions and Priorities for Actions

Audit Opinions

Audit work is based on sampling transactions to test the operation of systems. It cannot guarantee the elimination of fraud or error. Our opinion is based on the risks we identify at the time of the audit.

Our overall audit opinion is based on 5 grades of opinion, as set out below.

| Opinion | Assessment of internal control | |--------------------|------------------------------------------------------------------------------------------------| | High Assurance | Overall, very good management of risk. An effective control environment appears to be in operation. | | Substantial Assurance | Overall, good management of risk with few weaknesses identified. An effective control environment is in operation but there is scope for further improvement in the areas identified. | | Reasonable Assurance | Overall, satisfactory management of risk with a number of weaknesses identified. An acceptable control environment is in operation but there are a number of improvements that could be made. | | Limited Assurance | Overall, poor management of risk with significant control weaknesses in key areas and major improvements required before an effective control environment will be in operation. | | No Assurance | Overall, there is a fundamental failure in control and risks are not being effectively managed. A number of key areas require substantial improvement to protect the system from error and abuse. |

Priorities for Actions

| Priority | Description | |----------|-----------------------------------------------------------------------------| | Priority 1 | A fundamental system weakness, which presents unacceptable risk to the system objectives and requires urgent attention by management. | | Priority 2 | A significant system weakness, whose impact or frequency presents risks to the system objectives, which needs to be addressed by management. | | Priority 3 | The system objectives are not exposed to significant risk, but the issue merits attention by management. | Where information resulting from audit work is made public or is provided to a third party by the client or by Veritau then this must be done on the understanding that any third party will rely on the information at its own risk. Veritau will not owe a duty of care or assume any responsibility towards anyone other than the client in relation to the information supplied. Equally, no third party may assert any rights or bring any claims against Veritau in connection with the information. Where information is provided to a named third party, the third party will keep the information confidential.