Source document: https://docs-lawep.s3.us-east-2.amazonaws.com/1697639701534.pdf (Australia)
Chapter 3 Australian Crime Commission performance measurement This chapter considers the ACC's performance against its Key Performance 3.1 Indicators (KPIs).

It examines the measurement tools utilised to inform the KPIs and to track performance over time.
Key Performance Indicators 3.2 The Outcomes and Outputs Framework (the framework) provides the basis for the government's approach to budgeting and reporting for public sector agencies and the means by which the Parliament appropriates funds in the annual budget context.1 In 2010, the Joint Committee on Public Accounts and Audit observed that measuring key aspects of an agency's performance is a critical part of the Government's Outcomes Framework.2 Within the context of the framework, KPIs are 'established to provide information (either qualitative or quantitative) on the effectiveness of programs in achieving objectives in support of respective outcomes'.3 The Department of Finance (Finance) has provided the following guidance for 3.3 agencies in developing KPIs: Agencies should focus on reporting a strategic and meaningful level of performance indicators, demonstrating the link between the program performance indicators and the outcome.4 3.4 Agencies are required to provide a relevant, informative and useful range of performance indicators that can be tracked over time.
In advice to entities on developing KPIs, Finance recommended that agencies use both qualitative and quantitative information to measure program performance and provided the following definitions: 1 2 3 4 Australian National Audit Office, Audit Report No.

23 2006-07, Application of the Outcomes and Outputs Framework, 2007, p. 15.
Joint Committee of Public Accounts and Audit, Inquiry into the Auditor-General Act 1997, Report 419, December 2010, p. 20, www.aph.gov.au/Parliamentary_Business/Committees/House_of_Representatives_Committees ?url=jcpaa/agact/report.htm (accessed 13 June 2013).

Australian National Audit Office, Audit Report No.


5 2011-12, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, September 2011, p. 9, www.anao.gov.au/Publications/Audit-Reports/2011- 2012/Development-and-Implementation-of-KPIs-to-Support-the-Outcomes-and-Programs- Framework (accessed 13 June 2013).
Department of Finance, Guidance for the Preparation of the 2012-13 Portfolio Budget Statements, March 2012, p. 38, www.finance.gov.au/budget/budget-process/portfolio-budget- statements.html (accessed 19 March 2014).

10 Quantitative: This type of reporting is represented by numbers or percentages in a table.

Qualitative: This type of reporting is represented by narrative text.
Agencies should identify aspirational goals or milestones that are intended to be achieved by the program.5 Finance noted that KPIs must be designed to be 'capable of signalling to 3.5 government, Parliament and the community whether programs are delivering intended results'.6 Further, consistent, clear reporting on performance provides an important record of an agency's 'progress towards meeting government policy objectives, how well public money is being spent and whether planned achievements are on track'.7 3.6 A performance audit report from the Australian National Audit Office (ANAO) titled Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework emphasised the importance of an 'appropriate mix of qualitative and quantitative KPIs including targets against which progress towards program objectives could be assessed'.8 The ANAO noted that a tendency to rely on qualitative KPIs reduces the ability of an agency to measure the results of program activities over time.
Whereas: A mix of effectiveness KPIs, that place greater emphasis on quantitative KPIs and targets, would provide a more measurable basis for performance assessment.9 The ANAO argued that because KPIs are statements of the pre 3.7 expected impacts of a program, it is important that they are: defined and • • • • specific—so as to focus on those results that can be attributed to the particular intervention/program; ‐ measurable—include quantifiable units or targets that can be readily compared over time; achievable—realistic when compared with baseline performance and the resources to be made available; relevant—embody a direct link between the program’s objective and the respective effectiveness KPI; and Department of Finance, Performance Information and Indications, October 2010, p. 7, www.finance.gov.au/financial-framework/financial-management-policy- guidance/performance-information-and-indicators.html (accessed 19 March 2014).
Department of Finance, Performance Information and Indications, October 2010, p. 2.

Department of Finance, Performance Information and Indications, October 2010, p. 1.

Australian National Audit Office, Audit Report No.

5 2011-12, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, September 2011, p. 53.

Australian National Audit Office, Audit Report No.


5 2011-12, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, September 2011, p. 53.
• timed—include specific timeframes for completion.10 ACC's outcome and program structure 3.8 The ACC's outcome and program structure are set out in its annual report: 11 Outcome 1 Reduction in the threat and impact of serious and organised crime, through analysis of and operations against national criminal activity, for governments, law enforcement agencies and private sector organisations.
Outcome strategy Collaborate with law enforcement and related government agencies and private industry as part of the national effort against serious and organised crime.11 3.9 The ACC's single outcome is underpinned by two programs: • • Program 1.1.1—Strategic criminal intelligence services, the performance of which is measured by two KPIs; and Program 1.1.2—Investigations and intelligence operations into federally relevant criminal activity, which are measured by six KPIs.12 Program 1.1.1 3.10 Program 1.1.1 has a set of deliverables to meet the overall aim that: intelligence services are designed to provide The ACC's criminal Commonwealth, state and law enforcement and relevant government agencies with the criminal intelligence necessary to effectively and efficiently disrupt serious and organised criminal activity and reduce the vulnerabilities posed to the Australian community.13 territory The following table lists Program 1.1.1's KPIs, their targets and results for 3.11 2012-13.
10 Australian National Audit Office, Audit Report No.

5 2011-12, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, September 2011, pp 15–16.

11 ACC, Annual Report 2012-13, p. 32.

12 ACC, Annual Report 2012-13, pp 59; 112.

13 ACC, Annual Report 2012-13, p. 261.

12 Program 1.1.1—ACC KPIs and performance 2012-13 14 KPI 2012-13 target 2012-13 result 1.
Strategic intelligence products 90% 100% align with ACC Board endorsed National Criminal Intelligence Priorities.

2.

Partner agencies agree or strongly that the ACC's intelligence enhances their understanding of serious and organised crime.

90% 90% of senior executives of partner agencies agree/strongly agree 3.12 In relation to the first KPI, the annual report noted that a combined 1866 analytical and tactical intelligence products were produced during 2012-13.
Of these products, 100 per cent align with the National Criminal Intelligence Priorities (NCIPs), including all 16 of the ACC strategic products in the Picture of Criminality in Australia suite.15 The second KPI is measured by stakeholder feedback and the explanatory 3.13 note in relation to it states that: Each partner agency has its own role and priorities, and each has different needs for and uses of criminal intelligence.
We continue to work with our stakeholders to deepen our understanding of their needs and improve the value of our intelligence from their perspective.16 Program 1.1.2 3.14 The overall aim of program 1.1.2 is as follows: services by providing unique The ACC's investigations and intelligence operations underpin its criminal intelligence intelligence collection capabilities.
ACC investigations are conducted in partnership with law enforcement agencies with the objective of disrupting and deterring federally relevant serious and organised criminal activity.

In 2012-13, the ACC, under the guidance of its Board, will further focus its coercive powers determinations to more comprehensively address emerging issues in the organised crime environment.17 14 ACC, Annual Report 2012-13, p. 60.

15 ACC, Annual Report 2012-13, p. 61.

16 ACC, Annual Report 2012-13, p. 61.


17 ACC, Annual Report 2012-13, p. 261.

Program 1.1.2—ACC KPIs and performance 2012-1318 KPIs 2012-13 result 13 1.

Partner agencies agree, or strongly agree, that the ACC's contribution enhances efforts to combat serious and organised crime.

2.

Partner agencies agree, or strongly agree, that the ACC's coercive powers are effective.

3.

Targeted ACC investigations and operations are aligned with ACC Board priorities and approved by the ACC Board.

4.
4.

The ACC's operational intelligence and contribution to joint intelligence investigations and operations enhance the efficiency and/or effectiveness of law enforcement efforts to disrupt and deter serious and organised crime, as measured by stakeholder feedback.

5.

The activities of targeted criminal entities are disrupted as a result of ACC intelligence, investigations and operations, and activity is undertaken to confiscate proceeds of crime.
100% of managers and senior executives agree (target was 80%) 80% (target was 90%) 100% of senior executives of partner agencies agree/strongly agree 100% of senior executives of partner agencies agree/strongly agree 26 disruptions 185 people charged 332 charges laid 38 people convicted $83.87 million proceeds of crime restrained $26.02 million proceeds of crime forfeited $159.98 million tax assessments issued 34 firearms seized $16.34 million in cash seized $589.52 million estimated street value of drugs seized $84.74 million value of the illicit drug precursors production seized.19 potential of 6.
Availability of the Australian Criminal Intelligence Database and the Australian Law Enforcement Intelligence Network.

>99% (target was 98%) 18 ACC, Annual Report 2012-13, p. 112.

19 ACC, Annual Report 2012-13, p. 115.

14 3.15 The availability of ACID and ALEIN achieved greater than 99 per cent in 2012-13 and 2011-12.
In 2012-13, there were 1 302 439 searches on ACID compared to 331 664 in 2011-12, a fourfold increase.20 The ACC noted that this was 'due primarily to increased multiple-entity searches by the ACC-led National Criminal Intelligence Fusion Capability'.21 that responded the government has The committee notes 3.16 the recommendation made in its last report22 that the ACC review and re-examine its KPI concerning the ACID and the ALIEN.
In response, the government stated that it accepts the recommendation and that the ACC is currently undertaking a scoping study to determine a replacement system for ACID and to improve ALEIN.

In addition, the ACC has provided a new KPI for ACID and ALEIN.

The performance of ACID and ALEIN will now be measured against 'provision of a national criminal intelligence database and analytical tools, which facilitate the sharing and analysis of criminal intelligence across jurisdictions.
'23 to The committee will carefully examine the ACC's 2013-14 Annual Report with 3.17 a view to determining the extent to which the change in KPI has resulted in improved performance information.

Challenges in measuring ACC performance 3.18 The annual report highlights some of the complexities in measuring ACC results.
It notes that most of the work undertaken by the ACC is classified, or has long-term results often being realised years after the ACC's initial involvement, and once court decisions, law reforms and changes in community behaviour have concluded or been implemented.24 3.19 During its examination, the committee asked the ACC to comment on the difficulties surrounding the measurement of the performance of the ACC, as much of the work undertaken by the ACC is utilised by other agencies.
Mr Jevtovic stated that the ACC is eager to demonstrate the worth of investment in the organisation for crime fighting, but that it is difficult to monitor how much of the information is used: …we might produce 2500 information reports, which we disseminate to Australian law enforcement over the country.

Once we disseminate that 20 ACC, Annual Report 2012-13, p. 110.

21 ACC, Annual Report 2012-13, p. 110.


22 Joint Parliamentary Committee on Law Enforcement, Examination of the Australian Crime Commission Annual Report 2011-12, September 2013, p. 16.

23 Government Response to the Parliamentary Joint Committee on Law Enforcement report: Examination of the Annual Report of the Australian Crime Commission 2011-12, March 2014, p. 1, www.aph.gov.au/~/media/Committees/Senate/committee/le_ctte/annual/2013/ACC/gov_respon se/gov_response.pdf (accessed 31 July 2014).
24 ACC, Annual Report 2012-13, p. 5.
15 information, there is no technical solution to monitoring which component of our 2500 products has actually been accessed or used if the people using it do not tell us…that is the reality and we accept that.25 3.20 However, the level of reporting by the ACC was improved through changes made to the ACC's Performance Measurement framework, as recommended by the committee in its last report.26 This involved adapting the framework to: • • more accurately reflect both quantitative and qualitative results—capturing short-term and long-term, direct and indirect impacts of activities; and measure the performance against the new Strategic Plan 2013–18, which highlights the evolution of serious and organised crime and the need to continually adapt and develop the right capabilities to respond.27 The annual report notes that these changes have had a positive outcome on 3.21 performance: Our revised indicators enable us to better self-assess our performance, to complement feedback from our stakeholders which we intend to gather more regularly throughout each reporting year.
This will enable us to report in new and different ways on the value we add to the national fight against serious and organised crime.28 3.22 The revised performance management framework will require the ACC to design new systems for collecting performance data, which will be implemented over the course of the five year Strategic Plan.
The committee will examine the ACC's 2013-14 Annual Report with a view to determining the efficiency of the new systems for collecting and reporting on performance data over the course of the Strategic Plan.

Stakeholder surveys 3.23 The annual report notes that stakeholder relationships are critically important to the ACC's success in combating serious and organised crime.

The ACC uses stakeholder surveys to help inform ACC reporting obligations.
In 2012-13, the ACC than engaging market revised researchers to gather data through stakeholder survey forms, the ACC conducted its research in-house with a wider range of qualitative interviews.

According to the ACC, its stakeholder research methodology.

Rather 25 Mr Paul Jevtovic, Acting CEO, ACC, Committee Hansard, 5 March 2014, pp 4–5.

26 Joint Parliamentary Committee on Law Enforcement, Examination of the Australian Crime Commission Annual Report 2011-12, September 2013, p. 22.
27 ACC, Annual Report 2012-13, p. 138.

The ACC developed the Strategic Plan 2013–18 over the course of the reporting period.

The report was developed following a comprehensive review of the sector as well as staff and stakeholder engagement (ACC, Annual Report 2012-13, p. 121).

In the review by the Chief Executive Officer, it is noted that the 'new strategic plan and associated roadmap for the future include important steps to cement our collaborative approach and take it to the next level.'
(ACC, Annual Report 2012-13, p. 22).

28 ACC, Annual Report 2012-13, p. 139.

16 the result provided 'more in-depth information for analysis, [and] a more focused indication of how [the ACC] performed', while reducing costs.29 Committee view 3.24 The committee appreciates the complexities involved for the ACC in developing meaningful qualitative KPIs that can be measured over time.

The committee also acknowledges that the nature of some of the ACC's work may not be directly quantifiable.
3.25 In its previous report the committee recommended that the ACC Annual Report 2012-13 should include information on progress made towards establishing a balance of quantitative and qualitative KPIs, which can be measured over time.30 The committee notes that the government has accepted this recommendation,31 and that the ACC has amended its key performance indicators for 2013-14 for both its Portfolio Budget Statement and its annual report.32 3.26 The new KPIs for outcome 1 include: • • • • • • • the ACC produces useful intelligence that identifies and provides insights on new and emerging serious and organised crime threats.
the ACC fills intelligence gaps through the identification of vulnerabilities and indicators of serious and organised crime.

the ACC collects and maintains national intelligence holdings of serious and organised crime threats and targets.

the ACC interprets and analyses national holdings to create a national serious and organised crime intelligence picture.
the ACC informs and influences hardening of the environment against serious and organised crime; the ACC influences or enables the disruption, disabling or dismantling of serious and organised crime; and the ACC participates in or coordinates collaboration in joint investigations and operations to prevent and disrupt serious and organised crime.33 29 ACC, Annual Report 2012-13, p. 25.
30 Joint Parliamentary Committee on Law Enforcement, Examination of the Australian Crime Commission Annual Report 2011-12, September 2013, p. 22.

31 Government Response to the Parliamentary Joint Committee on Law Enforcement report: Examination of the Annual Report of the Australian Crime Commission 2011-12, March 2014, p. 2, www.aph.gov.au/~/media/Committees/Senate/committee/le_ctte/annual/2013/ACC/gov_respon se/gov_response.pdf (accessed 31 July 2014).

32 ACC, Annual Report 2012-13, p. 138.


33 Attorney-General's Department, Portfolio Budget Statement 2013-14, Australian Crime Commission, p. 87.

3.27 The committee notes these new KPIs and will closely examine the relevant information included in the ACC's 2013-14 annual report, to ensure that it provides a more measurable basis for performance assessment.

17