Nagoya Journal of Higher Education Vol.1 (2001)

On the Use of Performance Indicators
in Japan’s Higher Education Reform Agenda
Darrell R. LEWIS*IKEDA Terumasa**
Halil DUNDAR***<Outline>
This paper explores the use of performance indicators as the principle
mechanism for assessing accountability in the reform of higher education in Japan. We review the (a) historical arguments in Japan for
employing performance indicators and the uses of such indicators
elsewhere in the world, (b) factors calling for their use and the notionof accountability in higher education, (c) nature, concerns and controversies with performance indicators in higher education, (d) realitythat performance indicators often lead to performance funding andthe use of government budgets as policy tools, and (e) problematic
issues in performance-based funding. All of these dimensions haveimportant implications for the current scene in Japan.

1. Introduction
In Japan, after more than 15 years of discussion and debate
between faculty, administrators and appointed officials within the
University Council and the Association of National Universities, and***Professor, College of Education and Human Development,
University of Minnesota***Professor, Center for the Studies of Higher Education,
Nagoya University***Education Economist and Project Manager, The World Bank

67

within the national Ministry of Education, Science, Sports andCulture [MESSC], the use of performance indicators has finallyarisen within the context of assisting reformation of the higher edu cation system. With the beginning of the 21st century it has been
proposed that each higher education institution in the country willdevelop appropriate "self-monitoring and self-evaluation" materials,
procedures and reports. Yet, little has been said about what criteria will be used, what performance will be monitored and assessed,
what purpose(s) will be served, and who will do such monitoring
and assessment. Much discussion has arisen with respect to theneeds for reform in the context of decentralization and the needs
for greater independence and flexibility for each of the public
national universities (MESSC, 1995, 1999; University Council, 1998),
as well as how to improve and restructure the curriculum, instruction and research infrastructure (MESSC, 1997), but little attention
has been given to how one goes about conducting the process inJapan and for what purposes. This paper is intended to help in
answering these questions.

2. Historical Arguments and Context in Japan
Since the mid-1980s there have been essentially two basic arguments for decentralizing and deregulating the national public universities in Japan. The first has been the argument that suchdecentralizing with appropriate public accountability will lead to
improved quality by introducing direct market competition fromthe many private schools already operating outside of governmental support and much regulation. This argument has not only been
coming from the private sector of higher education, but also fromthe executive branch of the national government. Even an advisory group of economists recently published a report under the auspices of the national Economic Planning Agency supporting greater
privatization of the public national universities. They argued that
more market competition between the national public and private
68

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

universities, along with more transparency and accountability,
would be essential for improving the quality of Japanese higher
education (Yonezawa, 1998).
A second related argument for greater decentralization andaccountability emphasizes the necessity to make information on thequality of universities available for more informed consumer choice
and decision-making. In brief, both arguments are being made so
as to induce greater effort in the public universities towards quality
enhancement and greater information available to consumers aboutsuch quality. Obvious in both of these arguments is the need formore and better information about the outcomes resulting from thepublic universities. So, the question really swings around what it isthat needs to be reported and who makes judgments about thisperformance in Japan.
In 1986, following recommendations by the National Council on
Educational Reform, the MESSC established a national University
Council as an advisory group of experts that included representatives from the universities, industry and labor unions. In the context of earlier admonitions about the need for the public national
universities to become more independent and self-reliant, theUniversity Council recommended the introduction of a "self-monitoring and self-evaluation" system for all of the higher education
systems, including the national public, local public, and private
schools. The council argued that continuous self-monitoring andself-evaluation would be essential for revitalizing universities andfor ensuring that universities were fulfilling their public responsibilities (Yonezawa, 1998). These recommendations were first put into
effect in the 1991 MESSC regulations on Standards for the
Establishment of Universities that required universities to makeefforts to enhance and maintain self-evaluation systems for teaching
and research.
Further urgings for self-monitoring and self-evaluation wereagain made within MESSC's report in 1995 on university reform.
Under this mid-1990s system, both internal and external assess69

ment programs were to be designed and carried out by each individual institution. The University Council strongly supported thisset of requirements and judged it to be consistent with institutional
preferences for academic freedom and autonomy. The latest
MESSC (1999, p. 17) report has estimated that by October, 1998 atleast 533 (or about 86 percent) of the country's 622 universities
were "involved" in some way with "self-monitoring and self-evaluation," but only 15 percent had introduced some form of external
assessment (Yonezawa, 1998). Although the Japan University
Accreditation Association had published guidelines for such selfassessment in 1991 and had in place a "voluntary external evaluation program" in 1996, most of the leading universities chose to
design their own unique approaches and only a few of the universities elected to submit themselves to external assessment. It
appears that almost everyone in Japan's higher education supports
advancing quality and is even willing to participate in self-evaluation, but few wish to have their institution submitted for external
assessment and review.
As calls for the "privatization" of the public national university
system continue to arise in both the private sector and the national
committee charged with making recommendations for restructuring the national government, the MESSC and the Association of
National Universities have come out strongly opposing these recommendations. Now, in the past year in response to these renewed
concerns about public higher education in Japan, both the ministry
and the University Council have come forward with plans fordeveloping a more systematic evaluation and quality assurance program for all of public higher education. Moreover, the University
Council has recently recommended the establishment of a new centralized assessment agency that would gather and publish data necessary for making comparisons between both public and private
universities independent of both the MESSC and individual universities.
The University Council (1998) consistently has been calling for70

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

increasing the ability of each university to "independently respondto changes" --i.e., to become more independent, more autonomous,and more deregulated from central governmental agencies, butthey have not called for the total "privatization" of the public
national universities as some outside of the universities had hoped
they would. They likewise have called for the establishment of anevaluation system conducted by third parties independent of both
universities and the MESSC, and for greater institutional independence such that individual presidents could provide appropriate
decision-making and implementation strategies within their respective institutions. They have also extended a call for greater evaluation of the "educational activities" and "research activities" of eachprogram and university. Most important, they have argued "universities, as a part of society, must further inform the public of
their activities...(and) should appropriately allocate resources in amore objective and transparent way on the basis of detailed information on evaluation." Nothing could be clearer. Japan needs anappropriate set of performance indicators and publication mechanisms for reporting to the public on the outcomes of its universities.

3. Historical Use of Performance Indicators Elsewhere in theWorld
Higher education over the past twenty years has had difficulties
not only in Japan but across the world as well. It has come underintense scrutiny over the past two decades as the general pressures for institutional quality, accountability, productivity, andaffordability have increased (e.g., MESSC, 1995, 1999; Gaither,
Nedwek & Neal, 1994; Borden & Banta, 1994; Layzell, 1996; Cave,
Hanney, Henkel & Kogan, 1997). The qualities of institutions ingeneral and undergraduate programs in particular and the utilization of resources have become matters of public debate and scrutiny across most developed nations. This move towards greater
accountability, productivity and quality has resulted in a growing
71interest in the use of performance indicators for higher education
to improve accountability--i.e, to monitor the "public investment" inhigher education. In recent years, especially throughout the 1990s
in Japan, as public resources for higher education have become
even more constrained, terms like accountability, productivity,
restructuring and quality have been used frequently in discussionsof higher education. This has come about partly as a result of the
public's perceptions that they may not be receiving sufficient value
from their public institutions.
One outcome of this debate is the growing number of nations
who have decided to adopt accountability reporting systems requir ing public institutions to assess and report their performance on acommon set of measures. All of this is particularly true in thecases of the United States (Burke & Serban, 1998), Canada(Advanced Education and Career Development, 1997), New Zealand(Lord, Robb & Shanahan, 1998), the United Kingdom, Australia, theNetherlands, Finland, Denmark, Norway, Sweden (Cave, Hanney,
Henkel & Kogan, 1997) and Japan (University Council, 1998), exceptthat no one has written about the need for a common set of mea sures in Japan that could be used for comparisons across institu tions.
It seems useful to review some of the basic concepts and principles of performance indicators to gain a better understanding of
how such measures and methods can contribute to accountability,
productivity, and quality improvement in higher education. The
purpose of this paper is to provide an overview of the literature tobetter understand the adequacy of presently used performancemeasures, methods and possible options, as well as their implications for Japan as it moves forward with its strategic plans forreforming public higher education. We believe that the paper provides a comprehensive review of performance indicator approaches
and major issues in their current use. The paper also includes
some practical recommendations and principles to guide the use ofhigher education performance indicators in Japan.
72

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

4. Factors Calling for Performance Indicators
Why are performance indicators increasingly being used inhigher education? What are the major factors that have triggered
the rapid growth of performance indicator systems across themany countries? Quite simply, the major factors have been competition for public funds, the need for greater efficiency and quality,
and the need for public accountability.
Competition for public funds: Higher education must increasingly compete for diminishing public funds. In some nations thereal values of tax revenues have declined through either inflation
or recession, while at the same time they are experiencing competing demands for public dollars. Such demands are coming from anumber of diverse sectors such as health care, transportation, social
services, corrections (i.e., prisons), and school reform movements.
In the case of Japan, the private higher education sector that currently is delivering education to over 75 percent of all students inhigher education also has been pushing for the central authorities
to "level the playing field" so that they might be better able to compete effectively with the public national universities.
Calls for greater efficiency and quality in delivering services:As demand and rising enrollments have rapidly expanded the public sector of higher education, both parents and public authorities
are questioning whether more effective competition might induce
greater efficiency as well as quality within the public sector ofhigher education.
Calls for greater accountability: Most governments hold all
public institutions, including those in higher education, accountable
for the use of public funds. This concern about accountability is anatural and appropriate outcome of public funding. Policymakers
demand more quantitative data about how well higher education is
performing and how public resources are used to produce thedesired outcomes and outputs in all of its dimensions of instruction,
research and outreach.
73

5. The Notion of Accountability in Higher Education
It may be necessary to briefly examine the notion and role ofaccountability in the development of performance measures.
Accountability is defined as demonstrating the worth and use of
public resources. Higher education in most countries has been
faced with greater demands to demonstrate its worth and toaccount for its use of public resources, partly as a result of fierce
competition for tightened state funds and partly as a result of otherrestructuring taking place throughout the public sector.
Policymakers have also criticized public institutions as being inefficient, incompetent, unproductive, oversized, overspecialized, andunder-focused on undergraduate education, while excessively
emphasizing graduate education and research, and being unresponsive to meeting other nationwide priorities and goals. Such criticisms have persuaded some politicians to charge that public colleges and universities do not deserve the amount of public support
that they have had in previous decades.
In response to such concerns, many countries have required
their public institutions to report on their achievements and failures to central state authorities on a set of performance indicators
to assess and monitor "public investment" in higher education. As
a logical extension of such a system of performance indicators,
many have also reexamined funding processes for public higher
education in order to target financial resources towards quality
improvement and greater accountability. There is a common belief
among many policymakers in developed countries that current
funding policies are obsolete and non-responsive to national economic realities and educational goals. Such policies have been criticized as having little relationship to institutional performance andproductivity. Policymakers have maintained that while current
public funding policies provide powerful tools for financial accountability, they have often failed to provide incentives to achieve public goals and objectives. As a result, many countries and states,74

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

such as those within the United States, the United Kingdom, theNetherlands, Australia, Finland and Sweden, also have attempted touse performance funding for the purposes of accountability andquality improvement in higher education (Cave et al., 1997).
It should be noted that accountability in higher education is by
no means a new concept in higher education despite the increased
concerns in recent years. National and state governments havealways held public institutions accountable. But the traditional
focus in accountability has been usually on financial accountability
that required public institutions to account for how they spend thefunds provided by governments. The question was "were thefunds spent appropriately?" However, in recent years, there has
been a change in the concept of accountability in higher education.
New accountability policies have increasingly focused on performance and outcomes. Public policymakers are interested in the
return on investment and they ask, "what did we achieve by spending public dollars in higher education and how well was it used"(Layzell, 1996).

6. What are Performance Measures?
Performance measures or indicators are typically defined as factual or opinion information, usually in quantitative forms (e.g., ratios,
percentages, ranks, and so forth) but also in qualitative forms as
well, about various aspects of the functioning of higher education
institutions and for various purposes-- e.g., monitoring, evaluation,and resource allocation (see, for example, Kells, 1992; Sizer, Spee &Bormans, 1992; Cave et al., 1997). Performance measurement
reflects the view that higher education needs to be more responsive to state concerns and more accountable to a broader constituency that includes students, employers, parents, and the general public.
Measures usually provide information about the resources(inputs), characteristics of the educational production (process), and75

outputs or outcomes at various levels of the higher education systems (e.g., system, institutional, or college) and allow institutions tocompare their relative position in key strategic areas to peers, topast performance, or to some standard or reference point. Selectedindicators range from simple quantifiable indicators such as student/faculty ratios or costs per student to more qualitative indicators such as student satisfaction measured by surveys or assessing
the quality of research and scholarship activity. Types, numbers,
and purposes of measures vary greatly by nation or state and by
institution. For example, while some institutions report about 250
specific measures, others report fewer than a dozen measures.
Nonetheless, the actual number of measures typically ranges
between 15 and 25; and most nations, states and institutions usuallyshare a common core of measures despite individual differences. It
should be noted that this consistency is more related to the availability of certain data rather than with broad consensus about what
is most important across the institutions. A comprehensive but by
no means complete list of potential performance measures in higher
education in the United States is provided in Borden and Banta(1994). The list includes about 280 specific measures in 21 areas,
ranging from admissions to teaching/learning. Another usefulsource would be the 1997 annual report on performance indicators
by the University of Minnesota (Office of the Executive Vice
President, 1997). Similar lists and examples of such performancemeasures are provided by Burke (1997) and Ruppert (1994) for theUnited States, by Cave and his colleagues (1997) and Johnes andTaylor (1990) for the United Kingdom, and by Lord and her colleagues (1998) for New Zealand.
The major contribution of the performance measurement
approach has been to provide a framework to examine data andlink the data with institutional priorities to facilitate strategicchanges. Performance measures provide indicators for achievement in relation to goals and for variation from expected levels ofactivity. They are useful for monitoring and evaluating perfor76

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

mance, and for guiding management and policy decisions.
Performance indicators can be powerful policy tools to help inform
higher education decision making when the indicators are also wellintegrated with other planning and funding policies. Their purposes vary by country system or institution but generally they havemuch broader views of performance assessment than earlier formsof student assessment. They focus not only upon students but also
on other strategic issues faced by the institutions such as research,outreach, administrative and management effectiveness and efficiency.
Performance measures have multiple uses. One should notlose sight of the multiple purposes for which performance measures
might be employed within higher education. Many institutions, forexample, use individual faculty performance indicators to assist in
making decisions about annual salary increases in those institutions
with a merit pay system. Other institutions use unit or program
performance indicators for resource allocation decisions within colleges (Lewis & Kallsen, 1995) or between colleges within universities (Dolence & Norris, 1994; Massy, 1996). Still others use performance indicators for reporting to their boards of trustees (Office of
the Executive Vice President, 1997). And, of course, the most common form of performance indicators have been used to report tothe state or nation about performance on those measures that areimportant to the institution's mission and the states priorities.
In almost all cases, the development of performance measures
has led to a transition from essentially a regulatory internal review
and resource reallocation role to one of providing information to theconsuming public. Cave and his colleagues (1997) have pointed outthat this has had several consequences. First, performance measurement has ceased to be only a centralized monolithic systemserving only the institution and state, but has become a joint product of both the institution and public sector funding organizations
and a variety of more specialized, possibly private sector, organizations. The development of consumer guides such as the U.S News77

and World Report annual issue on colleges and universities,
Barron's Guide to American Colleges, and The Times Good
University Guide in the United Kingdom are examples of this tendency.
It is also likely that the new purchasers of information about
higher education institutions will want it in a variety of different
forms. The information needs of prospective students will differ
materially from both public and private funders of research andparties interested in external services and outreach programs.
Thus, the development of performance measures must be broadly
conceived and understood that its elements must be accessible tomultiple constituencies for multiple purposes.

7. Concerns and Controversies with Performance Measures
Despite their increasing popularity, numerous concerns exist
about whether performance measures can accomplish their broad
and ambitious goals. These concerns need to be addressed before
adopting a set of performance indicators. Some of these concerns
include the following issues.
First, and most important, there are serious issues of validity
and reliably in the selection and application of performance indicators (Kells, 1993). The most prevalent concern about validity is that
often the measures selected are those most readily available orthose that are the easiest to collect, rather than those most important to the mission or goals of the program or institution.
Moreover, some performance indicators might result in deflecting
attention from more critical issues or other unintended outcomes
since considerable time and resources are often spent for collecting
data through a variety of surveys and other instruments.
There may also be a lack of a common set of measures for similar institutions since each institution usually attempts to develop its
own measures. This event makes difficult a comparison of institutions based on their performances. Certainly for accountability to78

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

parents and students there needs to be some minimum number ofcommon indicators such that public judgments can be made whencomparing across institutions (Linke, 1992).
Types of performance measures vary greatly because thisdepends on the unit developing the measures. For example, whilemost legislators or governing boards generally focus on input andoutput measures from readily available quantifiable data such as
the number of students served, degrees granted, retention, andcompletion rates and per student expenditure, institutions are moreinterested in process and outcome related measures such as student experiences and faculty scholarship and research accomplishments which are difficult to quantify in terms of simple statistics,
counts, or indicators. Since easily quantifiable information is notnecessarily the most informative and useful for decision-making andquality improvement, such a focus becomes a particular concern forseveral areas in which no readily available data may exist.
The purposes of performance measures also vary depending on
the unit that develops them. For example, while legislatively mandated performance measures are usually intended to require higher
education institutions to demonstrate accountability and achievement of their missions and goals, institutionally developed measures
are often designed to influence the institution's priorities, monitor
the process, communicate its achievements and success, andimprove its quality.
Moreover, there is often a tension between the demand foraccountability and institutional autonomy. An appropriate balance
between the legitimate need for information and public accountability and institutional autonomy is a particular concern to many institutional administrators and faculty who are skeptical about performance measures due to concerns about intrusion of the state ininstitutional autonomy.
While there is an increasing use of quantitative (objective) measures, the use of qualitative measures needs to be also emphasized.
However, the difficulty in measuring and collecting the data79

remains as a major impediment in using such measures. At theleast, every developed nation that is conducting graduate programs
needs an external agency to assess the quality of each of its programs. We are not here recommending accreditation standards,
although in some program areas it may be worthwhile to have minimum standards, but what we are recommending is national assessment of the quality of each graduate program based on bothnational rankings by external peers and by quantitative measuressimilar to what one finds in the National Research Council's doctoral program assessments in the United States (Goldberger, Maher &Flattatu, 1995).
The publication of performance data (particularly in the form of
ranked and comparative data) can become a controversial issue inhigher education, since they can be taken out of context and misused. Nevertheless, it is essential that the publication of performance data take place in order to insure appropriate public
accountability.
Finally, there might be disagreement between the government
and institutions on the use of the indicators. While there is anincreasing interest among public policymakers in the use of performance measures for funding, many in higher education institutions
oppose such uses. A set of nation-wide measures may not be consistent with individual institutional goals and directions. The connection between indicators and selective funding even within theinstitution is not as clear as is often assumed.

8. Developing Performance Indicators often Leads to
Performance Funding
Initially performance reporting was used to demonstrate institutional and public accountability and no attempt was made to link
the indicators directly to a budget. However, institutional assessment without a link to some form of funding had only limited success and was not as far reaching as many policy makers desired or80

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

anticipated. During the 1980s, several countries began to use performance measures both for incentive funding and as rewards forenhanced quality and directed assistance on national priorities (e.g.,the United Kingdom, several states in the United States, theNetherlands, and Australia). Despite the lack of a direct link
between assessment and funding in many programs, an indirect
link has always existed to the extent that students would enroll at
institutions with good performance, which in turn led to increased
appropriation to support that enrollment.
Across the world there is an increasing recognition of the use of
performance measures to guide public planning and budgetarydecisions. Many policymakers in almost all of these countries haveconcluded that the current funding mechanisms require a majorchange; from growth in funding, principally through enrollment
based formula funding, toward funding performance and results.
Numerous public bodies in these countries have linked budget allocations explicitly to the results of performance measurement inorder to influence directly the behavior of institutions to achieve
the public's goals and priorities. For example, as of the fall of 2000
almost all of the 50 states in the United States had some form of
performance measures in place, and over 30 of these states had
some form of performance funding programs taking place in their
public higher education systems. The rapid growth in the numberof such funding programs during the 1990s in both North America
and Northern Europe demonstrates the strong international interest in systems that link performance to budgeting. The major purposes of both performance indicators and funding programs havebeen to improve accountability, enhance the quality of undergraduate education and to achieve state- and nation-wide goals and priorities in higher education.
As performance funding in higher education has become an
important policy tool in North America, Northern Europe, and theSouth Pacific countries of Australia and New Zealand, it is also
expected that such approaches will increasingly be examined and81possibly used across other developed countries as well. One suchcountry at the present time is Japan as it moves forward with its
own higher education reform agenda. However, it should be notedat the outset that few studies have examined the implementationand efficacy of various funding and resource allocation mechanisms
in general and performance funding in particular (Nedwek, 1996).
Most of the countries with performance funding in place are struggling to develop a public system that will be effective, efficient, andfair in addressing the general concerns within the current reform
initiatives underway.
Performance funding amounts are still rather low: In fact, mostof the public systems are conducting performance and incentive
funding on an annual recurring basis only at the margin and onlywith about one to five percent of the public's allocations to higher
education. The only exception is England where it is alleged thatthe central government is allocating up to 10 percent of its recurring allocation to public higher education based on performancefunding procedures within their new reform initiatives (Cave et al.,1997). In brief, performance funding is still one of the smallest components of most current public funding budgets for higher education.
Direct state appropriations in the United States, for example,
clearly constitute a significant portion of the total revenue of all
public institutions. However, by the early 1990s less than one-halfof all revenues of public universities came from state and local public appropriations. The contribution of state and local appropriations to the sum of total revenues at local four-year (62.2%) and twoyear colleges (67.4%) was far more important (Blasdell, et al., 1993).
So in reality, we are noting that only about one-half of one to five
percent of public institutional revenues is being allocated via performance or incentive funding mechanisms in a country with astrong preference for performance funding.
An important development in both the United States andEngland over the past two decades has been a significant shift82On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

downward in the relative contribution of state funding to the overall revenue of public institutions, while the relative importance oftuition and fees has increased. For example, in the United States
the overall state contributions to public research universities had
declined from about 45% of total current fund revenues in the mid1980s to less than one-third by the mid-1990s. On the other hand,the contribution of tuition and fees to total revenue had increased
from 13% in 1978 to over 18% by the mid-1990s. Thus, to somedegree the increase in tuition and fees as a revenue source has offset the decline in state support. The federal government also provides substantial support to higher education in the form of student
aid and research grants and contracts.

9. Using Governmental Budgets as Policy Tools
The budget has increasingly become a policy tool to ensure theachievement of accountability and productivity as well as nationwide priorities in higher education across the developed world.
Many policymakers see the utility of the budget as a fiscal tool foraccountability and that it reflects state priorities. As a result, newfiscal policies increasingly attempt to tie funding to specific outcomes and performance (Minnesota Higher Education Coordinating
Board, 1993).
There is a diverse set of allocation mechanisms in higher education for direct public operational subsidies. At one extreme, thebureaucratic (e.g., line item budgeting) approach requires that each
expenditure item and staffing be approved by the funding agency
and considers all staff members as civil servants. Such restrictions
are usually justified as a means of ensuring accountability frominstitutions; for instance, prescribed staffing patterns are supposed
to ensure quality control. However, such strict control of institutional expenditures focuses primarily on financial accountability. In
other words, institutions account for how they spend public funds
provided by the state with little focus on program outcomes. This83

approach provides little institutional autonomy or authority. At theother extreme, public agencies can allocate resources through"block grants" and allow institutions to have maximum flexibility inallocating state resources based on their own strategic directions
and goals. The degree of state control over institutional funding
and accountability mechanisms vary between these two extremes
from country to country. Clearly Japan currently lies closest to theextreme of state control.
Two other common approaches to public funding for higher
education are incremental (historical) and formula approaches andboth are found within North America and Northern Europe. The
incremental approach generally assumes the continuation of existing activity. Thus, it provides little policy context for strategicplanning. The funding level is determined based on the prioryear's allocation after some judgmental or analytical adjustments(e.g., enrollment change, inflation). The formula approach, for exam ple, emerged as the method of choice for distributing most state
funds for higher education during the 1960s and 1970s in the
United States. Using objective criteria, public funds are allocatedamong all the public institutions. The major goal is to provide
equal public funding to similar institutions engaged in similar activity or achieving similar outcomes.
Since these approaches provided little accountability in the useof public funds, several new approaches emerged during the 1990s.
These were student funding through governmental scholarships
and student aid (i.e., vouchers), strategic (incentive) funding, contractual funding for specific services or work product, and performance based funding. When one of these approaches has been
adopted, it has been typically added as a supplement to the current
approach in use, as in the case of performance based funding
described above. The principal goal of all four of these new
approaches is to hold public colleges and universities more accountable in their use of public resources and make them more responsive to the stated needs of public policy.
84

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

Although public funding mechanisms vary from country tocountry depending upon their on-going academic and political culture, there are a number of desirable characteristics of an effective
funding mechanism. These basic characteristics include serving
broad public policy objectives, being directive and forward looking,providing equitable distribution among institutions, providing
income predictability, stability and flexibility, and being practicaland transparent. These characteristics can also have various
weights relative to their importance depending upon the prioritiesof the public sector. Since a single funding mechanism often fails to
include all these characteristics, a mix of various funding mechanisms is often recommended for university funding.
It is important to note that in no case has any developed country cut its public national universities loose and told them tobecome private institutions. However, in many cases the state has
encouraged diversification of revenue sources, including seeking
additional funds from the private sector including raising tuition
and fees. In all cases the states and nations have acknowledged
the important public role of public universities as they have contributed to the social capital of the country that extent beyond justskill preparation for jobs. These externalities have frequently beenaddressed in the literature and are readily recognized (e.g., Becker& Lewis, 1992).
While all countries make some forms of subsidies to public higher education institutions and use budgets for accountability and inachieving their priorities and goals, the degree of state control overbudgets and how the budgets are used to achieve these goals
varies substantially by country. Even within some countries, nosingle method applies to every state and to every institution. Many
countries and institutions have a variety of features with several
allocation mechanisms. More important, the question remains to beanswered as to what extent governments should use budgets to set
educational policy. What strategies are most successful in achieving results?
85

Special initiatives and incentive grants: Some states use publicbudgets to address accountability and quality concerns by implementing "non-formula, special initiatives." Such initiatives include
endowed chairs, special funding for centers of excellence, and competitive or incentive grants for various campus initiatives. These
non-formula funds come as additional funding as the governmental
agencies support certain initiatives on different campuses. The
main emphasis is on quality improvement and addressing the needsof the nation or region by providing non-formula funds for special
initiatives. This has been used quite extensively in both the UnitedStates and Scandinavia during the past two decades and is currently underway in Japan through targeted scholarship and researchfunding (MESSC, 1997).
Accountability reporting on performance: A growing number ofcountries and states require public institutions to develop andreport on a system of state or nation-level performance measures.
This reporting is called "accountability reporting on performance"
using reporting cards, or assessment reports. Its purpose is todemonstrate institutional accountability, but no attempt is made tolink the indicators directly to the budget. Although the actual
results of assessments are not tied to funding, indirect links
between the funding and performance results exist to the extent
that better performing institutions will have more students andresources, which in turn leads to increased state appropriations to
support these enrollments and other activities. Moreover, somenations and states include strong budgetary sanctions for institutions that fail to comply. Institutions are also required to reportannually on progress made in carrying out productivity improvements and the budget process is identified as the key strategy foraddressing priorities and making productivity improvements.
Performance based funding: An increasing number of countries
and states have adopted or are considering adopting performancebased funding schemes that provide a direct link between performance and public sector funding. These allocation mechanisms are86On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

perceived as "rational" allocation techniques. In such systems,
funds are allocated on the basis of performance criteria. There aremany different approaches to performance funding which range
from very strong ties to very loose connections between funding
and performance measures.
Merged budgetary and academic plans: Finally, some states
have combined budget and academic plans with the hope that sucha combination would result in improved quality and greater
accountability. Such approaches have been used in a number of
institutions (e.g., Wisconsin in the United States) over the past
decade in an effort to improve quality and provide greater accountability. The thrust of this approach is institutional flexibility inmanaging and reallocating its funds provided by the state inachieving the mission, goals, and objectives identified by the public
sector's higher education master plan and institutional strategicplans. These plans serve as the essential foundation for a new
funding system. While in the past the link between plans and funding had been at best peripheral, this approach suggests a wellestablished link between the two. The state's role is to provide ahigher education master plan in consultation with its institutions
and then allocate state funds as "block grants." Performance measures are used to monitor results. Each institution has maximum
flexibility in determining how to allocate its resources to achieve
the goals and objectives identified in the state's master plan andinstitutional strategic plans. The critical issue is to develop a carefully designed linkage between planning and budgeting.
While the first three approaches rely on the government as thecatalyst, the last one stresses the importance of institutional autonomy and decision-making in achieving the desired results from public investment in higher education. These four approaches are by
no means independent. No single approach applies to all countries
and institutions. Indeed, most countries and states employ a combination of these approaches to select their best features.

8710. What is Performance Funding?
Performance funding is simply the allocation of public funding
based on a set of performance criteria. The focus is on outcomesand performance. The general purpose is to ensure that public
institutions will meet a range of public priorities and expectations.
The specific purposes of performance funding are to achieve
statewide goals and objectives from higher education institutions;achieve greater accountability in the use of public funds; andimprove the quality of education, research and services of higher
education institutions.
Over the years, new approaches to budgeting have beendevised that allow institutions to have autonomy over decision making, yet ensure greater accountability over the use of public funds.
Some of these methods include management by objectives, zerobased budgeting, program planning budgeting systems, total quality management, and strategic planning (see, for example,
Caruthers, et al., 1994). The most recent interest has been on performance funding. Across the many nations there has been agrowing emphasis on the formal assessment of institutional performance and use of public funding as an incentive and reward forhigh performance achievement of the stated goals. The use of performance data has also differed from past practice. While the focus
in the past has been on the process of reporting data, the current
emphasis is on critical analysis of what the data mean for the purpose of change (i.e., the product) and how public funds can be used
for this purpose.
Despite the growing use of various forms of performance funding methods, there seems to be little consensus on elements andapproaches. A growing number of countries and states have beenusing financial incentives for enhancing quality in higher education
as well as greater accountability. A 1983 survey in the UnitedStates, for example, found that 13 states were changing theirfinancing structure to promote quality. A more recent 1996 survey88

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

found that at least 24 states now have explicit incentives in theirbudgeting process to improve the quality of higher education as
well as for providing greater accountability and responding to statepriorities (Layzell, 1999).
Some may ask why performance based funding policies are necessary since most nations provide funding based on their desired
goals and priorities. The budget does reflect public values andstate priorities for higher education. During the 1960s through the1970s and on into the 1980s in both Japan and in most of the rest of
the developed world, statewide goals for higher education
addressed access. Budget mechanisms, primarily funding formulas,
were prevalent as a means of promoting growth in higher education and equity in the allocation of resources. In Japan the principal mechanism was the government's facilitation and assistance inthe expansion of private higher education.
Beginning in the mid-1980s, there has been a change in thefocus of higher education goals from continuing to promote equity
in access and funding to a value driven system that focuses on student and institutional performance and improvement. The budget
has begun to be viewed as an instrument to control expenditureswith the intent of improving efficiency.
Problematic issues in performance based funding: Despite theirgrowing popularity, performance funding is not a new idea. It has
been tried in several countries and states and sometimes even discarded because of its failure to achieve its intended purposes. The
appropriateness of performance based funding models for higher
education has not been clearly established in all countries that have
tried them. It has even been alleged that no successful models of
performance based funding exist (Bateman & Elliott, 1994). While
the merits of performance based funding remain unclear, many arestill skeptical about its use. Therefore, the potential effects of the
proposed funding methods need to be carefully examined.
Nevertheless, it is predicted that more countries will be attempting
to implement such policies, given their increased concerns about89accountability, quality, and productivity in higher education.
Unfortunately, there are only a limited number of studies examining funding techniques in general and the efficacy of performancefunding methods in specific terms (Layzell, 1996). Most have been
only descriptive and not very analytical.
Nonetheless, our review of the literature suggests several
lessons. Performance based approaches to funding requires careful
program design and implementation. Funding objectives and themeasures to provide funding must be very explicit. Measurement
must be an on-going effort. Failure in the implementation of performance funding policies often results from complexities in the designof such programs. Such initiatives can have the greatest potentialof success when carried out as part of an overall plan of higher
education development, not simply used ad hoc to respond to special interest pressures, or a changed fiscal or political climate.
Some existing or newly created performance measures may not
be appropriate for funding purposes since the cause and effect relationship between performance and funding is not always clear.
More importantly, such systems could cause institutional administrators to focus only on the "rewards" of reaching measurable goals
rather than on the management of institutions or achieving thebroader goals and objectives of the institution as a whole.
There must be a linkage between planning, performance measures, and budgeting to have a maximum effect. Yet, a particular
problem with nation-wide performance measures has been theirfailure in successfully linking measures to institutional planning andfunding. Some evidence in the literature also suggests that linking
performance measurement to planning and budgeting has notproven effective in practice, despite the fact that several carefully
designed attempts have been made (Bateman & Elliott, 1994).
While many countries and states have attempted to link funding
directly to planning and performance, it is not clear whether performance indicators can be both an accountability device for charting
the past and a planning tool for meeting higher education needs for90

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

the future.
Institutional self-governance and academic freedom have been
important principles in the academe throughout the past century.
They are widely regarded throughout the developed world as necessary principles for success. Some public institutions have evenhistorically considered themselves autonomous from governmental
policy formulation. Some contrarian stakeholders in higher education even believe that public institutions do not need to justify theiractivities to government or society at large. While institutional
autonomy and academic freedom are critical parts of the modern
university, it is usually accepted that they also bring both a privilege and a responsibility. Any institution using public funds has an
obligation to show how those funds are used and what they haveaccomplished. Performance funding mechanisms can combine institutional autonomy and accountability to ensure adequate performance, but the problem must be addressed on all campuses.
It is unclear whether schools should be rewarded or punished ifthey accomplish the stated objectives or fail to accomplish them.
Bateman and Elliot (1994 (pp.50-51)) note, "it is not at all clear that
reducing funding for poorly performing institutions will cause themto improve... Nor is it clear that additional funding is the mosteffective motivator for higher education institutions". According toconventional employee management theory, people are motivatedby the intrinsic satisfaction of a job (e.g., the opportunity forachievement, challenge, contributions, excitement, and personal
recognition). To have a successful incentive program, incentives
must be meaningful to individuals, not just to an institution. Yet,the basic structure of public performance funding systems reward
and recognize groups through institutional incentives and not
through recognizing individual incentives. The question remains as
to whether such "collective rewards" can effectively motivate individual faculty and staff to achieve greater performance. The
answer to this question is not yet known (Gaither, Nedwek, & Neal,
1994, p.25). How an institution internally uses performance91rewards, once received, to motivate individual learning, creativity,and initiative becomes a critical issue. Many institutions were
struggling with ways to define internal reward structures (Lewis &Becker, 1979).
Much of the debate on performance funding is on measurement,
not on the concept of the funding. One of the main lessons learned
from attempts to implement performance based funding systems isthat there must be effective measures of both performance andquality. Public systems wishing to use criteria for quality or performance in the budget process must deal with definitions of quality and effectiveness. Yet, most have found it difficult to define ormeasure such dimensions. "Bad data are worse than no data whenindicators become the primary tool for managing higher educationor allocating resources" (Gaither, et al., p.22).
It is recommended that measures be simple and easy to administer. While there are certain advantages to this approach, simple
measures (many of which are routinely available from institutions)
may not reflect the intended goals and objectives that the state
wants to achieve. Performance funding often reflects past efforts
to show improvement on important issues that have already beenassessed to be successful; or it may reflect factors that are not mission related or outside the control of the institution.
One of the most controversial issues associated with performance funding is determining the amount of performance funding
to be allocated to institutions. Performance funding can be threatening to institutions. For this reason, it is recommended that performance funding operate at the margin. It should be small enoughto not threaten institutions, but big enough to be taken as a serious
effort by institutions. When the amount of performance dollars isseen as a significant amount by institutions, it certainly gets attention and the system by which it is distributed will have major influence on changing institutional behavior. Yet, it is not clear what
portion of the budget should be involved in performance funding.
While some have suggested not more than 2 to 3 percent, others92

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

have recommended 5 to 10 percent. Some have argued that there
should be a gradual increase in the size of the performance funding
budget component after starting with a small percentage.
Timing is also an important factor in establishing baselines, collecting outcome information, and measuring improvement. Thereis often a two- to four-year time lag between performance and funding. This can be a particular problem for allocating resources forplanning objectives. Moreover, the amount of time required todevelop these programs, measure their performance, and reportthem can be high.

11. In Summary
This paper has reviewed the expressed need for effective performance indicators in the reform efforts of Japan's public national
university system and we concur with the final published set of
public recommendations by the MESSC. We have also reviewed
the international literature with respect to both performance indicators and performance funding in higher education. From this literature, we have several important recommendations for Japan.
First, despite some limitations of performance measures, theidentification of meaningful indicators for determining, assessing, orreporting on measurable goals and outcomes is viewed as a goodand useful idea. They can be used to document the past and they
can inform the future when they are developed within a policy
framework ensuring that they are policy relevant and purposeful.
Individual institutions for both internal review and reallocation can
use them. They can be used by decision makers in the private sector (including parents and students) for choice making. And thegovernment or the Ministry of Education for public accountability
and inter-institutional assessment can use them, as the need for
public accountability is very clear in Japan. They might also be
used for incentive and performance based funding by Ministry, but
the jury is still out on this latter issue and we do not have the93answers. We can only frame the issues, problems and opportunities as we have done in this paper.
Nevertheless, a recent study on performance indicators has
noted that:
The trend is clear and irreversible: Higher education institutions must start developing better ways to judge more adequately
how well they are doing. Higher education has been reluctant todevelop performance indicators because it is believed that the mission of higher education is too diverse to measure and short-term
measurement might be inadequate to measure long-term success.
If the members of the academy-- faculty, academic leaders, students-- do not participate in the process of developing and improving the use of performance indicators, however, external organizations will force some form of indicators on them (Gaither, Nedwek,& Neal, 1994, p. xi).
Finally, we have found that the experiences within many othercountries suggest that performance measures can be especiallyeffective when they are linked to institutional or nationwide planning and resource allocation efforts. Perhaps, the most importantuse of performance measures is their use for funding purposes
within the institution itself that might serve as a catalyst for institutional change to improve its quality and achieve its goals and priorities. Some have argued that many previous campus-driven
assessment approaches have not been successful in meeting
accountability demands largely because they have lacked incentives in the form of performance funding. Thus, many are now predicting an increased use of performance measures as a basis forfunding decisions by public policy makers (Gaither, 1997; Folger &Jones, 1993; Serban, 1998; Cave et al., 1997). Yet, the experiences inmany countries suggest caution in moving too quickly. There aremany "unresolved" issues regarding performance measurements
and their use for funding. The belief that performance indicators
can provide accurate information on institutional functioning
appears to have weakened in recent years, with early optimism94

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

having become more tempered by debate and experience (Yorke,1997). Perhaps a bit more use of "self-monitoring and self-evaluation" along with more incentive based funding on top of existingcore funding might be in order across the entire system of higher
education in Japan.

References
Advanced Education and Career Development (1997) Key Performance
Indicators Reporting Manual for Alberta Post-secondary Institutions,
Edmonton, AB: Author.
Albright, B. N. (1996) From Business as Usual to Funding for Results,
Columbus, OH: Ohio Higher Education Commission(http://www.bor.ohio.gov/herc/albright96.html).
Bateman, M. and Elliot, R. W. (1994) "An attempt to implement performance-based funding in Texas higher education," M. Epper (ed), Focus
on the Budget: Rethinking Current Practice, Denver: Education
Commission of the States.
Becker, W. & Lewis, D. (1992) The Economics of American Higher
Education, Boston: Kluwer Academic Publishers.
Blasdell, S. W., McPherson, M.S. & Schapiro, M. O. (1993) "Trends in revenues and expenditures in U.S. higher education," M.S. McPherson, M.
O. Schapiro and G. C. Winston (eds), Paying the Piper: Productivity,
Incentives, and Financing in U.S. Higher Education, Ann Arbor, MI:
The University of Michigan Press.
Borden, M.H. & Banta, T.W. (eds) (1994) Using Performance Indicators
to Guide Strategic Decision Making (New Directions for Institutional
Research, Number 82), San Francisco: Jossey-Bass Publishers.
Burke, J. (1997) Performance Funding Indicators: Concerns, Values, andModels for Two- and Four-Year Colleges and Universities, Albany,
NY: Rockefeller Institute.
Burke, J., and Serban, A. (eds) (1998) Performance Funding for Public
Higher Education: Fad or Trend?, New Directions for Institutional
Research, Number 97, Francisco: Jossey-Bass Publishers.
Caruthers, J. K., Marks, J. L. and Walker, J. K. (1994) "Important safeguards in funding processes for public higher education," R. M. Epper95(ed), Focus on the Budget: Rethinking Current Practice, D e n v e r :Education Commission of the States.
Cave, M., Hanney, S., Henkel, M. and Kogan, M. (1997) The Use of
Performance Indicators in Higher Education, London: Jessica Kingsley
Publishers.
Dolence, M. G. and Norris, D. M. (1994) "Using key performance indicators to drive strategic decision making," New Directions for
Institutional Research, 82, pp.63-80.
Dundar, H. and Lewis, D. (1998) "Determinants of research productivity
in higher education," Research in Higher Education, 39(6), pp.607-631.
Folger, J. & Jones, D. P. (1993) Using Fiscal Policy to Achieve State
Education Goals: State Policy and College Learning, D e n v e r :Education Commission of the States.
Gaither, G. (1997) "Performance indicator systems as instruments foraccountability and assessment," Assessment Update, 9(1), 1-2, 14-15.
Gaither, G. , Nedwek, B. P. and Neal, J. E. (1994) Measuring Up: The
Promises and Pitfalls of Performance Indicators in Higher Education,
ASHE-ERIC Higher Education Reports, Volume 23, Washington, DC:
George Washington University.
Goldberger, M., Maher, B. and Flattatu, P. (1995) Research-Doctorate
Programs in the United States, Washington, DC: National Academy
Press.
Johnes, J. and Taylor, J. (1990) Performance Indicators in Higher
E d u c a t i o n , Buckingham: The Society for Research into Higher
Education and Open University Press.
Kells, H. R. (1992) "An analysis of the nature and recent development of
performance indicators in higher education," Higher Education
Management, 4(2), pp.131-138.
Kells, H.R. (1993) The Development of Performance Indicators forHigher Education, Paris: Organisation for Economic Cooperation andDevelopment.
Layzell, D.T. (1996) "Developments in state funding for higher education". J.C. Smart (ed.), Higher Education: Handbook of Theory andResearch (Volumn XI), New York: Agathon Press.
Layzell, D. T. (1999) "Linking performance to funding outcomes at thestate level for public institutions of higher education," Research inHigher Education, 40(2), pp.233-246.

96

On the Use of Performance Indicators
in Japan's Higher Education Reform Agenda

Lewis, D. and Kallsen, L. (1995) "Multiattribute evaluations: An aid inreallocation decisions in higher education," The Review of Higher
Education, 18(4), pp.437-465.
Lewis. D. and Becker, W. (eds) (1979) Academic Rewards in Higher
Education. Cambridge, MA: Ballinger Publishing.
Linke, R. D. (1992) "Some principles for application of performance indicators in higher education," Higher Education Management, 4 ( 2 ) ,pp.194-203.
Lord, B. R., Robb, A. J. and Shanahan, Y. P. (1998) "Performance indicators: Experiences from New Zealand and Tertiary Institutions," Higher
Education Management, 10(2), pp.41-57.
Massy, W. F. (1996) Resource Allocation in Higher Education, Ann
Arbor: University of Michigan.
MESSC (1995) Why University Reform is Needed, Tokyo: MESSC(http://www.monbu.go.jp/hakusyo/1995eng/c1.html)
MESSC (1997) Improving the Infrastructure for Scientific Research,
Tokyo: MESSC (http://www.monbu.go.jp./hakusyo/1997eng/c2s1.html).
MESSC (1999) Japanese Government Policies in Education, Science,Sports and Culture, 1999: Educational Reform in Progress, Tokyo:
MESSC.
Minnesota Higher Education Coordinating Board (1993) State Funding
and State Goals: Linking Post-Secondary System Appropriations to
Outcomes, Minneapolis: Minnesota Higher Education Coordinating
Board.
Nedwek (1996) "Public policy and public trust: The use and misuse of
performance indicators in higher education", J.C. Smart (ed.), Higher
Education: Handbook of Theory and Research (Volumn XI), New
York: Agathon Press.
Office of the Executive Vice President (1997) University of Minnesota
1997 Institutional Performance Report, Minneapolis: University of
Minnesota (http://www.opa.pres.umn.edu/critmeas/perf97/perf97.htm).
Ruppert, S. S. (ed.) (1994) Charting Higher Education Accountability: A
Sourcebook of State-level Performance Indicators, Denver: Education
Commission of the States.
Serban, A. M. (1998) "Performance funding criteria, levels and methods,"
New Directions for Institutional Research, 97, pp.61-67.
Sizer, J. E., Spee, A., and Bormans, R. (1992) "The role of performance97

indicators in higher education," Higher Education, 24(2), pp.133-156.
University Council (1998) A Vision for Universities in the 21st Century
and Reform Measures, Toyko: MESSC (http://www.monbu.go.jp/
series-en/00000015/).
Yonezawa, Akiyoshi (1998) "Further Privatization in Japanese Higher
Education," International Higher Education, Fall.
Yorke, M. (1997) "The elusive quarry of total quality in higher education," Tertiary Education and Management, 3(2), pp.145-156.

98