AC 2007-746: DEVELOPMENT AND IMPLEMENTATION OF A BALANCED SCORECARD FOR ENGINEERING DISTANCE LEARNING PROGRAMS AT VIRGINIA TECH

For more than twenty years, engineering distance learning programs have provided postbaccalaureate education for working engineers and scientists. The programs are offered in various engineering disciplines that include civil engineering, computer science, electrical engineering, and industrial engineering. State of the art technology that includes interactive videoconferencing and online delivery methods are used to deliver classes to students. The programs did not have a performance indicator to help monitor and evaluate its performance. The Office of Distance Learning and Computing (ODLC) took the initiative to develop a balanced scorecard as a tool to monitor and indicate the performance of the programs. The initiative was driven by the continuous improvement process for one of its distance learning programs, the Commonwealth Graduate Engineering Program (CGEP). After successfully developing and implementing a balance scorecard for CGEP, ODLC expanded the balance scorecard to incorporate all its distance learning programs including Master’s of Information Technology (MIT) program, National Institute of Aerospace (NIA) program, and School of Biomedical Engineering and Sciences (SBES) program. This paper reviews development, implementation, and maintenance phases of the balance scorecard. Guidelines and lessons learned throughout these processes are presented. The process of selecting a software program to help maintain the balance scorecard is discussed. Introduction Balanced scorecard was first introduced in the early 1990s by Dr. Robert Kaplan of Harvard Business School and Dr. David Norton, a president of a Massachusetts consulting firm. Balanced scorecard is a tool that “translates an organization's mission and strategy into a comprehensive set of performance measures that provides the framework for a strategic measurement and management system”. 1 The balanced scorecard has replaced the traditional performance measurements that only concentrate on financial and accounting measures. These traditional measures fail to address many issues that businesses should be concerned with and fail to monitor multiple dimensions of performance. 2 Traditional measures provide insufficient and distractive reports for managers to use to make decisions. Numerous studies indicate the limitations and ineffectiveness of the traditional financial performance measures. Kaplan and Norton 3 pointed out that financial measures only focus on the past and are unable to reflect current value-added actions. Financial measures fail to include other critical factors such as customer satisfaction, employee satisfaction, and the quality of products or services. 4 Financial measures only represent one perspective of an organization’s performance and focus only on the short term goal. 5 On the other hand, the balanced scorecard measures four perspectives of the organizations. The four balanced scorecard perspectives are customers, internal process, learning and growth, and financial. 1 The balanced scorecard helps translate mission, vision statements, and the organization’s strategy which are in words to measurements that help clarify and communicate the direction of the organization to all of its members. 6 The Balanced Scorecard Detail of the four balanced scorecard perspectives are presented below: 1,3 1. Customer perspective: How do customers see us? P ge 12501.2 This perspective captures how companies provide their products and services from the customer’s point of view. Satisfying customers is a top priority for businesses. Managers need to transform their customer service missions to measure issues that matter to customers. These measures usually fall into time, quality, performance, service, or cost categories. 2. Internal perspective: What must we excel at? Since internal processes are what drive companies to ensure customer satisfaction, it is important for companies to focus on their internal process performance. Companies need to identify their niche to maintain their competitive edge. Therefore, companies need to determine in what they must excel. 3. Innovation and learning perspective: Can we continue to improve and create value? Companies need to continue improving their performance in order to meet customers’ expectations and excel in internal processes. Continuous learning behavior of the organization is important to ensure that the organization is moving forward and remaining competitive in a high competition environment. 4. Financial perspective: How do we look to shareholders? Kaplan and Norton 3 identified that the “financial performance measures indicate whether the company’s strategy, implementation, and execution are contributing to bottom-line improvement”. This category usually consists of profitability, growth, and shareholder value. This perspective denotes the traditional performance measures. Measurement System Development Process ODLC used the measurement system development process (MSDP) to develop a balanced scorecard for its distance learning programs. The MSDP consists of 6 main steps and a central process for creating the infrastructure for the measurement system, as shown in Figure 1. Brief explanation of these steps is presented below (adapted from Rentes & Van Aken, 2000; adapted from Van Aken & Coleman, 2001), 7, 8 Figure 1: The measurement system development process (MSDP). 8 P ge 12501.3 Create an infrastructure for the measurement system In this step, the organization needs to create an infrastructure to support the measurement system. This step includes forming a team to design and execute the performance measurement system and developing a communication method between the design team and stakeholders to share information. 1. Define the need for measurement It is important that an organization clearly defines its need for a performance measurement system. A clear purpose will likely lead the organization to successful development and implementation processes for a performance measurement system. Even though a clearly defined purpose cannot guarantee success, it will help the organization rethink whether the performance measurement system is the right solution for the problem. 2. Define what we do In this stage, a target system is defined. An Input/Output Analysis will be used in this stage to provide a system scan with details of the target system to ensure that the performance measurements are thorough. An Input/Output Analysis is composed of suppliers, inputs (information, equipments, material, etc.), value adding processes, outputs (product or services), customers, and desired outcomes. The mission and vision statements if not existing will be developed in this stage. Moreover, a SWOT Analysis (Strengths, Weaknesses, Opportunities, and Threats) should be developed in this stage to examine the organization’s internal and external environments. Strengths and Weaknesses analyses focus on an organization’s internal environment assessment, while Opportunities and Threats analyses focus on the organization’s external environment. 3. Define what we must excel In this stage, key performance areas (KPAs) will be developed. KPAs are the balanced scorecard’s perspectives. However organization can modify these perspectives to fit the needs and the strategy of the organization. Note that some researchers might refer to KPAs as critical success factors, key success factors, critical result areas, and key result areas. 4. Define how we know if we’re successful In this stage, the metrics will be developed under each KPA. A set of 10-12 metrics is recommended for a unit within a larger organization, and a set of 15-20 metrics is recommended for the entire organization. 5. Implementing the measurement system In this stage, a metric development matrix and a visibility board will be identified and developed. The metric development matrix will contain information about the metric specification; consisting of the metrics, the operational definitions and/or formulas, the purposes of the metrics, and the metric owners; the portrayal design :consisting of the portrayal frequencies, the types of data, and the portrayal tools; the data collection plan: consisting of the tracking tools, availability of the data, the data collection responsibilities, the data collection tools, the data collection frequencies; and utilization: consisting of the implementation dates and metric goals. The organization can also add other information that is useful to implement the performance measurement system to the metric development matrix. The metrics will then be portrayed on the visibility board. 6. Utilize the measurement system After collecting and portraying data on the visibility board, the performance measurement system will be assessed and evaluated. The results can lead to many change initiatives in the organization. Next, the organization can review the impact of the initiatives. The organization can examine the performance measurement system, asking such things as whether the initiatives have led to good decision making, whether the performance P ge 12501.4 measurement system has an impact on organizational performance, and whether it has been easy to execute. The organization can then identify and develop a new set of initiatives. Balanced Scorecard in Higher Education The balanced scorecard is widely used and accepted as a valuable tool in the industry sector. Recently, the balanced scorecard has also been used in not-for-profit organizations and academic institutions. One of the reasons that the balanced scorecard is not as widely used in this type of organization might be because of the perception on financial perspective. Financial perspective, as one of the components of the balanced scorecard might mislead one to think that the balanced scorecard is only useful for for-profit organizations. However, looking back at how the balanced scorecard was originated, the balanced scorecard was developed to replace the traditio