# Product Lifecycle Management: Measuring What Is Important – Product Lifecycle Implementation Maturity Model

Abram J. Walton, Cynthia L. Tomovic, Michael W. Grieves

► To cite this version:

Abram J. Walton, Cynthia L. Tomovic, Michael W. Grieves. Product Lifecycle Management: Measuring What Is Important – Product Lifecycle Implementation Maturity Model. 10th Product Lifecycle Management for Society (PLM), Jul 2013, Nantes, France. pp.406-421, 10.1007/978-3-642-41501-2_41. hal-01461876

HAL Id: hal-01461876

https://inria.hal.science/hal-01461876

Submitted on 8 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Product Lifecycle Management:
Measuring what is Important – Product Lifecycle
Implementation Maturity Model

Abram L. J. Walton, Ph.D.¹, Cynthia L. Tomovic, Ph.D.²,
Michael W. Grieves, EDM³

¹Florida Institute of Technology, USA

²Old Dominion University, USA

³Florida Institute of Technology, USA

awalton@fit.edu

**Abstract.** Industry reports that benefits of PLM are difficult to assess because the same benefit can be expressed as a function of time, cost, quality, or any combination. Based on a review of the PLM literature in an earlier study, a PLM Process Model and an initial list of PLM related metrics was generated and later confirmed through interviews with experienced PLM users. In the current study, the original PLM Process Model was refined and the list of metrics was subjected to an exploratory factor analysis in which specific metrics were found to be related to one of four factors: Inputs, Processes, Outputs, and Outcomes. Based on the results of this study, a Product Lifecycle Implementation Logic Model was developed that serves as a program-level guide in helping to quantify PLM performance in support of meeting organizational strategic goals.

**Key Words:** PLM; product lifecycle management; innovation; performance measurement; metrics

# 1 Introduction

Product Lifecycle Management (PLM) is an integrated, information-driven approach comprised of people, processes/practices, and technology. It serves to integrate information across all phases of a product's lifecycle and its environment including product ideation, design, manufacturing, distribution, support, and retirement from use [1], [2]. Over the last decade, PLM has become an integral part of the global manufacturing landscape [3].

The concept of PLM emerged in the late 1990's-early 2000's. Its specific aim was to move information sharing beyond the engineering phase of a product by providing a shared platform for the creation, organization, and dissemination of product-related
information across the extended enterprise [4]. In essence, PLM is to provide a
holistic approach to managing product information [1], [5].

In today's marketplace, global competition is forcing manufacturing industries to reduce costs and time associated with product development, manufacturing, and mass customization. PLM is viewed often times as an investment in technology that supports operations and provides companies an overall competitive advantage in these areas [6, 7]. However, if companies were to view PLM as a means to achieve overarching organizational goals, rather than as simply as an investment in technology in support of operations, companies would come to appreciate PLM as an investment that improves and enhances all facets of the enterprise, including the bottom line [8], [9], [10], [11].

As companies begin to think about implementing PLM, even in its earliest stages, they need to think about what metrics to employ when measuring the actual benefits of a PLM-driven solution [12], [13]. Building on earlier work that focused on developing a PLM Process Model and developing a list of PLM-related metrics [11], this paper seeks to empirically further verify the importance of the initial list of PLM-related metrics by conducting a confirmatory factor analysis.

## 2 Significance of the Problem

Although some companies and engineering firms may posit that PLM is merely an extension of prior Product Data Management (PDM) efforts, since 2001 a number of businesses have launched new PLM initiatives based on the understanding that, if successfully implemented, PLM leads to enterprise-wide efficiencies and opportunities [11], [14]. Although founded on the main tenets of Just-in-Time (JIT) and Lean principles, PLM has a different orientation to Lean [2], [15]. According to Grieves [2], [15], PLM focuses on identifying and capturing wasted resources associated with time, energy, and materials, and the subsequent reallocation of these captured resources in support of product and process improvements and innovations that, ultimately, result in new revenue streams [11], [16], [17]. Manufacturing companies that have implemented new PLM systems and processes beyond those of traditional PDM systems and processes are reporting gains as much as: 20% increases in design productivity, 50-80% reductions in the time required to modify complex designs, 50% increases in time to explore more design options, improving the capability of conducting numeric control programming up to 10 times faster, machining up to 35% faster, 60% reductions in pallet manufacturing time, and 40% decreases in the errors found at the final assembly stage, as well as other benefits [18]. Nonetheless, even with these gains, industry continues to report that the benefits of PLM are difficult to assess because the same benefit can be expressed as a function of time, cost, quality, or a combination thereof. According to Shah and Ward [8], some of the confusion associated with PLM is due to the lack of a consistent definition as to what constitutes a PLM system. Given that this is the case, it follows then that there is also an inconsistent way to measure the impact of PLM. This lack of consistency is problematic as, according to Walton [5], unless PLM-related metrics are purposefully and strategically developed, PLM initiatives may result in nothing more than an instantiation of PDM principles.
If companies are to continue investing significant portions of their IT budgets in PLM-related technology, infrastructure, training and support, they must be able to derive valid and reliable data that measures the impact of their PLM investments [5], [11], [16], [17]. Moreover, without being able to accurately assess the impact of PLM on the bottom line, or its direct and indirect influences on cost-savings and revenue-generation, risk mitigation strategies often employed during IT installations and upgrades may fall short of providing a holistic foundation for future PLM investments [5], [11], [16], [17]. While the more traditional performance measures of net income, operating income, and revenue, will always be relevant and important in determining the profitability and successes of an organization, new PLM-related metrics must be identified to more accurately account for the costs and potential impacts associated with PLM initiatives [19].

Shah and Ward [8] found that there is significant overlap in performance attributes with studies associated with Lean Manufacturing when compared to PLM. According to Grieves [1], [2], this is because the main tenets of a PLM system diverge from Lean by including more holistic measures of data-attributes such as data singularity, data correspondence, data cohesion, data traceability, data reflectivity, and cued availability of data. Due to these differences, companies cannot use traditional Lean manufacturing-related metrics alone when attempting to measure the impact of PLM. Rather, accurate and valid PLM-specific metrics reflective of an entire PLM system need to be developed.

## 3 PLM Assessment Process Model

This paper further analyzes the PLM Assessment Process Model and PLM Metrics Framework proposed and explicated by Tomovic [16], and further refined by Walton [5] (Figure 1).

<figure><img src="image_2.png" /><figcaption>Fig. 1 PLM Assessment Process Model & Strategic Technology Road Mapping [5]</figcaption></figure>
Figure 1 indicates how various processes and business functions intersect and influence an organization's strategy, outcomes, and the development of the measurement standards. Effective management of the product lifecycle assumes measurement of relevant metrics derived from business goals and objectives. Furthermore, understanding how PLM processes and systems contribute to an organization's strategic plan and initiatives, and monitoring the impact of PLM outcomes in this context, minimizes misaligned spending on PLM initiatives. The business strategy provides the organization's definition of success; it is from these definitions of success that targeted, key performance indicators and metrics are identified. Accurate collection, reporting, assessment, and analysis of these key performance indicators provide feedback in the PLM Process Model and are measured against the initial strategic goals.

Ultimately, all metrics should be tied to business objectives related to growth, revenue, and profitability; in this manner, organizations will be better able to see where money is being effectively spent to support and further justify their PLM initiative. Each organization's unique instantiation of their PLM processes ultimately determines the metrics that should be measured and impacts the execution of an organization's strategic plan; no ubiquitous and definitive set of metrics will apply to all organizations. Nonetheless, a common, core set of metrics may be identifiable, which is the focus of this study.

# 4 Methodology

*Prior Study.* In prior studies [5], [11], [16], [17], numerous PLM metric frameworks were developed based on a review of the literature, and their level of importance and priority were later confirmed through qualitative interviews and focus groups with key PLM users. This line of research sought to develop and validate key PLM metrics and then link those metrics to a framework that would assist industry in their use and interpretation of data. Metrics were defined and a web based PLM metrics survey was developed and pilot tested. During these studies, the following research objectives were met: 1) pilot tested the PLM metrics survey with an initial set of mature PLM users; 2) established and distributed an electronic version of the survey; 3) codified and analyzed preliminary data in the aggregate; 4) reviewed findings with industry-based PLM focus group; 5) revised the PLM metrics survey, and 6) retested the survey by expanding the pool of participants.

*Current Study.* Expanding on the previous studies, this project worked toward the development of a program-level guide for quantifying PLM performance in support of organizational strategic goals. A PLM Logic Model was derived and validated. The following key questions were considered: 1) What metrics are appropriate in the measurement of PLM performance? 2) What are the key features of a balanced performance measurement system? 3) How can a performance measurement system help to drive desired business and human capital results? 4) How can companies determine if a global, regional, or local approach is appropriate? and 5) What are the potential outcomes and benefits from an effective performance measurement system?
## 4.1 Data Collection and Development of the Instrument

In the data collection phase of this study, a mixed methods approach was employed whereby surveys and open-ended interviews were conducted with mature PLM users and focus groups to identify and examine the impact of PLM. According to the literature, when attempting to measure the impact of a phenomenon, a mixed method approach is more likely to results in more reliable data than in a single method alone [20], [21], 22].

There are multiple authors that have created, tested, and published models that delineate a suggested methodology when creating a survey instrument to be used in business and social science research [20], [23], [24]. In accordance with the suggested literature, the first step in this study was to identify key literature related to metrics for PLM. This literature supported the development of a framework in which two main tenets emerged, waste reduction and innovation [11]. Centering all metrics on these objectives led to the development of a multi-tiered metrics framework, which was used as the primary tool in the creation, and categorization of subsequent metrics.

This study extended the analysis and evaluation of the initial data set and descriptive statistics delineated in Tomovic [11], [16], [17]. The initial data collection, survey methodology, and participant recruiting methods were as follows: 1) 67 metrics were identified from the literature that measured the impact of PLM within an organization; 2) the metrics framework and initial set of 67 metrics were validated using focus groups that included key industrial representatives and practitioners; 3) the findings from the focus groups resulted in the refinement of the target areas used in the metrics framework; 4) through further analysis, interviews, and literature synthesis, a series of additional Key Performance Indicators and an additional 113 metrics were added to the overall list of 180 metrics; and 5) a few select follow-up interviews were conducted with key individuals in participating firms.

Based on yet further study and practitioners’ reviews, and through additional literature reviews, and interviews, the framework was used to identify an additional 170 metrics, resulting in a total of 350 PLM metrics. The resulting outcome of 350 metrics exemplified the fact that there was no consistency in the number, depth, breadth, or overall application of metrics in the PLM space. To resolve these issues, researchers then conducted focus groups and a pilot study with advisory board members from the Center for Advanced Manufacturing at a major university. Once again, concerns were addressed regarding the scope of the metrics framework, and it was refined accordingly, reducing the number of metrics to create a ‘global’ list of approximately 60 high-level or strategic metrics, from which the remaining metrics could be rolled-out like a multi-tiered, hierarchical dashboard.

The development of the data collection instrument for this research resulted in a web-based survey. The 65-item survey instrument was professionally designed and tested for face validity according to the methods prescribed by [20]. The survey design was simplistic and straightforward. Based on the focus groups, interviews, it comprehensive design, and the all-encompassing nature of the survey questions, the survey instrument was determined to have content validity. The completed survey instrument was submitted to both academic-experts and industry-based focus groups for final pilot testing in order to ensure no topic had been neglected.
The final survey was composed of 8 high-level organizational variables and 57 PLM-specific metrics, totaling 65 items. In order to validate the PLM metrics identified and refined in steps one and two of this study, participants were asked to assess, for each survey-item, whether the metric was currently "in use"; and, to indicate whether the metric was considered important, or not, in assessing the overall impact of PLM. As delineated through clear instruction in the Metrics survey, in order for a participant to consider the metric "in use", the item had to pass the following criteria: 1) be collected at least on an annual basis; 2) be utilized by all members of top management; 3) be stored in a manner that ensures availability to numerous appropriate people in the organization; and 4) have a standard method for calculation. With regard to measuring importance, it was assumed that if the participant indicated that the item was "in use", the item was considered important. Additionally, respondents had the option of indicating that an item was either "not in use, but important:" or "not used, not important".

## 4.2 Participants

Various members of the research team attended several international conferences, workshops, and educational conferences, such as Partners for the Advancement of Collaborative Education (PACE) and the North American PLM Summit. At each venue potential participants were identified and subsequently contacted regarding their willingness to take the PLM metrics survey. Moreover, each participant was given the option to invite other PLM practitioners to take the survey, thus resulting in snowball sampling; often times, others included were those associated with one of the initial invitee's own organization or network affiliation, or were based on other professional contacts (e.g., professional associations, clients, etc.). The web-based survey was ultimately distributed to over 150 participants, of which 50 participants or 33% completed the survey.

# 5 Data Analysis

The initial analysis and descriptive statistics related to the data collected in this study was previously published [11]. Briefly, the majority of the respondents (40%) were from Aerospace and Automotive industries, with a majority of companies having implemented PLM for between 5-10 years. Furthermore, the majority of companies responding to the survey had gross sales revenue in excess of $1 billion per year.

The majority of people responding to the surveys were in research and development or engineering (20%), followed by management (16.4%). Based on specific trade or job-related knowledge, individuals from these two departments were probably the most appropriate to respond to the survey as Subject Matter Experts (SMEs); those from Human Resources were represented by only two respondents. The individual respondents to the survey held a wide range of jobs, from general management, marketing, sales, R&D, manufacturing, IT, HR, and engineering. Descriptive statistics delineated the perceived usefulness and importance, or lack thereof, for each metric with regard to whether or not each respondent and/or his or
her company employed that metric. The data analysis presented in this paper sought
to extend the investigation and evaluation of the initial set of PLM-related metrics by
conducting an exploratory factor analysis (EPA). The EPA allowed researchers to
investigate whether or not groupings of metrics were easily developed, such that a
more holistic framework could be created should a practitioner decide to employ
these metrics in a meaningful way.

**5.1 Exploratory Factory Analysis**

An Exploratory Factor Analysis (EFA) was conducted using the software package SPSS to identify PLM-metrics related factors based on the PLM Metrics survey [25]. The extraction method used was Principal Components Analysis (PCA), which is the default method of extraction in SPSS. Several EFAs were conducted, with the final set containing four factors retained for rotation. Of the various models tested through SPSS, the four factor set was chosen to be retained because metrics used in manufacturing tend to focus on Input, Process, Output, and Outcomes, and these categories are typically normalized to enable comparisons between phases of production [26]. Regarding Input and Output metrics, they are commonly applied because they are highly specific and directly related to a specific end point in a process [27].

Input metrics assessed level of capital, labor, and time, while Process metrics assessed the extent to which the inputs are being processed and the extent to which appropriate processes were being implemented. Output metrics included the product or service delivery and implementation targets for PLM, including, for example completion performance, resource optimization, change control and change capacity, configuration management, project or product quality metrics, among others [28]. Outcome metrics assessed the changes and/or benefits resulting from PLM activities. Output metrics included such results as waste reduction, innovations and new products, continuous improvement and sustainable green manufacturing.

The EFA did not reveal an Impact factor. Impact metrics measure long-term outcomes such as the return on investment, which many of the companies surveyed, have not yet realized to date. It is therefore hypothesized that future research will find the last few items listed in the Outcome Factor to be Impact Metrics. However, results for this study resulted in only a four-factor model.

**5.2 Reliability (4-factors)**

Cronbach's alpha for the study was calculated using SPSS factor structure suggested by Chelladurai [29]. Reliability of the 4-factor structure in the study ranged from .941-.953 with a mean of .978. Table 1 lists the 4 factors items included in each factor and factor alphas for the study sample (n=58).
<table><caption>Table 1. Scale of PLM Metrics Reliability Statistics</caption><thead><tr><td></td><td>Number of items</td><td>Cronbach's Alpha on Standardized items</td></tr></thead><tbody><tr><td>Factor 1</td><td>12</td><td>.941</td></tr><tr><td>Factor 2</td><td>9</td><td>.904</td></tr><tr><td>Factor 3</td><td>16</td><td>.945</td></tr><tr><td>Factor 4</td><td>21</td><td>.953</td></tr><tr><td>Total scale</td><td>58 items</td><td>.978</td></tr></tbody></table>

## 5.3 Product Lifecycle Implementation Maturity Model (PLIMM)

As shown by the results in the preceding section, many organizations in the manufacturing industry continue to face difficulties related to data and information access, and the value-measurement of their respective information systems. As a result of this study a Product Lifecycle Implementation Maturity Model was developed. The Product Lifecycle Implementation Maturity Model (PLIMM) helps organizations determine their level of maturity in terms of employing and measuring PLM. The PLIMM was based on the Capability Maturity Model (CMM) literature. According to Wang [30], CMM was originally developed to help the Department of Defense in acquiring appropriate software. The Capability Maturity Model (CMM) approach has been used successfully by many organizations as a basis for assessing relative maturity of practices in various areas, including: data management, warehousing, and governance maturities [31]. For this reason, CMM was adapted in order to create the PLIMM. Furthermore, the metrics identified in this paper and validated through the survey results were parsed into their respective categories to fit within the new PLIMM model. In other words, although the metrics derived from this study may benefit any company wishing to better understand their successes related to PLM, not all companies have processes that are mature enough to benefit from or even provide accurate data enough to capitalize on the PLM metrics. By parsing the metrics according to a maturity model, companies can employ the framework regardless of their own current maturity level associated with their PLM processes. Therefore, even if benchmarking metrics have yet to be collected, companies can use the framework to identify the most appropriate starting point when broaching the subject of identifying and employing PLM metrics.

The PLIMM model presents the performance indicators for each level, providing a broad perspective for assessing PLM capacity. At Level 1, Inputs, the performance indicators delineate resource needs, and whether appropriate amounts of resources have been invested. Level 2, Processes, indicators determine if appropriate processes are being employed; in other words, it helps companies determine if PLM is being implemented and employed properly. Level 3, Outputs, indicators demonstrate the extent to which outputs meet customers' needs and requirements. Level 4, Outcomes, indicators demonstrate whether or not the desired results are being achieved. Hypothetically, at the highest level, Level 5, Impact, the indicators demonstrate success by measuring the return on a PLM investment. Table 2 illustrates the Product
Lifecycle Implementation Maturity Model. Further research is needed to confirm whether Level 5 metrics, Impact metrics, are statistically a separate factor from Level 4, Outcome metrics. Even though only a four-factor model was confirmed in this study, for illustration purposes, Level 5 metrics are included in Table 2, but at this point may be better thought of as belonging to Level 4.

<table><caption>Table 2. Product Lifecycle Implementation Maturity Model</caption><thead><tr><th></th><th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Hypothetical Level 5</th></tr></thead><tbody><tr><th>Metric Types</th><td>Inputs (Have appropriate resource been invested?)</td><td>Processes (Have appropriate processes been implemented?)</td><td>Outputs: (How efficient are the processes?)</td><td>Outcomes: (How effective are the processes?)</td><td>Impact (What is the ROI?)</td></tr><tr><th>Metrics</th><td>Money Time People Technology Infrastructure</td><td>Ideation Concept Development Requirements Management Design Engineering Quality Regulatory Sourcing and procurement Manufacturing and Launch Distribution Quote/order generation Field Service Dispose Reuse</td><td>Requirements-traceability Visualization Device master record control Ideas & concepts Design capture & accessibility Change control Capacity configuration management Metrics Cost of Risk Product development & prototype Resource optimization Product Quality</td><td>Generation of new business Software integration Globalization Cost performance Larger market share Cost reduction Design reuse</td><td>Waste reduction Innovation New Products New Practices New Processes Continuous improvement Sustainable Green manufacturing</td></tr><tr><th>Data Availability</th><td>Limited</td><td>Moderate</td><td>Moderate</td><td>Limited</td><td>Limited</td></tr><tr><th>Collection Difficulty</th><td>High</td><td>High</td><td>Moderate</td><td>High</td><td>Very High</td></tr><tr><th>Collection Automation</th><td>None</td><td>Low</td><td>Moderate</td><td>Low</td><td>Very low</td></tr></tbody></table>

### 5.3.1 Level 1: Input Metrics

At Level 1, Input metrics, organizations are still at a very immature level of metrics collection. At this level the focus is on measuring the extent to which appropriate resources are being invested in PLM, with broad measurement of overall success. The resource inputs driving the applicability of metrics include capital, labor, and time [11], [16], [17]. Specific metrics measuring resources are listed in Table 3.

<table><caption>Table 3. Input Metrics</caption><thead><tr><th>Metric</th><th>Metric</th></tr></thead><tbody><tr><td>Average cash expense cost per product/project</td><td>Average manufacturing engineering development cost per project/product</td></tr><tr><td>Average manufacturing capital cost per product/project</td><td>Average planning/design cash expense cost per product/project</td></tr></tbody></table>
<table><tr><td>Average manufacturing cash expense per product/project</td><td>Average planning/design cost per product/project</td></tr><tr><td>Average manufacturing development cost per project/product</td><td>Average planning/design development cost per product/project</td></tr><tr><td>Average manufacturing engineering capital cost per product/project</td><td>Number of customers captured by new products*</td></tr><tr><td>Average manufacturing engineering cash expense cost per product/project</td><td>Number of new customers captured by new products</td></tr><tr><td>Number of responses to RFP's</td><td>Cost of tool design/redesign</td></tr><tr><td>Total number of new customers*</td><td></td></tr></table>

For research purposes involving such specific and confidential information, the availability of data is limited and difficult to collect. Most organizations do not typically automate or share data collection on their investment efforts; as such information is highly confidential. Therefore, while this research focused on what metrics are appropriate for PLM instantiations, no data was collected regarding specific levels of pre-and-post performance measures associated with any savings or revenue generated through the use of a PLM system. Two of the metrics that converged at this level, ‘number of customers captured by new products’ and ‘total number of new customers’ may be hypothetically deemed Level 5, Impact metrics, which will need to be confirmed in future studies.

### 5.3.2 Level 2: Process Metrics

The next level of metrics maturity is the Processes metrics. These metrics allow organizations to examine their procedures and then, in turn, more effectively control processes. The focus is on assessing the appropriateness of the processes being implemented within the design, build, service, and retirement/reuse/recycle phases of PLM. Table 4 lists the metrics that can be measured to determine the effectiveness of these PLM processes.

<table><caption>Table 4. Process Metrics</caption><thead><tr><td>Metric</td><td>Metric</td></tr></thead><tbody><tr><td>Amount of time required for manufacturing</td><td>Cost per manufacturing engineering error</td></tr><tr><td>Average capital cost per project/product</td><td>Cost per manufacturing error</td></tr><tr><td>Average development cost per project/product</td><td>Number of business processes re-engineered</td></tr><tr><td>Cost per planning and design errors</td><td>Number of parts re-used</td></tr></tbody></table>

### 5.3.3 Level 3: Output Metrics

Level 3 are the Output metrics, the focus of which is on whether the PLM process is efficient in producing products or services. At this level, data is beginning to

<footnote>(* Potentially a Level 5 Impact metric)</footnote>
become more accessible and organizations are beginning to automate their data collection efforts. However, collection of data on the processes is still quite difficult despite the fact that there is greater automation of data collection, which should make data more accessible. Table 5 outlines metrics that are used to measure the Output indicators. In future studies, overall revenue and market share may be better defined as a Level 5, Impact Metric.

<table><caption>Table 5. Output Metrics</caption><thead><tr><td>Metric</td><td>Metric</td></tr></thead><tbody><tr><td>Amount of inventory</td><td>Number of RFP's won</td></tr><tr><td>Amount of personnel output</td><td>Number of product prototypes built</td></tr><tr><td>Amount of time for break-even for new product introductions</td><td>Number of pre-production design changes</td></tr><tr><td>Amount of time required for manufacturing engineering</td><td>Number of suppliers meeting requirements</td></tr><tr><td>Amount of time required for product planning and designing</td><td>Number post-production design changes</td></tr><tr><td>Market share*</td><td>Overall revenue*</td></tr><tr><td>Number of engineering change orders</td><td>Revenue from new products less than 3 years old</td></tr><tr><td>Number of manufacturing engineering errors</td><td>Time to market for new products</td></tr><tr><td>Number of planning and design errors</td><td>Number of manufacturing errors</td></tr><tr><td>Time to market for product improvements</td><td></td></tr></tbody></table>

### 5.3.4 Level 4: Outcome Metrics

Outcome metrics help the organization determine if the production processes are effective. At this level the metrics help to determine if there are any changes and/or benefits resulting from the overarching PLM activities. The collection and reporting of effectiveness metrics are difficult to automate, thus limiting the availability of data. Performance indicators at this level focus on the extent to which procedures and controls have been integrated into the systems. Table 6 lists metrics for measuring outcomes of PLM investments. Again, based on conjecture of future studies, some of the metrics may be better thought of as Level 5, Impact Metrics.

<table><caption>Table 6. Outcome Metrics</caption><thead><tr><td>Metric</td><td>Metric</td></tr></thead><tbody><tr><td>Hours of downtime</td><td>Number of new products</td></tr><tr><td>Number of processes documented in regards to the "support" of products</td><td>Number of processes documented in regards to the "disposal" of products</td></tr><tr><td>Amount of time to develop new ideas</td><td>Number of product recalls</td></tr></tbody></table>

<footnote>(*Potentially a Level 5 Impact metric)</footnote>
<table><tr><td>Number of applications, operating systems, and DBMS integrated</td><td>Reallocation of saved manufacturing engineering processes time*</td></tr><tr><td>Number of collaborative research ventures</td><td>Reallocation of saved planning and designing process time*</td></tr><tr><td>Number of liability lawsuits</td><td>Number of simulated tests</td></tr><tr><td>Number of new industry initiatives supported</td><td>Number of warranty claims</td></tr><tr><td>Number of new product functions or features</td><td>Reallocation of saved manufacturing process time*</td></tr><tr><td>Number of new product ideas evaluated</td><td>Number of product failures</td></tr><tr><td>Number of simulated prototypes</td><td></td></tr></table>

(*Potentially a Level 5 Impact metric)

### 5.3.5 Hypothetical Level 5: Impact Metrics

At Level 5, Impact, metrics measure the impact of PLM investment by determining if there are positive changes in the manufacturing situation or other enterprise divisions. Impact metrics allow organizations to determine if there is a long-term return on their PLM investments. Metrics at this level measure such results as enterprise-wide waste reduction, innovations in new processes, practices, and products, continuous improvement, and sustainable green manufacturing [11], [16], [17].

As a result of the Exploratory Factor Analysis, the metrics variables were categorized into only four factors; whereas those that would hypothetically be Level 5, Impact, are typically confounded by other factors. For instance, measurement of waste reduction is also implied in the process measures, where improved processes would result in a reduction of waste. The same is true for continuous improvement. However, three metrics on reallocation of savings that were included at the Outcomes level can appropriately be used to measure Impact, as these effectively measure the return on investment.

## 6 Discussion and Implications

The purpose of this study was to further refine a PLM Assessment Model (Figure 1) and identify, develop and integrate PLM metrics into a Product Lifecycle Implementation Maturing Model (Table 2). The PLIMM framework developed supports organizations as they define PLM metrics and measure how effectively they are being implemented across the enterprise. Initially, through an iterative process of synthesizing literature, interviewing key PLM stakeholders, and applying scholarly research methods, a large number of metrics were identified. It became quickly apparent, however, that there is a myriad of overlapping metrics, many of which measure overlapping benefits. The final results of the survey produced and validated PLM metrics that are most widely used and considered to be important to industry for measuring PLM initiatives.
A number of metrics were identified as being ‘used and important’ in the PLM performance measurement process. Overall revenue was considered by a significantly large proportion of the respondents as being ‘used and important’ as a metric. Market share and productivity were also identified as important measures. Crucial to the PLM process is time and cost measures, as evidence by the number of other metrics identified as ‘used and important’.

As previously discussed, many organizations have yet to fully implement all phases of their PLM initiative, and thus the issue of return on investments remains elusive. Nonetheless, as a normal course of business, executives must justify their current and future PLM investments as a function of organizational performance, both at the tactical and strategic levels. When designing a PLM process, it is critical that organizations first consult their strategic plan and conduct a PLM benefits analysis as it relates to achieving their strategic goals. While important, defining metrics that accurately measure the impact of PLM is no easy task. The difficulty, in large part, arises from the need to filter standard business-process improvements from the overall impact of PLM on cost-savings and revenue-generation in an organization. The ultimate goal of PLM is to pay dividends through a significant return on investments and increases in innovations. However, without direct measures, most respondents do not seem to have a clear picture of whether, or how, PLM is impacting their bottom line.

As organizations continue to migrate past the more segregated Product Data Management (PDM) software and move towards more holistic and integrated PLM enterprise systems, each organization will inevitably struggle to identify the most appropriate starting point for metrics development and deployment. Thus, the Product Lifecycle Implementation Maturity Model, which integrates and categorizes metrics, may prove helpful as organizations attempt to identify the most appropriate, realistic, and feasible starting point for their metric-development initiatives.

# 7 Conclusions, Limitations of the Study, and Future Research

In this study, researchers extended upon earlier PLM metrics-related studies [5], [11], [16], [17], to provide program-level guidance for quantifying PLM performance in support of organizational strategic goals. In this study, an Exploratory Factor Analysis (EFA) was conducted, a complex procedure further exacerbated by the imperfections of “real world” data [26], [32], [33]. While principal components with varimax rotation and the Kaiser criterion are the norm, they are not optimal; particularly when data do not meet assumptions, as is often the case with social science and business related data [26]. The most replicable results are still obtained by using large samples [34]. Ideally, there would be at least a 5:1 subject to item ratio. The diversity in our data set was limited with the majority of participants coming from very large industries with revenue streams of over one billion dollars.

Future research initiatives could make the results more generalizable and applicable to a broader spectrum of industries; both horizontally across industry-types and vertically within various-sized organizations. Researchers for this study have already begun to identify additional participants. In future studies, not only will online surveys be conducted, but more interviews with industry executives and employees
will be conducted to determine what metrics are appropriate for ongoing PLM
performance measurement. Arguably, as PLM systems continue to evolve, solution
providers are ‘building better islands’, but they are still building islands [5]. The
metrics associated with each of these software islands will necessitate further
investigation until at which point PLM systems evolve to a point where all systems
are fully and tightly integrated, and where data flows consistently, accurately, and
without error between PLM subsystems. As PLM systems continue to evolve, it is
anticipated there will be a much broader spectrum of participants available for future
study which will necessitate a secondary exploratory factor analysis to incorporate the
new data, followed later by studies employing confirmatory factor analysis.

In future work researchers plan to test the refined PLM Assessment Process Model and the Product Lifecycle Implementation Maturing Model. Further testing will attempt to resolve the practical and theoretical issues in the implementation of the PLIMM that will support performance metrics. The literature will need to be reviewed continually, validity of metrics will need to be verified continuously, and the participant pool will need to be expanded.

Summarizing the PLIMM, at Level 1, Inputs, of the Maturity Model, companies should focus on defining and measuring inputs. At this level, the focus is on measuring the extent to which appropriate resources are being invested in the system. Companies at Level 2, Processes, should focus on process measures such as whether a process is in control or not. At this level, companies should focus on measuring features of procedures and processes. At Level 3, Outputs, companies should focus on measuring whether products and services produced are meeting performance standards for example. Companies at Level 4, Outcomes, should focus on measuring outcomes such as improvements in production processes, and capturing wasted resources, time energy, and material. The highest level, Level 5, Impacts, concerns optimization and measuring impact and return on investment with an eye towards reallocating resources for the purpose of generating new and innovative ideas and technologies that result in new products, new markets, and impact on bottom line [35]. Clearly, defining appropriate PLM-related metrics becomes increasingly complex and difficult to collect as an organization matures in its PLM implementation. Nonetheless, given the sizable investments PLM requires, it is imperative that the impact of those investments be understood in the context of meeting and potentially exceeding organizational goals.
References

1. Grieves, M.: Product Lifecycle Management: Driving the Next Generation of Lean Thinking. The McGraw-Hill Company, Inc., New York (2006)

2. Grieves, M.: Virtually Perfect: Driving Innovative and Lean Products through Product Lifecycle Management. Space Coast Press, Cocoa Beach (2011)

3. Subrahmanian, E., Rachuri, S., Fenves, S. J., Foufou, S., Sriram, R. D.: Product Llifecycle Management Support: A Challenge in Supporting Product Design and Manufacturing in a Networked Economy. Int J Prod Life Cycle Man. 1, 4-25 (2004)

4. Ameril, F. and Dutta, D.: Product Lifecycle Management: Closing the KnowledgeLoops. Computer-Aided Design and Applications. 2, 577-590 (2005)

5. Walton, A.: Product Lifecycle Management: Strategic Technology Road Mapping. Ph.D. Dissertation, Purdue University, West Lafayette, (2009)

6. Arena Solutions, Inc.: Why PLM? available at:
http://www.arenasolutions.com/why/plm/index.html (accessed 12 November 2008)

7. Williams, A.: Cost control: trim manufacturing costs during product design. CADalyst. (2005) available at:
http://findarticles.com/p/articles/mi_m0BLL/is_5_22/ai_n15394545 (accessed 10 January 2009).

8. Shah, R., Ward, P.T.: Defining and Developing Measures of Lean Production. J Ops Man, 25, 785-805 (2007)

9. CIMdata: PLM Benefits Appraisal Guide. CIMdata Report, July, (2002) available at:
http://www.cimdata.com/publications/desc_benefits_appraisal.html (accessed 15 January 2009)

10. CIMdata: Product Lifecycle Management: Empowering the Future of Business. CIMdata Report, June 1, (2006) available at:
http://www.cimdata.com/publications/PLM_Definition_0210.pdf (accessed January 15 2009)

11. Tomovic, C.L., Ncube, L.B., Walton, A., and Grieves, M.: Development of Product Lifecycle Management Metrics: Measuring the impact of PLM", Int J Mfg Tech Man. 19, 3/4, pp. 167-179 (2010)
12. Mercer, P.: ERP #66: Implementing an Effective Global Performance Measurement System (2007) available at: http://www.mercer.com/summary.htm?siteLanguage=100&idContent=1286845 (accessed 12 December, 2008)

13. CIMdata: PLM and ERP Integration: Business Efficiency and Value. CIMdata Report, March 8, : (2003) available at: http://www.cimdata.com/publications/complimentary_white_papers.php (accessed January 15 2009)

14. Stark, J.: Product Lifecycle Management: 21st Century Paradigm for Product Realization. Springer, London (2004)

15. Grieves, M.W.: PLM--Beyond Lean Manufacturing. Mfg Eng. 130, 3, 23-25 (2003)

16. Tomovic, C., Ncube, L.B., and Walton, A.: Development of Product Lifecycle Management Metrics: Measuring the Impact of PLM. Paper presented at the International Conference on Product Lifecycle Management, July 9-11, Beijing, China (2008)

17. Tomovic, C.L., Walton, A., Ncube, L., Grieves, M., Birtles, B., and Bednar, B.: Measuring the Impact of Product Lifecycle Management: Process Plan, Waste Reduction and Innovations Conceptual Frameworks, and Logic Model for Developing Metrics. In: Tomovic, M. and Wang, S. (eds.) Product Realization: A Comprehensive Approach, 2-4. Springer, New York (2008)

18. IBM: PLM Challenges and Benefits: Results of the IBM Online Diagnostic Tool. (2004) available at: ftp://ftp.software.ibm.com/software/plm/uk/plm_diagnostic_exec.pdf (accessed 29 November, 2008)

19. Barchan, M.: Measuring Success in a Changing Environment. Strat & Leadership. 27, 3, 12-15 (1999)

20. Borg, W.R., Gall, M.D.: Educational Research. Longman, New York (1989)

21. Tesch, R.: Qualitative Research: Analysis Types and Software Tools., Falmer, New York (2002) http://www.bing.com/search?q=Tomovic%2C+Walton%2C+Ncube%2C+Grieves%2C+Birtles+and+Bednar&src=IE-SearchBox

22. Bourque, L.B., Fielder, E.P.: How to Conduct Self-Administered and Mail Surveys. Sage, London (1995)

23. Rea, L.M., Parker, R.A.: Designing and Conducting Survey Research: A Comprehensive Guide. Jossey-Bass, San Francisco (2007)
24. Sekaran, U.: Research Methods for Business: A Skill Building Approach.
John Wiley & Sons, Inc, Hoboken, (2003)

25. Velicer, W.F. and Jackson, D.N.: Component Analysis Versus Common Factor Analysis: Some Issues in Selecting an Appropriate Procedure.
Multivariate Behav Res. 25, 1, pp. 1-28 (2009)

26. Costello, A.B., Osborne, J.W.: Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from your Analysis.
Practical Ass, Res, & Eval. 10, 1-9 (2005)

27. Committee on Metrics for Global Change Research, Climate Research Committee, National Research Council: Thinking Strategically about the Appropriate use of Metrics for the Climate Change Science Program.
National Research Council of The National Academies, The National Academies Press (2005) available at:
http://books.nap.edu/openbook.php?record_id=11292&page=R1 (accessed 14 February 2009)

28. PTC: The Definition and Implementation of PLM Technologies. PTC Executive Briefing, PTC Aerospace and Defense. (2004) available at: http://www.ptc.com/WCMS/files/25853en_file1.pdf (accessed 20 October 2008).

29. Chelladurai, P., Inglis, S. E., & Danylchuck, K. E. (1984). Priorities in intercollegiate athletics: Development of a scale. Res Quart for Exerc and Sport, 55(1), 74-79

30. Wang, J.: Capability Maturity Model with Information Systems Development. IS6840 Information Systems Analysis. University of Missouri at St. Louis (2003) available at:
www.umsl.edu/~sauter/analysis/6840.../wang/IS6840FinalPaper.doc
(accessed 7 March 2009).

31. Enterprise Warehousing Solutions, INC.: Implementing a Maturity Model.
(2008) available at: http://www.ewsolutions.com/education/cmm-training/implement-cmm (accessed 10 April 2009)

32. Suhr, D.: Reliability, Exploratory & Confirmatory Factor Analysis for the Scale of Athletic Priorities. Paper presented at SUGI 28 SAS Conference Proceedings: Western Users of SAS Software, San Jose, California (2002)

33. Suhr, D.D.: Exploratory or Confirmatory Factor Analysis? SUGI 31 SAS Conference Proceedings: Western Users of SAS Software. San Jose, California (2005)
34. Fabrigar, L.R., Wegener, D.T., MacCallum, R.C., and Strahan, E.J.: Evaluating the Use of Exploratory Factor Analysis in Psychological Research. Psychol Meth. 4, 3, 272-299 (1999)

35. Chew E., Clay, A., Hash, J., Bartol, B. and Brown, A.: Guide for Developing Performance Metrics for Information Security: Recommendations of the National Institute of Standards and Technology. NIST Special Publication. 800-80 (2006)