Abstract:
A method and apparatus for measuring, managing and improving the quality and performance of supplied products and services across a large and potentially very diverse organization or enterprise. The invention provides a more accurate and comprehensive view of a supplier&#39;s overall quality and performance record than conventional systems and methods because it features real-time, interactive access to both quantitative and qualitative performance data throughout the entire life of the supplied product or service. The invention provides this real-time interactive access to the quality and performance data at multiple levels of detail according to user preference. In addition, the invention utilizes supplemental supplier performance indicators, such as customer satisfaction survey results and contract and diversity compliance statistics, to bring key supplier quality and performance issues into sharper focus, and provides corrective action plans to improve supplier quality, performance and accountability throughout and across an entire organization or enterprise.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of Art 
     The present invention relates generally to the field of quality and performance assurance, and more particularly to the subject of measuring, managing and improving the quality and performance of products and services supplied by suppliers both inside and outside of an organization or enterprise. Still more particularly, the present invention relates to automated, computer network-based systems and methods for measuring, managing and improving the quality and performance of supplied products and services across large and potentially very diverse organizations and enterprises. 
     2. Related Art 
     Measuring, managing and improving the quality and performance of supplied products and services usually comprises one of the most significant problems large and diverse organizations, such as national and international manufacturing firms, national defense contractors, telecommunications providers, and state university systems, must face and solve in order to achieve their business, growth, profit, savings and/or budget objectives. In some contexts, the organization&#39;s ability to measure, control and improve the quality and performance of supplied products and services across the entire organization may be a crucial factor to the organization&#39;s success or failure in a competitive or economically-distressed industry. Consequently, companies, businesses and organizations are constantly searching for ways to measure, manage and improve the quality and performance of products and services supplied by outside vendors. 
     Various conventional quality assurance and quality management systems and procedures have been introduced in an attempt to address and reduce quality and performance problems. Due to a variety of shortcomings, however, conventional quality assurance and quality management systems and procedures have not resulted in much success. 
     To begin with, conventional supplier performance and quality management systems typically rely primarily—if not entirely—on the results of “satisfaction surveys” conducted by the vendor or the organization after the product or service which is the subject of the survey has been deployed. In other words, the satisfaction surveys are typically conducted only during the “post-deployment” stage of the life of the product or service, when all of the engineering, testing installation, integration and debugging problems have already been solved. Moreover, such surveys usually address only a particular product line of a supplier, rather than multiple or all product lines for that supplier. 
     The value of almost any satisfaction survey depends substantially on when the survey is conducted. It has been found, however, that satisfaction survey results obtained only after the deployment of the product or service usually do not provide an accurate view of supplier quality. By the time a post-deployment satisfaction survey is conducted and the results become widely available, memories have faded, details are forgotten, letters, documents and e-mails disclosing and discussing quality and performance issues, and the failures and successes that arose during the “pre-deployment” stage (i.e., before the product or service was finally ready for deployment) are often lost and/or forgotten. A division-level manager now basking in the glow (and obtaining the rewards) of being able to produce twice as many widgets, for instance, often forgets or underrates the impact of numerous engineering, testing, delivery and installation problems he had to endure before the responsible product or service was finally put into production. 
     Not surprisingly, then, snapshot survey results obtained only during post-deployment typically fail to tell the complete story of a supplier transaction and typically fail to shed light on many quality-related issues that could benefit from analysis and improvement. Moreover, because they represent only a snapshot measurement of quality during the post-deployment stage, conventional quality systems usually focus only on relatively obvious, short-term supplier performance issues, such as missed product or service delivery dates. 
     Another problem with customer satisfaction surveys is that they only ask the survey participants to grade, rank or assign a “score” to a quality or performance metric. As such, these surveys provide only quantitative results (e.g., discrete performance or quality “grades” or “scores”) from the survey participants. But different individuals usually have different—sometimes substantially different—views and opinions when it comes to assigning a quantitative quality or performance grade to a particular product or service. Conduct or performance that one person considers “excellent,” may be considered by another person—even another person who works very closely with the first person—as merely “acceptable.” Thus, asking these two people to grade or rank the quality of a product or service on a scale ranging from “poor,” “acceptable,” “good,” “very good” to “excellent,” for example, sometimes leads to inadequate, confusing or misleading results. In this respect, even quantitative grades and scores can be very subjective. 
     Conventional supplier quality management systems and processes suffer from another problem in that they typically cannot be applied to the whole of a large organization or enterprise having a multiplicity of diverse sub-organizations, departments or divisions which obtain products and services from that supplier. Rather, they can only be applied at the level of the sub-organization, department or division that provides the data. Such sub-organizations, departments and, divisions may (and usually do) have substantially different quality requirements, grading systems and problem-reporting procedures. Consequently, the results of satisfaction surveys and/or quality reports produced by individual and diverse departments or divisions are not all that useful for measuring, managing and improving supplier quality across an entire organization, which makes it extremely difficult to generate concrete cross-organization level corrective action plans to improve supplier quality. 
     For all of these reasons, conventional supplier performance and quality management systems lack the continuity, consistency and objectivity required by most organizations to achieve significant, long-term supplier quality and performance improvements across the entire organization. Accordingly, there is a need for systems and methods for measuring and managing supplier quality and performance issues on a level appropriate for an entire organization or enterprise, and for the entire life of the product or service, including both the pre- and post-deployment stages. There is a further need for such systems and methods to incorporate both quantitative and qualitative data concerning quality and performance, as well as supplemental supplier performance indicators, such as the suppliers&#39; compliance or noncompliance with critical contract provisions and diversity objectives associated with delivering the product or service. Further still, these methods and systems need to include components and processes for creating, tracking and resolving quality and performance corrective action plans on an on-going basis. 
     SUMMARY OF INVENTION 
     The present invention addresses these and other problems associated with conventional supplier quality and performance management systems. This invention comprises a process of measuring and managing supplier quality throughout the life of the product or service. It includes mechanisms for acquiring and managing qualitative and quantitative measurements during pre-deployment and post-deployment phases, together with contract and diversity compliance measurements, as well as logistics measurements, to provide for a comprehensive assessment of supplier accountability. The invention uses customer satisfaction surveys to bring customers&#39; response into focus, and follows up with corrective action plans to establish a mutually beneficial supplier relationship. By repeating this process at regular intervals, the invention enables better management and continuous improvement of performance and quality of diverse product and service lines. 
     In general, the invention is a process and system for managing performance information in an organization for a supplied product or service. The process comprises the steps of: (1) identifying a set of performance metrics pertaining to a product or service, the set of performance metrics including both pre-deployment stage and post-deployment stage performance metrics; (2) receiving a division-level quantitative data point and a division-level qualitative data report for each performance metric from each division in the organization; (3) producing an organization-level quantitative score for the product or service responsive to the division-level quantitative data point; (4) producing an organization-level qualitative assessment for the product or service responsive to the division-level qualitative report; and (5) generating an overall performance score for the product or service responsive to the organization-level quantitative score and the organization-level qualitative assessment. 
     The pre-deployment stage in the life of a product or service may include multiple pre-production activities for the product or service, including without limitation, engineering, system testing, delivery, installation and integration. The post-deployment stage may include many or all other phases in the life of a product or service, including without limitation, operation, production, ramp-down, phase out and retirement. The set of performance metrics may include, for example, the degree of compliance with a key contract provision, the degree of compliance with a diversity commitment, the extent to which the product or service is broken, under repair, or otherwise unavailable, the degree to which the product or service failed to conform to design specifications, the extent to which repairs are required for related products as a result of receiving the product or service, the extent to which the product or service is delivered late, the extent to which the product or service fails during testing, the degree to which the product or service meets expectations, the reliability for the product or service, and the maintainability of the product or service. Other performance metrics known and used by those skilled in the art may also be used with the invention. The division-level quantitative data point may comprise, for instance, a grade, rank or score for a performance metric, or a value representing a count, price, cost, or period of time for a performance metric. The division-level qualitative data report may comprise responses to a customer satisfaction survey, for example, but it may also include anecdotal data and or personal experiences with the supplier or the supplied products or services. 
     In some embodiments, the identifying step includes the step of recognizing performance metrics already being used by a division, while other embodiments include defining new performance metrics for a division that are appropriate for use and analysis by an entire enterprise. In addition, the organization-level quantitative data point may be derived according to a specified grading criteria, such as an organization or enterprise-wide standard, an industry standard, or a government standard. 
     In some embodiments, the process includes the steps of determining which of the plurality of divisions possesses information related to the performance metric and establishing a procedure for collecting the division-level quantitative data point and the division-level qualitative data report from those divisions. Such collection procedures may optionally include “pre-processing” the quantitative and qualitative data in order to convert it to a format appropriate for the entire organization or enterprise. This pre-processing step frequently makes it unnecessary in some contexts for organizations and enterprises using the invention to impose new and unfamiliar data collection and formatting restrictions on the divisions who provide the quality and performance data. 
     Ideally, each division in the organization or enterprise that submits the quality and performance data concerning the supplied product or service also provides regular updates for the data according to a specified schedule, such as once per month, once per quarter, or upon reaching certain life-cycle milestones. 
     In another embodiment, the process further includes the step of receiving, for each performance metric in the set of performance metrics, a quantitative data point and a qualitative data report from the supplier, for both a pre-deployment stage and a post-deployment stage of the product or service life-cycle. In this embodiment, the invention takes into account the quantitative and qualitative views of the suppliers, as well as the views of those supplied. Still further, the process may include creating a summary based on the overall performance score and the organization-level quality assessment, distributing the summary to each division in the plurality of divisions, and receiving verifications and approvals of the summary from each division in the organization that participated in providing the underlying data. Preferably, the summary is then published on an interconnected data communications network, such as the Internet or on the organization or enterprise&#39;s intranet, with one or more user-activatible controls configured to display detailed quantitative and qualitative data in response to user input. The process may further include providing a copy of the summary to the supplier, and requesting a corrective action plan, which could then be approved and monitored by the organization. 
     In order to develop a complete picture of a supplier&#39;s performance record for a supplied product or service, the present invention, unlike conventional systems, incorporates supplemental performance indicators, such as the supplier&#39;s record of compliance with key contract provisions (e.g., delivery dates, response times, processing speeds, fitness for a particular purpose, etc.) and key diversity objectives (e.g., a minimum level of subcontracting with women and minority-owned businesses). 
     The invention also incorporates logistics into the supplier performance analysis. Logistics include supplied product and service attributes such as the total number of discrepancies, defects, freight and/or shipping issues that arise during the pre- and post-deployment stages of the product or service life-cycle. While these attributes rarely amount to a breach of contract, they quite frequently have a substantial impact on the quality and performance of the supplied product or service, and therefore should not be overlooked in a robust supplier performance management system. Accordingly, the process may also include the steps of identifying and collecting such supplemental performance and logistics data from the divisions of the organization, and including the data in the division-level and organization-level quantitative grades and qualitative reports. 
     In another aspect of the invention, an online supplier performance and accountability computer system for an organization is provided. The system comprises: (1) a first memory storage area having at least one region for storing a plurality of division-level quantitative grades provided by a plurality of divisions in the organization, the plurality of quantitative grades pertaining to a set of performance metrics for the supplied product or service; (2) a second memory storage area having at least one region for storing a plurality of division-level qualitative reports provided by the divisions, the plurality of division-level qualitative reports also pertaining to the set of performance metrics; (3) an executive user interface configured to receive input from a user identifying the supplier; and (4) an electronic report card generator configured to retrieve and display, responsive to the input received by the executive user interface, an organization-level quantitative score responsive to the division-level quantitative grade, and an organization-level qualitative report responsive to the division-level qualitative report. The first memory storage area and the second memory storage area may or may not be the same. At least a portion of the plurality of division-level quantitative grades and a portion of the division-level qualitative reports are provided by the plurality of divisions before the product or service is deployed. 
     In this aspect of the invention, the electronic report card generator may comprise a database management application server configured to retrieve the division-level quantitative grade from the first memory storage area and the division-level qualitative report from the second memory area. The electronic report generator may also include a Web server configured to display quantitative data and qualitative data to the user according to a standard protocol understood by the user&#39;s terminal device. 
     In some embodiments, the invention is configured to include a third memory storage area (which may or may not be the same as the first and second memory storage areas) having at least one region for storing an executive summary. In these embodiments, an administrative component is included to create the executive summary based on the plurality of division-level quantitative grades stored in the first memory storage area and the plurality of division-level qualitative reports stored in the second memory storage area, and to store the executive summary in the third memory storage area. The executive user interface is optionally configured to display the executive summary to the user at various levels of detail in response to user input and according to user preference. 
     A business logic processor may be included in the system to control aspects such as the exact timing supplier report cards are published and/or made available for users to review. It would be a business logic processor, for example, that prevents corrective action plans generated by suppliers from being published before they are approved by the organization. 
     The administrative component is further configured, in a preferred embodiment, to display a user-activatible document attachment control configured to attach an electronic document to the executive summary. With this functionality, for example, the user can attach electronic spreadsheets, images and text data to the executive summaries, which further explain or illustrate points included in the summaries. The system may further include, as part of the administrative component, for example, an authentication facility configured to determine whether the user is authorized to view or modify an executive summary, and a pre-processor configured to convert an original division-level quantitative grade or division-level qualitative report into a specified format prior to storing such data in the first or second memory storage areas. 
     In yet another aspect, a computer-readable storage medium encoded with a computer-executable program to display a supplier performance report card is provided. In a preferred embodiment, the supplier performance report card comprises a pre-deployment stage score, a post-deployment stage score, a logistics score and the results of a customer satisfaction survey. The pre-deployment stage score may comprise, for example, a value indicating a performance level for the product or service during a system integration, testing, engineering, delivery, or installation stage, while a post-deployment score may comprise a value indicating a level of performance achieved during a production or maintenance stage for the supplied product or service. 
     Alternatively, the supplier performance report card may comprise an organization-level quantitative score for a plurality of performance metrics pertaining to the supplied product or service, an organization-level qualitative report for the performance metrics, and an executive summary responsive to the organization-level quantitative score and the organization-level qualitative report. In this aspect, as in each of the other aspects, the plurality of performance metrics includes both a pre-deployment stage performance metrics and post-deployment stage performance metrics. In this case, the organization-level quantitative score is based on the division-level quantitative grades for the performance metrics, and the organization-level qualitative report is based on the division-level qualitative reports for the performance metrics. 
     The division-level quantitative grades and the division-level qualitative reports for the performance metrics may include responses to customer satisfaction surveys, and may be provided by the divisions both before, during and after the product or service is deployed. The division-level quantitative grades and the division-level qualitative reports may also constitute updates for prior division-level quantitative grades and prior division-level qualitative reports, respectively. The organization-level qualitative report may also be based on a plurality of customer satisfaction survey responses provided by a single division of the organization, or a plurality of customer satisfaction survey responses provided by a plurality of divisions of the organization. 
     In a preferred embodiment, the supplier performance report card may further include a request for a corrective action plan, in which case the program encoded in the storage medium may be further configured to display the request in response to an input by a user. The supplier performance report card may further include a corrective action plan, which can also be displayed to a user in response to input. Preferably, the program is further configured to display the division-level quantitative grades and/or the division-level qualitative reports underlying the requests and the corrective action plans in response to a selection of a user-activatible control displayed with the request or plan, or upon the entry of a sequence of characters by the user. 
     In embodiments, the supplier performance report card may further include a logistics report, a contract compliance report, a diversity compliance report, or a combination of two or more of these reports, which can also be displayed in response to the selection of a detail control or upon entry of a sequence of characters by the user. 
     In yet another aspect, the invention comprises a user interface for a supplier performance management system. The user interface includes a first display region comprising an executive summary of a supplier performance report card, where the executive summary includes an organization-level quantitative score, and an organization-level qualitative report for a performance metric pertaining to the product or service. A second display region in the user interface includes a user-activatible control configured to display a division-level quantitative grade, and a third display region includes a user-activatible control configured to display a division-level qualitative report. In this user interface, the organization-level quantitative score is based on the division-level quantitative grade, and the organization-level qualitative report is based on the division-level qualitative report. The division-level qualitative report preferably comprises a response to a customer satisfaction survey provided by a division of the organization. The user interface may further comprise a fourth display region comprising a user-activatible control configured to display a corrective action request or a corrective action plan. 
     FEATURES AND ADVANTAGES OF THE PRESENT INVENTION 
     It is a feature of the present invention that it provides real-time, interactive access to quality and performance measurements for supplied products and services for an organization having potentially many diverse divisions and many diverse methods of measuring and collecting quality and performance data. 
     It is another feature of the present invention in that it provides access to qualitative as well as quantitative supplier performance data. 
     It is a third feature of the invention that these measurements are collected, analyzed and published during both the pre-deployment and post-deployment stages of the life of the product or service, thereby providing a more accurate and complete picture of supplier performance. 
     It is another feature of the present invention that it provides a uniform standard for measuring supplier performance throughout a whole organization or enterprise. 
     It is yet another feature of the invention that it provides an electronic report card for a supplier that is accessible to users via an interconnected data communications network, such as the Internet or an intranet. 
     It is still another feature of the invention that users can view organization-level executive summaries of supplier quality and performance reports, and, if desired, drill down to view the supporting details, such as individual division-level quantitative grades or individual division-level customer satisfaction survey results. 
     It is still another feature of the invention that it provides a mechanism for attaching electronic documents, such as text files, spreadsheets, images, etc., to the executive summaries. 
     It is still another feature of the invention in that it provides for the creation, tracking and resolution of corrective action plans to improve supplier quality and performance. 
     It is still another feature of the invention that it incorporates measurements for logistics and supplemental performance indicators, such as contract and diversity compliance. 
     The present invention has significant advantages over conventional systems. Among other things, the invention provides a more accurate view of supplier performance because it incorporates pre-deployment quantitative and qualitative measurements and analysis, as well as post-deployment measurement and analysis. Moreover, the measurements and analysis are more consistent and trustworthy because they are received from a variety of internal sources and/or departments who work directly with the suppliers, and because each of these sources contributes directly to the overall quality and performance score. Another advantage of the invention is that it allows for collection of quality and performance data from a multiplicity of divisions or departments without dictating the format in which such data is collected and provided. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
         FIG. 1  shows a high-level block diagram of a supplier performance accountability system configured according to one embodiment of the present invention. 
         FIG. 2  depicts a flow diagram illustrating the steps performed in an embodiment of the present invention. 
         FIGS. 3 through 7  depict exemplary graphical user interface screens that may be used in embodiments of the present invention to provide access to supplier performance electronic report cards. 
         FIG. 8  depicts an exemplary format for a customer satisfaction survey report generated by an embodiment of the present invention. In this example, the report contains both a table and a graph illustrating trends in customer satisfaction survey results for a particular quarter. 
         FIG. 9  depicts an exemplary graphical user interface screen for the administrative component of an embodiment of the present invention. 
         FIG. 10  depicts a process flow diagram illustrating the flow of data and the steps performed by a system configured to operate in accordance with one embodiment of the present invention, particularly with respect to pre-processing pre- and post-deployment measurement data. 
         FIG. 11  depicts a process flow diagram illustrating the flow of data and the steps performed in collecting and processing responses to customer satisfaction surveys and corrective action plans according to an embodiment of the present invention. 
         FIG. 12  depicts a process flow diagram illustrating the overall operation and flow of data in a supplier performance and accountability system configured to operate in accordance with the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     With reference now to the figures, a detailed discussion of certain embodiments of the present invention is presented. Notably, the present invention may be implemented using software, hardware or any combination thereof, as would be apparent to those of skill in the art. Accordingly, the illustrative figures and examples below are not meant to limit the scope of the present invention or its embodiments or equivalents. 
       FIG. 1  shows a high-level block diagram of a supplier performance accountability system configured according to one embodiment of the present invention. In the embodiment depicted in  FIG. 1 , the Supplier Performance and Accountability System  150  comprises a Web Server  115 , an Administrative Component  120 , an Electronic Report Card Generator  125 , an Executive User Interface  130 , a Pre-processor  135 , a Database Management Application  140 , and a Memory  145 . In a preferred embodiment, Supplier Performance and Accountability System  150  is coupled to and in communication with one or more division systems (shown as Division System  155  in  FIG. 1 ) via Data Communications Path  175 , and one or more user terminals (shown in  FIG. 1  as User Terminals  110 A,  110 B and  110 C) via Data Communications Paths  111 A,  111 B and  111 C. Each of these components is described in more detail below. 
     Web Server  115  typically is a program that, using the client/server model and the World Wide Web&#39;s Hypertext Transfer Protocol (HTTP), serves the files that form Web pages to Web users (whose computers contain HTTP clients that forward their requests). Every computer on the Internet that contains a Web site must have a Web server program. In this case, users (such as organization executives) wishing to view electronic supplier report cards residing on Supplier Performance and Accountability System  150  log onto User Terminals  110 A,  110 B and  110 C, and connect to Web Server  115  via Data Communications Paths  111 A,  111 B and  111 C. In a preferred embodiment, Data Communication Paths  111 A,  111 B and  111 C comprise wired Internet connections, wireless Internet connections, or both. A user terminal for accessing the system may comprise a personal laptop computer system or desktop computer system (shown as User Terminals  110 A and  110 B in  FIG. 1 ), a web-enabled wireless telephone or personal digital assistant (shown as User Terminal  110 C in  FIG. 1 ), or any other web- or browser-enabled device configured to communicate with a Web server over a data communications network. 
     There are a variety of Web server programs available on the market that may be suitably adapted for use with the Supplier Performance and Accountability System  150  depicted in  FIG. 1 . The Web server program known as “Windows NT Server®,” for instance, available from Microsoft Corporation (www.microsoft.com) of Redmond, Wash., is just one example of such a program. And although the components of Supplier Performance and Accountability System  150 , as depicted in  FIG. 1 , may be located in the same physical computer system and accessed via a single Web server, such as Windows NT Server®, it should be apparent to one skilled in the art that these components also may be housed in physically separate computer systems, which may (or may not) exist at locations physically remote from each other. Such alternative configurations would not depart from the scope of the present invention. Web Server  115  may also be comprised of a combination of Web servers, application servers and database servers. For brevity and ease of understanding, however, such alternative combinations are not shown in  FIG. 1 . 
     Administrative Component  120  provides administrative user interface screens (such as the exemplary user interface screen described below with reference to  FIG. 9 ) and functionality that allows administrative users of the system to perform administrative functions. Such administrative tasks may include, for example: (1) retrieving, reviewing, editing and publishing supplier performance report cards and the results of customer satisfaction surveys; (2) creating and modifying executive summaries; and (3) uploading and attaching supporting electronic documents, such as text files, images, spreadsheets; etc. 
     In a preferred embodiment, Administrative Component  120  would also include a user Authentication Facility  122 , which would allow administrative users to configure and control a user&#39;s access to the system. Preferably, the authentication process is customized according to the particular products and services or the particular suppliers under consideration. Using methods and application programs known to those in the art, users are assigned various access rights depending on their individual responsibilities as they relate to receiving or using supplied products and services. In a preferred embodiment, initial access is provided upon “logging in” with a default password, which is then changed to a user-defined password for enhanced security. Upon authentication, various user-interface screens, applications and tools become available, depending on the users&#39; defined access rights. 
     Similar to the Administrative Component  120 , Executive User Interface  130  provides access to user interface screens, such as the exemplary user interface screens described below with reference to  FIGS. 3-7 , and functionality that allows users to log in and view electronic report cards for a supplier. Both Administrative Component  120  and Executive User Interface  130  are coupled to and in communication with Electronic Report Card Generator  125 , which generates organization-level scores and reports based on the division-level quantitative and qualitative performance and quality data stored in and retrieved from Memory  145  by Database Management Application  140 . 
     In practice, Electronic Report Card Generator  125  receives and interprets requests for supplier performance information from an administrative user via Administrative Component  120 , or from an executive user via Executive User Interface  130 , and sends those requests to Database Management Application  140 . Using standard database querying methods known to those of skill in the art, Database Management System  140  retrieves the appropriate division-level quantitative and qualitative data from Memory  145  and passes it back to Electronic Report Card Generator  125 , where it is compiled and formatted into a supplier report card. Electronic Report Card Generator  125  then passes the supplier report card to Web Server  115 , where it is formatted into a Web page (hyper-text markup language) and transmitted to User Terminals  110 A,  110 B and  110 C via Data Communications Paths  111 A,  111 B and  111 C. 
     The system may also include a Business Logic Processor  136 , which may be located in Electronic Report Card Generator  125 , Database Management Application  140 , or elsewhere in the system. Such a business logic processor might be configured, for example, to prevent creation of an executive summary before all of the divisions who need to contribute quantitative and qualitative data to Memory  145  have provided division-level performance data. 
     Pre-processor  135  is an optional component of Supplier Performance and Accountability System  150 , which, if necessary, converts quantitative data points and qualitative data reports from the format used by the division providing the data into a format acceptable to Supplier Performance and Accountability System  150 , and, more specifically, to a format expected by Database Management Application  140 . Suppose, for example, Division System  155  in  FIG. 1  has installed and uses its own quality and performance problem tracking apparatus (comprising Memory  165  and Problem Tracking System  160  in  FIG. 1 ), which produces reports containing quality and performance data, some of which is required by Supplier Performance and Accountability System  150  to produce organization-level supplier report cards. In a preferred embodiment, Pre-processor  135  may be configured and programmed with intelligence, through the use of filters, templates and parsing algorithms, for example, to receive via Data Communications Path  175  a division-specific supplier performance report generated by Problem Tracking System  160  of Division System  155 , to parse the division-specific performance report, and to create a new performance report containing only the data required by Supplier Performance and Accountability System  150 . The new report, or perhaps just the data from the new report, may then be stored in Memory  145  by Database Management Application  140 , and retrieved later for viewing and analysis under the control of Electronic Report Card Generator  125 . Preferably, Division System  155  is configured to provide, and Supplier Performance and Accountability System  150  is configured to receive, such division-specific supplier performance reports on a regular basis throughout the entire life for the product or service in question, including both the pre-deployment and post-deployment stages. The operation of a pre-processor, according to one embodiment of the present invention, is described in more detail below with reference to  FIG. 10 . 
     It should be noted that although they are shown as separate physical components in  FIG. 1 , the Supplier Performance and Accountability System  150  could be implemented by combining Administrative Component  120 , Electronic Report Card Generator  125 , Executive User Interface  130 , Pre-Processor  135 , Database Management Application  140  and business logic processor (not shown) into one program or processor. 
     In preferred embodiments, supplier report cards may also be accessed and/or downloaded through a file transfer protocol (ftp) port (not shown in  FIG. 1 ) in Supplier Performance and Accountability System  150 . Thus, not all of the supplier report cards provided by the system are delivered through Web Server  115 . If User Terminal  110 B, for example, has installed a standard ftp client, and is therefore capable of sending an ftp download request, then the system will respond by transmitting an electronic report card to User Terminal  110 E via an ftp connection over an ftp port. In preferred embodiments, the division-specific performance reports received from Division System  155 , for example, may also be transferred to Supplier Performance and Accountability System  150  via an ftp connection. 
       FIG. 2  contains a high-level flow diagram illustrating the steps performed in an embodiment of the present invention to generate an overall performance score. First, in step  210 , a set of performance metrics pertaining to the product or service in question is identified. The set includes both pre- and post-deployment performance metrics so that performance may be evaluated from the perspective of the entire life of the product or service. Table 1 below, for instance, illustrates the kind of pre- and post-deployment performance metrics that may be identified in this step, as well as examples of grading criteria that could be used, as described below, to generate organization-level grades for such metrics in an embodiment of the invention. 
     
       
         
               
             
               
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 Performance Metrics and Grading Criteria 
               
             
          
           
               
                 Category 
                 Measurement 
                 Description 
                 Guideline 
               
               
                   
               
               
                 Pre-Deployment: SIT 
                 Lab entry - delivery 
                   
                 A - if delivery met original 
               
               
                   
                   
                   
                 date committed by vendor. 
               
               
                   
                   
                   
                 B - if delivery is within 2 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                   
                   
                   
                 C - if delivery is within 3 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                   
                   
                   
                 D - if delivery is within 4 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                   
                   
                   
                 F - if delivery exceeds 4 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                 Pre-Deployment: SIT 
                 Lab entry - test 
                   
                 A - if all lab entry checklist 
               
               
                   
                 results 
                   
                 items are met. 
               
               
                   
                   
                   
                 B - if in the test team&#39;s view 
               
               
                   
                   
                   
                 only minor deviations are 
               
               
                   
                   
                   
                 noted. All major deliverables 
               
               
                   
                   
                   
                 (release notes, test plans, etc.) 
               
               
                   
                   
                   
                 are met. 
               
               
                   
                   
                   
                 C - if only 1 major item is not 
               
               
                   
                   
                   
                 met. 
               
               
                   
                   
                   
                 D - if more that 1 major item is 
               
               
                   
                   
                   
                 not met. 
               
               
                   
                   
                   
                 F - if product is rejected. 
               
               
                 Pre-Deployment: SIT 
                 Lab entry - 
                   
                 A - if vendor consistently 
               
               
                   
                 responsiveness 
                   
                 exceeds expectations with 
               
               
                   
                   
                   
                 respect to the relevant 
               
               
                   
                   
                   
                 responsiveness dimensions 
               
               
                   
                   
                   
                 B - if vendor consistently meets 
               
               
                   
                   
                   
                 expectations with respect to 
               
               
                   
                   
                   
                 the relevant responsiveness 
               
               
                   
                   
                   
                 dimensions 
               
               
                   
                   
                   
                 C - if vendor is inconsistent in 
               
               
                   
                   
                   
                 their various responses 
               
               
                   
                   
                   
                 D - if vendor appears to try 
               
               
                   
                   
                   
                 very hard and expend effort, 
               
               
                   
                   
                   
                 but is generally ineffective in 
               
               
                   
                   
                   
                 many responsiveness 
               
               
                   
                   
                   
                 dimensions 
               
               
                   
                   
                   
                 F - if vendor is chronically poor 
               
               
                   
                   
                   
                 in setting and meeting 
               
               
                   
                   
                   
                 expectations in the relevant 
               
               
                   
                   
                   
                 responsiveness dimensions 
               
               
                 Pre-Deployment: SIT 
                 Testing cycles - 
                   
                 A - if delivery met original 
               
               
                   
                 delivery 
                   
                 date committed by vendor. 
               
               
                   
                   
                   
                 B - if delivery is within 2 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                   
                   
                   
                 C - if delivery is within 3 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                   
                   
                   
                 D - if delivery is within 4 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                   
                   
                   
                 F - if delivery exceeds 4 weeks 
               
               
                   
                   
                   
                 of original date committed by 
               
               
                   
                   
                   
                 vendor. 
               
               
                 Pre-Deployment: SIT 
                 Testing cycles - test 
                   
                 A - if product has no troubles 
               
               
                   
                 results 
                   
                 that would prevent 
               
               
                   
                   
                   
                 deployment or cause 
               
               
                   
                   
                   
                 Operations workarounds 
               
               
                   
                   
                   
                 (There are no critical or major 
               
               
                   
                   
                   
                 troubles. The release 
               
               
                   
                   
                   
                 submitted makes it to field 
               
               
                   
                   
                   
                 deployment.) 
               
               
                   
                   
                   
                 B - if product makes it to field 
               
               
                   
                   
                   
                 deployment with no critical 
               
               
                   
                   
                   
                 troubles, but requires an 
               
               
                   
                   
                   
                 Operations workaround(s). 
               
               
                   
                   
                   
                 (There are no critical troubles 
               
               
                   
                   
                   
                 but there are one or more 
               
               
                   
                   
                   
                 major troubles requiring 
               
               
                   
                   
                   
                 workarounds). The release 
               
               
                   
                   
                   
                 submitted makes it to field 
               
               
                   
                   
                   
                 deployment. 
               
               
                   
                   
                   
                 C - if product makes it to field 
               
               
                   
                   
                   
                 deployment with one critical 
               
               
                   
                   
                   
                 trouble that requires an 
               
               
                   
                   
                   
                 acceptable Operations 
               
               
                   
                   
                   
                 workaround. (There is one 
               
               
                   
                   
                   
                 critical and/or majors but 
               
               
                   
                   
                   
                 workarounds enable field 
               
               
                   
                   
                   
                 deployment). 
               
               
                   
                   
                   
                 D - if product has at least one 
               
               
                   
                   
                   
                 critical trouble without an 
               
               
                   
                   
                   
                 effective Operations 
               
               
                   
                   
                   
                 workaround such that 
               
               
                   
                   
                   
                 deployment is prevented until 
               
               
                   
                   
                   
                 the vendor provides an 
               
               
                   
                   
                   
                 additional release for testing. 
               
               
                   
                   
                   
                 (There are critical or major 
               
               
                   
                   
                   
                 troubles which prevent field 
               
               
                   
                   
                   
                 deployment). 
               
               
                   
                   
                   
                 F - if significant features are 
               
               
                   
                   
                   
                 not functioning, troubles are 
               
               
                   
                   
                   
                 numerous, and/or product 
               
               
                   
                   
                   
                 stability is such that SIT would 
               
               
                   
                   
                   
                 not recommend 
               
               
                   
                   
                   
                 advancement to field test or 
               
               
                   
                   
                   
                 FOA. (There are numerous 
               
               
                   
                   
                   
                 critical and or major troubles 
               
               
                   
                   
                   
                 and the system under test 
               
               
                   
                   
                   
                 requires major re-work). 
               
               
                 Pre-Deployment: SIT 
                 Testing cycles -  
                   
                 A - if vendor consistently 
               
               
                   
                 responsiveness 
                   
                 exceeds expectations with 
               
               
                   
                   
                   
                 respect to the relevant 
               
               
                   
                   
                   
                 responsiveness dimensions 
               
               
                   
                   
                   
                 B - if vendor consistently meets 
               
               
                   
                   
                   
                 expectations with respect to 
               
               
                   
                   
                   
                 the relevant responsiveness 
               
               
                   
                   
                   
                 dimensions 
               
               
                   
                   
                   
                 C - if vendor is inconsistent in 
               
               
                   
                   
                   
                 their various responses 
               
               
                   
                   
                   
                 D - if vendor appears to try 
               
               
                   
                   
                   
                 very hard and expend effort, 
               
               
                   
                   
                   
                 but is generally ineffective in 
               
               
                   
                   
                   
                 many responsiveness 
               
               
                   
                   
                   
                 dimensions 
               
               
                   
                   
                   
                 F - if vendor is chronically poor 
               
               
                   
                   
                   
                 in setting and meeting 
               
               
                   
                   
                   
                 expectations in the relevant 
               
               
                   
                   
                   
                 responsiveness dimensions 
               
               
                 Pre-Deployment: SIT 
                 Documentation 
                   
                   
               
               
                 Pre-Deployment: EFI  
                 % Jobs w/out 
                 % jobs without outages 
                 Weight: 15% 
               
               
                   
                 Service Outages 
                 out of total jobs 
                 A = 100% 
               
               
                   
                   
                   
                 B &gt;= 99.75% 
               
               
                   
                   
                   
                 C &gt;= 99.50% 
               
               
                   
                   
                   
                 D &gt;= 99.00% 
               
               
                   
                   
                   
                 F &lt; 99.00% 
               
               
                 Pre-Deployment: EFI 
                 Audits - Verizon 
                 % conforming audits out  
                 Weight: 10% 
               
               
                   
                   
                 of total audits by 
                 A &gt;= 95.75% 
               
               
                   
                   
                 Verizon 
                 B &gt;= 88% 
               
               
                   
                   
                   
                 C &gt;= 82% 
               
               
                   
                   
                   
                 D &gt;= 76% 
               
               
                   
                   
                   
                 F &lt; 76% 
               
               
                 Pre-Deployment: EFI 
                 Audits - Vendor 
                 % conforming audits out  
                 Weight: 5% 
               
               
                   
                   
                 of total audits by 
                 A &gt;= 95.75% 
               
               
                   
                   
                 Vendor 
                 B &gt;= 88% 
               
               
                   
                   
                   
                 C &gt;= 82% 
               
               
                   
                   
                   
                 D &gt;= 76% 
               
               
                   
                   
                   
                 F &lt; 76% 
               
               
                 Pre-Deployment: EFI 
                 Specs &amp; Central 
                 % specs and CO 
                 Weight: 10% 
               
               
                   
                 Office Records @ 
                 records out of total jobs 
                 A &gt;= 95.75% 
               
               
                   
                 GMOP 
                   
                 B &gt;= 88% 
               
               
                   
                   
                   
                 C &gt;= 82% 
               
               
                   
                   
                   
                 D &gt;= 76% 
               
               
                   
                   
                   
                 F &lt; 76% 
               
               
                 Pre-Deployment: EFI 
                 Specs &amp; CO 
                 % specs and CO 
                 Weight: 10% 
               
               
                   
                 Records @ GMOP 
                 records requiring no 
                 A &gt;= 95.75% 
               
               
                   
                 Not Requiring 
                 changes out of total 
                 B &gt;= 88% 
               
               
                   
                 Changes 
                 jobs 
                 C &gt;= 82% 
               
               
                   
                   
                   
                 D &gt;= 76% 
               
               
                   
                   
                   
                 F &lt; 76% 
               
               
                 Pre-Deployment: EFI 
                 End of Job Survey  
                 % surveys received out 
                 Weight: 5% 
               
               
                   
                 Received 
                 of total surveys 
                 A &gt;= 95.75% 
               
               
                   
                   
                   
                 B &gt;= 88% 
               
               
                   
                   
                   
                 C &gt;= 82% 
               
               
                   
                   
                   
                 D &gt;= 76% 
               
               
                   
                   
                   
                 F &lt; 76% 
               
               
                 Pre-Deployment: EFI 
                 End of Job Survey 
                 % jobs conformed out 
                 Weight: 15% 
               
               
                   
                 Results 
                 of total surveys 
                 A &gt;= 95.75% 
               
               
                   
                   
                   
                 B &gt;= 88% 
               
               
                   
                   
                   
                 C &gt;= 82% 
               
               
                   
                   
                   
                 D &gt;= 76% 
               
               
                   
                   
                   
                 F &lt; 76% 
               
               
                 Pre-Deployment: EFI 
                 % Orders 
                 % orders completed on 
                 Weight: 30% 
               
               
                   
                 Completed on 
                 time out of total orders 
                 A &gt;= 95.75% 
               
               
                   
                 Time 
                   
                 B &gt;= 88% 
               
               
                   
                   
                   
                 C &gt;= 82% 
               
               
                   
                   
                   
                 D &gt;= 76% 
               
               
                   
                   
                   
                 F &lt; 76% 
               
               
                 Post-Deployment: 
                 Customer 
                 1. Customer outage 
                 A = 99.999-100% 
               
               
                   
                 Availability - Supplier 
                 minutes per month. 
                 B = 99.998% 
               
               
                   
                 Attributable (a) Customer 
                 2. Total Minutes of 
                 C = 99.997% 
               
               
                   
                 Outage Minutes, (b) Total 
                 Availability per month. 
                 D = 99.996% 
               
               
                   
                 Minutes of Availability, % 
                 3. Percentage of 
                 F &lt; 99.996% 
               
               
                   
                 Availability = (1 − a/b) * 100 
                 Availability. 
                   
               
               
                 Post-Deployment: 
                 Maintainability - 
                 Problem reports can 
                 A = 100% 
               
               
                   
                 Responsiveness of 
                 include written problem 
                 B = 95-99.9% 
               
               
                   
                 Technical Support. 
                 reports, billing and 
                 C = 90-94.9% 
               
               
                   
                 Number of Problem 
                 maintenance capabilities, 
                 D = 85-89.9% 
               
               
                   
                 Reports (NPR) - % of 
                 telephone calls that  
                 F &lt; 85% 
               
               
                   
                 critical and major 
                 are directly entered as 
                   
               
               
                   
                 problems closed on 
                 a problem report. Any  
                   
               
               
                   
                 time. % time meets 
                 report of a problem  
                   
               
               
                   
                 requirements. % 
                 after general availability 
                   
               
               
                   
                 reports closed/ 
                 of each release regardless 
                   
               
               
                   
                 due to be closed. 
                 of whether related to 
                   
               
               
                   
                   
                 hardware, software, 
                   
               
               
                   
                   
                 documentation, no 
                   
               
               
                   
                   
                 trouble found, procedural 
                   
               
               
                   
                   
                 or duplicate. 
                   
               
               
                 Post-Deployment: 
                 Feature Availability: 
                 # Critical and/or High 
                 A = 0 
               
               
                   
                 # Critical and/or 
                 Priority Features where 
                 B = 1 
               
               
                   
                 High Priority 
                 Supplier not in sync with 
                 C = 2 
               
               
                   
                 Features where 
                 Verizon needs - As per 
                 F &gt; 2 
               
               
                   
                 Supplier not in Sync 
                 Service and Architecture 
                   
               
               
                   
                 with Verizon Needs -  
                 Features Schedule of 
                   
               
               
                   
                 As per Service 
                 Deliverables 
                   
               
               
                   
                 and Architecture 
                   
                   
               
               
                   
                 Features Schedule 
                   
                   
               
               
                   
                 of Deliverables 
                   
                   
               
               
                 Post-Deployment: 
                 Reliability - Initial 
                 Initial Return Rate (IRR) 
                 Percent of Circuit Pack types 
               
               
                   
                 Return Rates (IRR): 
                 for units shipped 1 to 6 
                 with actual return rate less 
               
               
                   
                 (a) Circuit Pack 
                 months. Determine the 
                 than predicted return rate: 
               
               
                   
                 Returns, (b) IRR 
                 percentage of up to 5 
                 A = 100% or zero return rate 
               
               
                   
                 Population, (c) 
                 circuit pack types with 
                 B = 80-99% 
               
               
                   
                 Actual Initial Return 
                 the highest return rate 
                 C = 60-79% 
               
               
                   
                 Rate, (d) Predicted 
                 for a three-month 
                 D = 40-59% 
               
               
                   
                 Return Rate. 
                 period (current month 
                 F = 0-39% 
               
               
                   
                 (a/b) * 100 = c, 
                 and previous 2 months) 
                   
               
               
                   
                 percent of Circuit 
                 ending in the report 
                   
               
               
                   
                 Pack Types having 
                 month. Each circuit 
                   
               
               
                   
                 (c) less than (d) 
                 pack on the list must 
                   
               
               
                   
                   
                 have had at least 9 
                   
               
               
                   
                   
                 during the three-month 
                   
               
               
                   
                   
                 period. 
                   
               
               
                 Post-Deployment: 
                 Reliability - One 
                 One Year Return Rate 
                 Percent of Circuit Pack types 
               
               
                   
                 Year Return Rates 
                 (YRR) for units shipped 7 
                 with actual return rate less 
               
               
                   
                 (YRR): (a) Circuit 
                 to 18 months. 
                 than predicted return rate: 
               
               
                   
                 Pack Returns, (b) 
                 Determine the 
                 A = 100% or zero return rate 
               
               
                   
                 YRR Population, (c) 
                 percentage of up to 5 
                 B = 80-99% 
               
               
                   
                 Actual Yearly 
                 circuit pack types with 
                 C = 60-79% 
               
               
                   
                 Return Rate, (d) 
                 the highest return rate 
                 D = 40-59% 
               
               
                   
                 Predicted Return 
                 for a three-month 
                 F = 0-39% 
               
               
                   
                 Rate. (a/b) * 100 = c, 
                 period (current month 
                   
               
               
                   
                 percent of Circuit 
                 and previous 2 months) 
                   
               
               
                   
                 Pack Types having 
                 ending in the report 
                   
               
               
                   
                 (c) less than (d) 
                 month. Each circuit 
                   
               
               
                   
                   
                 pack on the list must 
                   
               
               
                   
                   
                 have had at least 12 
                   
               
               
                   
                   
                 returns during the three- 
                   
               
               
                   
                   
                 month period. 
                   
               
               
                 Post-Deployment: 
                 Feature Patch 
                 Feature Patch Quality is 
                 A = 97.00-100% 
               
               
                   
                 Quality (FPQ 0): The 
                 the percentage of 
                 B = 93.00-96.99% 
               
               
                   
                 most recent 
                 offical feature patches 
                 C = 89.00-92.99% 
               
               
                   
                 dominant release 
                 that are determined to 
                 D = 85.00-88.99% 
               
               
                   
                 reported. (a) # 
                 be defective. 
                 F &lt; 85.00% 
               
               
                   
                 defective feature 
                   
                   
               
               
                   
                 patches per month, (b) Total 
                   
                   
               
               
                   
                 # feature patches available 
                   
                   
               
               
                   
                 for general release for 
                   
                   
               
               
                   
                 most recent dominant release 
                   
                   
               
               
                   
                 reported. c = (1 − a/b) * 100 
                   
                   
               
               
                 Post-Deployment: 
                 Reliability - # of FCC  
                 1 monthly failure or 
                 A = 0 FCC event 
               
               
                   
                 events, Supplier attributable: 
                 more =F; 30,000 or &gt; 
                   
               
               
                   
                 1 monthly failure or more = 
                 lines OOS (Out of 
                 F &gt;= 1 FCC event 
               
               
                   
                 F; 30,000 or greater  
                 Service) for 30 minutes 
                   
               
               
                   
                 lines OOS (Out of Service)  
                 or greater 
                   
               
               
                   
                 for 30 minutes or greater. 
                   
                   
               
               
                 Post-Deployment: 
                 Software Update 
                 A variety of new 
                 A = 97.00-100% 
               
               
                   
                 Quality (SWU 0) - 
                 products have been 
                 B = 93.00-96.99% 
               
               
                   
                 The most recent 
                 developed that use an  
                 C = 89.00-92.99% 
               
               
                   
                 dominant release 
                 alternative approach to 
                 D = 85.00-88.99% 
               
               
                   
                 reported. This 
                 install new 
                 F &lt; 85.00% 
               
               
                   
                 measurement is 
                 generic/releases and 
                   
               
               
                   
                 used to evaluate 
                 maintenance software 
                   
               
               
                   
                 the level of 
                 (point or dot releases) 
                   
               
               
                   
                 defective software  
                 into the product. SW 
                   
               
               
                   
                 updates with the 
                 updates replaces the 
                   
               
               
                   
                 goal of minimizing  
                 existing code with new 
                   
               
               
                   
                 associated risks. 
                 software. The 
                   
               
               
                   
                   
                 mechanism used to 
                   
               
               
                   
                   
                 install the generic/release 
                   
               
               
                   
                   
                 and the point or dot  
                   
               
               
                   
                   
                 releases are essentially 
                   
               
               
                   
                   
                 the same. SW Update 
                   
               
               
                   
                   
                 Quality (SWU) quantifies 
                   
               
               
                   
                   
                 the percentage of updates 
                   
               
               
                   
                   
                 that are defective. 
               
               
                   
               
             
          
         
       
     
     Next, in steps  215  and  220 , respectively, a division-level quantitative data point (such as the number of system outages for the product or service) and a division-level qualitative report (such as anecdotal comments about the testing or installation of a product or service, or the results of a customer satisfaction survey) are received from one or more divisions in the organization. Ideally, the quantitative data point and the qualitative data reports are received by means of a data communications path in an interconnected data communications network, such as the one depicted in  FIG. 1 . The data may also be delivered by more traditional means, however, such as by typed reports delivered by hand or mail, for instance, without departing from the scope of the present invention. 
     An organization-level quantitative score is then produced from the division-level quantitative data point, step  225 . In a preferred embodiment, this step is accomplished by processing the division-level quantitative data point according to a grading criteria provided by the organization, the industry as a whole, or a local or national government or regulatory body, such as the Federal Communications Commission. The grading criteria may specify, for instance, that a certain number of critical or major problems associated with the product or service that were resolved within a specified maximum time frame warrants a grade of “A.” See the column entitled “Guidelines” in Table 1, above, for examples of grading criteria that could be used for these purposes in an embodiment of the invention. 
     The effect of this step, essentially, is to generate quantitative performance data that is “normalized” according to a standard set of business requirements that spans an entire organization, not just one division. Performing this step provides a way to reconcile problems created by receiving performance grades for the same metrics from two divisions that use different grading scales. If a division submits grades using a grading scale of “0” to “10,” for example, performing this step converts such grades to the organization-level grading scale, which might be on a scale of “A” to “F.” So, essentially, this step results in the generation of quantitative data points, such as grades, that are meaningful, from a relative standpoint, for the entire organization. 
     While the processing of the division-level data points according to a specified grading criteria like the criteria depicted in Table 1 above may be performed by a human being without departing from the scope of the present invention, an automated processor operating under the control of grading criteria provided in an electronic template (i.e., a spreadsheet) is the preferred method. Such electronic templates and processors may be incorporated, for example, in Pre-processor  135 , which was discussed above with reference to  FIG. 1 . Indeed, multiple and distinct templates may be used in cases where different divisions have different quality and performance requirements. By installing these templates in a processor running at the organization level, divisions would not be required to change their performance and quality data collection procedures in order to participate in the organization-wide supplier quality analysis. 
     Returning now to  FIG. 2 , the next step, step  230 , is to produce an organization-level qualitative assessment based on the division-level qualitative report provided by the division. This step is accomplished, for example, by obtaining responses to customer satisfaction surveys, by obtaining specific anecdotal comments and opinions from users of the product or service, by collecting information concerning the current status of key provisions of the product or service contract, by collecting information from the supplier concerning its compliance with diversity objectives, by analyzing trends evident from the responses to the customer satisfaction surveys, by collecting details (beyond the hard numbers) concerning product or service outages, and by identifying and following up on missing and/or inconsistent quantitative data points. Table 2 below illustrates the types of questions that could be asked in a customer satisfaction survey utilized in an embodiment of the present invention. 
     
       
         
               
             
               
               
             
           
               
                 TABLE 2 
               
               
                   
               
               
                 Sample Customer Satisfaction Survey Questions 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 Customer 
                 What rating best describes the level of satisfaction relative to this supplier consistently 
               
               
                 Satisfaction Survey 
                 meeting product requirements (includes system availability, hardware reliability, 
               
               
                 Question No. 1 
                 number of trouble reports, and/or number of software patches/defective patches) 
               
               
                 Customer 
                 What rating best describes the level of satisfaction relative to this supplier 
               
               
                 Satisfaction Survey 
                 communicating product or service changes to you in a timely manner (product or 
               
               
                 Question No. 2 
                 service related)? 
               
               
                 Customer 
                 What is your level of satisfaction with Product Quality received from this supplier? 
               
               
                 Satisfaction Survey 
                   
               
               
                 Question No. 3 
                   
               
               
                 Customer 
                 What is your current level of satisfaction with the supplier&#39;s ability to meet your present 
               
               
                 Satisfaction Survey 
                 requirements (includes delivery/system turnover, and invoice accuracy and 
               
               
                 Question No. 4 
                 timeliness) 
               
               
                 Customer 
                 What rating best describes this supplier&#39;s product evolution process to meet your 
               
               
                 Satisfaction Survey 
                 needs (including the development and introduction of new product and/or software 
               
               
                 Question No. 5 
                 features/releases, etc.)? 
               
               
                 Customer 
                 What rating best describes this supplier&#39;s service support (inside sales, customer 
               
               
                 Satisfaction Survey 
                 service and technical support, including response to issues during emergencies and 
               
               
                 Question No. 6 
                 resolution time)? 
               
               
                 Customer 
                 Was the supplier in “Default” of this contract? 
               
               
                 Satisfaction Survey 
                   
               
               
                 Question No. 7 
                   
               
               
                 Customer 
                 Was the supplier in noncompliance with any of the “Critical Contract Provisions”? 
               
               
                 Satisfaction Survey 
                   
               
               
                 Question No. 8 
                   
               
               
                 Customer 
                 Has this supplier agreed to all of the Critical Contract Provisions as part of their current 
               
               
                 Satisfaction Survey 
                 contract? 
               
               
                 Question No. 9 
               
               
                   
               
             
          
         
       
     
     By combining and cross-checking all of these qualitative elements with qualitative data received from other divisions concerning the same product or service, a single qualitative assessment that takes all these factors into account is produced. Although not required, this step may be performed by a human being having experience with the divisions providing the division-level qualitative reports, and/or experience with the suppliers, products or services in question. 
     As illustrated by step  235  in  FIG. 2 , the next step is to generate an overall performance score for the supplier based on the organization-level quantitative score and the organization-level quality assessment produced in steps  225  and  230 . The overall performance score is typically embodied in an executive summary of the quantitative and qualitative data provided by the divisions. 
     Finally, as illustrated by  FIG. 2 , each of the above steps is repeated on a regular basis throughout the entire life of the product or service. How frequently these steps should be repeated typically depends on the quality and performance requirements of the organization using the invention. 
       FIGS. 3 through 7  depict exemplary user interface screens that may be displayed in embodiments of the present invention to provide access to the supplier performance report cards generated by, for example, Electronic Report Card Generator  125  in  FIG. 1 .  FIG. 3 , for instance, shows an introductory screen containing buttons configured to display an executive summary performance report (Button  310 ), a pre-deployment performance report (Button  320 ), a post-deployment performance report (Button  330 ), the results of a customer satisfaction survey (Button  340 ), a corrective action request (Button  350 ) and a corrective action plan (Button  360 ). 
       FIG. 4  depicts an example of a user interface screen containing an executive summary according to the present invention. This screen contains at least three regions, each configured to provide performance information or access to performance information concerning a supplied product or service. In this example, Region  410  provides a control (Button  412 ) configured to display the results of a customer satisfaction survey. An example of such results is shown in the report depicted in  FIG. 8 . Region  420  contains controls (Buttons  422  and  424 ) configured to display corrective action documents (i.e., the corrective action request and a corrective action plan). Region  430  of the user interface screen in  FIG. 4  shows an example of an executive summary concerning the performance of a supplier. Notably, a control (Button  440 ) is also provided, which allows the user to drill down to user interface screens containing more detailed quantitative and qualitative data. 
       FIGS. 5 and 6  show exemplary user interface screens for displaying organization-level data for both quantitative and qualitative pre-deployment measurements for a supplied product or service. These screens contain quantitative data points (i.e., grades), indicated in  FIG. 5  by reference number  510 , and qualitative data, indicated in  FIG. 5  by reference number  520 .  FIGS. 5 and 6  also show how certain user interface screens may contain user-activatible buttons (reference number  530  in  FIGS. 5 and 610  in  FIG. 6 ) configured to display additional quantitative and qualitative data to the user, if selected.  FIG. 7  shows an exemplary user interface screen for displaying quantitative and qualitative post-deployment measurements. Similar to  FIGS. 5 and 6 , this screen also contains controls ( 710 ) configured to show more detailed data upon selection by a user. 
       FIG. 8  shows examples of the kind of tables ( 810 ) and charts ( 820 ) that could be used with the present invention to display qualitative data or summaries of qualitative data provided by the divisions. In this example, the table ( 810 ) and chart ( 820 ) summarize the percentage of the number of survey responses marked “Outstanding” or “Very Good.” 
       FIG. 9  illustrates an example of a user interface screen that might be generated by an administrative component of a supplier performance and accountability system configured in accordance with the present invention. As shown in  FIG. 9 , this screen contains controls and buttons that, among other things, allows the user to send emails ( 910 ), retrieve and review survey responses ( 920 ), update performance metric measurements and summaries ( 930 ), upload corrective action plans and supporting documents ( 940 ) and to review and approve publication for supplier performance report cards ( 950 ). 
     With reference to  FIGS. 10 ,  11  and  12 , a more detailed description of the flow of data and the steps performed in a supplier performance and accountability system of the present invention is now provided. 
       FIG. 10  illustrates, for example, the flow of data and the steps performed by a supplier performance and accountability system (such as Supplier Performance and Accountability System  150  described above with reference to  FIG. 1 ), with particular focus on the steps performed by Pre-processor  135 . First, at step  1002 , during a pre-deployment stage in the life of the product or service, division-level performance measurement data for the product or service is collected, preferably from each division in the organization that receives or uses the product or service in question. The pre-deployment performance measurement data is transferred to an application server typically serving as a host for the supplier performance and accountability system. As illustrated in  FIGS. 5 and 6 , such pre-deployment performance measurement data may include, for example, quantitative and qualitative division-level grades the product or service achieved during testing for the product or service. The data may be collected through electronic means, such as transmitting electronic reports via one or more data communications network connections between a division-level problem reporting/quality control computer system and the application server, or through more traditional means, such as by compiling and delivering to the organization one or more written reports, which are then manually entered on the application server. As illustrated at step  1004  of  FIG. 10 , performance measurement data generated during a post-deployment stage (e.g., grades the product or service achieved after deployment in areas such as availability, maintainability and reliability, as shown in  FIG. 7 ) are also collected (preferably from each division in the organization) and transmitted to the application server hosting the supplier performance and accountability system. 
     In a preferred embodiment, and as shown at step  1006 , logistics performance data, embodied, for example, in supplier compliance report files, are also transmitted from the divisions to the application server (over an ftp connection, for instance) on a monthly basis. Logistics performance data may include supplier compliance reports concerning incorrect or improper product or service labeling and packaging, misdirected or late product or service deliveries, shipping damage, etc. 
     When the division-level pre-deployment and post-deployment grades are received on the application server, the next steps (steps  1008  and  1010  in  FIG. 10 ) are to calculate organization-level pre- and post-deployment grades, collect and summarize qualitative data (such as comments and opinions of individual subject matter experts in the divisions who work with the supplied products or services), and save such grades, comments and opinions on the server. The organization-level pre- and post-deployment grades may be based on, for example, standard measurement criteria established by the organization, the industry as a whole, or even the government. In some embodiments, collecting and summarizing qualitative data, such as the comments and opinions of subject matter experts in the divisions, may be performed by a human quality expert in the organization. 
     The final step in  FIG. 10  is to calculate and generate an overall performance report card for the supplied product or service (step  1018 ). But before performing the final step, the application server determines, as shown in steps  1012  and  1014 , whether logistics performance data has been received and is therefore available for the overall performance analysis. If it is determined, for example, that no logistics performance data (e.g., supplier conformance reports) are available on the application server, the system may be configured to initiate an ftp session to immediately receive such logistics performance data, as illustrated in  FIG. 10  by the arrow indicating a return of control to step  1006 . 
     When the logistics performance data (e.g., supplier conformance reports) become available on the application server, the data is processed and aggregated, as in step  1016 , according to a standard measurement criteria, to generate an organization-level logistic grade, which is then saved for the overall supplier performance calculation of step  1018 . As part of step  1016 , qualitative data relating the logistics measurements, such as the comments and opinions of individual experts working with the supplied products or services, are also collected, summarized and saved at the application server level. 
     Finally, at step  1018 , the pre-deployment grades and comments, the post-deployment grades and comments, and the logistics grades and comments are all used to generate the overall report card for the supplier. 
       FIG. 11  contains a process flow diagram illustrating the flow of data and steps performed—at various levels of the system—to process responses to customer satisfaction surveys and generate corrective action plans according to one embodiment of the present invention. As shown in  FIG. 11 , the system may include steps performed by computer systems at Application Server Level  1160 , steps performed by individuals and/or computer systems at Organization Level  1170 , steps performed by individuals and/or computer systems at Division Level  1180 , and steps performed by individuals and/or computer systems at Supplier Level  1190 . 
     The first step, step  1102 , is to collect pre- and post-deployment data from the divisions. This step was described in more detail above with reference to  FIG. 10 , and, more specifically, with reference to steps  1002 ,  1004 ,  1008  and  1010  of  FIG. 10 . Next, the pre- and post-deployment measurement data are checked for completeness in steps  1103  and  1104 . If the data is complete, a request is typically made, at step  1106 , for the division to participate in a customer satisfaction survey (CSS). If the data is incomplete, additional pre- and post-deployment measurement data may be collected (as illustrated by the arrow indicating a return of control to step  1102 ). After the division completes the customer satisfaction survey (CSS), step  1108 , the survey results are reviewed by the organization, as illustrated in steps  1110  and  1112 . In a preferred embodiment, the organization then generates an executive summary of any major performance issues, along with a request for corrective action (CAR), step  1114 , and revises any previously-discovered major issues and corrective action requests that already reside on the application server, step  1116 . Next, in steps  1118  and  1120 , the organization sends a notification to the divisions to review and provide modifications for the executive summary. Any such modifications received from the divisions are then uploaded to the application server (step  1122 ), which uses the modifications at step  1124  to generate and save electronic versions of a supplier performance report card (ERC), customer satisfaction survey report (CSS) and corrective action request (CAR). When the ERC, CSS and CAR are complete (as determined by the organization at step  1125 ), they are released for review and published on the application server. See steps  1126  and  1128 . In addition, these files may be sent, along with a request for a corrective action plan (CAP), to the supplier (step  1130 ) at Supplier Level  1190 . If in this occurs, the supplier reviews the ERC, CSS and CAR (step  1132 ), clarifies and resolves any performance-related issues or confusion about the product or service, if necessary (steps  1134  and  1135 ), and develops and sends a corrective action plan (CAP) back to the organization. See step  1136 . 
     The organization reviews the corrective action plan and uploads it to the application server, step  1138 , where it is published for access by the division, step  1139 . The division reviews the corrective action plan (CAP), step  1140 , and determines whether the plan meets the division&#39;s needs and expectations for the product or service. See step  1142 . If the division&#39;s needs and expectations are met, the division may accept the correction action plan (CAP) and notify the organization of the acceptance, as shown in step  1144 . If, on the other hand, the division does not accept the corrective action plan, the division sends a notification to that effect to the supplier and, if applicable, invokes any contractual remedies the division may have available with respect to the performance of the product or service. See step  1146 . The supplier would then be expected to resolve the problems associated with the unaccepted corrective action plan, and send a new corrective action plan back to the organization, as shown in step  1148 . 
     Finally,  FIG. 12  shows a flow diagram illustrating the overall process flow for a supplier performance and accountability system configured to operate according to an embodiment of the present invention. As illustrated by  FIG. 12 , the process advantageously incorporates performance and quality measurement activities at multiple stages and levels, including Pre-deployment  1250 , Logistics  1260 , Application Server Level  1270 , Organization Level  1280  and Post-Deployment  1290 , to provide a more accurate and comprehensive account of supplier performance. Generally, although not necessarily, the steps depicted in  FIG. 12  occur in the relative order they appear as one scans  FIG. 12  from left to right. 
       FIG. 12  shows, for example, that the first step in the overall process is to perform a request for pre-deployment measurement data, step  1202 , and make a request for post-deployment measurement data, step  1203 . Next, as shown in step  1204 , pre-deployment measurement data is provided. This information typically comes from a division of the organization, but may also be provided by a supplier or another part of the organization. The next step, step  1205 , is to update the pre-deployment measurement data on the application server. By this time, the post-deployment measurement data requested in step  1203  may be available to the organization because it has been provided, for example, by the division or a supplier. See step  1206 . If so, then the post-deployment data is also updated on the application server, as illustrated in step  1207 . While these steps are taking place, various problem reporting machines and applications at the division level, for example, may be configured, in a preferred embodiment, to generate and to transmit logistics measurement data (such as monthly vendor compliance reports) to Application Server Level  1270  via an ftp connection, as illustrated by steps  1210  and  1211  of  FIG. 12 . 
     As Application Server Level  1270  receives pre-deployment measurement data, post-deployment measurement data and logistics measurement data from the divisions, a pre-processing application, such as Pre-Processor  135  of  FIG. 1 , for example, may be configured to read, parse and format the data for use by the organization or application server for generating an overall electronic report card (ERC), as shown in steps  1208 ,  1209 ,  1212  and  1214 . Next, in step  1216 , the organization reviews the electronic report card (ERC) in order to determine (at steps  1218 ,  1220  and  1222 , respectively) whether the pre- and post-deployment data, as well as the logistics measurement data, are valid and complete. These determinations are preferably, although not necessarily, made by a human quality and performance expert at the organization level, who may rely on having access to a large body of quantitative and qualitative performance data received from multiple divisions in the organization, as well as perhaps a great deal of personal experience dealing with the divisions reporting the information and the particular products or services in question. 
     If, at step  1218 , the pre-deployment data does not appear to be valid or complete, then the system may be configured to return to step  1202 , where additional pre-deployment measurement data is requested, and the whole process is repeated. Likewise, if the post-deployment is invalid or incomplete in step  1220 , control passes back to step  1203 , where additional post-deployment data is requested. If, at step  1222 , the logistics measurement data appears invalid or incomplete, the system may be configured to return to step  1211 , thereby initiating another ftp session to receive updated logistics measurements from the division. In each of these cases where the data is determined to be invalid or incomplete, a new electronic report card will be generated at Application Server Level  1270  and reviewed by the organization. Finally, when the pre- and post-deployment measurement data, as well as the logistics measurement data, are determined to be valid and complete, the organization approves and publishes the electronic report card (ERC), as shown in step  1224  of  FIG. 12 . 
     The above-described preferred embodiments are intended to illustrate the principles of the invention, but not to limit its scope. Various other embodiments, as well as modifications and equivalents to these preferred embodiments may occur to those skilled in the art upon reading the present disclosure or practicing the claimed invention. Such variations, modifications and equivalents are intended to come within the scope of the invention and appended claims.