Manufacturing Extension Programs: Manufacturers' Views About Delivery and Impact of Services
Published: 1996-03-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO surveyed manufacturers' views regarding the impact of manufacturing extension programs (MEP), focusing on: (1) factors contributing to the positive impact of overall business performance reported by the majority of survey respondents; (2) whether companies' expectations were met regarding MEP impact on specific business performance indicators; and (3) whether companies thought that MEP actually demonstrated attributes they valued most.
section_title: What GAO Found
paragraphs: GAO found that: (1) the results of its survey could not be applied to all MEP participants or all MEP service categories; (2) companies that supplemented MEP assistance with their own resources, implemented more MEP recommendations, were small and relatively new, and did not pay fees for MEP assistance were more likely to view the MEP program positively; (3) the source of MEP funding did not influence companies' views of the assistance's impact; (4) National Institute of Standards and Technology (NIST) officials believed that NIST support improved MEP programs' efficiency and effectiveness and made MEP services more widely available; (5) about two-thirds to three-quarters of the companies that expected MEP assistance to enhance specific business indicators believed that the results met or exceeded their expectations; and (6) over 90 percent of the companies rated staff expertise, timely assistance, and reasonably priced MEP service fees and project proposals as important MEP features and most were satisfied with their specific MEP programs in these areas.

section_title: Letter
paragraphs: Manufacturing extension programs (MEP) offer manufacturers assistance in modernizing or upgrading their operations, often with state and federal funding. The National Institute of Standards and Technology (NIST) manages federal MEP funding through its Manufacturing Extension Partnership Program, also known as MEP. In this report, MEP collectively refer to all state, federal, and university manufacturing extension programs.
In the current climate of federal budget reductions, Congress is reevaluating its funding of MEP through NIST. One issue Congress is considering is whether MEP services have helped improve companies’ business performance. This report responds to your request that we obtain manufacturers’ views regarding the impact of MEP services on their business performance and the factors that affected the impact and delivery of MEP services.
To identify the impact of MEP services on manufacturers’ business performance, we conducted a national survey of manufacturers who had received substantive services from MEP in 1993. The survey asked manufacturers to assess the impact of MEP assistance on various aspects of their business performance. In our August 1995 briefing report to you, we summarized the overall impact of MEP assistance on the business performance of manufacturers we surveyed, and presented the views of a number of companies that had not used MEP services. We reported that about 73 percent (or 389) of 535 respondents indicated that they believed MEP assistance had positively affected their overall business performance. About 15 percent (or 82) reported they believed MEP assistance had not affected their overall business performance. In addition, most respondents reported that MEP assistance had positively affected their use of technology in the workplace (about 63 percent), the quality of their product (about 61 percent), and their customers’ satisfaction (about 56 percent). This report presents our conclusions from further analysis of our national survey results. These results cannot be generalized to all manufacturers that used MEP.
Our objectives for this report were to (1) identify factors that may have contributed to the positive impact on overall business performance reported by the majority of survey respondents; (2) determine whether companies’ expectations were met regarding the impact of MEP assistance on specific business performance indicators, such as manufacturing time frames and labor productivity; and (3) determine whether companies thought that MEP actually demonstrated attributes they valued most, such as MEP staff expertise, timely assistance, and reasonably priced fees. We did not verify either positive or negative effects of MEP assistance reported by manufacturers, and we did not evaluate the operations or management of specific federal or state programs.
subsections: section_title: Background
paragraphs: The primary mission of MEP is to give “hands-on” technical assistance to small- and medium-sized manufacturers trying to improve their operations through the use of appropriate technologies. MEP engage in a variety of activities to assist small- and medium-sized manufacturers, often in partnership with other business assistance providers such as Small Business Development Centers, community colleges, and federal laboratories. MEP offer a wide range of business services, including helping companies (1) solve individual manufacturing problems, (2) obtain training for their workers, (3) create marketing plans, and (4) upgrade their equipment and computers. MEP assistance focuses on small- and medium-sized manufacturers because research by the National Research Council and others has indicated that these companies lack the resources necessary to improve their manufacturing performance.
MEP funding typically comes from a variety of sources, which may include federal and state government agencies, universities, private industry, and fees. Between fiscal years 1988 and 1994 Congress appropriated a total of $141.7 million (in 1994 dollars) to MEP through NIST. Fiscal year 1995 appropriations were $104 million. State or local agencies are to provide matching funds for NIST grants to individual MEP. A 1995 Battelle Memorial Institute report estimated that states collectively spent about $57.7 million specifically on MEP in fiscal year 1994. That same fiscal year, federal MEP spending was $66 million. We were not able to determine the amount of MEP funding from other sources of support, including universities, private industry, and users’ fees.
section_title: Scope and Methodology
paragraphs: This report analyzes data from questionnaires we sent to 766 manufacturers that had completed at least 40 hours of MEP assistance in 1993 in one or more of four service categories. We obtained the names of these manufacturers from the directors of 57 MEP in 34 states. A total of 551 manufacturers (72 percent) completed and returned the questionnaire. We also interviewed eight manufacturers who had received MEP services and were given tours of their manufacturing facilities in Maryland, Georgia, North Carolina, and South Carolina. Appendix II provides more details on our scope and methodology.
In assessing the impact of MEP services on their companies’ overall business performance, 13 percent of survey respondents reported an extremely positive impact, 59 percent reported a generally positive impact, and 15 percent reported no impact. (Less than 1 percent of respondents (0.2 percent) said the assistance had had a negative impact.) We analyzed the likelihood of the companies reporting that the impact of MEP assistance on their overall business performance was extremely positive, somewhat positive, or not positive, depending on various company and program characteristics identified through the survey. We analyzed how the reported impact of MEP assistance related to the companies’ reported age, 1994 gross sales, and number of permanent employees. Also, we analyzed the reported impact of MEP assistance in relation to the companies’ activities associated with the assistance—whether they made financial investments, spent staff time, implemented recommendations, and paid for the assistance. In addition, we analyzed how the reported impact of MEP assistance varied according to whether programs received NIST funds.
We used logistic regression techniques to determine which factors were statistically significant in predicting the reported impact of MEP assistance on companies’ overall business performance. The strength of these particular statistical techniques is that they allowed us to estimate the individual influence of each factor on the reported impact, both before and after the influences of all other relevant factors identified in the survey were controlled. Appendix III provides more detailed information on our methodology, the models tested, and the results obtained.
We used simple frequency distributions to determine whether the companies’ expectations were met regarding the impact of MEP assistance on specific business performance indicators and to analyze whether MEP demonstrated the attributes most valued by the companies.
Results from our work cannot be generalized to all companies that used MEP because our questionnaire covered only companies that had completed at least 40 hours of MEP assistance. In addition, our results do not apply to all MEP services because we limited our analysis to four MEP service categories.
Since we did not evaluate the operations or management of specific federal programs, we did not obtain agency comments on this report. However, on February 12, 1996, we discussed a draft of this report with NIST officials, including the Director of the NIST Manufacturing Extension Partnership Program. He agreed with the technical accuracy of the report and offered minor clarifications, which we incorporated into the report where appropriate.
We did our work primarily in Los Angeles, New York, San Francisco, and Washington, D.C., from February 1995 to January 1996 in accordance with generally accepted government auditing standards.
section_title: Factors That Influenced the Impact of MEP Assistance on Companies’ Overall Business Performance
paragraphs: We analyzed several factors related to company and program funding characteristics to determine whether they influenced the companies’ assessment of the impact of MEP assistance on their overall business performance. We found that several company characteristics—relating to company level of involvement with MEP assistance, and company size and age—influenced the companies’ assessment of the impact of MEP assistance on their overall business performance. However, the program funding characteristic we examined—whether the program received NIST funds—did not influence the companies’ assessment of the impact of MEP assistance.
We found that the level of companies’ involvement played an important role in determining the outcome of MEP assistance. The manufacturers that had made financial investments in their company as a result of MEP assistance were 2.5 times more likely than those that did not to report an “extremely positive” impact on their overall business performance, as opposed to a “generally positive” impact. They also were 5.6 times as likely to report a generally positive impact as opposed to a “neutral” or “negative” impact. Companies whose staff spent relatively more time in activities related to MEP assistance were 1.7 times more likely to report an extremely positive impact of MEP assistance on their overall business performance, as opposed to a generally positive impact.
Furthermore, the relatively small companies, which research has indicated are most in need of modernization assistance, were most likely to report that their overall business performance was improved by MEP assistance. According to the National Research Council, small- and medium-sized manufacturers generally lack the expertise, time, money, and information necessary to improve their manufacturing performance. We found through our survey that the companies whose fiscal year 1994 gross sales were less than $1 million were 3.1 times more likely to assess the impact of MEP assistance as extremely positive, as opposed to generally positive. Likewise, the companies started since 1985 were 2.0 times as likely as the older companies to report an extremely positive effect of MEP assistance on their overall business performance, as opposed to a generally positive effect.
Our visits to manufacturers provided examples of how MEP assistance benefited growing companies. A furniture manufacturer said his company needed MEP assistance to make fewer mistakes in the growth process. This manufacturer said he used MEP experts to help identify and correct environmental and worker safety hazards, so the facility would comply with federal workplace standards. At a company that makes molded plastics, the company president said that the company needed MEP assistance to guide its rapid growth. MEP helped this company with strategic management, planning, worker training, and quality improvement.
Our survey revealed no significant differences in how the companies viewed the impact on their overall business performance of MEP that did and did not receive NIST funds. MEP funding typically comes from a variety of sources, which may include federal and state government agencies, universities, industries, and fees. The combination of funding sources varies across programs, but our analysis revealed no significant distinction in how the companies assessed the impact of MEP that did and did not receive NIST funds. Specifically, MEP that received NIST funds were equally as likely as other MEP to have their impact on business performance rated positively by the companies. In commenting on our analysis, NIST officials said that, given the manufacturers’ positive responses to our survey, they expected no difference in how the manufacturers viewed the impact of MEP that did and did not receive NIST funds. Moreover, they said that the function of NIST funding is to help MEP serve more clients, with a wider variety of services. Also, they said that they believed NIST support improves programs’ efficiency and effectiveness, which are dimensions of MEP that our survey did not address.
section_title: Improvements to Specific Business Performance Indicators Met Most Companies’ Expectations
paragraphs: As part of our analysis, we compared what the companies said they expected from MEP assistance to the results they reported. We found that most of the companies (between 61 and 77 percent) reported that MEP assistance met or exceeded their expectations for improvements to specific business performance indicators, such as manufacturing time frames, the quality of market research, and sales to new and repeat customers. However, between 23 and 39 percent of the companies reported that their expectations were not met for improvements to these indicators.
Our survey results indicate that equipment modernization and plant layout assistance improved manufacturing time frames for most of the companies expecting these improvements (see fig. 1). In particular, the survey results indicate that equipment modernization and plant layout assistance met or exceeded the expectations of a substantial number of the companies for reducing cycle times—the times required by machines or work stations to fully complete their sequence of operations (77 percent)—and setup time—the time it takes to prepare equipment for changes to production (76 percent). In addition, the assistance met a large number of the companies’ expectations for improvements to worker output (76 percent). However, about 30 percent of companies we surveyed that received equipment modernization and plant layout assistance reported that they did not have their expectations met for reductions to manufacturing lead time, the time it took them to process an order, from start to finish, after design approval.
Several companies commented on how MEP assistance affected their efforts to improve plant layout and modernize equipment. One manufacturer that we visited said the company was able to solve problems with congestion and redundant product movement on the plant floor after implementing MEP plant layout recommendations. The company was rewarded with faster production and lower costs. Another manufacturer responding to our survey commented that, by modernizing equipment and improving plant layout, the company was better able to meet its delivery schedules and, thus, satisfy its customers’ needs.
Most of the companies that received product design and development assistance reported in our survey that they achieved anticipated improvements to quality (see fig. 2). In particular, large proportions of these companies reported fewer incomplete product development projects (77 percent) and improved quality of market research (71 percent). Most of the comments we received regarding product design and development assistance were positive. For example, one respondent commented that the assistance it received made it possible for the company to develop a process that it could not have developed on its own. However, not all companies shared such views. One respondent wrote that it took too much management time to work with MEP consultants, and he felt that the company had educated the consultants, and not vice versa.
Our survey also indicated that most companies’ expectations for reduced product design and development time frames were satisfied. Seventy percent of the companies reported they received anticipated reductions in the time needed to get new products to market. One survey respondent commented about the importance of MEP assistance in getting a new product to market, noting that the assistance helped the company to overcome equipment problems, which freed the company to market new machine technology. Despite positive assessments such as these, our survey results show that product design and development assistance met fewer of the companies’ expectations for reducing costs of product development (66 percent) and increasing access to new customers (65 percent), compared to other business performance indicators.
Between 61 and 74 percent of the companies we surveyed that expected quality improvement assistance to bolster specific business performance indicators were satisfied with the results they received (see fig. 3). A substantial percentage of the companies had their expectations fulfilled regarding increased sales to repeat customers (74 percent) and new customers (69 percent). However, our results indicate that, for 39 percent of the companies, quality improvement assistance did not meet expectations for reducing rework and scrap levels.
Customer satisfaction was an important goal of the companies seeking quality improvement assistance. Ninety-four percent of the companies we surveyed regarding quality improvement assistance said they sought the assistance in order to enhance their competitive position in the marketplace. In interviews, several manufacturers told us that they undertook quality improvement initiatives in order to retain and attract customers. They said that an increasing number of customers had high expectations for the quality of products. For example, at a foundry we visited, the company president said that many customers of foundry products were reducing the number of suppliers and were working on a closer, more long-term basis with the remaining suppliers. He said that this new customer-supplier relationship put more emphasis on quality than ever before and that it was extremely important to guarantee quality in order to retain customers. Another survey respondent said that the company was “forced” to comply with a quality assurance program by its customers, even though customers rejected virtually none of its products.
section_title: MEP Had Features Companies Valued
paragraphs: Most of the companies that responded to our survey were satisfied with the service delivery features of the program they used. The companies ranked MEP staff expertise, timeliness, and affordability as the features most important to them. A majority of the companies (80 percent or more) also responded that they were satisfied that their program demonstrated each of the service delivery features they deemed important (see fig. 4).
About 93 percent of the companies responding to our survey rated staff expertise as an important attribute for MEP in general, and 88 percent of respondents said they were satisfied with the expertise of the staff at the specific program they had used. In our visits to manufacturers, they cited several examples of how MEP staff expertise benefited their company.
A manufacturer of heavy agricultural equipment said its three staff engineers were fully occupied solving day-to-day manufacturing problems, with no time to address the “big picture.” The company used MEP experts to support company efforts to develop innovations to keep the company moving forward.
A manufacturer of souvenir and collectible items was considering investing in over $600,000 worth of advanced production equipment. The manufacturer told us that MEP located a consultant who had the expertise to provide the company with an independent opinion about whether the equipment under consideration was appropriate for the company’s needs.
A hosiery mill had installed advanced knitting machines but continual machinery breakdowns had cut productivity by 70 percent. A senior company official told us that MEP brought experts in training, engineering, and human resources to help the company reverse this decline and benefit from the machinery upgrade.
Most respondents looking for timely and affordable assistance said they found it through MEP. About 92 percent of the survey respondents rated timely assistance as an important MEP attribute, and 83 percent said they were satisfied with the timeliness of the assistance provided by the program they had used. Ninety-one percent of respondents rated reasonably priced MEP service fees and project proposals as important MEP attributes, and most were satisfied with the price of fees and proposals costs at their own program. Eighty percent of respondents who paid fees were satisfied that the fees were reasonable, and about 81 percent of respondents were satisfied that their program had project proposal costs within their financial means. Three hundred twenty-eight survey respondents (60 percent) paid a fee for MEP assistance. Of those, 58 percent said that the value added or worth of the assistance was worth more than what they paid for it, 27 percent said the assistance was worth about what they paid, and 11 percent said the assistance was worth less than the fee they had paid.
As agreed with you, unless you announce the contents of this report earlier, we plan no further distribution until 14 days after the date of this letter. At that time, we will send copies to the Director of NIST, the Secretary of Commerce, and the Chairmen and Ranking Minority Members of congressional committees that have responsibilities related to these issues. Copies also will be made available to others upon request.
The major contributors to this report are listed in appendix IV. Please contact me at (202) 512-8984 if you have any questions concerning this report.
section_title: Questionnaire With Aggregate Responses
section_title: Objectives, Scope, and Methodology
paragraphs: At the request of Chairwoman Constance A. Morella of the Subcommittee on Technology, House Committee on Science, we obtained manufacturers’ views regarding the impact of manufacturing extension programs’ (MEP) services on their business performance and the factors that affected the impact of MEP services.
In August 1995, we reported that most manufacturers responding to our questionnaire believed MEP assistance had positively affected their overall business performance. Our objectives for this report were to analyze (1) the factors that may have contributed to the positive impact of MEP assistance on companies’ overall business performance; (2) the question of whether companies’ expectations were met regarding the impact of MEP assistance on specific business performance indicators, such as manufacturing time frames and labor productivity; and (3) the issue of whether MEP actually demonstrated attributes that companies indicated they valued most, such as MEP staff expertise, timely assistance, and reasonably priced fees. We did not verify either positive or negative impacts reported by manufacturers.
To identify manufacturers that had used MEP services to survey regarding the services’ impact on their business performance and the factors that had affected the services’ impact, we (1) developed criteria for the type of MEP our study would include, (2) located all MEP that fit our criteria, and (3) asked these MEP for their cooperation in supplying names of clients that met our survey criteria (described in the following paragraphs).
Since the term “MEP” could include a variety of programs and organizations, we consulted MEP literature and MEP experts to develop a set of criteria to use in identifying programs to include in our study. For the purpose of our study, we considered programs to be relevant if their primary function was to provide direct technical assistance to individual manufacturers, using program staff or supervised consultants. We defined “technical assistance” as one or more of the following activities: providing access to and encouraging the use of innovative and/or off-the-shelf manufacturing technologies and processes; disseminating scientific, engineering, technical, and management providing access to industry-related expertise and capability in university research departments; and transferring advanced manufacturing (i.e., cutting edge) technologies and techniques to companies.
Our definition excluded business assistance programs such as the Small Business Administration’s Small Business Development Centers; business incubators; financial assistance, funding, and grant programs; joint research ventures with universities and/or federal laboratories; on-line technical data base services; and industry networks.
We located 80 MEP that met our criteria for inclusion and had been established before 1994. We used reports from the National Governor’s Association, the Northeast-Midwest Institute, and the Battelle Memorial Institute in Ohio that contained references to existing MEP as the basis for identifying programs that would possibly fit our criteria. We confirmed and updated information in these reports by conducting structured telephone interviews with all programs that we believed matched our criteria. We interviewed officials from a total of 114 programs in 40 states. Eighty of them met our criteria for inclusion and had been established before January 1994.
Fifty-seven of the 80 MEP that qualified for our study supplied us with the names of clients that met our survey criteria. Thirteen of these MEP received NIST funding for fiscal year 1994, accounting for 36 percent of survey respondents. In an effort to determine if the qualified programs that provided client information differed from the qualified programs that did not, we compared the two sets of programs on the basis of program age, total funding, federal funding, and type of administration. The results of the comparisons indicated that there were no significant differences between MEP that did and did not provide client data.
We asked the 57 participating MEP to select from their records all manufacturers that met specific criteria that we developed in consultation with MEP officials and MEP evaluation experts. The client had to meet the following criteria: It had to be a manufacturing facility, which means that its products had to belong to one or more of the manufacturing categories in the Department of Commerce’s Standard Industrial Classification codes. Our survey excluded nonmanufacturing facilities, such as service providers or farmers. It had to have received at least 40 hours of MEP assistance in 1993. Thus, when the facility received our survey in early 1995, at least 1 year would have elapsed since the MEP assistance ended. MEP evaluation experts have told us that 1 year would have been sufficient time for facilities to be able to gauge the value of the assistance and its impact on their business performance. Experts also have told us that 40 hours would have been enough assistance to have had a potential effect on a manufacturer’s business performance. It had to have completed assistance in one or more of the four categories defined in the following paragraph. In cases in which a manufacturer completed more than one type of assistance, we asked the MEP official to choose the primary assistance provided to the manufacturer (i.e., the assistance requiring the most MEP time and/or resources).
We did not verify the client information MEP provided against the programs’ records.
The assistance categories we included in our survey involved the following: Quality improvement. Technical assistance in planning, developing, and implementing a quality system to help a manufacturer attain higher quality standards.
Equipment modernization and plant layout. The evaluation and analysis of plant layout and equipment to determine the most efficient means of manufacturing or assembly through reorganization of the process flow through the facility, and/or upgrading, reconfiguring, or replacing manufacturing equipment.
Product design and development. Services to support the creation, enhancement, or marketing of a manufacturer’s product.
Environmental or energy assessment. Assessment of hazardous materials, discharge, waste products, energy use, and other environmental effects within a manufacturing operation.
We chose these four assistance categories because they share important characteristics. They are types of assistance that MEP typically offer clients, so our survey potentially could include clients from most MEP. Also, the four types of assistance are defined in a similar way by most MEP, according to MEP officials. Other MEP services (such as worker training and strategic business planning) may vary considerably from one program to another.
Finally, we selected types of assistance that were directed at clients’ manufacturing operations. MEP clients receiving operations-related assistance were able to tell us (1) how they expected the assistance would affect their operations and/or performance and (2) whether or not these expectations were met. Other types of MEP assistance—examples are material engineering, electronic data exchange, and computer upgrading—have effects on manufacturers’ operations that are less visible and less easily measured. As a result, manufacturers may have difficulty determining the expected and actual impact of these types of services on their business operations and performance.
We designed four questionnaires, each focusing on one assistance category. In designing our survey questions, we obtained input from National Institute of Standards and Technology (NIST) and MEP officials, MEP evaluation experts, and managers at manufacturing facilities. We also reviewed client surveys that MEP used.
Each questionnaire contained identical questions to obtain background information about the respondent and to get respondents’ views on the impact of MEP services on their business performance and the factors affecting the impact of MEP services. However, the four surveys also had unique questions asking about the expected and actual outcomes of the assistance, because each type of assistance focuses on a different aspect of manufacturers’ operations. We tailored these questions to ask about the kind of impacts that reasonably could be expected to result from the particular kind of assistance received.
As part of our survey development, we tested all four surveys with manufacturers who had received MEP assistance in Texas, Iowa, New York, and Kansas. We chose those states in order to cover diverse areas of the country where MEP are located. We also interviewed eight manufacturers who had received MEP services and were given tours of their manufacturing facilities in Maryland, Georgia, North Carolina, and South Carolina. We visited these southern states because MEP directors had agreed to arrange for us to meet selected clients. We asked the manufacturers about their experiences with MEP services and the impact of those services on their business performance.
Our final surveys initially were mailed to a total of 843 manufacturers from February 1995 through March 1995. Follow-up mailings were made through May 1995. Each manufacturer was sent one survey, based on MEP information on the primary type of service the manufacturer had received.
The primary reason manufacturers did not respond to our survey was their inability to recall MEP assistance they had received. We wrote letters asking the nonrespondents why they did not return our survey. We received responses from 60 companies out of 274 nonrespondents. About 33 percent told us that no one at their facility could recall the assistance received in 1993 and/or that we had addressed the survey to a person who no longer worked at the facility. On the basis of this information, in addition to other information provided by our nonrespondents, we reduced our survey population from 843 to 766.
We obtained an overall response rate of 72 percent across all four surveys. Response rates varied from a low of 63 percent for the environmental/energy survey to a high of 76 percent for the quality improvement survey.
Our analysis of the companies that did and did not respond to our survey found nothing to indicate that our results would have been different if the nonrespondents had completed our questionnaire. The respondents and nonrespondents were similarly distributed across different geographic locations and different MEP.
Since we did not evaluate the operations or management of specific federal programs, we did not obtain agency comments on this report. However, on February 12, 1996, we discussed a draft of this report with NIST officials, including the Director of the NIST Manufacturing Extension Partnership Program. He agreed with the technical accuracy of the report and offered minor clarifications, which we incorporated into the report where appropriate.
We did our work primarily in Los Angeles, New York, San Francisco, and Washington, D.C., from February 1995 to January 1996 in accordance with generally accepted government auditing standards.
section_title: Technical Appendix: Loglinear and Logistic Methodologies and Analysis Results
paragraphs: We used logistic regression techniques to determine which factors were statistically significant in predicting the reported impact of MEP assistance on companies’ overall business performance. We began our analysis by considering nine factors that may have affected how the manufacturers we surveyed assessed the impact of MEP assistance on their overall business performance. The factors included the following characteristics of those manufacturers: (1) the number of permanent employees as of January 1, 1995, (2) the number of hours company staff devoted to MEP assistance, (3) the year the company started operating, (4) the company’s gross sales in fiscal year 1994, (5) whether the company paid any fees for MEP assistance, (6) whether the company made any financial investments as a result of the assistance, (7) whether the assistance included recommendations, and (8) the percentage of MEP recommendations the company implemented. We also considered whether the company used a program that had received NIST funds. These factors all are listed in the first column of table III.1.
Odds ratios indicating the effects of the different factors on the odds of MEP being assessed as: Extremely Positive vs.
Generally Positive vs.
0 = 100 or more; 1 = 20 - 99; 2 = less than 20 0 = less than 100; 1 = 100 - 250; 2 = more than 250 0 = before 1985; 1 = since 1985 0 = over $1 million; 1 = under $1 million 0 = no; 1 = yes 0 = no; 1 = yes 0 = no; 1 = yes 0 = few or none; 1 = some; 2 = all or almost all 0 = yes; 1 = no Number of permanent employees was dropped from the multivariate analysis because of its strong association with gross sales. Each of these two indicators of company size were significantly related to assessments when the other indicator was ignored. However, when we controlled for gross annual sales, the effect of number of permanent employees was not statistically significant.
The percentage of recommendations implemented was dropped from the multivariate analysis because there were too few responses to perform the analysis. Only 70 percent of the companies received recommendations and provided information on the percentage of recommendations implemented.
Some of these factors had many categories. We used loglinear methods to determine which of those categories differed with respect to companies’ assessment of the overall impact of MEP. We combined the categories that were not significantly different from one another. The categories which ultimately were contrasted with one another are given in the second column of table III.1. For the purpose of our analysis, the factors were used as the independent variables.
We used simple bivariate logistic regression models to estimate the individual influence of each factor on the reported impact of MEP assistance, without controlling for the influence of all other relevant factors identified in the survey. We estimated which of the nine factors, as categorized in Table III.1, were related to (1) the odds on the overall impact of MEP being assessed as extremely positive versus generally positive and (2) the odds on the overall impact of MEP being assessed as generally positive versus negative or neutral. Our bivariate estimates are given as odds ratios in the third and fifth columns of table III.1.
As can be seen in that table, seven of the nine factors had a significant relationship to the likelihood that companies assessed the impact of MEP assistance as extremely positive, as opposed to generally positive. In addition, four of the nine factors were significantly related to the odds of companies assessing the impact of MEP assistance as generally positive, as opposed to neutral or negative.
The bivariate odds ratios have a straightforward interpretation. The odds ratio gives an estimate of how each factor, as categorized in column 2 of Table III.1, affected companies’ assessment of MEP assistance. For example, the companies with 20-99 employees were more than twice as likely as the companies with 100 or more employees to assess the impact of MEP as extremely positive as opposed to generally positive. Likewise, the companies with less than 20 employees were more than twice as likely as the companies with 20 to 99 employees to assess the impact of MEP as extremely positive, as opposed to generally positive. Similar interpretations can be given to the other odds ratios in the table. The bivariate odds ratios are estimates that do not take into account the effects of other variables.
We also undertook multivariate analysis of the data. Multivariate analysis also estimated the individual effect of each factor on the reported impact of MEP assistance, but it controlled for the influence of all other relevant factors. It is necessary to control for the influence of multiple factors because some factors are associated with others, making it impossible to isolate their individual effect on the dependent variable. Our multivariate analysis did not include two factors used in the bivariate analysis: the number of permanent employees and the percentage of recommendations companies had implemented.
The odds ratios in the fourth and sixth columns of table III.1 provide the results of multivariate analysis. Odds ratios that are marked by an asterisk represent statistically significant effects. Five factors had significant effects on the odds of whether programs were assessed extremely positively as opposed to generally positively: (1) the number of company staff hours devoted to the assistance, (2) when the company started operating, (3) the company’s 1994 fiscal year gross sales, (4) whether the company paid any fees for the assistance, and (5) whether the company made any financial investments as a result of the assistance. Only two factors had significant effects on whether assessments were generally positive as opposed to neutral or negative: (1) the number of company staff hours devoted to the assistance and (2) whether the company made any financial investments as a result of the assistance.
Many of the significant effects from the multivariate analysis are quite sizable. For example, the companies that made financial investments were 2.5 times as likely as those that had not made financial investments to assess the impact of MEP assistance as extremely positive, as opposed to generally positive. The companies that made financial investments also were 5.6 times as likely as companies that had not made financial investments to assess the impact of MEP assistance as generally positive, as opposed to neutral or negative. Other odds ratios can be similarly interpreted.
Our letter report features the results of the multivariate analysis. The multivariate estimates may differ from the bivariate estimates because the multivariate analysis controlled for the effects of all other factors when estimating the influence of one factor. Bivariate analysis estimates the influence of one factor without controlling for the effects of other factors. In general, the multivariate and bivariate estimates for each factor are similar, with two exceptions.
The first exception is company staff hours devoted to MEP assistance. In the bivariate analysis, this factor was unrelated to whether companies assessed the impact of MEP assistance as extremely positive versus generally positive. However, multivariate results indicate that company staff hours were significantly related to companies’ assessment of the impact of assistance as extremely positive, as opposed to generally positive. We believe that the significance varies because of a relationship between company size and the number of company staff hours spent on MEP assistance. In particular, larger companies devoted more staff hours to the program. In order to accurately assess the independent influence of company staff hours, we needed to control for company size. Our multivariate model controls for company size by including the variable that measures gross sales. Therefore, the multivariate model provides a more accurate assessment of the impact of company staff hours, independent of company size.
The second exception was the factor measuring whether MEP assistance included recommendations. In our bivariate analysis, this variable was significantly related to both extremely positive and generally positive assessments. However, its significance disappeared in our multivariate analysis. Companies receiving recommendations were more likely to devote more staff hours to the program and to make financial investments as a result of MEP assistance. Therefore, when the multivariate analysis controlled for company staff hours spent on the assistance and financial investments made as a result of the assistance, the effect of recommendations was rendered insignificant.
section_title: Major Contributors to This Report
subsections: section_title: General Government Division, Washington, D.C.
paragraphs: Susan S. Westin, Assistant Director Douglas Sloane, Supervisory Social Science Analyst Stuart Kaufman, Senior Social Science Analyst Barry L. Reed, Senior Social Science Analyst Rona Mendelsohn, Senior Evaluator (Communications Analyst)
section_title: Los Angeles Regional Office
paragraphs: Patrick F. Gormley, Assistant Director Amy L. Finkelstein, Evaluator-in-Charge Edward Laughlin, Senior Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.