|  **Economic         Assessment Office (EAO)** ACCELERATION OF TECHNOLOGY           DEVELOPMENT          BY THE          ADVANCED TECHNOLOGY PROGRAM:          The Experience of 28 Projects Funded in 1991 Frances Jean Laidlaw, Ph.D.          Industry Consultant

Office           of Economic AssessmentAdvanced Technology ProgramNational Institute of Standards and TechnologyTechnology AdministrationU.S. Department of Commerce

Table of Contents
EXECUTIVE SUMMARY
ACKNOWLEDGMENTS
 CHAPTER 1: INTRODUCTION: DELINEATING THE STUDY 
**1.1         About the Advanced Technology Program (ATP)**
**1.2           Is ATP Meeting its Mandate to Accelerate the Development and Commercialization           of Technology ?**
**1.3           What is Cycle Time ?**
**1.4           Why Reduce Cycle Time ?**
**1.5           Cycle-Time-Related Findings of Previous Studies of ATP**
**1.6           Further and More Focused Study of Impact on Applied Research Cycle Time           Warranted**
 CHAPTER 2: RESEARCH DESIGN 
**2.1         Research Methodology**
**2.2           Sampling Methodology**
**2.3           Research Questions on Cycle Time**
**2.4           Research Question on Applied-Research Performance Measures**

CHAPTER 3: RESEARCH RESULTS**3.1           Interpretation of Results**
**3.2           The Importance of Reducing Applied Research Cycle Time**
**3.3           Reasons to Reduce Applied Research Cycle Time**
**3.4           Impact of ATP Participation on Applied Research Cycle Time**
**3.5           Flow-Through of ATP's Impact on Applied Research Cycle Time to Later           Stages in the Technology Development Life Cycle**
**3.6           Economic Value of Reducing Applied Research Cycle Time By One Year**
**3.7           ATP Effects that Helped Reduce Applied Research Cycle Time**
**3.8           Carryover of Cycle-Time Improvements Resulting from ATP Participation           to Non-ATP Applied Research Projects**

LIST OF REFERENCES
END NOTES
 APPENDIX 1: SELECTED BIBLIOGRAPHY ON CYCLE TIME
APPENDIX 2: ATP PROJECTS AND INTERVIEWEES
 APPENDIX 3: ABSTRACT

 EXECUTIVE SUMMARY

The Advanced Technology         Program (ATP) is a unique partnership in which the Federal government         and private industry jointly fund research and development projects that         have high technical risk and commensurately large potential for creating         broad-based benefits for the United States. The ATP was created by the         Technology Competitiveness Act of 1988, amended by the American Technology         Preeminence Act of 1991 (P.L. 100-418 and P.L. 102-245). Industry conceives         and proposes research projects to the ATP. The ATP evaluates the proposed         projects for their potential to add to the scientific knowledge base and         to contribute to the nation's economy, and provides multi-year funding         to those projects that score high in rigorous competitions. Research awards         are made both to industry-led joint ventures and to single companies.         If the research projects are successful, the companies are expected to         undertake subsequent commercialization activities and to promote the diffusion         of the enabling technologies developed into multiple application areas.         The ATP focuses on accelerating the development of those technologies         which are likely to have benefits extending well beyond the direct award         recipients. In economic terminology, it focuses on funding technologies         for which the social return is likely to be high and far in excess of         the private return on investment. 

The ATP acts as a         partner rather than a contractor in the research project. In addition         to the funding, awardees often receive technical advice from the National         Institute of Standards and Technology (NIST) scientific staff and advice         from ATP's technical and business monitors who provide project oversight         for the ATP throughout the project life. In addition, proposers receive         constructive feedback upfront from ATP's peer review process on their         technical and business plans. The ATP emphasizes integrated planning across         all aspects of the project by encouraging participants to establish a         team that includes people who are involved with the development of the         technology - and people who will be responsible for the eventual application         of the technology in products or processes that can be commercialized.         Although the ATP provides funding only for research, it requires that         upfront planning be done for down-stream activities such as marketing,         production, and distribution; and encourages that the voice of the customer         or end-user be inserted into the process as soon as possible. 

The ATP has an evaluation         program to ensure that the projects it funds are on track, to determine         their short- and longer-run impacts, and to improve the program's effectiveness.         The 1993 Government Performance and Results Act (GPRA) requires evaluation         of Federal programs, and the ATP has emphasized evaluation from its start         in 1990. In support of that effort, the ATP engages the services of a         variety of academic and consulting economists, as well as its own staff.         This is one such evaluation study. 

Since the ATP-funded         projects are multi-year research projects and most of the approximately         300 projects the ATP has funded to date were funded only in the last several         years, it is premature to attempt to measure empirically the long-term         impacts of the program. However, some of the earlier funded projects have         recently completed or are nearing completion of their research phase,         and it is beginning to be feasible to conduct preliminary assessments         of some of the effects. 

The study investigated         the impact that participation in the ATP has had on the applied research         cycle time of a group of program participants from 1991 through the spring         of 1996. Cycle-time impacts are of keen interest because one of the ATP's         legislated mandates is to assist U.S. businesses in creating and applying         generic technology for rapid commercialization. 

Earlier surveys found         that a majority of program participants believed that participation in         the ATP had helped them to reduce cycle time and that it was important         to do so - but the earlier surveys which addressed a variety of performance         metrics did not provide details on why cycle-time reduction is of special         importance to the companies; on how participation in ATP helped them to         reduce their applied research and technology development cycle time and         get to market more quickly; or on whether there were time effects beyond         the ATP project. 

Structured telephone         interviews with the primary investigators of the 28 projects funded by         the ATP in 1991 resulted in the following principal findings: 
-  The sample included           18 single-company projects, led by 13 small for-profit firms, 3 medium           or large companies, and 2 non-profits. In addition, the sample included           10 joint ventures, 4 led by non-profits, 3 led by small companies, and           3 led by large companies.    Twenty-seven (27)           or 96% of the 28 company interviewees estimated that participating in           ATP had helped their companies to reduce their technology development           cycle time anywhere from 30% to 66% with fourteen estimating a 50% reduction           (most typically from a projected six-year cycle down to a three-year           cycle). 
 - These program             participants spoke of a stage in the technology development life cycle             that is not usually reflected in the literature . . . that is the             "capitalization stage," wherein cash infusions are particularly critical             to further progress. 

 - The interviewees               attributed their cycle-time reduction to ATP's requirements for               disciplined and integrated project planning and management; achievement               of a critical mass of resources through ATP funding; attraction               of additional financial support because of ATP's "Halo Effect;"               greater project stability because of a long-term and strategic commitment               to solving the technical problem; and ATP's emphasis on collaboration.               
 Twenty-four (24)           or 86% of the 28 interviewees expected the positive impact on cycle           time already experienced in the applied-research stage to flow through           to later stages in the technology-development cycle (the product development,           production, and marketing stages), thereby causing them to enter the           marketplace more quickly. 
 Fifteen (15) or           54% of the 28 interviewees were able to quantify their "ball-park" estimates           of the economic value of reducing cycle time by just one year, and these           estimates ranged from one million to several billions of dollars, with           a median value of $5.5 million. 
 Twenty-four (24)           or 86% indicated that participation in ATP resulted in cycle-time improvements           that carried over to other technology development projects outside of           ATP. They spoke of adapting specific "ATP practices" to related projects;           application of methodologies and processes used or developed in the           ATP project to the firm as a whole; adoption of a cultural bias favoring           speedier processes; and taking advantage of the positioning provided           by the enabling technologies developed in their ATP projects to accelerate           the development of a whole series of related applications. 

 - These findings               on carry-over impacts suggest that ATP's positive impacts on participant               cycle time may extend beyond the specific project funded, resulting               in broader and longer-term benefits to awardees than might have               been anticipated. 

 - These findings               suggest that ATP may be fostering self-perpetuating, institutionalized               practices that reduce cycle time. 

The economic vitality         of a country depends increasingly on the ability of its companies to harness         and use technical capabilities rapidly and effectively. Since applied         research is a source of new technologies and product innovations, accelerating         research cycle time holds potential for accelerating commercialization         of research results and, thereby, promoting the economic growth of the         United States and enhancing our quality of life. 

From the findings         of this study, it appears that the ATP is helping to overcome two types         of economic efficiency problems related to speed to market: 
-  the difficulty           of obtaining funding at all to undertake long-run, high-risk, enabling           technology development; and    the difficulty           of implementing coordinated, collaborative R&D management practices           needed to speed the conduct of R&D and the commercialization of           the resulting technology. 
ACKNOWLEDGMENTS

The author wishes         to thank all those who contributed so many excellent ideas and suggestions.         First is Rosalie Ruegg, the Director of the Advanced Technology Program's         Economic Assessment Office, who made a significant contribution to the         research at the beginning of the project, by helping me to improve the         series of interview questions; and at the end, by helping me to clarify,         focus, and shorten the final document. 

Next, appreciation         is due to Richard Spivack and Ernesto Robles, colleagues in the Economic         Assessment office, who helped me run chi square and other statistical         analyses on a computerized statistical package, SPSS. Richard Spivack         served as reviewer, reading initial and final drafts of the report and         providing helpful comments. In addition, Jeff Harrison, a research librarian         at the National Institute of Standards and Technology, was of great assistance         in my literature search. 

Special thanks are         due to the 28 program participants who gave so generously of their time         and thoughts in the structured interviews: Doug Armstrong, Conrad Balazs,         Don Burland, Calvin Carter, Trevor Castor, Jack Corley, Stephen Demko,         Thomas Edman, Joseph Engleberger, Bruce Gambol, David Gibson, Scott Goodwin-Johannason,         Arnold Hagler, Carl Johnson, Nassar Karam, Glenn Kephart, Gretchen King,         Jim McNeely, John Parker, John Pollinger, John Rowell, Howard Saunders,         Mike Sellberg, John L. Simonds, Ernie Vahala, Bill Waddell, David Welch,         and Arthur Yang. 

Finally, I would like         to acknowledge the contribution of the Washington Editorial Review Board         Reader, Gregory Tassey, a Senior Economist in NIST's Program Office. CHAPTER I: INTRODUCTION:           DELINEATING THE STUDY

1.1 About the Advanced Technology         Program (ATP)

The Advanced Technology         Program (ATP), which is administered by the National Institute of Standards         and Technology, an agency of the Department of Commerce, is a unique partnership         between government and private industry to accelerate the development         of high-risk, generic technologies that promise significant commercial         pay-offs and widespread benefits for the economy. The ATP was created         in 1988 with the passage of the Omnibus Trade and Competitiveness Act         of 1988 (P.L. 100-418), which directed that the National Institute of         Standards and Technology (NIST) expand its outreach beyond laboratory         research and development to include a rigorously competitive ATP and the         Manufacturing Extension Partnership Program. 

The ATP provides funds         for the early phases of technology development through cooperative research         agreements with single businesses or industry-led joint ventures and research         consortia. Awards to individual companies may not exceed $2 million and         the period of Federal funding may not exceed three years. Projects by         joint ventures may run as long as five years, with no limit on the amount         of funding . Single companies that receive awards are reimbursed for part         or all of the direct costs of their proposed research but must pay for         all overhead costs. Joint ventures, which consist of two or more companies,         are reimbursed for less than half of their direct and overhead costs;         the companies must provide more than 50 percent of the total funding for         their project. (Some of these requirements are expected to change for         future competitions.) 

The goals of the ATP         as stated in its enabling legislation are "to assist U.S. business in         creating and applying the generic technology and research results to (1)         commercialize significant new scientific discoveries and technologies         rapidly and (2) refine manufacturing technologies." As restated in the         Federal Register on July 14, 1990, the ATP will assist U.S. businesses         to improve their competitive position and promote U.S. economic growth         by accelerating the development of a variety of pre-competitive generic         technologies by means of grants and cooperative agreements. 

Through 1996, the         ATP had announced 288 awards and committed $989 million for advanced technology         development. The participating organizations (the awardees) had committed         over $1 billion dollars in matching funds.

1.2 Is ATP Meeting its Mandate         to Accelerate the Development and Commercialization of Technology ?

One of ATP's evaluation         interests lies in gaining a better understanding of intra-firm impacts         of ATP-funded projects. This study contributes to that area by providing         a more in-depth look at the impact of ATP on the awardees' applied research         cycle times.(1)

The specific purpose         of this study was to evaluate the impact that the ATP is having on applied         research cycle time of companies participating in the program, with the         larger objective of determining if the ATP is successfully meeting its         legislated mandate to accelerate the development and the commercialization         of technology. The study focuses on applied research cycle time, because         high-risk applied research is largely what the ATP funds, and at this         early stage of the ATP, that is the period of time most feasible to evaluate.(2) Although there is a large body of literature         and an established discipline that focuses on improving production cycle         time and even product-development cycle time, this is not the case for         applied-research cycle time. The investigation of applied-research cycle         time is a relatively new and emerging discipline. 

1.3 What is Cycle Time ?

Cycle time, according         to Daniel Jordan, is the following: 

"Cycle time is the         time required to complete a particular process, such as the product development         process, from start to finish. Cycle time is distinct from timeliness.         It should be viewed from the customer's perspective in that it includes         time required for testing, validation, modifications and rework. Although         not all aspects of cycle time may be under an organization's control,         it is important to know where the time is spent in order to best identify         opportunities for cycle-time reduction."(3)

Similarly, Hewlett-Packard         defines cycle time as the period of time from the starting point when         the opportunity occurs to the point when the customer is satisfied.(4)

Different terms are         sometimes used to connote the cycle time concept. These terms include         "Time Line," "Cycle Time," "Time to Market," "Speed," and "Slip Rate."         

It is relatively easy         to define the starting points and ending points for the "downstream" functions         of manufacturing, sales, and distribution. It is more difficult to define         the starting points and the ending points for the "upstream" functions         of applied research and technology development. One can not state with         any certainty the starting and ending points for applied research. No         matter what one posits, it will be challenged, debated, and discussed.         

Hewlett-Packard has         instituted a program at the beginning of each new product development         cycle called the "Imaginative Understanding of User Needs (IU2N)."         During this time, the organization pulls together its resources and defines         the product that will proffer the most competitive response to the opportunity.         With reference to Figure 1, the goal is to cut the time between "To" and         "Ts" as a strategy to increase return on investment.(5)

**Figure 1.           Hewlett-Packard Model of Product-Development Cycle Time**

Within the company         setting, the main source of cycle-time reduction strategies has traditionally         been operations. Typically, cycle-time reduction (CTR) started in manufacturing         and then migrated "downstream" to sales and distribution. A question is         whether the ATP can start at the applied research end and stimulate time         reductions that will propagate forward, from the research cycle through         the production and marketing cycles. 

1.4 Why Reduce Cycle Time         ? 

*Cycle-time         reduction is viewed as increasingly important because it enhances global         competitiveness.*In the 1970s, Japanese companies         adopted a civilian equivalent of maneuver warfare in manufacturing called         "cycle-time analysis," and "slaughtered" American competitors. (6)

According to Groves,         an Ernst and Young study concluded that Japan is the only country that         emphasizes the routine use of both cycle-time analysis and process simplification.         About half of the Japanese businesses use these practices more than 90         percent of the time. Fewer than 25 percent of the businesses in the United         States, Canada, and Germany use process cycle-time analysis or process         simplification to improve their business processes.(7)

In a 1988 Harvard         Business Review article titled "Time - The Next Source of Competitive         Advantage," George Stalk warned that unless United States companies reduce         their new product development and introduction cycles from 36-48 months         to 12-18 months, Japanese manufacturers would easily out-innovate and         out-perform them.(8)         In order to reduce cycle time, Stalk said that domestic firms must learn         to develop manufacturing processes and products simultaneously to collapse         time and ensure better manufacturability, an approach often referred to         as simultaneous or concurrent engineering. (9)

McGrath, Anthony and         Shapiro have stated, "If manufacturing and total quality management were         viewed as the industrial battlegrounds of the late 1970s and 1980s, product         development and cycle-time reduction are the battlegrounds of the 1990s.         The advantages that come from reducing time to market and consistently         developing better products are so significant that they will shift the         competitive balance in favor of companies that can achieve them first.         A company that can efficiently introduce more new products, react faster         to market and technology changes, and develop superior products will win         battles with its competitors. The key to achieving these advantages is         improving the product development process and getting to market more quickly.         This is why it will become the battleground of the 1990s."(10)

*Cycle-time reduction         helps a company achieve its goals.* As Table 1 shows, high performance         companies report faster time-to-market and have more aggressive cycle-time         reduction goals than low-performance companies. 

 Table 1. Cycle-Time Performance         of High Versus Low Performance Companies
 Performance             Level **Today Months **Goal             Months**1.             Product                 Improvement
HI               
LO
8.5               
10.0
5.3               
5.72.             Product Line                 ExtensionHI             
LO11.6             
13.47.9             
9.03.             New Product                 LinesHI             
LO22.8             
25.814.9             
18.2***Source: Mercer Consulting Company presentation,       1995.***

By being first to         market, a company favorably impacts its goals as they relate to sales,         sales growth, development costs, total costs, profits, market position,         market share, customer loyalty, competitors, industry standards and intellectual         property rights.(11)         In industries dependent on patent protection for high margin, an additional         year before the product comes off patent may be extremely valuable. Informal         standards, such as the "form factor" or "feel of the product," can also         be importantly related to cycle time. Gary Tooker, president of Motorola,         pointed out that the form factor for the company's portable hand-held         phone established a standard in the industry for "feel" because it was         first to market. 

Though most of the         articles to date on reducing R&D cycle time focus more on the "D"         (development) than the "R" (research), there is at least one article that         highlights industry's increasing interest in reducing applied research         cycle time. Bob Burkhart's article "Reducing R&D Cycle Time" for the         Industrial Research Institute (IRI) states that research at the "fuzzy         front end" carries high uncertainty; and successful output cannot be readily         scheduled, as with the development phase. Only after identifying the potential         causes for uncertainty in the decisions and activities of the research         phase might barriers be reduced or removed by applying appropriate solutions;         for as knowledge grows, risk recedes and the capacity to advance in a         shorter time frame materializes. Bob Burkhart's research for the article         came from the IRI's ten-member Management Research Team which used common         total quality management (TQM) tools to identify 45 major causes of uncertainty         in research in eight different categories, as shown in Table 2. "Customer         requirements not defined" and "delays in decision" were the most frequently         encountered causes of uncertainty. Forty potential solutions to causes         of uncertainty in research were also identified, as shown in Table 3.         The solutions outlined by this industry team are consistent with ATP's         emphasis on integrated business and technical planning and management.(12)

 Table 2. Causes of Uncertainty         in Research **Technical             Processes (T)**            1. Not invented here             2. Science insufficient             3. Effort insufficient             4. Core competency mismatch             5. Skill mismatch             6. Customer interface insufficient             7. Product feature mismatch             8. Technical planning insufficient             9. Technical support insufficient             10. Manufacturing capability insufficient             11. Financially unfeasible             12. Economically unfeasible             13. Timing inappropriate**Management             Style**            1. Management planning insufficient             2. Management effort insufficient             3. Management support insufficient             4. Management review insufficient             5. Business practices restrictive             6. Finances inadequate             7. Reward systems inappropriate             8. External technical interfaces inadequate**Business             Processes (B)**            1. Risk assessment insufficient             2. Market strategy inadequate             3. Business strategy inadequate             4. Technical strategy inadequate             5. Priorities change             6. Facilities unavailable**Market**            1. Customer needs definition inadequate             2. Requirements inadequately defined             3. Market intelligence insufficient             4. Sales capability insufficient             5. Distribution capability insufficient             6. Value chain understanding insufficient**People/Culture**            1. Customer revalidation insufficient             2. Company's culture inappropriate             3. Teamwork inadequate             4. Creative encouragement inadequate             5. Employee profile incomplete **Competitor**            1. Competitor intelligence insufficient             2. Competitor's capabilities unknown             3. Competitor's strategies unknown**Communication**            1. Information flow inadequate             2. Technical services inadequate**External             efforts**            1. Regulatory impact unanticipated             2. Public sentiment misunderstood             3. Global diversity misunderstoodSource: Bob Burkhart,       "Reducing R&D Cycle Time," Research Technology Management, May-June       1994, pp. 27-32.

Table 3. Potential         Solutions to Uncertainty in Research              1. Enhance customer and market research resources and efforts             21. Outsource economically unfeasible products             2. Use market and competitor intelligence databases and consultants             22. Utilize valid product termination criteria             3. Assign staff to monitor competitors, patents, literature, society             meetings, universities, federal labs, etc.             23. Practice open communication, participative management, and employee             empowerment             4. Use formal documentation of customer and vendor interactions             24. Implement competitor countermeasures             5. Use multi functional teams to routinely validate needs/requirements             25. Utilize appropriate individual/team reward systems             6. Establish on-going interface with customers' and vendors' R&D             organizations             26. Match personnel acquisition and utilization with strategies and             priorities             7. Conduct market, customer and competitor training for staff             27. Establish opportunities for continuous professional education             8. Establish on-going interface with universities, federal labs, consultants,             research institutes, etc.             28. Employ continuous improvement/TQM processes             9. Conduct needs discovery training             29. Utilize technical gatekeepers             10. Conduct communications, team building and facilitator skill training             30. Utilize dedicated product stewards             11. Utilize technical team brainstorming             31. Utilize flexible budgeting             12. Maintain balance of long- and short-range R&D efforts in core             technologies             32. Maintain positive contacts with influential groups, officials,             agencies, etc.              13. Utilize SBU and R&D interfaces in project management             33. Maintain central reporting and data archiving system             14. Validate project criteria against company strategy             34. Account for teamwork in performance management system             15. Routinely evaluate marketing, manufacturing, and technology functional             strategies             35. Utilize dedicated and competent regulatory, environmental and             legal staffs             16. Routinely evaluate match of resources to projects             36. Initiate early examination of potential liabilities and regulatory             issues              17. Initiate early prototype development             37. Maintain up-to-date response plan             18. Acquire businesses with the needed knowledge/resources             38. Maintain positive community outreach efforts             19. Develop joint ventures, alliances and cooperative and contract             research agreements             39. Maintain proactive industry group involvement             20. Plan and execute coordinated market entry             40. Practice parallel project management***Source: Ibid.***

As total quality management         (TQM) practices were thought to render the most significant strategic         advantages in the 1970s and 1980s, cycle-time reduction is believed to         be the new wave that will carry world-class organizations from the end         of this millennia into the next. As total quality management became a         common practice, so will the practice of reducing cycle time. There is         a new recognition that in the same way that a shorter product development         cycle enhances a firm's ROI, reducing R&D cycle time helps a company         to increase its ROI. 

1.5 Cycle-Time-Related Findings         of Previous Studies of ATP 

Previous studies found         that participants reported that the program had the effect of shortening         their research and development (R&D) cycle time, enabling them to         accelerate the development and commercialization of the technology. 

Solomon Associates,         in a 1993 report, found that sixty-nine percent (69%) of the first group         of ATP awardees reported a significant shortening of the R&D cycle.         (13)

Silber & Associates,         in a 1996 report, found for 125 ATP participants, funded between 1990         and 1992 the following:(14) (1) 95% of participants           believed that the ATP award accelerated their progress. 
 (2) 74% anticipated           shaving off a minimum of two years from the R&D cycle. 
 (3) 81% described           speed to market as critically important or very important. 
 (4) 66% anticipated           entering the market before the window of opportunity closes (not all           respondents were involved in commercializing products or processes derived           from their ATP-funded technology, for example, a given joint venture           member might have a specialized research role only). 
1.6 Further and More Focused         Study of Impact on Applied Research Cycle Time Warranted

A review of these         findings from earlier studies found that program participants believed         that participation in the ATP program had supported them in their efforts         to accelerate their technology development, but the studies -- which addressed         a broad and diverse set of performance metrics -- did not provide details         on cycle-time reduction. They did not go into *why* program participants         felt that reducing cycle time was important; *how* participation         in ATP had helped participants to reduce their cycle time; whether reducing         applied research cycle time translated into shortening time to market;         and whether there was carry-over to other operations. Nor did these surveys         assign an economic value to reducing cycle time. The primary purpose of         this case study was to gain a clearer understanding of these and other         relevant issues pertaining to acceleration of technology development and         commercialization. CHAPTER 2: RESEARCH DESIGN

 2.1 Research Methodology         

Michael Quinn Patton's         1987 book, How to Use Qualitative Methods in Evaluation, provided         useful guidance on the research methodology.(15)

To support the research         design, the case study method was paired with user surveys. Both methods         are viewed as appropriate to evaluate R&D projects that took place         in the past, and both have their own strengths and weaknesses. For example         case studies offer the advantage of providing detailed illustrations of         the relationship between R&D and its impacts, but the disadvantage         of generally providing no way to "add up" the results statistically for         a more than anecdotal picture. User surveys, in contrast, generally provide         more statistical evidence of impact across members of a group, but often         lack the richness of detail needed to understand underlying relationships.         (16) Pairing         these methods allows the researcher to enrich the statistical treatment         with detailed information about the hows and whys of applied research         project acceleration. 

This post hoc study         assesses the impacts on cycle time of participating in the ATP, based         on a telephone survey of the twenty-eight (28) 1991 ATP awardees. The         twenty-eight awardees surveyed include 18 single company awardees and         the lead organization of each of 10 joint venture projects. Together,         the 28 projects in the sample comprise 10% of the projects the ATP has         funded to date. 

Nonparametric data         on applied research activities, and parametric data on applied research         cycle times were collected via telephone interviews. The collection of         nonparametric data was intended to provide insights into how and why participation         in ATP had reduced applied research cycle time for ATP program participants,         while collection of parametric data was intended to allow the quantification         and statistical evaluation of how much the cycle time had been reduced         - and hence, to assess the degree of impact. 

The interviews were         structured so that the same set of questions were asked in the same order,         so that themes could be easily discerned. The questions were asked in         open-ended style and the interviewer simply recorded their responses without         trying to match their comments to previously coded possible responses.         The coding was performed after the interviews were completed. 

Because interviewees         were not provided with advance copies of the interview questions, their         responses were more "top of head" and spontaneous in nature and tone.         Interviewees were encouraged to be candid and forthright with their responses.         They were assured that their individual responses would be without company         or personal attribution. 

2.2 Sampling Methodology

The study focused         on 1991 awardees because the majority, having been awarded cost-sharing         grants for a three-year period, were at or near the end of their active         participation in the ATP program. Sufficient time had transpired to allow         the impact of participating in ATP on applied research cycle time to have         materialized and become apparent to program participants. 

Table 4. Number of         ATP Proposals, Number of Awards and Monetary Value of Awards, 1990 - 1996Year             No.             of Proposals            Submitted No.             of            Awards Total             Awards             [$ million]1990             2491146**1991****271****28****93**1992             14021481993             25229601994             595883091995             7031034141996             308819Totals             **2518****288             (11%)****989**

Table 5. Taxonomy         of 1991 Interviewees 
**Technology             and Type of Lead Organization****Organization****Ten             (10) Advanced Materials** 3 Small Companies
Cree             Research, Inc.             Nanophase Technologies Corp.             Spire Corp.
 3 Medium/Large               Companies
Armstrong             World Industries             Allied Signal Aerospace             IBM Corp.
 3 Medium/Large               Companies                     Leading JVs
Ford             Motor Co.             Honeywell             Westinghouse Electric Corp.
 1 Non-Profit               Research Institute
Michigan             Molecular Institute 
**Nine (9) Electronics**
4 Small Companies
American             Superconductor Corp.             AstroPower             Iterated Systems             X-Ray Optical Systems
 2 Small Companies                                   Leading JVs
Conductus,             Inc.             Spectra Diode Laboratories
2 Non-Profit               Consortia                     Leading JVs
American             Scaled-Electronics Consortium             National Storage Industry Consortium
1 Non-Profit               Consortium
The             Microelectronics Center of North Carolina
**Five             (5) Biotech**
 5 Small Companies
Aastrom             Biosciences             Aphios             Biosym Technologies             BioSys             Engineering Animation
**Four             (4) Manufacturing**
 1 Small Company
Transitions             Research Corp
 3 Non-Profit               Consortia                    Leading JVs
Auto             Body Consortium             National Center for Manufacturing Sciences             South Carolina Research Authority

As shown in the taxonomy         in Table 5, the 1991 ATP awardees can be classified in four major technology         arenas: advanced materials, electronics, biotech and manufacturing technologies.         Interviewees included those from small for-profit companies, medium- and         large-sized for-profit companies, as well as non-profit industry consortia.         ATP defines small business in the same way as the Small Business Administration         - that is, as a business having fewer than 500 employees, but in fact,         the majority of the small companies in the sample, and in ATP as a whole,         are quite small, having fewer than 50 employees. (Details - project title,         participant's name, location, and contact information are included in         Appendix 2.) 

2.3 Research Questions on         Cycle Time

In evaluating the         impact that the ATP program had on company applied research cycle time,         the following eight questions were asked of the principal investigator         for each project: 

 (1) How important         is it for your company to reduce cycle time (time to market) ? 
 (2) Why is it important           for your company to reduce applied research cycle time ? 
 (3) How much (by           what percent change) did participation in the ATP affect your ATP project's           applied research cycle time ? 
 (4) Do you expect           the impact on cycle time in the applied research stage to flow through           to other stages in the technology development life cycle ? 
 (5) Can you give           a ballpark estimate of the economic value of reducing your cycle time           by one year ? 
 (6) How were the           cycle time improvements achieved ? (i.e., what did you do differently           as a result of participating in ATP ?) 
 (7) Did the cycle           time improvements (that were a result of participating in ATP) carryover           to other technology development projects outside of the ATP project           ? 

 2.4 Research Question on         Applied-Research Performance Measures

An eighth question         was also asked: "What applied research performance measures are tracked         and assessed across applied research projects ?" 

This open-ended question         was asked in order to discover if and how companies were measuring applied         research cycle time and/or other applied research performance measures.         
CHAPTER 3: RESEARCH RESULTS

3.1 Interpretation of Results

The research results         follow, organized as responses to the questions listed in section 2.3         preceding. The research results are descriptive in that they describe         the impact that the ATP program had on the 28 awardees3.6 from the 1991         competition, as related by the principal investigator of each project's         lead organization. 

Despite limitations,         descriptive analysis allows a researcher to analyze data and communicate         the results without attempting to make a general statement or inference         that goes beyond the particular individuals studied. In this study, descriptive         analysis illustrates the Advanced Technology Program's impact on a group         of awardees' applied research cycle time and suggests future lines of         inquiry.

**Research Results and Analyses**
3.2 The Importance of Reducing         Applied Research Cycle Time
**96%               STATED IT WAS VERY IMPORTANT FOR THEIR COMPANY TO REDUCE CYCLE TIME**

Twenty-seven (27)         or 96% of the 28 interviewees indicated that it was "very important" for         their companies to reduce cycle time, while one (4%) indicated that it         was "important." Specific terms used by the interviewees to answer the         question, "How important is it for your company to reduce cycle time ?"         - all of which were characterized by the author as denoting "very important"         - were the following: 
 (1) "Very important."           
 (2) "Extremely           important." 
 (3) "Obviously           very important." 
 (4) "One of our           most important corporate imperatives." 
 (5) "Critically           important." 
 (6) "Critical to           our future." 
 (7) "Absolutely           critical." 
 (8) "A must." 

These results are         consistent with findings from an open literature review that found that         cycle-time reduction was viewed as important because it enhances a company's         global competitiveness and helps a company to better achieve its performance         goals. These results are also consistent (though not identical) with the         findings from the recent survey conducted by Silber & Associates.         (17)

3.3 Reasons to Reduce Applied         Research Cycle Time
**THE               MOST FREQUENTLY GIVEN REASON TO REDUCE CYCLE TIME WAS TO MEET COMPETITION**

When the twenty-eight         interviewees were asked "Why is it important for your company to reduce         cycle time ?", most gave more than one reason. **As a result, there         are more than 28 responses.**

There is obvious overlap         among the types of reasons given, but in order to preserve the flavor         of the responses, these have been kept separate rather than merging them         under umbrella headings. 

**(1) Meet Competitive         Challenge [12 interviewees]**

 Reducing cycle           time and being first to market with technological innovations and new           products provides one with a competitive advantage. One interviewee           stated, "Hitting the market first - with a viable product - gives you           a chance to be more competitive and successful. It gives you a significant           leg-up." 

 Another interviewee           said that his company had "faced very strong competition from the Japanese.           They have a much shorter cycle time." He went on to say, "we may have           a similar cost structure but our cycle times are two to three times           longer. It is critical that we reduce cycle time if we are to be competitive           with them." 

**(2) Satisfy           Customers [9 interviewees]**

 Nine interviewees           all mentioned the following points relating to the impact of reducing           applied research cycle time on customer satisfaction: 

 Enhancing customer           satisfaction. 
 Reacting to voice           of customer more quickly. 
 Meeting wants and           desires of customer. 

**(3) Attract           Additional Capital [8 interviewees]**

 Many of the program           interviewees spoke of a stage in the technology development life cycle           that is not reflected in the literature - that is the "capitalization           stage." Cash infusions are critical to these innovators and inventors           who need capital to fund their high-risk technologies. 

 Eight interviewees           stated that reducing applied research cycle time helps them to acquire           "elusive and shrinking" R&D dollars. 

 One interviewee           pointed out that it is only possible to acquire venture capital and           financial support for a new technology and/or new product that have/has           well-defined risk, well-defined value, and well-defined time to market.           He stated, "for new technology development, you can only have so many           years and so many dollars available for early stage development until           a project must be self-supporting. You must clear the threshold and           you must show that you have something real. If you don't hit the threshold,           you don't get the funds." 

 A second interviewee           stated, "reduced cycle time shows that you are on the forefront of technology           development, establishes credibility with venture capitalists and ensures           future financing." 

 A third interviewee           expressed this concern, "if we don't get products out soon, venture           capitalists will lose interest and pull out. We have a critical need           to get something out in the next decade." 

**(4) Enhance           Technology Development Process [7 interviewees]**

 Seven interviewees           stated that reducing cycle time enhances the technology development           process by accelerating the development of technology to the point where           it can be utilized, leveraged and commercialized. "Results are more           likely to be realized." 

**(5) Reduce           Costs [6 interviewees]**

 Six interviewees           cited cost reductions from cycle-time reduction. 

 One interviewee           stated, "Reducing cycle time saves dollars in a big, big way. When you           get a product to market six months earlier, it generates many, many           millions of dollars in cost savings." Other interviewees said that with           a reduced cycle time:  The development             cost is lower.             The initial cost of new and improved versions is lower.             The labor cost is reduced.             The cost of capital is lower because of the time value of money. 

**(6) Survive           [5 interviewees]**

 Five interviewees           stated that cycle time and time to market affects their ability to survive.           
 One interviewee           stated, "Cycle time is an issue we wrestle with. We've learned that           the superior technology doesn't always win the game . . . Sometimes,           it's the company with the lower cycle time." 

 A second interviewee           stated, "The faster, the better. Anything that will help us do it faster,           the better off we are." 

 A third interviewee           stated, "It means 'life' or 'death' for us. 'Go' or 'no go.' Reducing           cycle time is necessary to being a viable organization. Fast, we survive;           slow, we don't. Entering the market during the window of opportunity           is a 'make' or 'break' decision - not only for us in this product class,           but for us as a company." 

**(7) Enhance           Quality [5 interviewees]**

 Five individuals           pointed out that production quality is enhanced by shorter cycle times.           Intuitively, one might expect production quality to be enhanced by longer           cycle times, not shorter cycle times. Though the responses may seem           counter-intuitive, one interviewee explained that "Quality is married           with cycle time. The two metrics are related. Especially in the hand-off           from design to manufacturing. Process is dramatically improved. It becomes           a collaborative environment. Manufacturers get to use design at the           time of design rather than post hoc. Quality of the product is enhanced           by shorter design and engineering lead time - engineers are able to           introduce changes that enhance the design or the product. In the past,           with the older, longer cycles, engineers would have to compromise."           

**(8) Increase           Return on Investment (ROI) [5 interviewees]**

 Five interviewees           stated that reducing cycle time increases return on investment. 

 Several interviewees           pointed out that reducing cycle times will help their companies more           quickly achieve long-term, industry-wide results necessary to hitting           their corporate financial benchmarks. 

**(9) Respond           to Changing Market [3 interviewees]**

 Three interviewees           stated that reducing cycle times would make it easier to survive in           their "turbulent," "highly unstructured" market places. 

**(10) Dominate           Market [3 interviewees]**

 Three interviewees           reflected one interviewee's thought that: "If you're the first one there           with a bug-free technology, you have a chance of establishing market           dominance - and can then set the bar for everyone else." 

**(11) Reduce           Risk [3 interviewees]**

 Three interviewees           thoughts were captured by one who said: "Greatly reducing cycle time           on the initial product and on subsequent products increases net present           value, and decreases uncertainty and risk." 

**(12) Grow           [2 interviewees]**

 Two interviewees           reflected the sentiment that "Cycle-time reduction allows us to develop           new business and new jobs." 

**(13) Increase           Sales Volume [2 interviewees]**

 Two interviewees           stated that when you're the first to market and one year in advance           over your competition, sales volume goes up. 

**(14) Increase           Market Share [2 interviewees]**

 Two interviewees           reflected sentiments that for any technology, it's important to reduce           applied research cycle time if you don't want to lose your chance at           increasing market share - and those who are first to market have a differential           advantage in terms of increasing market share. 

**(15) Combat           Perceived Weakness [2 interviewees]**

 Two interviewees           stated that applied research cycle-time performance was perceived as           a weakness in their respective companies. 

 The first stated,           "Reducing applied research cycle time is critical in starting a new           company. Part of the reason we merged with another company is that we           didn't do a good job on applied research cycle time." 

 The second stated,           "Though our company is recognized for having great technology performance,           we are also recognized for our poor applied research cycle time. For           the last one-and-a-half-to-two years, reducing applied research cycle           time has been greatly emphasized and is now an important metric in our           company." 

**(16) Develop           Critical Capability [2 interviewees]**

 Two interviewees           acknowledged that their companies were trying to reduce applied research           cycle time to develop a critical capability. The first stated, "Applied           research cycle-time reduction is critical for our product class. Our           company is trying to develop this capability." The second interviewee           echoed this sentiment by stating: "It is absolutely critical to do it           as quickly as possible; to try something new; and to operate in a short           technology development and product development cycle time." 

**(17) Take           Advantage of Window of Opportunity [2 interviewees]**

 Two interviewees           respectively made the following statements: "Commercial windows are           shortening," and "If you take too long, you're obsolete before you come           out with the product." 

**(18) Collaborate           [2 interviewees]**

 Two interviewees           linked reducing applied research cycle times to enhancing collaboration.           Because applied research cycle-time reduction requires a systemic and           integrated approach, people from different sub-systems must work with           each other and end users. 

**(19) Choice           of Competitive Strategy [2 interviewees]**

 Two interviewees           mentioned that reducing applied research cycle time opens up strategic           choices. The first observed, "When companies are not first to market,           they beat their brains out with price cuts in order to establish market           share." The second observed, "Reducing cycle time enables companies           to choose a higher performance strategy, instead of a price-cutting           strategy. Higher performing companies tend not to engage in price wars."           

**(20) Maintain           Technical Leadership [1 interviewee]**

 One interviewee           felt that being first to market with a new technology was critical to           maintaining technical leadership. "Our company had a competitive advantage           - we were the technical leader and were perceived as such. We were known           for our science. But, we also needed to be first to market in order           to maintain position as technical leader." 

**(21) Establish           Intellectual Property Rights [1 interviewee]**

 One interviewee           pointed out that, "Technology has a finite protected life. The more           rapidly you can get into the marketplace, the longer you can operate           with U.S. patent protection and other forms of intellectual property           rights protection." 

**(22) Follow           Industry Practice [1 interviewee]**

 One interviewee           stated, "In our industry, applied research cycle times have been dropping;           cycle times are now measured in months instead of years." 

 The finding that         the most frequently given reason to reduce applied research cycle time         was to meet competition is consistent with the paramount finding from         the literature review that cycle-time reduction is important because it         enhances global competitiveness. 

3.4 Impact of ATP Participation         on Applied Research Cycle Time
**PARTICIPATION IN THE ATP REDUCED THE PROJECTS' APPLIED               RESEARCH CYCLE TIME BY 50% OR THREE YEARS**

After being         asked, **"How much (by what percent change) did participation in         the ATP affect your ATP project's applied research cycle time ?"**,         the median response of the twenty-eight interviewees was that participation         in ATP reduced their ATP project cycle time by 50% or three years (see         Table 6). The median response was that without ATP, the same project would         have taken six years. In terms of years saved, the range was from 1 to         2 years at the low end, to 10 years and more (infinity) at the high end.         In terms of percentage cut in cycle time, the range was 25% to 80%. 

Five of the 28 interviewees         challenged the question. They said that a more relevant question was,         **"Would you have done the applied research at all without ATP support         ?"**. They attributed the entire existence of the project to participation         in ATP; with an "infinite" impact on the applied research cycle time.         As one interviewee stated: 

"The business environment         is now more short sighted. It is more hesitant to put capital into technology         development projects; and to apply technology once it is developed. Given         the past three-to-four years, it's fair to say we wouldn't have started         or pursued this research without ATP funding." 

These five interviewees,which         included companies of all sizes, said they could not bound the reduction         in applied research cycle time because they would not have ever begun         without the ATP award. 

**Table 6. Impact         of ATP Participation on Applied Research Cycle Time**
             Rank Order by % Reduction              % Reduction              Number of Years Shorter (1)             small*unbounded(2)             small*unbounded(3)             medium/large*unbounded(4)             medium/large*unbounded(5)             medium/large*unbounded(6)             small75             to 80%10             years shorter(7)             medium/large66-75%             6             to 9 years shorter(8)             small50-66%             5             to 6 years shorter(9)             small50-60%             at             least 5 years shorter(10)             small50%             5             to 6 years shorter(11)             small50             % 5             years shorter(12)             small50%5             years shorter(13)             medium/large 50%             5             years shorter**(14)             small****50%             MEDIAN****3             years shorter**(15)             small50%             3             years shorter(16)             medium/large50%             3             years shorter(17)             medium/large 50%             3             years shorter(18)             medium/large 50%             3             years shorter(19)             medium/large 50%             3             years shorter(20)             medium/large50%             3             years shorter(21)             medium/large50%             2             years shorter(22)             medium/large 50%             2             years shorter(23)             small50%             1.5             years shorter(24)             small50%             1.25             years shorter.(25)             small33-50%1             - 2 years shorter. (26)             small33%             2             years shorter(27)             medium/large30%             3-5             years shorter.(28)             medium/large 25%             2             years shorter.* These companies       were unwilling to bound their estimated cycle-time reductions because they       said they wouldn't have ever started without the ATP award. 

 It is interesting         that many of the interviewees' responses to this question are so similar,         particularly since the interviewees come from different kinds of organizations,         different sizes of organizations, and different types of industries. 

 3.5 Flow-Through of ATP's         Impact on Applied Research Cycle Time to Later Stages in the Technology         Development Life Cycle 86%               OF THE INTERVIEWEES EXPECT THE IMPACT ON CYCLE TIME IN THE APPLIED               RESEARCH STAGE TO FLOW THROUGH TO OTHER STAGES IN THE TECHNOLOGY               DEVELOPMENT LIFE CYCLE

When asked, **"Do         you expect the impact on cycle time in the applied research stage to flow         through to other stages in the technology development life cycle ?"**,         twenty-four (24) or 86% of the 28 interviewees said yes. One interviewee         suggested that this is better described as a cascade effect, not a flow,         since it is not linear. He described it as a driving force that has a         broad effect. This implies that, at least in some cases, speeding up the         R&D may have a disproportional impact on the later stages in technology         development. 

Of the four interviewees         who did not give a clear yes to the question, one said **"probably         yes"**, but that he wasn't sure, and three said that it was not         applicable. Two of those provided the following reasons for the lack of         applicability of the question: 

"We only do technology         development. We get a commercial partner interested and transfer technology         to them. The intellectual property revenue stream seeds and funds internal         R&D." 

"The applied research           only advanced to the demonstration stage. That's when market analysis           revealed that new competitive challenges in the marketplace had rendered           our applied research obsolete. As a result, we did not advance the applied           research beyond the demonstration stage." 

A careful review of         the literature did not yield many articles that touched specifically on         the flow-through of applied research cycle-time savings to later stages         in the technology development cycle. Most of the R&D evaluations conducted         by industry, government, academia, and science over the past four decades         have focused on the short-term; few studies have directly linked research         inputs to research outputs and research outcomes. 

If there were neither         flow-through nor linkage between research cycle-times and the long-term         commercial outcomes, then there would be no market-place benefit associated         with reducing applied research cycle time. If, on the other hand, there         is a flow-through or linkage, then the potential for accelerated long-run         economic benefits - that as a consequence may be larger both in present         value terms and in nominal terms (due to competitive advantages) - exists         from shortening the research phase. Common sense would cause us to expect         a flow-through of time savings from the earlier stages to the later stages,         and, indeed, this study found that most of the interviewees expected the         impact on research time to flow through to later stages in the technology         development and commercialization cycle. Though this study makes the anticipated         linkage, the impact of earlier-stage time savings to the timing and size         of longer-term outcomes needs to be more fully explicated. 

 3.6 Economic Value of Reducing         Applied Research Cycle Time By One Year 79% OF THE INTERVIEWEES GAVE "BALLPARK ESTIMATES"               OF THE ECONOMIC VALUE OF REDUCING APPLIED RESEARCH CYCLE TIME BY               ONE YEAR

When asked, **"Can         you give a 'ballpark estimate' of the economic value of reducing applied         research cycle time by one year ?"**, twenty-two (22) or 79% of         the 28 interviewees gave either a quantitative or qualitative "ballpark         estimate." Fifteen (15) or 54% of the 28 interviewees gave a quantitative         estimate. Seven (7) or 25% of the 28 interviewees gave a qualitative         estimate. Some of the interviewees who provided quantitative responses         also provided qualitative responses. (18)

Even though interviewees         estimated that participating in the Advanced Technology Program helped         them to reduce their applied research cycle time anywhere from one to         10 years, all were asked to give the economic benefit associated with         just a one-year reduction. The estimates are therefore much more         conservative than if they had been asked to estimate the total value of         the realized cycle-time reduction. The estimates range from one million         dollars to "billions" for a one-year reduction in applied research cycle         time and appear to relate specifically to the direct economic values to         the company or JV member companies rather than to the potential broader         benefits that might be realized. The estimates in Table 7 are listed in         order of the size of the value, with the largest estimated value listed         first. The median estimate of the economic value of reducing the applied         research cycle time by just one year is $5 million to $6 million. 

**Table 7. Ballpark         Estimates of Economic Value of a One-Year Reduction in Applied Research         Cycle Time, In Order of Decreasing Value**
             Type of Organization              Economic Value of Getting to Market One Year Sooner              Nature of the Economic Value (1)             medium/large$100's             of millions to billionssales             revenue(2)             medium/large $1             Billion sales             revenue (3)             medium/large$100             to 200 millionsales             revenue(4)             small$15             to 250 million to ultimately ½ billion sales             revenue(5)             small$10             to 100 millionsales             revenue(6)             small$10             to 30 millionsales             revenue(7)             medium/large $15             millionsales             revenue**(8)             small****$5 mil. to           $6 mil**
**MEDIAN             VALUE****sales             revenue**(9)             small$5.2             million capital             cost savings(10)             medium/large $2             to 5 millionsales             revenue(11)             smallMillions             of dollarssales             revenue(12)             smallMillions             of dollarssales             revenue(13)             smallMillions             of dollars sales             revenue(14)             medium/large $2             millionsales             revenue(15)             small$1             to 2.25 millionsales             revenue and cost savings

The qualitative estimates         of cutting research by one year - like the quantitative estimates - centered         on sales revenue and cost savings. Five interviewees discussed financial         impacts associated with tripling the revenue stream, increasing sales,         reducing penalty costs, reducing environmental costs, reducing legal costs,         reducing cost of capital, and becoming profitable at an earlier time.         Eight interviewees provided qualitative perspectives on market impacts         in terms of commercializing applications, having a broader impact on the         whole industry, keeping the product current, accelerating results, and         making it possible to increase market share and establish market dominance.         

One interviewee -         speaking for a joint venture project - in discussing the value of reduced         cycle time stated: 

"Three companies have         commercialized applications of the technology developed in ATP. But there         has been a broader impact on the whole industry, which has advanced to         a higher level because of ATP. This would not have been taken on without         ATP's funding and risk-sharing." 

Another interviewee         - speaking for a single-company project - stated: 

"With the ATP project,         we will create a new market and be associated with it. This will lead         to an increase in our market share. If we came in a year later, competing         for market share would be more difficult. With the market share, we anticipate         generating a positive cash flow one year after product introduction."         

Six (6) or 21% of         the 28 interviewees who said, "No, I can not give a ballpark estimate,"         gave the following reasons: 

 (1) "It's very difficult,         if not impossible for me to do that." 

 (2) "It's hard           to quantify or qualify the gains." 

 (3) "As a component           manufacturer, we are an enabler and supporter of systems manufacturers.           As a result, we are not sure what level of economic value is generated."           

 (4) "That's very           hard. We sell 2-3 million systems not 1-2,000 products. We provide customers           with capabilities and solutions to problems. Though I cannot quantify           the benefits, I can say that they are significant and global." 

 (5) "It's nebulous.           The technologies we developed are not yet faced with competitive issues.           We offer new solutions and are looking for problems to solve." 

 (6) "The configuration           of the industry changed and the basis of competition moved from one           technology platform to another. As a result, the economic opportunity           disappeared and we did not commercialize the technology we were developing           in the Advanced Technology Program." 

It is perhaps surprising         that so many of the interviewees (a majority of 22 or 79% of the 28 interviewees)         were willing to give a "ballpark estimate" of the economic value associated         with reducing their applied research cycle time by one year. It was expected         that the majority of the respondents would not be able/willing to give         a "ballpark estimate". 

Not only did most         of the interviewees appear very comfortable providing estimates, it is         the researcher's opinion that if the remaining interviewees had been encouraged         to provide an estimate most would have done so. But since the intent was         to capture their initial responses to the question, freely-given, the         interviewees were not pushed to generate estimates. 

 3.7 ATP Effects that Helped         Reduce Applied Research Cycle Time 
THE               MOST-FREQUENTLY MENTIONED ATP EFFECT THAT HELPED REDUCE CYCLE TIME               WAS ATP'S REQUIRED PROJECT PLANNING AND MANAGEMENT. 

When the twenty-eight         (28) interviewees were asked: **"How were the cycle-time improvements         achieved, in other words, what did you do differently as a result of participating         in ATP?"**, they gave 58 answers. When the answers are grouped         by the type of ATP practice that helped interviewees reduce cycle time,         we find that there are five major ATP categories. Table 8 shows aggregate         total frequencies and percentages. 

**Table 8.         ATP Effects that Helped Interviewees to Reduce Cycle Time**
             ATP Effects that Helped Interviewees to Reduce Cycle time              Frequency of mention Percentage            (%) ATP's             Required Project Planning and Management1525.86%Achievement             of Critical Mass of Resources With ATP Funding 1220.69%Attraction             of Additional Financial Support through ATP "Halo Effect"1220.69%Greater             Project Stability Through Focus on Technical Problem1220.69%ATP's             Emphasis on Collaboration7             12.07%**TOTAL****58****100.00%**

**(1) ATP's Required         Project Planning and Management [15 interviewees]**

 For this sample           of 28 interviewees representing 28 projects from the 1991 competition,           the detailed project planning that ATP required and the project management           it provided to ensure that companies followed the project plan were           most frequently cited as the ATP effect that helped interviewees to           reduce cycle time. It was felt, in general, that the well laid-out plan           required by ATP from the companies lent stability to the applied research           program. According to one interviewee:           "Of greatest value, with ATP we had a well-laid-out plan, and followed           the plan without interruption. Without the ATP plan, the vagaries of           our business would have caused us to vary the plan. If we were not having           a good year financially, we would probably have pulled the plug - but           ATP's involvement lent stability to the research program." 

ATP's requirement           for an integrated business and R&D plan with its emphasis on concurrent           engineering seemed to speed things up. This was described by one interviewee:           
"For us, product             testing typically comes later in the product development cycle. ATP             wanted us to work early-on with potential customers and users. One             of our primary interests from the very beginning was to work with             potential users. We wanted to work in a 'true manufacturing envelope'             with 'true manufacturing equipment.' ATP made this happen." 

 Other project planning           and management techniques that were said to be important to cycle-time           reduction and attributed to the ATP involvement included: using a systematic           approach; developing definable time lines and value; bench marking and           selecting technology applications; integrating the voice of the customer;           assuring quality; and enhancing documentation procedures. 

**(2) Achievement           of a Critical Mass of Resources Through ATP Funding [12 interviewees]**

 ATP funding was           important to applied research cycle-time reduction because it enabled           interviewees to gain the critical mass of resources necessary to conduct           the applied research. This was mentioned by interviewees from small           for-profit companies, medium-to-large for-profit companies, and non-profit           consortia. For most interviewees, this meant increasing staffing, but           a couple mentioned material resources, and one said that the company           was able to more fully dedicate existing staff to the applied research           project. 

 An interviewee           from a large company stated: "Major companies are structured into decentralized           divisions - each little fiefdom (division) has a budget. Our division           didn't have adequate funding to pursue this project." 

An interviewee from           a small company stated, "ATP enabled us to acquire a critical mass of           resources. The corporation has limited resources. We looked around.           We wouldn't have had enough resources to reach critical mass to develop           and leverage resources. As a small company, we wouldn't have been able           to do the R&D, period." 

**(3) Attraction           of Additional Financial Support through ATP "Halo Effect" [12 interviewees]**

 Some interviewees           mentioned their improved ability to stimulate interest and acquire capital           to continue pursuing their advanced technologies as an important ATP           effect on cycle time. The additional infusion of capital was said by           some to have been critical. Many of the scientists and engineers appear           to have taken on an entrepreneurial role to support and speed the development           of their technology. 

 A few of the comments           about the effect of attracting additional capital on reducing applied           research cycle time follow: 
"There is a question             in our minds as to where we would have gotten funding, and a question             as to whether we would have survived. We didn't round up private funding             until after we got ATP funding. It's a high-risk project, and we had             previously operated by bootstrapping, which would not have lasted             long." 

"If we had not             had ATP funds, we would have had to attract capital. That would have             been difficult without ATP. With ATP, we got press and were able to             generate excitement. It easily halved the development time." 

"We were able             to leverage ATP participation to quickly acquire additional funding.             Selection as an ATP project resulted in the perception that we were             a viable organization. We were able to get new venture capital for             other technology projects, programs, and platforms." 

**(4) Greater           Project Stability Through Focus on Technical Issue [10 interviewees]**

 Six interviewees           commented that participating in the ATP enabled them to shorten the           applied research cycle time by allowing them to concentrate on solving           the technical problem. 
 One large company             interviewee said that: 

"The ATP contract             allowed an R&D group to do enough work on a technology to go to             other people in a large company with more than a prototype. The company             was reducing the R&D budget and manpower; and the ATP support             was critical because it allowed us to focus on the technology problem,             rather than the organizational problem." 

 Another large             company interviewee said that: 

"The technology             development wouldn't have happened without ATP. The ATP was really             a catalyst. The match was not only important from a financial standpoint             but from a strategic standpoint as well. What is most difficult for             us in a large bureaucracy is to get the snowball started and that's             what ATP helped us to do." 

 A small company             interviewee said that: 

"The ATP gave             us the ability to make certain mistakes in research - participating             allowed us to proceed in an orderly, "unpanicked" way." 

**(5) Enhanced           Collaboration [6 interviewees]**

 Six interviewees           said ATP enhancement of collaboration helped reduce the applied research           cycle time. Four interviewees explained that: 

"ATP brought strange           bedfellows together - who were competitors. ATP also brought members           of the supply chain together to define what they needed in the supply           chain. As a result, it helped us and other members of the supply chain           to move further, faster." 

"Perhaps most             importantly, ATP brought competing companies together. This would             not have otherwise happened. They never would have worked with each             other without ATP. Within mega-corporations, there is oftentimes much             personal competition between the senior executives. The benefit of             a joint venture is that senior executives from different companies             are sometimes more willing to share ideas with executives from other             companies than with peers in their own organization. This enhances             the innovative thinking and collaborative processes. The members of             the ATP joint venture were so compatible with each other that they             moved smoothly like a dance team - it got to the point that in meetings             that they were building on each other's words and finishing each other's             sentences." 

"Beyond enhancing             collaboration between competitors, ATP enhanced collaboration between             technology developers and technology purchasers. We developed the             technology in a joint venture partnership. Two of our partners didn't             want to buy R&D directly from us, but they wanted to invest in             our technology development under the auspices of ATP." 

"Through ATP,             we structured a mutual win-win with a joint venture partner that had             a complementary set of technology skills that enabled both of us to             develop the technology more quickly." 

Robert Cooper, in         a 1994 article, stated that the four factors that had the greatest impact         on time efficiency in new product development (in order of priority) were         (1) a strong organization and resource management, (2) a strong market         orientation, customer focus and execution of marketing plans, (3) strong         planning and decision making, and (4) strong technology and project management.         These are listed in Table 9 with an indication of the relative importance         of their impacts on time efficiency and time schedule. Of course, the         comparison with the ATP interviewee responses is not perfect: the focus         of Cooper's research was on product development, rather than the earlier         applied research stage which is the focus of this study. Furthermore,         he was concerned about factors the organization could bring to bear on         cycle time, whereas the focus of the responses in this study was on factors         the ATP brought to the equation. But the themes of planning, bringing         in business functions early, and strong management seem common to both.(19)

**Table 9. Drivers         of Cycle-Time Reduction in Product Development (Not Based on ATP Awardee)**
             Driver of Cycle-Time Reduction (rank ordered)              Impact on Time Efficiency              Impact on Adherence to the Time Schedule ***Strong             Organization***. A cross-functional team; accountable,             empowered, dedicated, clear leader, with top management support. Resource             Management. Strong             impact (0.316)Very             strong impact (0.527)***Strong             Marketing Function***. A strong market orientation and             customer focus; and quality of execution of marketing actions. Strong             impact (0.308)Very             strong impact (0.411)***Strong             Planning & Decision-Making***. Undertaking the homework             up-front - before development begins. Modest             impact (0.226)Strong             impact (0.478)***Strong             Technology Management.*** Quality of execution of technological             activities (technical assessment; development; lab tests; trial production;             production start-up, project management). Modest             impact (0.223)Strong             impact (0.331)***Source: Robert G. Cooper, "New Products: The Factors That Drive       Success," International Marketing Review, Vol. 11, No. 1, 1994, pp.       60-76.***

The finding that ATP's         required project planning and project management support was mentioned         more frequently than its funding support for this particular sample of         program participants, though not statistically significant, is suggestive.         It highlights one of the unusual characteristics of this governmental         program: it does more than disburse funds, it requires both R&D and         business advance project planning, and it monitors project progress against         both technical and business goals, over the multi-year project life. ATP         views the companies as partners in the technology development process         and takes an active role in overseeing how the projects are carried out.         As a result, the relationship between the "partners" (ATP and the awardee         organizations) entails a detailed technical and business review by ATP         of the proposed project, with feedback to the proposer; rigorous competition         among proposers against published selection criteria; and a kick-off meeting         where technical milestones and business goals are further reviewed and         detailed. The kick-off meeting is followed by quarterly reports and annual         reviews until the project is completed. Failure to perform can lead to         project termination. Then, there is post-project tracking of further developments.         This active participative role appears to be the reason why ATP was viewed         by the interviewees as strengthening their planning for technology development,         enhancing strategic focus, and providing stability to see the job through.         

It is interesting         to note that there are two types of acceleration implied in the responses.         One relates to overcoming delays in starting technology development projects.         The other relates to speeding up performance of the research once it is         underway. The ATP funding seems to have played a critical role in overcoming         the inability to get the projects off the ground. The advance technical         and business planning requirements, project oversight to hold it on course,         and research efficiency gains from collaboration - in addition to the         funding - seem to have been critical factors in speeding up the research         once the projects began. 

3.8 Carryover of Cycle-Time         Improvements Resulting from ATP Participation to Non-ATP Applied Research         Projects 
86%               STATED THAT CYCLE TIME IMPROVEMENTS THAT RESULTED FROM PARTICIPATING               IN ATP CARRIED OVER TO OTHER TECHNOLOGY DEVELOPMENT PROJECTS                

When the twenty-eight         interviewees were asked: **"Did the cycle-time improvements that         were a result of participating in the ATP carryover to other technology         development projects ?"**, twenty-four (24) or 86% of the 28 interviewees         said yes; three (3) or 11% said no; and one (1) or 3% said they didn't         know if cycle-time improvements "carried over." Most of the interviewees         explained ways in which the ATP-fostered cycle-time reductions were transferred         to other technology development projects. These responses are listed in         Table 10, grouped by nature of response and placed in order of total frequency         of mention. 

**Table 10.         Carryover of Cycle-Time Improvements to Other Projects**
               Type                Carryover of Cycle-Time Improvements                Frequency                Percentage              % (1)Yes               - Enabling, Generic, Precompetitive Technology932.14%(2)Yes               - Adopting "ATP Practice" to Related Projects621.44%(3)Yes-               Extended Adoption of New Methodologies and Processes414.29%(4)Yes-               Cultural Change27.14%(5)Yes               - A Little310.71%(6)No310.71%(7)Don't               Know13.57%**TOTAL**28               100.00%               

**(1) "Yes, Enabling,         Generic, Precompetitive Technology" [9 interviewees]**(20)

Nine interviewees,           or 32% of the sample, said that cycle-time improvements that were a           result of participating in ATP were carried over to other technology           development projects because the ATP resulted in an enabling technology           that had broader applications that allowed them to speed up other projects.           Two different interviewees explained this well: 

"Through ATP, we           built a better understanding of technology. We can spin-out products           at a faster rate because we have a fundamental understanding of core           technology with multiple applications." 

"It turned out             that the technology we were developing through ATP had utility for             other applications. It has become a technology platform for other             applications." 

**(2) Adopting           "ATP Practice" to Related Projects [6 interviewees]**

Six (6) interviewees,           or 21% of the sample, said that cycle-time improvements resulting from           their ATP participation were carried over to related projects within           the company, with other government organizations, and with industry           because they are applying ATP-required planning and project management           practices to other projects. One interviewee explained: 

"Our organization           has changed. We were once a small independent research entity - but           things have changed significantly. We've received a large contract from           industry. These independent development efforts with ATP and with industry           are similar in nature. The ATP cultural imperative and requirement that           experiments be written up in a rigorous fashion did have an impact -           we are giving the write-ups to an archivist; and we are using similar           methods on our industrial contract. We do not have a separate quality           assurance department, but following the ATP practice, we have embedded           quality into our day-to-day work habits." 

**(3) "Yes,           Extended Adoption of New Methodologies and Processes" [4 interviewees]**

Four (4) interviewees,           or 14% of the sample, reported that cycle-time improvements were enabled           by the adoption of new methodologies and processes resulting from their           ATP participation. Two interviewees explained . . . 

"Peer pressure being           what it is, once two groups come up with a way to reduce cycle time           - others immediately find out what is going on and apply the new methodology           to their own group." 

"We used ATP             funds to do process development. It was a building block for other             developments. We now have a reliable base to build on and benefit             from the cumulative effects." 

**(4) "Yes,           Cultural Change" [2 interviewees]**

Two (2) interviewees,           or 7% of the sample, believed that the cycle-time improvements that           were a result of participating in ATP resulted in a cultural change           that carried over to other projects. 

 Quoting one: 
"By doing this             on a faster speed in the ATP project, we built a culture that expects             to do things faster. By focusing on long-term needs, it allowed us             to better manage short-term needs." 

**(5) "Yes,           A Little." [3 interviewees]**

 Three (3) interviewees,           or 11% of the sample said, "Yes, the cycle-time improvements were carried           over 'a little,' in other words, to a minimum extent," but didn't elaborate           as to how. 

**(6) "No,           No Change." [3 interviewees]**

For the three (3)           interviewees, or 11% of the sample, who said that cycle-time improvements           were not carried over, they attributed the lack of carryover to internal           organizational dynamics. For example, one interviewee stated: 

"No - but our company           is going through a series of changes - we're restructuring every operation           into decentralized units. The company is breaking up into smaller units.           This reduces opportunity for cross-fertilization. In another organization           with a more centralized research system, more benefits could have been           derived. This is frustrating because the potential existed but could           not be realized." 

**(7) "Don't           know." [1 interviewee]**

One interviewee,           or 3.6% of the sample, said they didn't know if there was a carryover           to other projects. 

When ATP practices           result in cycle-time improvements that are carried over, we can say           that to some extent, these practices have been institutionalized. In           fact, this study found that the representatives for the lead-company           awardees thought that ATP-originated practices were, for the most part,           carried over, (hence institutionalized). Interviewees attributed the           carry-over to the enabling technologies that positioned the companies           to execute a number of subsequent spin-off activities faster; to the           use on related projects of ATP-acquired practices that foster project           acceleration; to the wider application of new methodologies and new           processes that reduce cycle time; and to cultural changes associated           with adopting a faster pace as a way of life. They discussed how participating           in the ATP program enabled them to remove organizational barriers to           cycle-time reduction. But beyond brief characterizations, they did not           provide much detail. To the extent that other work in the ATP-funded           companies was accelerated by the companies' participation in the ATP,           the benefits of ATP on the participating companies may be systemic in           nature and greater than anticipated. LIST OF REFERENCES 

Burkhart, Robert E.         "Reducing R&D Cycle Time." Research Technology Management,         May-June 1994, pp. 27-32. 

 Cooper, Robert G.         "New Products: The Factors That Drive Success." International Marketing         Review, Vol. 11, Nov. 1, 1994, pp. 60-76. 

 Groves, Ray. "Leadership         in Tomorrow's Global Marketplace." Vital Speeches of the Day, Dec.         15, 1991, pp. 144-146. 

 Inman, R. Anthony.         "Time-Based Competition: Challenges for Industrial Purchasing." Industrial         Management, Mar. 1992, p. 31. 

 Jordan, Daniel W.         "Reducing Project Durations Through Cycle Time Reductions." American         Association of Cost Engineers Transactions (AACE), 1993, pp. F.3.1-F.3.4.         
Machan, Dyan. "We're         Not Authoritarian Goons." Forbes, Oct. 24, 1994, pp. 246, 248.         

 McGrath, Michael         E., Michael T. Anthony and Amram R. Shapiro. Product Development: Success         Through Product and Cycle-Time Excellence. Boston: Butterworth-Heinemann,         1992. 

 Mulcaster, David         (Chairman). Methods for Assessing the Socioeconomic Impacts of Government         S&T. Ottawa, Ontario, Canada: Working Group on S&T Financial         Management and Mechanisms, May 1993, pp. ii-iii. 

"NIST At a Glance."         Guide to NIST, October, 1993, p.2. 

 Patterson, Marvin         L. "Accelerating Innovation: A Dip into the Meme Pool." National Productivity         Review, Autumn, 1990, pp. 409-418. 

 Patton, Michael Quinn.         How to Use Qualitative Methods in Evaluation. Thousand Oaks, Calif.:         Sage, 1987. 

 Ruegg, Rosalie T.         Guidelines for Economic Evaluation of the Advanced         Technology Program. Gaithersburg, Md.:U.S. Department of Commerce,         Technology Administration, National Institute of Standards and Technology         Report, NISTIR-5896, November 1996. 

 Silber & Associates.         Survey of Advanced Technology Program, 1990-1992         Awardees, Company Opinion About the ATP and Its Early Effects.         January 30, 1996. 

 Solomon Associates.         The Advanced Technology Program An Assessment of Short-term Impacts:         First Competition Participants. February 1993. 

 Stalk, George. "Time-The         Next Source of Competitive Advantage." Harvard Business Review,         July-Aug. 1988, pp.41-51 End Notes
1.          Rosalie T. Ruegg, Guidelines for Evaluation of         the Advanced Technology Program, (Gaithersburg, Md.: U.S. Department         of Commerce, Technology Administration, National Institute of Standards         and Technology Report, NISTIR-5896, November 1996). 
         2. The ATP         does not fund product development; it funds research for the purpose of         creating technology that will eventually lead to new and improved products         and services. And though the ATP does not fund streams of basic research,         it often funds projects that entail research tasks which are rather basic         in nature. Although it is an oversimplification to state categorically         that ATP funds applied research, that is predominantly what it funds;         and hence, for the purpose of this study, that is the term used. 

Although the ATP has         technology commercialization as a desired downstream outcome of the research         it funds - and some awardees are beginning to commercialize their technologies,         it is too early to assess directly the ultimate impact of the program         on commercialization. Hence, the study's focus on time changes in the         applied research cycle of ATP-funded projects is appropriate. 
         3. Daniel         W. Jordan, "Reducing Project Durations Through Cycle Time Reduction,"         American Association of Cost Engineers Transactions (AACE), 1993,         pp. F.3.1-F.3.4. 
****         4. Ibid.         
         5. Marvin         L. Patterson, "Accelerating Innovation: A Dip into the Meme Pool," National         Productivity Review, Autumn, 1990, pp. 409-418. 
         6. Forbes, Dec. 9, 1991 cited by Dyan Machan, "We're Not Authoritarian         Goons," Forbes, Oct. 24, 1994, pp. 246, 248. 
         7. Ray Groves, "Leadership in Tomorrow's Global Marketplace," Vital         Speeches of the Day, Dec. 15, 1991, p. 144-146. 
         8. George Stalk, "Time-The Next Source of Competitive Advantage," Harvard         Business Review, July-Aug. 1988, pp.41-51. 
         9. Anthony R. Inman, "Time-Based Competition: Challenges for Industrial         Purchasing," Industrial Management, Mar. 1992, p. 31. 
         10. Michael E. McGrath, Michael T. Anthony, and Amram R. Shapiro, Product         Development: Success Through Product and Cycle-Time Excellence, (Boston:         Butterworth-Heinemann, 1992), p. 3. 
         11. Ibid., pp. 6-7 and 11. 
         12. Robert E. Burkhart, "Reducing R&D Cycle Time," Research Technology         Management, May-June 1994, pp. 27-32. 
         13. Solomon Associates, The Advanced Technology Program, An Assessment         of Short-Term Impacts: First Competition Participants, February 1993.         
         14. Silber & Associates, Survey         of Advanced Technology Program, 1990-1992 Awardees, Company Opinion About         the ATP and Its Early Effects, January 30, 1996. 
         15. Michael Quinn Patton, How to Use Qualitative Methods in Evaluation,         (Thousand Oaks, Calif.: Sage, 1987). 
         16. David Mulcaster, Chairman, Methods for Assessing the Socioeconomic         Impacts of Government S&T (Ottawa, Ontario, Canada: Working Group         on S&T Financial Management and Mechanisms, May 1993), pp. ii-iii.         
         17. Silber & Associates, Survey of Advanced Technology Program,         1990-1992 Awardees, Company Opinion About the ATP and Its Early Effects,         January 30, 1996. 
         18. This is why there are more than 28 responses in total. 
         19. Robert G. Cooper, "New Products: The Factors That Drive Success,"         International Marketing Review, Vol. 11, No. 1, 1994, p. 60-76.         
         20. Note that this effect could also be viewed in the context of intra-firm         spillovers and economies of scope. 
 APPENDIX 1: SELECTED BIBLIOGRAPHY           ON CYCLE TIME 

Andel, Tom.         "Expandng Demand for Cycle Time Compression." Transportation &         Distribution, Oct. 1994, pp. 95-102. 

 Anderson, Duncan         Maxwell. "Time Warrior: He Shows How to Destroy Your Competition by Doing         Everything Faster." Success, Dec. 1991, pp. 38-41. 

 Armitage, Howard,         and Grant Russell. "Activity-based Management Information, TQM's Missing         Link." CMA Magazine, Mar. 1993, p. 7. 

 Atwater, J. Brian,         and Satya S. Chakravorty. "Does Protective Capacity Assist Managers in         Competing Along Time-Based Dimensions ?" Production & Inventory         Management Journal, Third Quarter, 1994, pp. 53-59. 

 Barlog Ray and Dana         Ginn. "Reduce Project Cycle Time." Chemical Engineering, July 1994,         pp. 133-135. 

Barrelle, Ann M. "An         Approach to Assist R&D Managers with Establishing Key Performance         Indicators." 1993 Management for Quality in Research and Development         Symposium, 
 Wilton, Conn.: Juran         Institute, 1993. 

 Bean, Thomas J. and         Jacques G. Gros. "R&D Benchmarking at AT&T." Research Technology         Management, July-Aug. 1992, pp. 32-37. 

 Beckert, Beverly         A., Vernie Knill, Perry Pascarella, George Weimer. "Integrated Manufacturing         VII." Automation, Apr. 1989, Special Supplement, pp. IM2 - IM20.         

 Bello, Mark and Michael         A. Baum. Foreword, Setting Priorities and Measuring Results at the         National Institute of Standards and Technology. Gaithersburg, Md:         NIST Report, January 31, 1994, p. 10. 

 Benson, Tracy E.         "IQSSM Defines the Gold." Industry Week, Jan. 20, 1992, p. 28.         

 Bergstrom, Robin         Yale. "Listening to the Shape of Change at Whirlpool Corporation." Production,         Sep., 1994, pp. 60-61. 

 Beruvides, M. G.         and D. J. Sumanth. "Knowledge Work: A Conceptual Analysis and Framework."         Productivity Management Frontiers - I, Amsterdam: Elsevier Science         Publishers B.V., 1987, p. 128. 

"Bringing Equality         Back to Quality at Corning." Management Decision, Vol. 32, No.         5, 1994, pp. 61-62. 

 Brinkerhoff, Robert         O. and Dennis E. Bressler. Productivity Measurement: A Guide for Managers         and Evaluators. Newbury Park, Calif.: Sage Publications, 1990, p.         39. 

 Brown, Edward A.         "Defining the Technology Generation Process." 1993 Management for Quality         in Research and Development Symposium, Wilton, Conn.: Juran Institute,         1993. 

 Brown, Mark G. and         Raynold A. Sevenson. "Measuring R&D Productivity." R&D Productivity:         Selected Papers from Research Management during the Period 1986-1990,         Washington, D.C.: Industrial Research Institute, 1990. 

 Chaudhry, Anil. "From         Art to Part." Computerworld, Nov 9, 1992. 

 Chen, Hong, J., Michael         Harrison, Avi Mandelbaum, Ann Van Ackere, and Lawrence M. Wein. "Empirical         Evaluation of a Queuing Network Model for Semiconductor Wafer Fabrication."         Operations Research, Mar. 1988. 

 Collier, Donald W.         "Measuring the Performance of R&D Departments, March 1977." R&D         Productivity: Selected Papers from Research Management during the Period         1977-1985, Washington, D.C.: Industrial Research Institute, 1985.         

"Copeland." Industry         Week, Oct. 17, 1994, pp. 37-38. 

 Cushnie, James A.         "Measuring Knowledge Work - 'How To'." Productivity Management Frontiers         - I, Amsterdam: Elsevier Science Publishers B.V., 1987, p. 147. 

"Cycle Time Reduction:         A Minute Saved Is A Minute Earned." Industrial Engineering, Mar.         1994, pp. 19-20. 

Denton, Keith D. "The         Service Imperative." Personnel Journal, Mar. 1990, pp. 66-74. 

Doyle, Michael F.         "Fundamentals of Strategic Supply Management." Purchasing World,         Dec. 1990, pp. 40-41. 

Dumbleton, John H.         Management of High-Technology Research and Development. Amsterdam:         Elsevier Science Publishers B.V., 1986, p. 19. 

Dutton, Barbara. "Best         In Class." Manufacturing Systems, Aug. 1988, pp. 14-19. 

Eisenhardt, Kathleen         M. "Speed and Strategic Choice: How Managers Accelerate Decision Making."         Reprinted from California Management Review, Vol. 32, Nov. 3, 1990,         pp. 1-2. 

"Fad or Fundamental         ? Best Practice Companies." Financial World, Sep. 17, 1991, pp.         34-35. 

Foster, Richard N.,         Lawrence H. Linden, Roger L. Whiteley, and Alan M. Kantrow. "Improving         the Return on R&D." R&D Productivity: Selected Papers from         Research Management during the Period 1986-1990, Washington, D.C.:         Industrial Research Institute, 1990, p. 6. 

Foster, Thomas A.         "Global Logistics Benetton Style." Distribution, Oct. 1993, pp.         62-66. 

Gault, Stanley C.         "Responding to Change." Research Technology Management, May-June         1994, pp. 23-26. 

Gupta, Ashok K., S.         P. Raj, and David Wilemon. "Managing the R&D and Marketing Interface."         Research Management, Mar.-Apr. 1987, pp. 38-43. 

Hales, Lee and Brian         Savoie. "Building a Foundation for Successful Business Process Reengineering."         Industrial Engineering, Sep. 1994, pp. 17-19. 

Hall, Bronwyn H. "Industrial         Research during the 1980s: Did the Rate of Return Fall ?" Brookings         Papers: Microeconomics 2, Washington D.C.: The Brookings Institution,         1993, pp. 308-309. 

 Hamel, Gary. "Forward         Thinking." Executive Excellence, Nov. 1994, pp. 5-6. 

"Hamilton Standard         Takes Top Honors in Concurrent Engineering." Machine Design, Apr.         18, 1994, pp. 18-24. 

Iversen, Wesley R.         "How Honeywell's Mino Will Speed Technology to Market." Electronics,         Apr. 28, 1988, p. 8. 

Kaplan, Robert S.         "In Defense of Activity-Based Cost Management." Management Accounting,         Nov. 1992, pp. 58-63. 

Keiser, Bruce A. "Revitalizing         the Quality Process in Research and Development." 1993 Management for         Quality in Research and Development Symposium, Wilton, Conn.: Juran         Institute, 1993. 

Kelly, Kevin and Peter         Burrows. "Motorola: Training for the Millenium." Business Week,         Mar. 28, 1994, pp. 158-162. 

Kozlowski, Theodore         R. "Implementing a Total Quality Process into Research and Development:         A Case Study." Management for Quality in Research and Development Symposium,         Wilton, Conn.: Juran Institute, 1993. 

Krehbiel, John, Jr.         "Finding a Better Way to Develop Products." Machine Design, Mar.         12, 1993, p. 90. 

"LEGO Systems Builds         U.S. Market with New Distribution Pieces." Material Handling Engineering,         Sep. 1994, pp. 3-6. 

Libby, Bill. "Reengineering,         by the Book." Manufacturing Systems, Apr. 1994, pp. 52-55. 

Lundgren, Bryan. "Special         Buys Need Standard Specs." Purchasing World, Dec. 1990, pp. 44/M3         to 45/M4. 

Maital, Shlomo. "Adapting         to the Unknowable." Across the Board, June 1994, pp. 59-60. 

Mandakovic, Tomislav         and Wm. E. Souder. "A Model for Measuring R&D Productivity." Productivity         Management Frontiers - I, Amsterdam: Elsevier Science Publishers B.V.,         1987, p. 139. 

Matsuo, Hirofumi,         Jen S. Shang, and Robert S. Sullivan. "A Crane Scheduling Problem in a         Computer-Integrated Manufacturing Environment." Management Science,         May 1991, p. 589. 

 McKenna, Joseph F.         "Ford Electronics." Industry Week, Oct. 18, 1993, pp. 35-36. 

Menger, Eve L. "Evolving         Quality Practices at Corning, Inc." Management for Quality in Research         and Development Symposium, Wilton, Conn.: Juran Institute, 1993. 

Meyer, Arnoud De and         Bart Van Hooland. "The Contribution of Manufacturing to Shortening Design         Cycle." R&D Management, July 1990, p. 230. 

Meyer, Marc H., Peter         Tertzakian, and James M. Utterback. "Metrics for Managing Research and         Development." Working Paper, Cambridge, Mass.: Massachusetts Institute         of Technology, April 1995, p. 3. 

Milas, Gene H. "Assembly         Line Balancing . . . Let's Remove the Mystery." Industrial Engineering,         May 1990, p. 31. 

Miller, John A. "Measuring         Progress Through Benchmarking." CMA Magazine, May 1992, p. 37.         

Mitchell, Donald L.         "Reducing Cycle Time." Across the Board, May, 1994, p. 56. 

Morgello, Clem. "George         Fisher of Motorola: The Quest for Quality." Institutional Investor,         Aug. 1991, pp. 45-46. 

Moser, Martin R. "Measuring         Performance in R&D Settings: September/October 1985." R&D Productivity:         Selected Papers from Research Management during the Period 1986-1990,         Washington, D.C.: Industrial Research Institute, 1990, pp. 35-36. 

Muller, E. J. "How         to Profit Using Third Parties." Distribution, May 1991, pp. 31-38.         

Muller, E. J. "Quick         Response Picks Up Pace." Distribution, June 1990, pp. 36-42. 

Muller, E. J. "Turbo         Logistics." Distribution, Mar. 1990, p. 32. 

Oberholtzer, David         M. "Why Reduce Cycle Times ?" Controls & Systems, Jan. 1992,         p. 66. 

Porter, John G., Jr.         "Post Audits - An Aid to Research Planning." R&D Productivity:         Selected Papers from Research Management during the Period 1977-1985,         Washington, D.C.: Industrial Research Institute, 1985, pp. 23-25. 

Przbylowicz, Edward         P. and Terence W. Faulkner. "Kodak Applies Strategic Intent to the Management         of Technology." Research Technology Management, Jan.-Feb. 1993,         p. 33. 

"Quick Response: Slow         but Inevitable." Material Handling Engineering, Sep. 1994, pp.         8-15. 

Quirmbach, Herman         C. "R&D: Competition, Risk and Performance." ISU Economics Reports         #26, Ames, Iowa: Iowa State University, March 1991, p. 1. 

Ranftl, Robert M.         Information from R&D Productivity Conference, Atlanta Georgia,         14-15 December 1983. Los Angeles: published by author, 1983. 

Ranftl, Robert M.         "Seven Keys to High Productivity: September/October 1986." R&D         Productivity: Selected Papers from Research Management during the Period         1986-1990, Washington, D.C.: Industrial Research Institute, 1990.         

Regis, Lesly. "Driving         Quality Up and Cycle Time Down with Design of Experiment." Feb. 1993,         pp. 54-58. 

Rothschild, Michael.         "Mapping Your Web." Forbes, Dec. 5, 1994, pp. 96, 100. 

Rummler, Geary A.         and Alan P. Brache. "Managing the White Space on the Organization Chart."         Supervision, May 1991, p. 12. 

Saxena, Mukul and         Rohinton K. Irani. "Knowledge-Based Modeling of Turbine Nozzles." Mechanical         Engineering, July 1993, pp. 84-90. 

Schainblatt, Alfred         H. "How Companies Measure the Productivity of Engineers and Scientists."         R&D Productivity: Selected Papers from Research Management during         the Period 1977-1985, Washington, D.C.: Industrial Research Institute,         1985, p. 35. 

Schmitt, Roland W.         "R&D in a Competitive Era." Research Management, Jan.-Feb.         1987, p. 15. 

Sheridan, John H.         "Lessons From the Best." Industry Week, Feb. 15, 1993, pp. 54-63.         

Sheridan, John H.         "'Little Things' Add Up to Competitiveness." Industry Week, May         17, 1993, p. 24. 

Singh, Dugesh K. "You         Can Use Simulation to Make the Correct Decisions." Industrial Engineering,         May 1991, pp. 41-42. 

Smith, Preston G.         and Donald G. Reinertsen. Developing Products in Half the Time.         New York: Van Nostrand Reinhold, 1991. 

Smith, Preston G.         and Donald G. Reinertsen. "Shortening the Product Development Cycle."         Research Technology Management, May-June 1992, pp. 44-49. 

Stahl, Michael J.         and Joseph A. Steger. "Improving R&D Productivity - Measuring Innovation         and Productivity - A Peer Rating Approach." R&D Productivity: Selected         Papers from Research Management during the Period 1977-1985, Washington,         D.C.: Industrial Research Institute, 1985, pp. 14-15. 

Stalk, George, Jr.,         and Thomas M. Hout. "Redesign Your Organization for Time-Based Management."         Planning Review, Jan.-Feb. 1990, pp. 5-9. 

Steele, Lowell W.         Managing Technology: The Strategic View. New York: McGraw-Hill         Book Company, 1989, p. 268. 

Steger, Joseph A.         Factors in Innovation, Productivity, Nonproductivity in Research: A         Review and Pilot Study.Troy, NY: Rensselaer Polytechnic Institute,         February 27, 1975, pp. 1-3. 

Thanedar, Shrinivas         P. Methods of Improving Productivity of Research and Development.MBA         Thesis, St. Louis, Mo.: Fontbonne College, 1987, p. 24. 

Thomas, Philip R.         "Reduce Cycle Time to Improve Quality." Executive Excellence, Mar.         1991. 

Thomas, Philip R.         "The Competitive Mindset." Executive Excellence, Oct. 1989. 

Thor, Carl G. "How         to Measure Organizational Productivity." CMA Magazine, Mar. 1991,         pp. 17-19. 

Velocci, Anthony L.,         Jr. "Pratt Synchronizes Its Supply Chain." Aviation Week & Space         Technology, June 7, 1993, pp. 154, 156. 

Vasilash, Gary S.         "What You Need to Know About Continuous Flow." Production, June         1993, p. 72. 

Venkatachalam, A.         R. "Design for Manufacturability: A Survival Strategy for the American         Manufacturing Industry." Industrial Management, May 1992, pp. 7-10.         

Villareal, Louis.         "Strategic Planning, Company Beliefs and Applied IE." Industrial Management,         Nov./Dec. 1991, pp. 2-4. 

Walters, J. E. Research         Management: Principles and Practice. Washington, D.C.: Spartan Books,         1965, pp. 44-45. 

Wasson, A. Robert.         "Understanding Customer Requirements for the Alcoa Technical Center."         1993 Management for Quality in Research and Development Symposium,         Wilton, Conn.: Juran Institute, 1993. 

Weil, Marty. "Flexible         Assembly System Ripens Quality and Economy." Manufacturing Systems,         Nov. 1990, pp. 20-22. 

Welter, Therese R.         "How to Build and Operate a Product-Design Team." Industry Week,         Apr. 16, 1990, pp. 35-50. 

Willyard, Charles         H. and Cheryl W. McClees. "Motorola's Technology Roadmap Process." Research         Technology Management, Sep.-Oct. 1987, p. 14. 

Wolff, Michael F.         "Managers at Work: Working Faster." Research Technology Management,         Nov.-Dec. 1992, p. 12. 
Xu, Xiaoling. Productivity         in The Scientific Research Unit. Masters Thesis, Tuscaloosa, Ala.:         University of Alabama, 1992, p. 13. 

Youssef, Mohamed A.         "Agile Manufacturing: A Necessary Condition for Competing in Global Markets."         Industrial Engineering, Dec. 1992, pp. 18-20. 
APPENDIX 2: ATP PROJECTS           AND INTERVIEWEES 
**1991             ATP Project Title and Interviewees**             SINGLE-COMPANY APPLICANTS Human             Stem Cell and Hematopoietic Expansion Systems
Aastrom Biosciences,               Inc. **[small for profit]**              P.O. Box 376               Ann Arbor, MI 48106              Doug Armstrong             313-930-5555             High Temperature Superconducting Racetrack Magnets for Electric Motor             Applications
American Superconductor               Corp. (ASC) **[small for profit]**              2 Technology Drive               Westborough, MA 01581 Bruce             Gambol             508-836-4200, x207Low             Temperature Viral Inactivation

Aphios **[small               for profit]**              3-E Gill Street               Woburn, MA 01801 Trevor             Castor             617-932-6933Thermal             Insulation Materials - Morphology Control and Processes for the Next             Generation of Performance.
Armstrong World               Industries**[medium/large for profit]**              Innovation Center R&D               2500 Columbia Ave               Lancaster, PA, 17603 Arthur             Yang             717-396-5201. Manufacturing             Technology for High Performance Optoelectronic Devices Based on Liquid             Phase Electro-Epitaxy 
AstroPower,               Inc. **[small for profit]**              Solar Park               Newark, DE 19716-2000 Jim             McNeely             302-366-0400A             Feedback-Controlled Metalorganic Chemical Vapor Deposition Reactor
Spire Corp.               **[small for profit]**              One Patriots Park               Bedford, MA 01730-2396 Nassar             Karam             617-275-6000, ext. 306Autonomous             Navigation in Quasi-Structured Environments
Transitions               Research Corp. **[small for profit]**              Shelter Rock Lane               Danbury CT 06810 Joseph             Engleberger             203-798-8988Development             and Applications of Density Functional Software for Chemical and Biomolecular             Modelings

Biosym Technologies,               Inc. **[small for profit]**              9685 Scranton Road               San Diego, CA 92121 Arnold             Hagler             619-546-5514U.S.             Self-Sufficiency in High-Quality Pyrethrin Production (Agridyne Technologies             was acquired by BioSys)

BioSys **[small               for profit]**              10150 Old Columbia Road               Columbia, MD 21046-1704 Dr.             Jeff Kelly             Integrated Force Array

The Center for               Microelectronics at MCNC **[non profit]**              P.O. Box 12889               3021 Cornwallis Rd.               Research Triangle Park, NC 27709-2889 Scott             Goodwin-Johannason             919-248-1964X-Ray             and Neutron Focusing and Collimating Optics

X-Ray Optical               Systems, Inc. **[small for profit]**              90 Fuller Road               Albany, NY 12205 David             Gibson             518-442-5250 Advancement             of Monocrystalline Silicon Carbide Growth Processes

Cree Research,               Inc.**[small for profit]**              2810 Meridian Pkwy, Ste 176               Durham, NC 27713 Calvin             Carter             919-361-5709 A             Three-Dimensional Database for Visualization of Human Physiology

 Engineering               Animation, Inc.**[small for profit]**              Iowa State Univ. Research Park               2625 N. Loop Dr.               Ames, Iowa              Mike Sellberg             515-296-6931Novel             Near-Net-Shape Processing of Engineered Ceramics

Garrett Ceramic               Components, a unit of Allied-Signal Aerospace **[medium/large               for profit]**              2525 W. 190th St.               Torrance, CA 90509 John             Pollinger             310-323-9500Polymeric             Switches for Optical Interconnects

IBM Corp. **[medium/large for profit]**              Almaden Research Center               650 Harry Road               San Jose, CA 95120 Don             Burland             408-927-1501High             Fidelity Digital Image Compression

Iterated Systems,               Inc.**[small for profit]**              5550-A Peachtree Pkwy, Ste 650               Norcross, GA 30092 Stephen             Demko             404-840-0633             770-840-0633Development             of Cost-Effective Routes to Compatibilize Polymers in a Commingled             Waste Stream 

Michigan Molecular               Institute **[independent research organization]**              1910 W. St. Andrews Rd.               Midland, MI 48640 Conrad             Balazs             517-832-5590Synthesis             and Processing of Nanocrystalline Ceramics on a Commercial Scale

Nanophase Technologies               Corp. **[small for profit]**              8205 So. Cass Ave., Ste 105               Darien, IL 60559 John             Parker             708-323-1200             Research Joint Ventures (JV)Development             of Advanced Technologies and Systems for Controlling Dimensional Variation             in Automobile Body Manufacturing
(Auto Body Consortium,               JV with ASC, Inc., CDI Transportation Group, Inc., Classic Design,               Delta Engineering, Detroit Center Tool, Inc., Efficient Engineering,               Perception, Pioneer Engineering, Progressive Tool & Industries               Co., Richard & Trute Tool & Die, General Motors, Chrysler,               and the University of Michigan.) 

Auto Body Consortium               **[non profit]**              2901 Hubbard               Ann Arbor, MI 48105-2467 Ernie             Vahala             313-741-5905             Hybrid Superconducting Digital System

(Conductus,               Inc., JV with TRW Inc., Hewlett-Packard, Stanford U., U.C. Berkeley)               

Conductus Inc.               **[small for profit]**              969 West Maude Ave.               Sunnyvale, CA 94086 John             Rowell             408-523-9408Cyclic             Thermoplastic Liquid Composite Molding for Automotive Structures

(Ford Motor               Co. JV with General Electric) 

Ford Motor Co.               **[medium/large for profit]**              Scientific Research Lab               M/S 3182, PO Box 2053               Dearborn, MI 48121-2053 Carl             Johnson             313-323-0399             Neural Network Control and Sensors for Complex Materials 

(Honeywell JV               with Hercules Aerospace, Sheldahl, 3M) 

Honeywell **[large for profit]**              3660 Technology Dr.               Minneapolis, MN 55418 Thomas             Edman             612-951-7514             Scalable High-Density Electronics Based on MultiFilm Modules 

(The American               Scaled-Electronics Consortium JV with the Microelectronics and Computer               Technology Corporation, and its member companies) 

Kopin Corp.               **[small for profit]**              695 Myles Standish Blvd.               Taunton, MA 02780 Glenn             Kephart             508-824-6696NCMS             Rapid Response Manufacturing

(NCMS JV with               Aries Technology, Cimflex Teknowledge Corp., Cimplex Corp., Dept.               of Energy Oak Ridge Y-12 Facility, Ford, GM, ICAD, Parametric Technology               Corp., Spatial Technology, Texas Instruments, and United Technologies.)               

National Center               for Manufacturing Sciences (NCMS)**[non profit]**              3025 Boardwalk               Ann Arbor, MI 48108 Bill             Waddell             313-995-4903             Ultra-High Density Magnetic Recording Heads

(NSIC JV with               Applied Magnetics Corp., Digital Equipment Co., Eastman Kodak Company,               Hewlett-Packard Corp., IBM, Quantum Corp., Storage Technology Corp.,               Carnegie Mellon University, University of Alabama, University of               California at San Diego, University of Minnesota, and Wasington               University.) 

National Storage               Industry Consortium. (NSIC) **[non profit]**              9888 Carroll Center Road               San Diego, CA 92126 John             L. Simonds             619-578-2436             PREAMP - Pre-Competitive Advanced Manufacturing of Electrical Products             

(SCRA JV with               Boeing, Digital Equipment Corp., Hewlett-Packard, GM/Hughes, Martin               Marietta, Arthur D. Little, Battelle, and Mentor Graphics.) 

South Carolina               Research Authority (SCRA) **[non profit]**              5300 International Boulevard               No. Charleston, SC 29418 Jack             Corley             803-760-3792             Monolithic Multiwavelength Laser Diode Array Spanning 430 to 1100nm

(Spectra Diode               Laboratories JV with Xerox Corp.) 

Spectra Diode               Laboratories, Inc. **[small for profit]**              80 Rose Orchard Way               San Jose, CA 95134-1356 David             Welch, VP, Business Development             408-943-9411             Plasma Technology for Low-Cost Diamond Production

(Westinghouse               Electric Corp. JV with SGS Tool Co.) 

Westinghouse               Electric Corp. **[large for profit]**              1310 Beulah Road               Pittsburgh, PA 15235-5098 Dr.             Howard Saunders             412-256-1960 
APPENDIX 3: ABSTRACT

**Proposed         for NIST Publication** (cite fully): Acceleration         of Technology Development by the Advanced Technology Program: The Experience         of 28 Projects Funded in 1991 

**Abstract:**         One of ATP's legislated mandates is to accelerate industry's commercialization         of new technologies. Earlier surveys found that a majority of program         participants believed that participation in the ATP had helped them to         do just that - but the earlier surveys which addressed a variety of performance         metrics did not provide details on why cycle-time reduction is of special         importance to the companies, or on how participation in the ATP helped         them to reduce their applied research and technology development cycle         time and get to market more quickly, or on whether there were any effects         beyond the ATP project. This survey, conducted via structured telephone         interviews estimated that participating in ATP had helped them to reduce         their technology development cycle time anywhere from 30% to 66% with         the median response being a 50% reduction (most typically from a projected         six-year cycle down to a three-year cycle). A little over half of the         interviewees provided quantitative estimates of the economic value of         reducing cycle time by a single year - the estimates ranged from $1 million         to several billion, with a median average value of $5.5 million. They         expected the positive impact on cycle time experienced in the applied-research         stage to flow through to later stages in the technology development cycle         (the product development, production, and marketing stages), thereby causing         them to enter the marketplace more quickly. These cycle-time improvements         were said to carry over to other technology development projects outside         of ATP. Interviewees spoke of adapting specific "ATP practices" to related         projects; applying methodologies and processes used or developed in the         ATP project to the firm as a whole; developing a cultural bias favoring         speedier processes; and taking advantage of the positioning provided by         the enabling technologies developed in their ATP projects to accelerate         the development of a whole series of related applications. 

**Key Words:**         acceleration of technology, applied research, cycle-time reduction, economic         evaluation, impact analysis, performance metrics, program evaluation,         research and development, technology transition.Date created:         October 23, 1997        Last updated:         May 13, 2003         ATP website comments: webmaster-atp@nist.gov • Technical        ATP inquiries: InfoCoord.ATP@nist.govPrivacy        policy / security notice / accessibility statement •  NIST        Disclaimer  • NIST        Information Quality StandardsNIST is an agency of the U.S.Commerce Department'sTechnologyAdministration.** |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|