**TECHNICAL NOTES FOR 1994**

SURVEY METHODOLOGY[^1]

####### Reporting Unit

The reporting unit for the Survey of Industrial Research and Development
is the enterprise, or company, defined as a business organization of one
or more establishments under common ownership or control. The survey
includes two groups of enterprises: (1) companies known to conduct R&D
and (2) a sample representation of companies for which information on
the extent of R&D activity is uncertain.

####### Frame Creation

The Standard Statistical Establishment List (SSEL), a Bureau of the
Census compilation that contains information on more than 3 million
establishments with paid employees, was the target population from which
the frame used to select the 1994 survey sample was created (see table
B-1 for target population and sample sizes). For companies with more
than one establishment, data were summed to the company level. The firm
was then assigned a single standard industrial classification (SIC) code
based on the activity of the establishment(s) having the highest dollar
value of payroll. This assignment was done on a hierarchical basis. The
enterprise was first assigned to the economic division (manufacturing or
nonmanufacturing) with the highest payroll, then to the two-digit SIC
code with the highest payroll within the assigned division, then to the
three-digit SIC code with the highest payroll within the assigned
two-digit industry.

The frame from which the survey sample was drawn included all for-profit
companies classified in nonfarm industries. For surveys prior to 1992,
the frame was limited to companies above certain size criteria based on
number of employees.[^2] These criteria varied by industry. Also, some
industries were excluded from the frame because it was believed that
these industries contributed little or no R&D activity to the final
survey estimates. For the 1992 sample, new industries were added to the
frame[^3] and the size criteria were lowered considerably and applied
uniformly to firms in all industries. As a result, nearly 2 million
enterprises with five or more employees were given a chance of
selection. For comparison, the frame for the 1987 sample included
154,000 companies of specified sizes and industries. The initial frame
used to select the 1994 sample was similar to the ones used to select
the 1992 and 1993 samples.

The frame ultimately used to select the 1994 sample differed from the
1993 frame in the following respects. First, the predetermination of
companies selected for the survey with certainty was limited to
companies with reported or estimated R&D expenditures of \$1 million or
more and all companies with 1,000 employees or more. For the 1993 frame,
external information about the likelihood that a company conducted R&D
was used to identify nearly 10,000 companies that were included in the
survey sample with certainty. Sources included the 1992 survey,
directories that include company information on R&D reported to the
Securities and Exchange Commission, commercially available directories
of R&D-performing companies, Department of Defense directories of
contracts awarded for R&D, and various publications and newsletters that
highlight firms conducting R&D. In the 1994 frame, these companies were
included along with R&D spending reported in the 1993 survey. Their
likelihood of selection in the 1994 sample was based on the level of
their R&D programs reported in 1993 rather than with certainty.

Other modifications to the 1993 frame for the 1994 survey were the
partitioning of the frame and the expanded use of simple random
sampling. Since the early 1980s, probability proportionate to size (PPS)
sampling had been used exclusively in the selection of new sample panels
for the R&D survey. An examination of 1992 survey results, however,
showed that the large influx of small companies into the frame that year
resulted in a disproportionate number of small companies being selected
for the sample, often with very large weights. These companies generally
reported little, if any, R&D activity. This disproportion was caused
primarily by the application of the minimum probability rule (see below
under "Sample Size and Weighting") which resulted in increased
probabilities of selection for several hundred thousands of these
smaller companies.

As a result, the 1994 frame was split into "large" and "small" company
partitions. PPS sampling continued to be used for the large partition.
However, more efficient simple random sampling (SRS) was used to sample
the small company partition. SRS is more efficient than independent PPS
sampling when little variability exists in the size of the variable
being measured.

Payroll was the basis for determining the partition between large and
small, and the analysis of payroll was performed for each of the 165
defined sampling strata. Within each stratum, companies were sorted by
payroll. Companies previously designated as certainties were defined as
large regardless of their payroll (that is, all predetermined
certainties were included in the PPS frame). With that parameter,
cumulative payroll values were computed until 90 percent of the stratum
total was reached. Companies contributing to the 90 percent share were
defined as large, and the remaining companies were defined as small. In
total, the large partition contained about 640,000 companies. The small
partition contained about 1.2 million companies. An additional benefit
of these design changes was a reduction in the maximum allowable weight
for selected companies.

####### Sample Selection

######## Probability Proportionate to Size

The distribution of companies by payroll and estimated R&D in the large
partition was skewed as in earlier frames. Because of this skewness, PPS
sampling used in previous designs was an appropriate selection technique
for this group. That is, large companies had a higher probability of
selection than did small companies. For this survey it would have been
ideal if company size could have been determined by its R&D
expenditures. Unfortunately, except for the companies that were in a
previous survey or for which there was information from external
sources, it was impossible to know the R&D expenditures for every firm
in the universe. Consequently, the probability of selection for most
companies was based on estimated R&D expenditures.

Since total employment was known for each company in the universe, it
was possible to use an already-observed relationship between employment
and R&D to estimate an amount for R&D expenditures to use as a measure
of size. This was the same strategy employed in all sampling operations
since 1981. For 1994 sampling, data collected in the 1993 survey were
used to derive this relationship separately for single-unit companies
and multiestablishment companies. The effect in all cases was to give
firms with a large number of employees higher probabilities of selection
since the assumption was that large companies were more likely to
perform R&D and that the amount of R&D was proportionate to the size of
the company.

Estimated R&D values were computed for companies in the small partition
as well. The aggregate of reported and estimated R&D from each company
in both the large and small partitions represented a total universe
measure of R&D expenditures. However, assigning R&D to every company
resulted in an overstatement of this measure. To adjust for the
overstatement, the universe measure was scaled down using factors
developed from the relationship of the universe measure of 1993 R&D and
the 1993 survey estimate. These factors, computed at levels
corresponding to published industry levels, were used to adjust
originally imputed R&D values so that the new frame total for R&D at
these levels approximated the 1993 published values. This adjustment
provided for better allocation of the sample among these levels.

######## Simple Random Sampling

In the small company partition, the use of simple random sampling
implied that each company within a stratum had an equal probability of
selection. Stratum definitions were the same as for the large partition.
The total sample allocated to the small partition was dependent upon the
total sample specified for the survey and upon the total sample
necessary to satisfy criteria established for the large partition. Once
determined, the allocation of this total by stratum was made
proportionate to the stratum's payroll contribution to the entire
partition.

######## Sample Stratification and Relative Standard Error Constraints

The particular sample selected was one of a large number of the same
type and size that by chance might have been selected. Statistics
resulting from the different samples would differ somewhat from each
other. These differences are represented by estimates of sampling error.
The smaller the sampling error, the more precise the statistic.

Primary concern was placed on the large company partition since it was
believed that nearly all of the R&D activity would be identified from
this sector. To control sampling error in the statistics resulting from
this portion of the frame, parameters were specified to allocate the
sample across various levels, or strata, that corresponded to industry
groupings. These parameters permitted the sample size to be varied to
achieve a desired level of sampling error for each stratum and were
assigned so that estimated errors of total R&D expenditures for
industries in these strata did not exceed certain levels. Sample sizes
among the strata were only constrained by the limit placed on the total
sample size dictated by the available budget.

For sample selections prior to 1992, the stratum designations were the
published industry categories. The sample was allocated across these
industry categories to provide high, medium, and low levels of
precision. For the 1992 sample the criteria for this allocation were
modified. In order to gather information to review and evaluate the
appropriateness of the published industry groupings, the allocation of
the sample was controlled for levels of industry detail below those
traditionally published. The result was that the frame was partitioned
into 140 manufacturing industry strata and 25 nonmanufacturing strata.
The manufacturing strata corresponded to the 140 three-digit industries
that comprised manufacturing. For nonmanufacturing, 12 strata
corresponded to three-digit nonmanufacturing industries that represented
a current level of publication or that had a high concentration of
scientists and engineers, and 12 strata corresponded to two-digit
nonmanufacturing industries where R&D activity was considered likely.
These nonmanufacturing strata thus identified newly emerging industries
or industries where improved coverage was desired. The final stratum was
the balance of nonmanufacturing industries that had not been included in
previous sampling frames or for which there was little indication of R&D
activity. This same stratification was used for the 1993 and 1994
samples.

For 1994, the following criteria for the relative standard error of
estimated R&D expenditures were established for the 165 strata of the
large company partition:

> a\. Relative sampling error not to exceed 2 percent: all 140
> three-digit manufacturing strata, 12 three-digit nonmanufacturing
> strata, and 3 two-digit nonmanufacturing strata.
>
> b\. Relative sampling error not to exceed 5 percent: the remaining 9
> two-digit nonmanufacturing strata, and the 1 stratum corresponding to
> the balance of nonmanufacturing.

These criteria, which differed from the criteria established for the
1993 survey, suggested a total sample size of approximately 17,600
companies from the large partition.

A limitation of the sample allocation process for the large partition
should be noted. The sampling errors used to control the sample size in
each stratum were based on a universe total that, in large part, was
improvised. That is, as previously noted, an R&D value was assigned to
every company in the frame, even though many of these companies actually
may not have had R&D expenditures. The value assigned was imputed for
the majority of companies in the frame and, as a consequence, the
estimated universe total and the distribution of individual company
values did not necessarily reflect the true distribution. Estimates of
sampling variability were nevertheless based on this distribution. The
presumption was that actual variation in the sample design would be less
than that estimated, because many of the sampled companies have true R&D
values of zero, not the widely varying values that were imputed using
total employment as a predictor of R&D. Previous sample selections
indicate that in general this presumption holds, but exceptions have
occurred when companies with large sampling weights have reported large
amounts of R&D spending. Thus, in general, the 2-percent and 5-percent
error levels described earlier are conservative. See table B-2 for a
list by industry of the actual standard error estimates for selected
items.

For the small partition, the same 165 strata were identified. Also
included was a separate stratum of approximately 8,700 companies that
could not be assigned to a stratum because of incomplete industry
identification in the Standard Statistical Establishment List (SSEL). In
1994, for the first time, a small number of companies was selected from
this group in the hopes that an accurate industry identification could
be obtained at a later point. The initial sample size specified for the
small company partition was 4,500 companies. The sample initially
allocated to a given stratum was proportionate to its share of total
payroll for the small partition.

In addition to sampling error, the estimates are subject to nonsampling
error. Errors are grouped into five categories: specification, coverage,
response, nonresponse, and processing. For detailed discussions on the
sources, control, and measurement of each of these types of error, see
the technical reports.[^4]

######## Sample Size and Weighting

The total sample size initially specified for the R&D survey was
approximately 22,000, and, as described above, was based primarily on
compliance with predetermined sampling error constraints established for
the large partition. The actual sample size was 23,541 companies which
differed from the target for several reasons. First, the frame for the
large partition was subjected to independent sampling. Each company in
the frame had an independent chance of selection, based on its assigned
probability, i.e., selection of a company was completely independent of
the selection of any other company. In independent sampling, sample size
itself is a random variable. Theoretically, a sample of size zero or a
sample the size of the entire universe is possible, but the
probabilities of these extremes are so small that these are nearly
impossible situations. The actual sample size is usually quite close to
the specified size. If there is too much deviation, the selection is
repeated.

Second, a minimum probability rule was imposed for both partitions. As
noted earlier, for the large partition, probabilities of selection
proportionate to size were assigned each company, where size is the
reported or imputed R&D value assigned each company. Selected companies
received a sample weight which was the inverse of their probability of
selection. Selected companies that ultimately report R&D expenditures
vastly larger than their assigned values can have adverse effects on the
statistics, which are based on the weighted value of survey responses.
To lessen the effects on the final statistics, the maximum weight of a
company was controlled by specifying a minimum probability that could be
assigned to the company. If the probability, based on company size, was
less than the minimum probability, then it was reset to this minimum
value. The consequence of raising these original probabilities to the
minimum probability was to raise the expected sample size. Similarly, a
maximum weight for each stratum was established for the simple random
sampling of the small partition. If the sample size initially allocated
to a stratum resulted in a stratum weight above this maximum value, then
the sample size was increased until the maximum weight was achieved. It
is likely that most of the difference between the size of the target
sample and the sample actually selected was because of these rules.

Third, a minimum sample size was established for each stratum of the
small partition. If the proportionately allocated sample size fell below
the minimum value for a given stratum, then the sample size was set
equal to this value.

Finally, between the time that the frame was created and the survey was
prepared for mailing, the operational status of some companies changed.
That is, they were merged with or acquired by another company, or they
were no longer in business. Before preparing the survey for mailing, the
operational status is updated to identify these changes. As a result,
the number of companies mailed a survey form is somewhat smaller than
the number of companies initially selected for the survey.

####### Survey Questionnaires 

Two questionnaires are used each year to collect data for the survey.
For large firms known to perform R&D, a detailed questionnaire, Form
RD-1L, is used to collect data for odd-numbered years and an abbreviated
version, Form RD-1S, is used to collect data for even-numbered years.
The questionnaires are cycled in this manner to reduce reporting burden
on survey respondents.

The Form RD-1L requests data on sales or receipts, total employment,
employment of scientists and engineers, expenditures for R&D performed
within the company with Federal funds and with company and other funds,
character of work (basic research, applied research, and development),
company-sponsored R&D expenditures in foreign countries, R&D performed
under contract by others, expenditures for pollution abatement and
energy R&D, detail on R&D by product field, Federal R&D support to the
firm by contracting agency, domestic R&D expenditures by State, and
foreign R&D by country. The Form RD-1S requests the same information
except for the last four items. Because companies receiving the Forms
RD-1L and RD-1S generally have participated in previous surveys,
computer imprinted data reported by the company for the previous year
are supplied for reference. Companies are encouraged to revise or update
this imprinted data if they have more current information.

To further limit reporting burden on small R&D performers and on firms
that are included in the sample for the first time, an even more
abbreviated form is used each year. Form RD-1A collects data only on
R&D, sales, employment, and operational status and includes a screening
item that allows respondents to indicate that they do not perform R&D
before completing the questionnaires. No prior-year information is
available since the majority of the companies have not reported
previously.

For the 1994 survey, about 2,700 companies that reported \$1 million or
more in R&D spending in the 1993 survey or had 1,000 employees or more
received Form RD-1S and nearly 20,800 received Form RD-1A. Of the 23,500
firms, 4,800 reported R&D expenditures.

####### Followup for Survey Nonresponse 

The 1994 survey questionnaires were mailed in March 1995, and recipients
were asked to respond within 60 days. Thirty days later, letters were
mailed to all survey recipients reminding them that their completed
questionnaire was due within the next 30 days. Copies of the Form RD-1A
and instructions were faxed to respondents who called a toll-free
telephone number indicated in the followup letters. After 60 days,
follow-up letters were sent to all nonresponding firms. Three additional
followup mailings were made to persistent nonrespondents, after 90, 120,
and 150 days.

In addition to the mailings, telephone followup was used to encourage
response from those firms ranked among the 300 largest R&D performers,
based on total R&D expenditures reported in the previous survey.
Telephone followup was also used for these firms during the initial data
edit phase of survey operations if data items were missing or unclear.
Table B-3 shows the number of companies in each industry or industry
group that received a questionnaire and the percentage that responded to
the survey.

####### Imputation for Item Nonresponse

For various reasons, many firms chose to return the survey
questionnaires with one or more blank items.[^5] For instance, the
internal accounting procedures of the firm may not have allowed it to
quantify the pollution-abatement expenditures portion of R&D. In
addition, some firms, as a matter of policy, refused to answer any
voluntary questions.[^6]

When respondents did not provide the requested information, estimates
for the missing data were made using imputation algorithms. In general,
the imputation algorithms computed values for missing items by applying
the average percentage change for the target item in the nonresponding
firm's industry to the item's prior-year value for that firm, reported
or imputed. This approach, with minor variation, was used for most
items.[^7] Table B-4 contains imputation rates for the principal survey
items.

####### Response Rates and Mandatory Versus Voluntary Reporting

Current survey reporting requirements divide survey items into two
groups: mandatory and voluntary. Response to four data items on the
questionnaires---total R&D expenditures, Federal R&D funds, net sales,
and total employment---are mandatory; response to the remaining items is
voluntary. During the 1990 survey cycle, NSF conducted a test of the
effect of reporting on a completely voluntary basis to determine if
combining both mandatory and voluntary items on one questionnaire
influences response rates. For this test, the 1990 sample was divided
into two panels of approximately equal size. One panel, the mandatory
panel, was asked to report as usual (four mandatory items and the
remainder voluntary), and the other panel was asked to report all items
on a completely voluntary basis. The result of the test was a decrease
in the overall survey response rate to 80 percent from levels of 88
percent in 1989 and 89 percent in 1988. The response rates for the
mandatory and voluntary panels were 89 percent and 69 percent,
respectively. Detailed results of the test were published in Research
and Development in Industry: 1990. For firms that reported R&D
expenditures in 1994, table B-5 shows the percentage that also reported
data for other selected items.

####### Character of Work 

Response to questions about character of work (basic research, applied
research, and development) declined in the mid-1980s, and, as a result,
imputation rates increased. The general imputation procedure described
above became increasingly dependent upon information imputed in prior
years, thereby distancing current-year estimates from any reported
information. Because of the increasing dependence on imputed data, NSF
chose not to publish character-of-work estimates in 1986. Consequently,
the imputation procedure used to develop these estimates was revised in
1987 for use with 1986 and later data and differs from the general
imputation approach. The new method calculates the character-of-work
distribution for a nonresponding firm only if that firm reported a
distribution within a five-year period, extending from two years before
to two years after the year requiring imputation. Imputation for a given
year is initially performed in the year the data are collected and is
based on a character-of-work distribution reported in either of the two
previous years, if any. It is again performed using new data collected
in the next two years. Thus, character-of-work estimates are revised as
newly reported information becomes available and are not final for two
years following their initial publication.

If no reported data are available for a firm, character-of-work
estimates are not imputed. As a consequence, only a portion of the total
estimated R&D expenditures are distributed at the firm level. Those
expenditures not meeting the requirements of the new imputation
methodology are placed in a "not distributed" category. Tables B-6, B-7,
and B-8 show the character-of-work estimates along with the "not
distributed" component for 1992, 1993, and 1994, respectively.

NSF's objective in conducting the survey has always been to provide
estimates for the entire population of firms performing R&D in the
United States. However, the revised imputation procedure would no longer
produce such estimates because of the "not distributed" component. So, a
baseline estimation method was developed to allocate the "not
distributed" amounts among the character-of-work components. In the
baseline estimation method, the "not distributed" expenditures are
allocated by industry group to basic research, applied research, and
development categories, using the percentage splits in the distributed
category for that industry. The allocation is done at the lowest level
of published industry detail only; higher levels are derived by
aggregation (just as national totals are derived by aggregation of
individual industry estimates) and result in higher performance shares
for basic and applied research and lower estimates for development\'s
share than would have been calculated using the previous method.[^8] The
estimates of basic research, applied research, and development provided
in section A of this report were calculated using the baseline
estimation method.

####### COMPARABILITY OF STATISTICS

This section summarizes the statistical revisions that have been made
because of changes in survey procedures and practices.[^9]

######## Revisions to Immediate Prior-Year Statistics

As has been the practice throughout the history of the Survey of
Industrial Research and Development, results from the current-year
survey are used not only to develop current-year statistics, but also to
revise immediate prior-year statistics. Changes to reported data can
come from three sources: from respondents (see discussion above under
"Survey Questionnaires"), from analysts involved in survey and
statistical processing, and from the industry reclassification process.
Respondents from companies that were in both the 1993 and 1994 surveys
may have revised previously reported data for 1993 because data were
received or developed too late to include in the initial estimates. In
follow-up conversations with respondents, analysts may have corrected
previously reported data or supplied missing data. Analysts also
examined the initial industry classification of companies and may have
made changes. For example, during processing for the 1993 survey
analysts found a few instances of industry reclassification because of
small fluctuations in payroll (see "Industry Shifts" below) and not true
shifts in company activity.

######## Year-to-Year Changes

Comparability from year-to-year may be affected by new sample design,
annual sample selection, industry shifts, and data revisions.

######## Sample Design

Changes to the sample design can affect comparability of year-to-year
estimates. By far the most profound influence on statistics from recent
surveys occurred when the new sample design for the 1992 survey was
introduced. Revisions to the 1991 statistics were dramatic (see Research
and Development in Industry: 1992 for a detailed discussion). The sample
design used for the 1992, 1993, and 1994 surveys are comparable in terms
of size and coverage. While the allocation of the sample was changed
somewhat, the design of the sample had little effect on the
comparability of the statistics for this three-year period.

######## Annual Sample Selection

With the introduction of annual sampling in 1992, more year-to-year
change have resulted than when survey panels were used. There are two
reasons why this is so. First, changes in classification of companies
not surveyed were not reflected in the year-to-year movement. The
wedging operation which was performed when a new sample was selected,
was a means of adjusting the data series to account for the changes in
classification that occurred in the frame (see the discussion on wedging
below). Second, yearly correlation of R&D data is lost when independent
samples are drawn each year.

######## Industry Shifts

The industry classification of companies is redefined each year with the
creation of the sampling frame. By redefining the frame, the sample
reflects current distributions of companies by size and industry. During
this process, a company may move from one industry into another because
of several factors: changes in a company's payroll composition, which is
used to determine the industry classification code (see discussion above
under "Frame Creation"), changes in the industry classification system
itself, or changes in the way the industry classification code is
assigned or revised during survey processing.

A company's payroll composition changes because of a number of events.
Among them are (1) the growth or decline of product or service lines;
(2) the merger of two or more companies; (3) the acquisition of one
company by another; (4) divestitures; or (5) the formation of
conglomerates. Since the introduction of annual sampling in 1992,
although unlikely, a company's data can be reclassified yearly. The
result is that a downward movement in R&D expenditures in one industry
is balanced by an upward movement in another industry from one year to
the next.

From time to time the standard industrial classification (SIC) coding
system, which is used by most Federal Government agencies that publish
industry statistics, is revised to reflect the changing composition of
U.S. industry. For statistics developed for 1988--91 from the 1988--91
surveys, companies retained the industry classifications assigned for
the 1987 sample. These classifications were based on the 1977 SIC
system. The last major revision of the SIC system was for 1987. This new
system was used to classify companies in the post-1991 surveys.

Finally, the method used to classify firms during survey processing was
revised slightly in 1992. Research has shown that the impact on
individual industry estimates has been minor.[^10] The current method
used to classify firms is discussed above under "Frame Creation."
Methods used for past surveys are discussed in the technical paper cited
below.[^11]

######## Data Revisions

Changes to reported data can come from two sources: from respondents
(see earlier discussion under Survey Questionnaires) and from analysts
involved in survey and statistical processing. Respondents from
companies that were in both the 1993 and 1994 surveys may have revised
previously reported data for 1993. Analysts, while performing followup,
may have corrected incorrectly reported or supplied missing 1993 data.
The industry-specific summary of changes in the 1993 R&D statistics
resulting from data revisions and industry shifts are presented in table
B-9.

######## Recent Survey Improvements[^12]

Before the 1992 survey, the sample of firms surveyed was selected at
irregular intervals.[^13] In intervening years, a panel of the largest
firms known to perform R&D was surveyed. For example, a sample of about
14,000 firms was selected for the 1987 survey. For the 1988 through 1991
studies, about 1,700 of these firms were annually resurveyed; the other
firms did not receive another questionnaire and their R&D data were
estimated. This sample design was adequate during the early years of the
survey because the performance of R&D remained concentrated in
relatively few manufacturing industries. However, as more and more firms
began entering the R&D-performing arena, the old sample design proved
increasingly deficient because it did not capture births of new
R&D-performing firms. The entry of fledgling R&D performers into the
marketplace was simply missed during panel years. Additionally,
beginning in the early 1970s, the need for more detailed R&D information
for nonmanufacturers was recognized. At that time, statistics for the
broad industry classifications, miscellaneous business services and
miscellaneous services, were added to the list of industry groups for
which statistics were published. By 1975, about 3 percent of total R&D
was performed by firms in nonmanufacturing industries.

During the mid-1980s, there was evidence that an increasing number of
nonmanufacturing firms were conducting a significant amount of R&D, and
again the number of industries used to develop the statistics for
nonmanufacturers was increased. Consequently, the annual reports in this
series for 1987 and since have included separate R&D estimates for firms
in the communication, utility, engineering, architectural, research,
development, testing, computer programming, and data processing service
industries; hospitals; and medical labs. Approximately 9 percent of the
estimated industrial R&D performance during 1987 was undertaken by
nonmanufacturing firms.

After the list of industries for which statistics were published was
expanded, it became clear that the sample design itself should be
changed to reflect the widening population of R&D performers among firms
in the nonmanufacturing industries[^14] and small firms in all
industries, to account better for births of R&D-performing firms and to
produce statistics that are generally more reliable. Beginning with the
1992 survey, NSF decided to (1) draw new samples with broader coverage
annually and (2) increase the sample size to approximately 23,000
firms.[^15] As a result of the sample redesign, for 1992, the reported
nonmanufacturing share was estimated to be 25 percent of total R&D.

######## Time Series Analyses

As discussed earlier, the statistics resulting from the survey are
better indicators of changes in, rather than absolute levels of, R&D
spending and personnel. Nevertheless, the statistics are often
considered as a continuous time series that has been prepared using the
same collection, processing, and tabulation methods. Such uniformity
during preparation has not been the case. Since the survey was first
fielded, improvements have been made to increase the reliability of the
statistics and to make the survey results more useful. To that end,
existing practices have been changed and new procedures have been
instituted. Preservation of the comparability of the statistics has been
an important consideration when improvements have been made, however.
Changes to survey definitions, the industry classification system, and
the procedure used to assign industry codes to multiestablishment
companies[^16] have had some, though not substantial, effects on the
comparability of statistics.[^17] The aspect of the survey that had a
greater effect on comparability was the selection of samples at
irregular intervals (i.e., 1967, 1971, 1976, 1981, 1987, 1992) and the
use of a subset or panel of the last sample drawn to develop statistics
for intervening years. As discussed earlier, this practice introduced
cyclical deterioration of the statistics.

As compensation for this deterioration, periodic revisions have been
made to the statistics produced from the panels surveyed between sample
years. Early in the survey's history, various methods were used to make
these revisions.[^18] After 1976 and until 1992 with the advent of
annual sampling, a linking procedure called wedging was used.[^19]
Simply described, in wedging, the two sample years on each end of a
series of estimates serve as benchmarks in the algorithms used to adjust
the estimates for the intervening years.

####### Wedging

######## Wedging Methodology

For a full discussion of the mathematical algorithm used for the wedging
process that linked statistics from the 1992 survey with those from the
1987 survey, see the technical memorandum cited below.[^20] In general,
the memorandum states that wedging

> *takes full advantage of the fact that in the first year of a new
> panel \[when a new sample is selected\], both current-year and
> prior-year estimates are derived. Thus, two independent estimates
> exist for the prior year. The estimates from the new panel are treated
> as superior primarily because the new panel is based on updated
> classifications \[the industry classifications in the prior panel are
> frozen\] and is more fully representative of the current universe (the
> prior panel suffers from panel deterioration, especially a lack of
> birth updating). The limitations in the prior panel caused by these
> factors are naturally assumed to increase with time, so that in the
> revised series, we desire a gradual increase in the level or revision
> over time which culminates in the real difference observed between the
> two independent sample estimates of the prior year. At the same time,
> we desire that the annual movement of the original series be preserved
> to the degree possible in the revised series.*

To that end, the wedging algorithm does not change estimates from sample
years and adjusts estimates from panel years, recognizing that
deterioration of the panel is progressive over time.

######## Wedged Versus Not-Wedged Statistics

One of the primary reasons for the decision to select a new sample
annually rather than at irregular intervals was to avoid applying global
revision processes such as wedging. Consequently, the 1992 survey was
intended to be the last one to employ the wedging procedure.

######## Revisions to Historical Statistics

Throughout the history of the survey, during regular survey processing,
all immediate prior-year statistics have been subject to revision with
results from the current year's survey. Changes to older statistics
usually have been limited to revisions because of changes in the
industry classification of companies caused by changes in payroll
composition detected when a new sample was drawn. Various methodologies
have been adopted over the years to revise, or backcast, the data when
revisions to historical statistics have become necessary.

Documented revisions to the historical statistics from post-1967 surveys
are summarized in *Research and Development in Industry: 1991* (NSF
94-325). Detailed descriptions of the specific revisions made to the
statistics from pre-1967 surveys are scarce. However, summaries of some
of the major revisions are included in the technical paper cited
below.[^21]

######## Comparisons to Other Statistical Series

The National Science Foundation (NSF) collects data on federally
financed R&D from both Federal funding agencies and performers of the
work (industry, Federal labs, universities, and other nonprofit
organizations). As reported by Federal agencies, NSF publishes data on
Federal R&D budget authority and outlays, in addition to Federal
obligations. These terms are defined below:[^22]

-   *Budget authority* is the primary source of legal authorization to
    enter into financial obligations that will result in outlays. Budget
    authority most commonly is granted in the form of appropriations
    laws enacted by Congress with the approval of the President.[^23]

-   *Obligations* represent the amounts for orders placed, contracts
    awarded, services received, and similar transactions during a given
    period, regardless of when the funds were appropriated or when
    future payment of money is required.

-   *Outlays* represent the amounts for checks issued and cash payments
    made during a given period, regardless of when the funds were
    appropriated or obligated.

For the reasons cited above, national R&D expenditure totals in NSF's
National Patterns of R&D Resources report series are constructed
primarily based on data reported by performers and include estimates of
Federal R&D funding to these sectors. But until performer-reported
survey data on Federal R&D expenditures are available from industry and
academia, data collected from the Federal agency funders of R&D are used
to project R&D performance. When survey data from the performers
subsequently are tabulated (as they are in this report), these
statistics replace the projections based on funder expectations.
Historically, the two survey systems have tracked fairly closely. For
example, in 1980 performers reported using \$29.5 billion in Federal R&D
funding, and Federal agencies reported total R&D funding between \$29.2
billion in outlays and \$29.8 billion in obligations.[^24] In recent
years, however, the two series have diverged considerably. For 1994,
performers reported \$60.2 billion in Federal R&D support, compared with
the \$66.2 billion to \$68.3 billion reported by Federal agencies.[^25]

The difference in the Federal R&D totals appears to be concentrated in
funding of industry (primarily aircraft and missile firms) by the
Department of Defense. Overall, industrial firms have reported
significant declines in Federal R&D support since 1990 (see table A-1),
while Federal agencies reported level or slightly increased funding of
industrial R&D. For 1994, Federal agencies reported \$31.7 billion in
total R&D obligations provided to industrial performers, compared with
an estimated \$22.5 billion in Federal R&D funding reported by
industrial performers.[^26] NSF is examining the causal factors of these
divergent trends.

####### SURVEY DEFINITIONS

######## *Cost per R&D scientist or engineer---*The arithmetic mean of the numbers of full-time equivalent (FTE) scientists and engineers engaged in the performance of R&D reported for January in two consecutive years divided into the total R&D expenditures of the earlier year, with the ratio attributed to the earlier year. For example, the mean of the numbers of FTE R&D scientists and engineers in January 1993 and January 1994 is divided into total 1993 R&D expenditures for a total cost per R&D scientist or engineer in 1993. 

######## *Employment, FTE R&D scientists and engineers*---Persons employed by the company during the January following the survey year who are engaged in scientific or engineering work at a level that requires knowledge of physical, life, engineering, or mathematical science equivalent at least to that acquired through completion of a 4-year college program with a major in one of those fields. The statistics in this report show the FTE employment. FTE employment is the number of scientists and engineers in the company who are assigned full time plus a prorated number employees working part-time on R&D. 

######## *Employment, total*---Number of persons domestically employed by R&D-performing companies in all activities during the pay period that includes the 12th of March.

*Federally funded R&D centers (FFRDCs)*---R&D-performing organizations
administered by industrial, educational, or other institutions on a
nonprofit basis, exclusively or substantially financed by the Federal
Government. R&D expenditures of the FFRDCs that are industry
administered are included with the Federal R&D data of the industry
classification of each of the administering firms. The
industry-administered FFRDCs included in the 1994 survey are listed as
follows.

FFRDCs supported by the Department of Energy:

Bettis Atomic Power Laboratory

Westinghouse Electric Corp.

West Mifflin, PA

Energy Technology Engineering Center

Rockwell International Corp.

Canoga Park, CA

Hanford Engineering Development Laboratory

Westinghouse-Hanford Corp.

Richland, WA

Idaho National Engineering Laboratory

EG&G Idaho, Inc.;

Westinghouse Electric Corp.

Argonne National Laboratory, West;

Rockwell International Corp.;

Idaho Falls, ID

Knolls Atomic Power Laboratory

General Electric Co.

Schenectady, NY

Oak Ridge National Laboratory

Martin Marietta Energy Systems, Inc.

Oak Ridge, TN

Sandia National Laboratories

Western Electric Co., Inc.---Sandia Corp.

Albuquerque, NM

Savannah River Laboratory

Westinghouse Electric Corp.

Aiken, SC

FFRDCs supported by the Department of Health and Human Services,
National Institutes of Health:

NCI Frederick Cancer Research Facility

Program Resources, Inc.

Frederick, MD

*Funds for R&D, company (and other)*---Cost of R&D actually performed
within the company and funded by the company itself or by other
non-Federal sources by contract, not including the cost of R&D supported
by companies but contracted to outside organizations such as research
institutions, universities and colleges, nonprofit organizations, or (to
avoid double-counting) other companies.

######## *Funds for R&D, Federal*---Receipts for R&D performed by the company under Federal R&D contracts or subcontracts and R&D portions of Federal procurement contracts and subcontracts. 

######## *Funds for R&D, total*---Operating expenses incurred by a company in the conduct of R&D in its own laboratories or other company-owned or -operated facilities including wages and salaries, materials and supplies, property and other taxes, maintenance and repairs, depreciation, and an appropriate share of overhead, not including capital expenditures. 

######## *Net sales and receipts*---Dollar values for goods sold or services rendered by R&D-performing companies to customers (outside the company), including the Federal Government, less such items as returns, allowances, freight, charges, and excise taxes. Domestic intracompany transfers and sales by foreign subsidiaries are excluded, but transfers to foreign subsidiaries and export sales to foreign companies are included. 

######## *Research and development*---Basic and applied research in the sciences and engineering and the design and development of prototypes and processes, excluding quality control, routine product testing, market research, sales promotion, sales service, other nontechnological activities or routine technical services, and research in the social sciences or psychology. 

######## *Basic research*---Original investigations for the advancement of scientific knowledge not having specific immediate commercial objectives, although such investigations may be in fields of present or potential interest to the reporting company. 

######## *Applied research*---Investigations for the discovery of new scientific knowledge having specific commercial objectives with respect to products or processes. (Applied research differs from basic research chiefly in terms of the objectives of the reporting company.) 

######## *Development*---Technical activities not routine in nature concerned with translating research findings or other scientific knowledge into products or processes. Not included are routine technical services to customers or other activities excluded above.

[^1]: Information for this section was provided by the Manufacturing and
    Construction Division of the Bureau of the Census, the collecting
    and compiling agent for the National Science Foundation. Copies of
    the technical papers cited can be obtained by contracting NSF's
    Research and Development Statistics Program in the Division of
    Science Resources Studies.

[^2]: See the Bureau of the Census technical memorandum entitled
    Evaluation of Total Employment Cut-Offs in the Survey of Industrial
    Research and Development, Nov 3, 1994.

[^3]: These industries are listed and discussed under Comparability of
    Statistics, later in this section.

[^4]: U.S. Department of Commerce, Bureau of the Census, *Documentation
    of Nonsampling Issues in the Survey of Industrial Research and
    Development,* RR94/03 (Washington, DC, Sept. 1994) and U.S.
    Department of Commerce, Bureau of the Census, *A Study of Processing
    Errors in the Survey of Industrial Research and Development,*
    ESMD-9403 (Washington, DC, Sept. 1994).

[^5]: For detailed discussions on the sources, control, and measurement
    of error resulting from item nonresponse, see the technical report:
    U.S. Department of Commerce, Bureau of the Census, *Documentation of
    Nonsampling Error Issues in the Survey of Industrial Research and
    Development,* RR94/03 (Washington, DC, Sept. 21, 1994). For a
    general discussion of the problems stemming from item nonresponse,
    see the technical report: National Science Foundation, *Estimating
    Basic and Applied Research and Development in Industry: A
    Preliminary Review of Survey Procedures,* NSF 90-322 (Washington,
    DC, 1990).

[^6]: All but four items---total R&D, Federal R&D, net sales, and total
    employment, which are included in the Census Bureau's annual
    mandatory statistical program---are voluntary. See further
    discussion under Response Rates and Mandatory Versus Voluntary
    Reporting, later in this section.

[^7]: For detailed descriptions and analyses of the imputation methods
    and algorithms used, see the technical report: U.S. Department of
    Commerce, Bureau of the Census, *An Evaluation of Imputation Methods
    for the Survey of Industrial Research and Development,* ESMD-9404
    (Washington, DC, Sept. 1994).

[^8]: See the NSF technical report cited previously for an explanation
    of the uncertainties in the data and to quantify their sensitivity
    to the choice of various possible imputation procedures.

[^9]: See also the technical paper U.S. Department of Commerce, Bureau
    of the Census, *Documentation of the Survey Design for the Survey of
    Industrial Research and Development: A Historical Perspective*
    (Washington, DC, 1995).

[^10]: The effects of recent changes in the way companies are classified
    during survey processing are discussed in detail in a Bureau of the
    Census technical memoranda entitled "Reclassification of Companies
    in the 1992 Survey of Industrial Research and Development for the
    Generation of the 'Analytical' Series" Oct. 25, 1994 and "Comparison
    of Company Coding Between 1992 and 1993 for the Survey of Industrial
    Research and Development" Nov. 3, 1994.

[^11]: U.S. Department of Commerce, Bureau of the Census, *Documentation
    of the Survey Design for the Survey of Industrial Research and
    Development: A Historical Perspective* (Washington, DC, 1995).

[^12]: See also National Science Foundation, SRS Data Brief, "1992 R&D
    Spending by U.S. Firms Rises, NSF Survey Improved" (NSF 94-325),
    (Arlington, VA, Sept. 9, 1994).

[^13]: During the early years of the survey, until 1967, samples were
    selected every 5 years. Subsequent samples were selected for 1971,
    1976, 1981, and 1987.

[^14]: For the 1992 survey, 25 new nonmanufacturing industry and
    industry groups were added to the sample frame: agricultural
    services (SIC 07); fishing, hunting, and trapping (09); wholesale
    trade---nondurables (51); stationery and office supply stores
    (5112); industrial and personal service paper (5113); groceries and
    related products (514); chemicals and allied products (516);
    miscellaneous nondurable goods (519); home furniture, furnishings,
    and equipment stores (57); radio, TV, consumer electronics, and
    music stores (573); eating and drinking places (581); miscellaneous
    retail (59); nonstore retailers (596); real estates (65); holding
    and other investment offices (67); hotels, rooming houses, camps,
    and other lodging places (70); automotive repair, services, and
    parking (75); miscellaneous repair services (76); amusement and
    recreation services (79); health services (80); offices and clinics
    of medical doctors (801); offices and clinics of other health
    practitioners (804); miscellaneous health and allied services not
    elsewhere classified (809); engineering, accounting, research,
    management, and related services (87); and management and public
    relations services (874).

[^15]: Annual sampling also remedies the cyclical deterioration of the
    statistics that results from changes in a company's payroll
    composition because of product line and corporate structural
    changes.

[^16]: For discussions of each of these, see the Bureau of the Census
    technical memorandum entitled, Wedging Considerations for the 1992
    Research and Development (R&D) Survey, June 10, 1994.

[^17]: See the Bureau of the Census technical memoranda entitled,
    Reclassification of Companies in the 1992 Survey of Industrial
    Research and Development (R&D) for the Generation of the Analytical
    Series, Oct. 25, 1994, and Effects of the 1987 SIC Revision on
    Company Classification in the Survey of Industrial Research and
    Development (R&D), Dec. 6, 1993.

[^18]: See U.S. Department of Commerce, Bureau of the Census, *Survey
    Design of the Survey of Industrial Research and Development: A
    Historical Perspective* (Washington, DC, 1995).

[^19]: The process was dubbed wedging because of the wedgelike area
    produced on a graph that compares originally reported statistics
    with the revised statistics that result after linking.

[^20]: Bureau of the Census technical memorandum, Wedging Considerations
    for the 1992 Research and Development (R&D) Survey, June 10, 1994.

[^21]: U.S. Department of Commerce, Bureau of the Census, *Survey Design
    of the Survey of Industrial Research and Development: A Historical
    Perspective* (Washington, DC, 1995).

[^22]: See also NSF, *Federal Funds for Research and Development: Fiscal
    Years 1994--96,* NSF 97-302 (Arlington, VA, 1997).

[^23]: See NSF, *Federal R&D Funding by Budget Function: Fiscal Years
    1994--96 (Budget Function),* NSF 95-342 (Arlington, VA, 1995).

[^24]: NSF, *National Patterns of R&D Resources: 1996,* NSF 96-333
    (Arlington, VA, 1996).

[^25]: Ibid.

[^26]: Ibid.