TECHNICAL NOTES FOR 1997

SURVEY METHODOLOGY[^1]

####### Reporting Unit 

The reporting unit for the Survey of Industrial Research and Development
is the company, defined as a business organization of one or more
establishments under common ownership or control. The survey includes
two groups of enterprises: (1) companies known to conduct R&D, and (2) a
sample representation of companies for which information on the extent
of R&D activity is uncertain.

######### Frame Creation 

The Standard Statistical Establishment List, a Bureau of the Census
compilation that contains information on more than 3 million
establishments with paid employees, was the target population from which
the frame used to select the 1997 survey sample was created (see table
B-1 for target population and sample sizes). For companies with more
than one establishment, data were summed to the company level. Each firm
was then assigned a single SIC code based on the activity of the
establishment having the highest dollar value of payroll. This
assignment was done on a hierarchical basis. The enterprise was first
assigned to the economic division (manufacturing or nonmanufacturing)
with the highest payroll, then to the 2-digit SIC code with the highest
payroll within the assigned division, then to the 3-digit SIC code with
the highest payroll within the assigned 2-digit industry.

The frame from which the survey sample was drawn included all for-profit
companies classified in nonfarm industries. For surveys prior to 1992,
the frame was limited to companies above certain size criteria based on
number of employees.[^2] These criteria varied by industry. Some
industries were excluded from the frame because it was believed that
they contributed little or no R&D activity to the final survey
estimates. For the 1992 sample, new industries were added to the
frame,[^3] and the size criteria were lowered considerably and applied
uniformly to firms in all industries. As a result, nearly 2 million
enterprises with 5 or more employees were given a chance of selection.
For comparison, the frame for the 1987 sample included 154,000 companies
of specified sizes and industries. The frame used to select the 1997
sample was similar to those used to select the 1992--96 samples.

######## Defining Sampling Strata 

A fundamental change initiated in 1995 and repeated for the 1996 and
1997 samples was the redefinition of the sampling strata. For the survey
years 1992--94, 165 sampling strata were established, each stratum
corresponding to one or more 3-digit-level SIC codes. The objective was
to select sufficient representation of industries to determine whether
alternative or expanded publication levels were warranted. For the 1997
survey, the strata were defined to correspond to publication-level
industry aggregations, and companies were assigned to strata based on
their 3-digit SIC codes. A total of 40 such levels were defined,
corresponding to the original 25 groupings of manufacturing industries
used as strata in sample designs before 1992 and to 15 new groupings of
nonmanufacturing industries.

######## Identifying Certainty Companies 

The criteria for identifying companies selected with certainty for the
survey have been modified since 1994. With a fixed total sample size,
there was some concern that representation of the very large
noncertainty universe by a smaller sample each year would be inadequate.
Before 1994, companies with 1,000 or more employees had been selected
with certainty, but it was observed that the level of spending varied
considerably and that many of these companies reported no R&D
expenditures each year. Beginning in 1995, these companies were thus
given chances of selection based only upon the size of their R&D
spending if they were in the previous survey or upon an estimated R&D
value if they were not. To further limit the growth in the number of
certainty cases occurring each year, the certainty criterion---the size
of their R&D spending---was raised for the 1996 survey from \$1 million
to \$5 million; it remained at that level for the 1997 survey.

######## Frame Partitioning 

The partitioning of the frame into "large" and "small" company
components and the use of simple random sampling (SRS) for the small
company partition were first introduced in the 1994 survey. A study of
1992 survey results showed that a disproportionate number of small
companies were being selected for the sample, often with very large
weights. These small companies seldom reported R&D activity. This
disproportion was a result of the minimum probability rule (discussed
later) used as part of the independent probability proportionate to size
(PPS) sampling procedure, which was the only procedure used prior to
1994. This rule increased the probability of selection for several
hundred thousand of these smaller companies. With SRS, these smaller
companies can be sampled more efficiently than with independent PPS
sampling since there is little variability in their size.

For 1995, total company payroll was the basis for the split between
large and small partitions. For each industry grouping, the largest
companies representing the top 90 percent of the total payroll were
included in the PPS frame. The balance of smaller companies comprising
the remaining 10 percent of payroll for the industry grouping were
included in the SRS frame. A benefit of this design change was a
reduction in the maximum allowable weight for selected companies
(weighting and maximum weights are discussed later).

For 1996 and 1997, total company employment was the basis for
partitioning the frame. The total company employment levels defining the
partitions were based on the relative contribution to total R&D
expenditures of companies in different employment size groups in both
the manufacturing and nonmanufacturing sectors. In the manufacturing
sector, all companies with total employment of 50 or more were included
in the large company partition. In the nonmanufacturing sector, all
companies with total employment of 15 or more were included in the large
company partition. Companies in the respective sectors with employment
below these values were included in the small company partition. In the
1997 survey, the large company partition contained about 540,000
companies; the small company partition contained about 1.3 million
companies. These counts were comparable to those in the 1995 (656,000
and 1.2 million, respectively) and 1996 (560,000 and 1.3 million,
respectively) partitions.

######## Identifying "Zero" Industries 

One final modification in frame development for 1996, which was repeated
for 1997, was the designation of "zero" industries in the large company
partition. Zero industries were those 3-digit SIC industries having no
R&D expenditures reported in the survey years 1992--94 (the years when
estimates by 3-digit SIC industry were formed). It was decided to keep
these industries in the scope of the survey but to draw only a limited
sample from them, since it seemed unlikely that R&D expenditures would
be reported. SRS was used to control the number of companies selected
within these industries.

####### Sample Selection 

A significant revision in the procedure for selecting samples from the
partitions changed the development and presentation of estimates from
the 1996 survey; this approach was repeated for the 1997 survey. A
sample of companies in the large company partition was selected using
PPS sampling in each of the 40 strata as in 1995. The sample of
companies in the small company partition was selected using SRS in only
2 strata rather than 40 as in 1995. Companies classified in
manufacturing industries were selected to represent the group of all
manufacturing industries rather than each manufacturing industry group.
Similarly, companies classified in nonmanufacturing industries were
selected to represent the group of all nonmanufacturing industries.

The purpose of selecting small companies from only two strata was to
reduce the variability in industry estimates attributed to the random
year-to-year selection of the companies in an industry and the
associated high sampling weights. Consequently, estimates for individual
industry groups are not possible from these two strata.[^4] Statistics
for the detailed industry groups are based only on the sample from the
large company partition. Estimates from the small company partition are
included in statistics for total manufacturing, total nonmanufacturing,
and all industries. For completeness, the estimates also are added to
the categories "other manufacturing" and "other nonmanufacturing."

######## Probability Proportionate to Size 

For 1995, the distribution of companies by payroll and estimated R&D in
the large partition of the sample was skewed as in earlier frames.
Because of this skewness, PPS sampling used in previous designs was an
appropriate selection technique for this group. That is, large companies
had a higher probability of selection than did small companies. It would
have been ideal if company size could have been determined by its R&D
expenditures. Unfortunately, except for the companies that were in a
previous survey or for which there was information from external
sources, it was impossible to know the R&D expenditures for every firm
in the universe. Consequently, the probability of selection for most
companies was based on estimated R&D expenditures.

**Imputing R&D.---**Since total payroll was known for each company in
the universe, it was possible for the 1997 survey to estimate R&D from
payroll using relationships derived from 1996 survey data. Imputation
factors relating these two variables were made for each industry
grouping. To impute R&D for a given company, the imputation factors were
applied to the company payroll in each industry grouping. A final
measure was obtained by adding the industry grouping components. The
effect, in general, was to give firms with large payrolls higher
probabilities of selection in accordance with the assumption that larger
companies were more likely to perform R&D.

Estimated R&D values were computed for companies in the small company
partition as well. The aggregate of reported and estimated R&D from each
company in both the large and small company partitions represented a
total universe measure of 1996 R&D expenditures. However, assigning a
value for R&D to every company resulted in an overstatement of this
measure. To adjust for the overstatement, the universe measure for the
1997 survey was scaled down using factors developed from the
relationship of the frame measure of 1996 R&D and the 1996 survey
estimate. These factors, computed at levels corresponding to published
industry levels, were used to adjust originally imputed R&D values so
that the new frame total for R&D at these levels approximated the 1996
published values. This adjustment provided for better allocation of the
sample among these levels.

######## Simple Random Sampling 

Only two major strata were defined for samples in the small company
partition---manufacturing and nonmanufacturing. The use of SRS implied
that each company within a stratum had an equal probability of
selection. The total sample allocated to the small company partition was
dependent upon the total sample specified for the survey and upon the
total sample necessary to satisfy criteria established for the large
company partition. Once determined, the allocation of this total by
stratum was made proportionate to the stratum's payroll contribution to
the entire partition.

######## Sample Stratification and Relative Standard Error Constraints 

The particular sample selected for each survey year was one of a large
number of the same type and size that by chance might have been
selected. Statistics resulting from the different samples would differ
somewhat from each other. These differences are represented by estimates
of sampling error. The smaller the sampling error, the more precise the
statistic.

**Controlling Sampling Error.---**The large company partition was of
primary concern, since it was believed that nearly all of the R&D
activity would be identified from this sector. To control sampling error
in the statistics resulting from this portion of the frame, parameters
were specified to allocate the sample across various levels, or strata,
that corresponded to the 40 industry groupings discussed earlier. These
parameters determined the sample size required to achieve a desired
level of sampling error for each stratum and were assigned so that
estimated errors of total R&D expenditures for industries in these
strata did not exceed certain levels. Sample sizes among the strata were
constrained only by the limit placed on the total sample size dictated
by the available budget.

**Sampling Strata and Standard Error Estimates.---**The practice, first
implemented in the 1995 survey and continued in the 1996 and 1997
surveys, of establishing sampling strata corresponding to published
industry groupings meant that more efficient samples could be selected
for these groups than had resulted when using the 165-strata design.
Even the expansion of the number of nonmanufacturing publication
groupings resulted in fewer sampling strata. The earlier designs defined
25 strata of 3-digit-SIC manufacturing industries, but published only
one category of nonmanufacturing industry. Beginning with the 1995
sample design, 15 nonmanufacturing strata were defined for sampling and
publication levels. Since there was no mandate in any ensuing year to
make a major reduction in the sample of 17,600 companies for the large
company partition selected under the 165-strata design, it was possible
to establish much tighter relative standard error constraints on the
smaller number of sampling strata. Thus, in 1997, 33 strata were
assigned a relative standard error constraint of 1 percent, while 7
strata were assigned a relative standard error constraint of 0.5
percent. These constraints resulted in an expected sample size of about
8,276 companies from the large company partition. The minimum
probability rule (discussed later) was adjusted so as to raise the
expected sample size closer to the 17,000 level.

A limitation of the sample allocation process for the large partition
should be noted. The sampling errors used to control the sample size in
each stratum are based on a universe total that, in large part, was
improvised. That is, as previously noted, an R&D value was assigned to
every company in the frame, even though most of these companies actually
may not have had R&D expenditures. The value assigned was imputed for
the majority of companies in the frame, and---as a consequence---the
estimated universe total and the distribution of individual company
values, even after scaling, did not necessarily reflect the true
distribution. Estimates of sampling variability were nevertheless based
on this distribution. The presumption was that actual variation in the
sample design would be less than that estimated, because many of the
sampled companies have true R&D values of zero, not the widely varying
values that were imputed using total payroll as a predictor of R&D.
Previous sample selections indicate that in general this presumption
holds, but exceptions have occurred when companies with large sampling
weights have reported large amounts of R&D spending. Thus, in general,
the 1 percent and 0.5 percent error levels described earlier are
conservative. See table B-2 for the actual standard error estimates for
selected items by industry.

For the 1995 small company partition, 40 strata were identified. Also
included was a separate stratum of approximately 6,260 companies that
could not be classified into an SIC code and therefore could not be
assigned to a stratum because of incomplete industry identification in
the Standard Statistical Establishment List. As was done for 1994, a
small number of companies was selected from this group in the hopes that
an accurate industry identification could be obtained at a later point.
The initial sample size specified for the small company partition was
5,500 companies. The sample initially allocated to a given stratum was
proportionate to its share of total payroll or the small partition. For
the 1996 and 1997 small company partitions, two strata (manufacturing
and nonmanufacturing) were identified, and a small number of companies
was selected from the group of unclassifiable companies. For 1997, a
final sample of 6,445 companies was selected from the small company
partition. The sample initially allocated to the two strata was
proportionate to its share of total payroll for the small company
partition.

**Nonsampling Error.---**In addition to sampling error, estimates are
subject to nonsampling error. Errors are grouped in five categories:
specification, coverage, response, nonresponse, and processing. For
detailed discussions on the sources, control, and measurement of each of
these types of error, see U.S. Bureau of the Census (1994b and 1994f).

######## Sample Size 

The target sample size initially specified for the 1997 survey was
25,000 companies and, as described above, was based primarily on
compliance with predetermined sampling error constraints established for
the large partition. The actual sample size was 23,417 companies. The
sample differed from the target for several reasons.

**Independent Sampling.---**First, the frame for the large company
partition was subjected to independent sampling. Each company in the
frame had an independent chance of selection based on its assigned
probability---i.e., selection of a company was completely independent of
the selection of any other company. In independent (or Poisson)
sampling, sample size itself is a random variable, and the actual sample
size will vary around the target or "expected" sample size.
Theoretically, a sample of size zero or a sample the size of the entire
universe is possible, but the probabilities of these extremes are so
small that these are improbable situations. In strata where the expected
sample size is more than 50, the actual sample probably will be within a
fairly narrow range so that increased variability is not a real problem.
However, in strata where the expected sample is small (i.e., less than
10), gross over- or undersampling of the strata is possible. In
practice, the size of the originally drawn sample is usually quite close
to the specified size. If there is too much deviation, however, the
selection can be repeated until it is closer to the target.

**Minimum Probability Rule.---**Second, a minimum probability rule was
imposed for both partitions in the sample. As noted earlier, for the
large company partition, probabilities of selection proportionate to
size were assigned each company, where size is the reported or imputed
R&D value assigned to each company. Selected companies received a sample
weight that was the inverse of their probability of selection. Selected
companies that ultimately report R&D expenditures vastly larger than
their assigned values can have adverse effects on the statistics, which
are based on the weighted value of survey responses. To lessen the
effects on the final statistics, the maximum weight of a company was
controlled by specifying a minimum probability that could be assigned to
the company. If the probability based on company size was less than the
minimum probability, it was reset to this minimum value. The consequence
of raising these original probabilities to the minimum probability was
to raise the expected sample size. Similarly, a maximum weight for each
stratum was established for the SRS of the small partition. If the
sample size initially allocated to a stratum resulted in a stratum
weight above this maximum value, then the sample size was increased
until the maximum weight was achieved. It is likely that most of the
difference between the size of the target sample and the sample actually
selected was due to the minimum probability rule.

**Changes in Operational Status.---**Third, between the time the frame
was created and the survey was prepared for mailing, the operational
status of some companies changed. That is, they were merged with or
acquired by another company, or they were no longer in business. Before
preparing the survey for mailing, the operational status was updated to
identify these changes. As a result, the number of companies mailed a
survey form was somewhat smaller than the number of companies initially
selected for the survey.

######## Weighting and Maximum Weights 

Weights were applied to each company record to produce national
estimates. Within the PPS partitions of the sample, company records were
given weights up to a maximum of 50; for companies within the SRS
partitions, company records were given weights up to a maximum of 250.

####### Survey Questionnaires 

Two questionnaires are used each year to collect data for the survey:
Form RD-1 in two versions for large firms, and Form RD-1A for small
firms and first-time reporters. For large firms known to perform R&D, a
detailed questionnaire, Form RD-1L, is used to collect data for
odd-numbered years; an abbreviated version, Form RD-1S, is used to
collect data for even-numbered years. The questionnaires are cycled in
this manner to reduce the reporting burden on survey respondents.

######## Questionnaire Content 

Form RD-1L, which was used for the 1997 survey, requests data on sales
or receipts; total employment; employment of scientists and engineers;
expenditures for R&D performed within the company with Federal funds and
with company and other funds; character of work (basic research, applied
research, and development); company-sponsored R&D expenditures in
foreign countries; R&D performed under contract by others; expenditures
for pollution abatement and energy R&D; detail on R&D by product field;
Federal R&D support to the firm by contracting agency; domestic R&D
expenditures by state; and foreign R&D by country. Form RD-1S requests
the same information except the last four items. Because companies
receiving Forms RD-1L and RD-1S generally have participated in previous
surveys, computerized imprinted data reported by the company for the
previous year are supplied for reference.

To further limit reporting burden on small R&D performers and firms that
are included in the sample for the first time, an even more abbreviated
form is used each year. Form RD-1A collects data only on R&D, sales,
employment, and operational status and includes a screening item that
allows respondents to indicate that they do not perform R&D before
completing the questionnaire. No prior year information is available
since the majority of the companies have not reported previously.

**New Items.---**Beginning in 1996, the collection of data on R&D
performed under contract by others was expanded. Previously, data were
collected only on nonfederally funded R&D performed under contract by
others. In 1996, data on contracted-out federally funded and total R&D
were collected to better measure the amount of R&D performed both within
and between companies.

######## Number of Questionnaires Sent 

For the 1997 survey, approximately 2,900 companies, including those that
reported \$5 million or more in R&D expenditures in the 1996 survey,
received Form RD-1L; and approximately 20,500 received Form RD-1A. Of
the 23,400 firms, about 3,700 reported R&D expenditures for 1997.

####### Followup for Survey Nonresponse 

The 1997 survey questionnaires were mailed in March 1998, and recipients
were asked to respond within 60 days. Thirty days later, letters were
mailed to all survey recipients reminding them that their completed
questionnaire was due within the next 30 days. After 60 days, followup
letters were sent to all firms that did not respond. Three additional
followup mailings were made to persistent nonrespondents after 90, 120,
and 150 days.

In addition to the mailings, telephone followup was used to encourage
response from those firms ranked among the 300 largest R&D performers,
based on total R&D expenditures reported in the previous survey. Table
B-3 shows the number of companies in each industry or industry group
that received a questionnaire and the percentage of companies that
responded to the survey.

######### Imputation for Item Nonresponse 

For various reasons, many firms chose to return the survey
questionnaires with one or more blank items.[^5] For instance, the
internal accounting procedures of the firm may not have allowed it to
quantify the character of work distribution of R&D. In addition, some
firms refused as a matter of policy to answer any voluntary
questions.[^6] When respondents did not provide the requested
information, estimates for the missing data were made using imputation
algorithms. In general, the imputation algorithms computed values for
missing items by applying the average percentage change for the target
item in the nonresponding firm's industry to the item's prior year value
for that firm, reported or imputed. This approach, with minor variation,
was used for most items.[^7] Table B-4 contains imputation rates for the
principal survey items.

######### Response Rates and Mandatory Versus Voluntary Reporting 

Current survey reporting requirements divide survey items into two
groups: mandatory and voluntary. Response to four data items on the
questionnaires is mandatory; response to the remaining items is
voluntary. During the 1990 survey cycle, NSF conducted a test of the
effect of reporting on a completely voluntary basis to determine if
combining both mandatory and voluntary items on one questionnaire
influences response rates. For this test, the 1990 sample was divided
into two panels of approximately equal size. One panel---the mandatory
panel---was asked to report as usual on four mandatory items with the
remainder voluntary; the other panel was asked to report all items on a
completely voluntary basis. The result of the test was a decrease in the
overall survey response rate to 80 percent from levels of 88 percent in
1989 and 89 percent in 1988. The response rates for the mandatory and
voluntary panels were 89 and 69 percent, respectively. Detailed results
of the test were published in the annual report resulting from the 1990
survey.[^8] For firms that reported R&D expenditures in 1997, table B-5
shows the percentage that also reported data for other selected items.

**Character of Work Estimates**

Response to questions about character of work (basic research, applied
research, and development) declined in the mid-1980s, and, as a result,
imputation rates increased. The general imputation procedure described
above became increasingly dependent upon information imputed in prior
years, thereby distancing current-year estimates from any reported
information. Because of the increasing dependence on imputed data, NSF
chose not to publish character of work estimates in 1986. The imputation
procedure used to develop these estimates was revised in 1987 for use
with later data and differs from the general imputation approach. The
new method calculated the character of work distribution for a
nonresponding firm only if that firm reported a distribution within a
5-year period, extending from 2 years before to 2 years after the year
requiring imputation. Imputation for a given year was initially
performed in the year the data were collected and was based on a
character of work distribution reported in either of the 2 previous
years, if any. It was again performed using new data collected in the
next 2 years. If reported data followed no previously imputed or
reported data, previous period estimates were inserted based on the
currently reported information. Similarly, if reported data did not
follow 2 years of imputed data, the 2 years of previously imputed data
were removed. Thus, character of work estimates were revised as newly
reported information became available and were not final for 2 years
following their initial publication.

Beginning with 1995, previously estimated values were not removed for
firms that did not report in the third year, nor were estimates made for
the 2 previous years for firms reporting after 2 years of nonresponse.
This process was changed because, in the prior period, revisions were
minimal. Estimates continue to be made for 2 consecutive years of
nonresponse and discontinued if the firm does not report character of
work in the third year.

If no reported data are available for a firm, character of work
estimates are not imputed. As a consequence, only a portion of the total
estimated R&D expenditures are distributed at the firm level. Those
expenditures not meeting the requirements of the new imputation
methodology are placed in a "not distributed" category. Tables B-6, B-7,
and B-8 show the character of work estimates along with the "not
distributed" component for 1995, 1996, and 1997, respectively. NSF's
objective in conducting the survey has always been to provide estimates
for the entire population of firms performing R&D in the United States.
However, the revised imputation procedure would no longer produce such
estimates because of the "not distributed" component. A baseline
estimation method was thus developed to allocate the "not distributed"
amounts among the character of work components. In the baseline
estimation method, the "not distributed" expenditures are allocated by
industry group to basic research, applied research, and development
categories using the percentage splits in the distributed category for
that industry. The allocation is done at the lowest level of published
industry detail only; higher levels are derived by aggregation, just as
national totals are derived by aggregation of individual industry
estimates, and result in higher performance shares for basic and applied
research and lower estimates for development's share than would have
been calculated using the previous method.[^9]

######### State Estimates 

Form RD-1L requests data on the cost of R&D performed for States in
which a company's R&D laboratories are located. An independent source,
the *Directory of American Research and Development,* published by the
Data Base Publishing Group of the R. R. Bowker Company, is used in
conjunction with survey results to estimate R&D expenditures by State
for companies that do not provide this information. The information on
scientists and engineers published in the directory is used as a proxy
indicator of the proportion of R&D expenditures within each State. R&D
expenditures by State are estimated by applying the distribution of
scientists and engineers by State from the directory to total R&D
expenditures for these companies. These estimates are included with
reported survey data to arrive at published estimates of R&D
expenditures for each State.

######### COMPARABILITY OF STATISTICS 

This section summarizes survey procedures and practices that may have
affected the comparability of statistics produced from the Survey of
Industrial Research and Development over time and with other statistical
series.[^10]

####### Revisions to Historical and Immediate Prior Year Statistics 

Revisions to historical statistics usually have been made because of
changes in the industry classification of companies caused, in turn, by
changes in payroll composition detected when a new sample was drawn.
Various methodologies have been adopted over the years to revise, or
backcast, the data when revisions to historical statistics have become
necessary. Documented revisions to the historical statistics from
post-1967 surveys are summarized in NSF (1994). Detailed descriptions of
the specific revisions made to the statistics from pre-1967 surveys are
scarce, but U.S. Bureau of the Census (1995) summarizes some of the
major revisions.

Changes to reported data can come from three sources: respondents,
analysts involved in survey and statistical processing, and the industry
reclassification process. Prior to 1995, routine revisions were made to
prior year statistics based on information from all three sources.
Consequently, results from the current-year survey were used not only to
develop current-year statistics, but also to revise immediate prior year
statistics. Beginning with the 1995 survey, this practice was
discontinued. The reasons for discontinuation of this practice were
annual sampling, continual strengthening of sampling methodology, and
improvements in data verification, processing, and nonresponse
follow-up. Moreover, it is not clear that respondents or those who
processed the survey results had any better information a year after the
data were first reported. Thus, it was determined that routinely
revising published survey statistics increased the potential for error
and often confused users of the statistics. Revisions are now made to
historical and immediate prior-year statistics only if egregious errors
are discovered.

####### Year-to-Year Changes 

Comparability from year to year may be affected by new sample design,
annual sample selection, and industry shifts.

######## Sample Design 

By far the most profound influence on statistics from recent surveys
occurred when the new sample design for the 1992 survey was introduced.
Revisions to the 1991 statistics were dramatic (see *Research and
Development in Industry: 1992* for a detailed discussion). While the
allocation of the sample was changed somewhat, the sample designs used
for the 1993--97 surveys were comparable to the 1992 sample design in
terms of size and coverage.

######## Annual Sample Selection 

With the introduction of annual sampling in 1992, more year-to-year
change has resulted than when survey panels were used. There are two
reasons why this is so. First, changes in classification of companies
not surveyed were not reflected in the year-to-year movement. Prior to
annual sampling, a wedging operation---which was performed when a new
sample was selected---was a means of adjusting the data series to
account for the changes in classification that occurred in the frame
(see the discussion on wedging later). Second, yearly correlation of R&D
data is lost when independent samples are drawn each year.

######## Industry Shifts 

The industry classification of companies is redefined each year with the
creation of the sampling frame. By redefining the frame, the sample
reflects current distributions of companies by size and industry. A
company may move from one industry to another due to either changes in
its payroll composition, which is used to determine the industry
classification code (see previous discussion under "Frame Creation");
changes in the industry classification system itself; or changes in the
way the industry classification code is assigned or revised during
survey processing.

A company's payroll composition can change because of the growth or
decline of product or service lines, the merger of two or more
companies, the acquisition of one company by another; divestitures; or
the formation of conglomerates. Although an unlikely occurrence, a
company's industry designation can be reclassified yearly with the
introduction of annual sampling in 1992. The result is that a downward
movement in R&D expenditures in one industry is balanced by an upward
movement in another industry from one year to the next.

From time to time, the SIC coding system, which is used by most Federal
agencies that publish industry statistics, is revised to reflect the
changing composition of U.S. industry. For statistics developed for
1988--91 from the 1988--91 surveys, companies retained the industry
classifications assigned for the 1987 sample. These classifications were
based on the 1977 SIC system. The last major revision of the SIC system
was in 1987. This new system has been used to classify companies in all
of the post-1991 surveys.

The method used to classify firms during survey processing was revised
slightly in 1992. Research has shown that the impact on individual
industry estimates has been minor.[^11] The current method used to
classify firms was discussed previously under "Frame Creation." Methods
used for past surveys are discussed in U.S. Bureau of the Census (1995).

####### Capturing Small and Nonmanufacturing R&D Performers[^12]

Before the 1992 survey, the sample of firms surveyed was selected at
irregular intervals.[^13] In intervening years, a panel of the largest
firms known to perform R&D was surveyed. For example, a sample of about
14,000 firms was selected for the 1987 survey. For the 1988--91 studies,
about 1,700 of these firms were resurveyed annually; the other firms did
not receive questionnaires, and their R&D data were estimated. This
sample design was adequate during the survey's early years because R&D
performance was concentrated in relatively few manufacturing industries.
However, as more and more firms began entering the R&D arena, the old
sample design proved increasingly deficient because it did not capture
births of new R&D-performing firms. The entry of fledgling R&D
performers into the marketplace was completely missed during panel
years. Additionally, beginning in the early 1970s, the need for more
detailed R&D information for nonmanufacturers was recognized. At that
time, the broad industry classifications "miscellaneous business
services" and "miscellaneous services" were added to the list of
industry groups for which statistics were published. By 1975, about 3
percent of total R&D was performed by firms in nonmanufacturing
industries.

During the mid-1980s, there was evidence that a significant amount of
R&D was being conducted by an increasing number of nonmanufacturing
firms; again, the number of industries used to develop the statistics
for nonmanufacturers was increased. Consequently, since 1987 the annual
reports in this series have included separate R&D estimates for firms in
the communication, utility, engineering, architectural, research,
development, testing, computer programming, and data processing service
industries; hospitals; and medical labs. Approximately 9 percent of the
estimated industrial R&D performance during 1987 was undertaken by
nonmanufacturing firms.

After the list of industries for which statistics were published was
expanded, it became clear that the sample design itself should be
changed to reflect the widening population of R&D performers among firms
in the nonmanufacturing industries[^14] and small firms in all
industries so as to account better for births of R&D-performing firms
and to produce more reliable statistics. Beginning with the 1992 survey,
NSF decided to (1) draw new samples with broader coverage annually, and
(2) increase the sample size to approximately 25,000 firms.[^15] As a
result of the sample redesign, for 1992 the reported nonmanufacturing
share was and continues to be estimated at approximately 25 percent of
total R&D.[^16]

######### Time-Series Analyses 

As discussed earlier, the statistics resulting from this survey are
better indicators of changes in, rather than absolute levels of, R&D
spending and personnel. Nevertheless, the statistics are often taken to
be a continuous time series prepared using the same collection,
processing, and tabulation methods. Such uniformity has not been the
case. Since the survey was first fielded, improvements have been made to
increase the reliability of the statistics and to make the survey
results more useful. To that end, past practices have been changed and
new procedures instituted. Preservation of the comparability of the
statistics has, however, been an important consideration in making these
improvements. Nonetheless, changes to survey definitions, the industry
classification system, and the procedure used to assign industry codes
to multi-establishment companies have had some, though not substantial,
effects on the comparability of statistics.[^17]

The aspect of the survey that had the greatest effect on comparability
was the selection of samples at irregular intervals (i.e., 1967, 1971,
1976, 1981, 1987, and 1992) and the use of a subset or panel of the last
sample drawn to develop statistics for intervening years. As discussed
earlier, this practice introduced cyclical deterioration of the
statistics. As compensation for this deterioration, periodic revisions
were made to the statistics produced from the panels surveyed between
sample years. Early in the survey's history, various methods were used
to make these revisions.[^18] After 1976 and until the 1992 advent of
annual sampling, a linking procedure called wedging was used.[^19] In
wedging, the 2 sample years on each end of a series of estimates served
as benchmarks in the algorithms used to adjust the estimates for the
intervening years.

For a full discussion of the mathematical algorithm used for the wedging
process that linked statistics from the 1992 survey with those from the
1987 survey, see U.S. Bureau of the Census (1994g). In general,
wedging---

> *takes full advantage of the fact that in the first year of a new
> panel \[when a new sample is selected\], both current-year and
> prior-year estimates are derived. Thus, two independent estimates
> exist for the prior year. The estimates from the new panel are treated
> as superior primarily because the new panel is based on updated
> classifications \[the industry classifications in the prior panel are
> frozen\] and is more fully representative of the current universe (the
> prior panel suffers from panel deterioration, especially a lack of
> birth updating). The limitations in the prior panel caused by these
> factors are naturally assumed to increase with time, so that in the
> revised series, we desire a gradual increase in the level or revision
> over time which culminates in the real difference observed between the
> two independent sample estimates of the prior year. At the same time,
> we desire that the annual movement of the original series be preserved
> to the degree possible in the revised series (U.S. Bureau of the
> Census, 1994).*

To that end, the wedging algorithm does not change estimates from sample
years and adjusts estimates from panel years, recognizing that
deterioration of the panel is progressive over time. One of the primary
reasons for deciding to select a new sample annually rather than at
irregular intervals was to avoid applying global revision processes such
as wedging. Consequently, the 1992 survey was intended to be the last
one affected by the wedging procedure.

######### Comparisons to Other Statistical Series 

NSF collects data on federally financed R&D from both Federal funding
agencies---using the Survey of Federal Funds for Research and
Development---and performers of the work, industry, Federal labs,
universities, and other nonprofit organizations---using the Survey of
Industrial Research and Development and other surveys. As reported by
Federal agencies, NSF publishes data on Federal R&D budget authority and
outlays, in addition to Federal obligations. These terms are defined
below:[^20]

-   *Budget authority* is the primary source of legal authorization to
    enter into financial obligations that will result in outlays. Budget
    authority most commonly is granted in the form of appropriations
    laws enacted by Congress with the approval of the president (NSF
    1998b).

-   *Obligations* represent the amounts for orders placed, contracts
    awarded, services received, and similar transactions during a given
    period, regardless of when the funds were appropriated or when
    future payment of money is required.

-   *Outlays* represent the amounts for checks issued and cash payments
    made during a given period, regardless of when the funds were
    appropriated or obligated.

National R&D expenditure totals in NSF's *National Patterns of R&D
Resources* report series are primarily constructed with data reported by
performers and include estimates of Federal R&D funding to these
sectors. But until performer-reported survey data on Federal R&D
expenditures are available from industry and academia, data collected
from the Federal agency funders of R&D are used to project R&D
performance. When survey data from the performers subsequently are
tabulated, as they are in this report, these statistics replace the
projections based on funder expectations. Historically, the two survey
systems have tracked fairly closely. For example, in 1980, performers
reported using \$29.5 billion in Federal R&D funding, and Federal
agencies reported total R&D funding between \$29.2 billion in outlays
and \$29.8 billion in obligations (NSF 1996b). In recent years, however,
the two series have diverged considerably. The difference in the Federal
R&D totals appears to be concentrated in funding of industry, primarily
aircraft and missile firms, by the Department of Defense. Overall,
industrial firms have reported significant declines in Federal R&D
support since 1990 (see table A-1), while Federal agencies have reported
level or slightly increased funding of industrial R&D (NSF 1996b). NSF
is examining the causal factors behind these divergent trends.

######### SURVEY DEFINITIONS 

*R&D funds per R&D scientist or engineer*---All expenditures associated
with the performance of industrial R&D (salaries, wages, and fringe
benefits paid to R&D scientists and engineers; materials and supplies
used for R&D; depreciation on capital equipment and facilities used for
R&D; and any other R&D costs) divided by the number of R&D scientists
and engineers employed. To obtain a per person expenditures on R&D for a
given year, the total R&D expenditures of that year are divided by an
approximation of the number of full-time equivalent (FTE) scientists and
engineers engaged in the performance of R&D for that year. For accuracy,
this approximation is the mean of the numbers of such FTE R&D-performing
scientists and engineers as reported in January for the year in question
and the subsequent year. For example, the mean of the numbers of FTE R&D
scientists and engineers in January 1996 and January 1997 is divided
into total 1996 R&D expenditures for a total cost per R&D scientist or
engineer in 1996.

*Employment, FTE R&D scientists and engineers*---People employed by the
company during the January following the survey year who are engaged in
scientific or engineering work at a level that requires knowledge of
engineering or of the physical, biological, mathematical, statistical,
or computer sciences equivalent at least to that acquired through
completion of a 4-year college program with a major in one of those
fields. FTE employment is the number of scientists and engineers in the
company who are assigned full time plus a prorated number of employees
working part time on R&D.

*Employment, total*---Number of people domestically employed by
R&D-performing companies in all activities during the pay period that
includes the 12th of March, the date most employers use when paying
first quarter employment taxes to the Internal Revenue Service.

*Federally funded R&D centers (FFRDCs)*---R&D-performing organizations
administered by industrial, educational, or other institutions on a
nonprofit basis, and exclusively or substantially financed by the
Federal Government. R&D expenditures of industry-administered FFRDCs are
included with the Federal R&D data of the industry classification of
each of the administering firms. The industry-administered FFRDCs
included in the 1997 survey are listed below.

FFRDCs supported by the Department of Energy:

Energy Technology Engineering Center,

Rockwell International Corp.,

Canoga Park, CA

Idaho National Engineering Laboratory,

Lockheed Martin Corp.,

Idaho Falls, ID

Oak Ridge National Laboratory,

Lockheed Martin Corp.,

Oak Ridge, TN

Sandia National Laboratories,

Lockheed Martin Corp.,

Albuquerque, NM

Savannah River Laboratory,

Westinghouse Corp.,

Aiken, SC

FFRDCs supported by the Department of Health and Human Services,
National Institutes of Health:

National Cancer Institute (NCI) Frederick Cancer Research Facility

Science Applications International Corporation

Advanced Bioscience Laboratories, Inc., Frederick, MD

*Funds for R&D, company and other*---The cost of R&D actually performed
within the company and funded by the company itself or by other
non-federal sources, not including the cost of R&D supported by
companies but contracted to outside organizations such as research
institutions, universities and colleges, nonprofit organizations,
or---to avoid double-counting---other companies.

*Funds for R&D, Federal*---Receipts for R&D performed by the company
under Federal R&D contracts or subcontracts and R&D portions of Federal
procurement contracts and subcontracts.

*Funds for R&D, total*---Operating expenses incurred by a company in
conducting R&D in its own laboratories or in other company-owned or
-operated facilities. These expenses include wages and salaries;
materials and supplies; property and other taxes; maintenance and
repairs; depreciation; and an appropriate share of overhead, not
including capital expenditures.

*Industrial R&D*---As used in this survey, R&D is the pursuit of a
planned search for new knowledge or understanding of the subject under
study. This search may have reference to a specific application (basic
research); the acquisition of knowledge or understanding to meet a
specific, recognized need (applied research); or the application of
existing knowledge or understanding toward the improvement of a present
product or process (development). In industry, *basic research* is the
pursuit of new scientific knowledge or understanding that does not have
specific immediate commercial objectives, although it may be in fields
of present or potential commercial interest; *applied research* is
investigation toward discovering new scientific knowledge that has
specific commercial objectives with respect to products, processes, or
services; and *development* is the systematic use of the knowledge or
understanding gained from research directed toward the production of
useful materials, devices, systems, or methods, including the design and
development of prototypes and processes. The survey covers industrial
R&D performed by people trained---either formally or by experience---in
engineering or in the physical, biological, mathematical, statistical,
or computer sciences and employed by a publicly or privately owned firm
engaged in for-profit activity in the United States. Specifically
excluded from the survey are quality control, routine product testing,
market research, sales promotion, sales service, and other
nontechnological activities; routine technical services; and research in
the social sciences or psychology.

*Net sales and receipts*---Dollar values for goods sold or services
rendered by R&D-performing companies to customers outside the
company---including the Federal Government---less such items as returns,
allowances, freight, charges, and excise taxes. Domestic intracompany
transfers and sales by foreign subsidiaries are excluded, but transfers
to foreign subsidiaries and export sales to foreign companies are
included.

######### REFERENCES 

National Science Foundation (NSF). 1956. *Science and Engineering in
American Industry: Final Report on a 1953--54 Survey.* NSF 56-16.
Washington, DC: U.S. Government Printing Office.

\-\--. 1960. *Science and Engineering in American Industry: 1956.* NSF
59-50. Washington, DC: U.S. Government Printing Office.

\-\--. 1990. *Estimating Basic and Applied Research and Development in
Industry: A Preliminary Review of Survey Procedures.* NSF 90-322.
Washington, DC.

\-\--. 1994. "1992 R&D Spending by U.S. Firms Rises, NSF Survey
Improved." SRS Data Brief. NSF 94-325. Arlington, VA.

\-\--. 1995. "1993 Spending Falls for U.S. Industrial R&D,
Nonmanufacturing Share Increases." SRS Data Brief. NSF 95-325.
Arlington, VA.

\-\--. 1996a. "1994 Company Funding of U.S. Industrial R&D Rises as
Federal Support Continues to Decline." SRS Data Brief. NSF 96-310.
Arlington, VA.

\-\--. 1996b. *National Patterns of R&D Resources: 1996.* NSF 96-333.
Arlington, VA.

\-\--. 1997a. "1995 U.S. Industrial R&D Rises, NSF Survey Statistics
Expanded to Emphasize Role of Nonmanufacturing Industries." SRS Data
Brief. NSF 97-332. Arlington, VA.

\-\--. 1997b. *Federal Funds for Research and Development: Fiscal Years
1995--97, Volume 45.* NSF 97-327. Arlington, VA.

\-\--. 1998a. "1996 U.S. Industrial R&D: Firms Continue to Increase
Their Investment." SRS Data Brief. NSF 98-317. Arlington, VA.

\-\--. 1998b. *Federal Research and Development Funding by Budget
Function: Fiscal Years 1996--98.* NSF 98-301. Arlington, VA.

\-\--. 1999. *National Patterns of R&D Resources: 1998.* NSF 99-335.
Arlington, VA.

U.S. Bureau of the Census. 1993. "Effects of the 1987 SIC Revision on
Company Classification in the Survey of Industrial Research and
Development (R&D)." Technical Memorandum. December 6.

\-\--. 1994a. "Comparison of Company Coding Between 1992 and 1993 for
the Survey of Industrial Research and Development." Technical
Memorandum. November 3. Washington, DC.

\-\--. 1994b. *Documentation of Nonsampling Issues in the Survey of
Industrial Research and Development.* RR94/03. Washington, DC.

\-\--. 1994c. *An Evaluation of Imputation Methods for the Survey of
Industrial Research and Development.* ESMD-9404. Washington, DC.

\-\--. 1994d. "Evaluation of Total Employment Cut-Offs in the Survey of
Industrial Research and Development." Technical Memorandum. November 3.
Washington, DC.

\-\--. 1994e. "Reclassification of Companies in the 1992 Survey of
Industrial Research and Development (R&D) for the Generation of the
'Analytical' Series." Technical Memorandum. October 25. Washington, DC.

\-\--. 1994f. *A Study of Processing Errors in the Survey of Industrial
Research and Development.* ESMD-9403. Washington, DC.

\-\--. 1994g. "Wedging Considerations for the 1992 Research and
Development (R&D) Survey." Technical Memorandum. June 10. Washington,
DC.

\-\--. 1995. *Documentation of the Survey Design for the Survey of
Industrial Research and Development: A Historical Perspective.*
Washington, DC.

[^1]: Information for this section was provided by the Manufacturing and
    Construction Division of the U.S. Bureau of Census, which collected
    and compiled these survey data for NSF. Copies of the technical
    papers cited can be obtained from NSF's Research and Development
    Statistics Program in the Division of Science Resources Studies.

[^2]: See U.S. Bureau of Census (1994d).

[^3]: These industries are listed and discussed later under
    "Comparability of Statistics."

[^4]: However, estimates for companies in these groups are included with
    their less detailed 2-digit SIC classifications in one table (table
    A-2). In all other tables, they are added to the "other
    manufacturing" and "other nonmanufacturing" classifications.

[^5]: For detailed discussions on the sources, control, and measurement
    of error resulting from item nonresponse, see U.S. Bureau of the
    Census (1994b). For a general discussion of the problems stemming
    from item nonresponse, see NSF (1990).

[^6]: All but four items---total R&D, Federal R&D, net sales, and total
    employment, which are included in the Census Bureau's annual
    mandatory statistical program---are voluntary. See further
    discussion under "Response Rates and Mandatory Versus Voluntary
    Reporting" later.

[^7]: For detailed descriptions and analyses of the imputation methods
    and algorithms used, see U.S. Bureau of the Census (1994c).

[^8]: NSF, *Research and Development in Industry: 1990* (NSF 94-304),
    John R. Gawalt (Washington, DC, 1994).

[^9]: See NSF (1990) for an explanation of the uncertainties in the data
    and to quantify their sensitivity to the choice of various possible
    imputation procedures.

[^10]: See also U.S. Bureau of the Census (1995).

[^11]: The effects of recent changes in the way companies are classified
    during survey processing are discussed in detail in U.S. Bureau of
    the Census (1994e and 1994a).

[^12]: See also NSF (1994, 1995, and 1996a).

[^13]: Until 1967, samples were selected every 5 years. Subsequent
    samples were selected for 1971, 1976, 1981, and 1987.

[^14]: For the 1992 survey, 25 new nonmanufacturing industry and
    industry groups were added to the sample frame: agricultural
    services (SIC 07); fishing, hunting, and trapping (09); wholesale
    trade---nondurables (51); stationery and office supply stores
    (5112); industrial and personal service paper (5113); groceries and
    related products (514); chemicals and allied products (516);
    miscellaneous nondurable goods (519); home furniture, furnishings,
    and equipment stores (57); radio, TV, consumer electronics, and
    music stores (573); eating and drinking places (581); miscellaneous
    retail (59); nonstore retailers (596); real estate (65); holding and
    other investment offices (67); hotels, rooming houses, camps, and
    other lodging places (70); automotive repair, services, and parking
    (75); miscellaneous repair services (76); amusement and recreation
    services (79); health services (80); offices and clinics of medical
    doctors (801); offices and clinics of other health practitioners
    (804); miscellaneous health and allied services not elsewhere
    classified (809); engineering, accounting, research, management, and
    related services (87); and management and public relations services
    (874).

[^15]: Annual sampling also remedies the cyclical deterioration of the
    statistics that results from changes in a company's payroll
    composition because of product line and corporate structural
    changes.

[^16]: See also NSF (1997a and 1998a).

[^17]: For discussions of each of these changes, see U.S. Bureau of the
    Census (1994g); for considerations of comparability, see U.S. Bureau
    of the Census (1994e and 1993).

[^18]: See U.S. Bureau of the Census (1995).

[^19]: The process was dubbed wedging because of the wedgelike area
    produced on a graph that compares originally reported statistics
    with the revised statistics that result after linking.

[^20]: See also NSF (1997b).