### Methodological Issues in Management Research

Lee Sechrest, PhD

Professor

### Department of Psychology

### University of Arizona

### Room 312

### Tuscon, AZ 85721

### 

White paper prepared for the Department of Veterans Affairs *Management
Research in VA Workshop*, sponsored by the HSR&D Management Decision and
Research Center

November 19-20, 2001

### Methodological Issues in Management Research

Managers want to make good decisions. Any decisions will, by definition,
be made on the basis of some presumed information. Even if a decision
were to be made by throwing dice, that process would almost certain stem
from "information" indicating that no better basis for the decision
could be discerned, e.g., that a randomly determined choice would be
likely to be better than a decision open to bias. At least to some
extent, it is axiomatic that the better the information, the better the
decisions.

It is useful to distinguish between data, facts, and information. Data
are simply observations, usually in the form of numbers thought to
represent some systematic process underlying them, i.e., a process
generating the numbers. Data do not mean anything or tell us anything
until they are interpreted in some way. Merely to have an observation
that on a particular day 43 patients were reported to have received a
particular service is not in itself meaningful. Facts are merely data
elevated in confidence to a point of suggested certainty. The
observation that 43 patients received a service may be registered as a
fact if it seems likely that the data are sufficiently trustworthy to
justify confidence that the data are correct. Facts come in all
varieties: numbers, declarative statements, equations, and so on. But
facts are not necessarily information For example, if one is told that
the land area of Bahrain is 231 sq.mi., that might be factual, but it
might not constitute information. A useful definition of information is
that it is any communication that reduces uncertainty with respect to
some decision. That Bahrain is 231 sq. mi. is information only if one is
in the position of having to make some decision involving that
information, e.g., about whether to invest in a manufacturing enterprise
in that country or whether, in a game, to choose Bahrain as the smallest
of the Arab states. Even the precise estimate for land area might not be
information if one already knows that Bahrain is very small and that is
all the precision of knowledge that one requires.

These distinctions are important in thinking about research because not
all research findings are necessarily factual, and the facts that are
acquired from research efforts are not necessarily information. The
reason is that research findings can be---must be---interpreted at
different levels. It may be, for example, that in a particular
organization more patients were processed by Unit A than by Unit B. That
could lead to a conclusion that the management style employed in Unit A
results in greater productivity than the management style employed in
Unit B. That, in turn, could lead to the conclusion that Management
Style A is superior to Management Style B. And so on. Reflection may
show, however, that the only undeniable "fact" available is that more
patients were [recorded]{.underline} as having been processed by Unit A
than by Unit B. The apparent better productivity of A could reflect no
more than errors in recording patients processed. Or the difference,
although real factually, might have happened simply by chance. The
difference might be real enough but also so small as to be
uninteresting. Or, even if the difference were real and sizable, there
is a considerable leap in the conclusion that it should be attributable
to differences in management style, and there is even a greater leap in
the conclusion that one management style should be considered generally
superior to the other.

Research, as suggested, may result in specific findings that can be
regarded as factual and that may represent information in the sense of
reducing uncertainty about decisions. Research may also be supportive of
more general theory about the phenomena of interest. Thus, research on a
particular management style in a particular context may be supportive of
a more general theory of or about management. Even though a particular
research project may be rather specific in many of its details, it may
still bear on very general principles of organizational management.
Conversely, a rather general theory may permit derivation of
recommendations concerning specific practices. Although the precision
either way, reasoning from specific findings to general theory or from
general theory to specific practices, may be limited, the process is
likely to result in better decisions that would be made by chance and
probably better than would be made overall by intuition.

Theory and research should reflect a continuing interplay of observation
and synthesis. It is possibile that theories may sometimes be better
than our facts, i.e., the results of specific research. Any one research
project is certain to be limited in important ways, at least in terms of
its generalizability to other types of persons, to other settings, to
different versions of the intervention, and so on. Moreover, any given
research project is subject to limitations resulting from a wide range
of errors that are inevitable. Measures are not perfect, interventions
are not perfectly implemented, individuals differ in their reactions to
the same set of conditions, reactions to interventions or other
arrangements may vary over the course of a day, or a week, or a month.
Thus, if there is a plausible, well-reasoned theory that tells us that a
particular intervention ought to decrease the time taken to do some
task, but the expected decrease does not happen, it may not necessarily
be the case that the theory is wrong. It may well be that the data just
were not up to the task of testing the theory.

In order to produce information that will be dependably useful to
managers, research must be carefully planned, carried out, and
interpreted. Good research does not just happen. It is the result of
deliberate application of well-tested methods. The aim of this paper is
to outline some of those methods and why they are important.

**Causal attribution**

The aim of research is, generally, to reveal causal connections between
variables, i.e., to demonstrate that one variable is a cause of another
variable. When the demonstration is sufficient, then we are, at least
potentially, in the position of being in control of the \"effect\"
(caused) variable. We may, if we are able to change the causal variable,
e.g., eliminate it, strengthen it, produce a change in the effect
variable. Research is sometimes carried out for descriptive purposes,
but the results of descriptive research are rarely of great interest in
their own right. For example, a manager may wonder whether costs of some
activity are higher on some days of the week than on others and may
assemble data relevant to that question. That costs are higher, let us
say, on Mondays than on other days may be a curious \"fact,\" but it is
of value only insofar as it suggests the possibility of another,
follow-on study that will explain the higher cost, i.e., that will
explain why the cost is higher, what causes the higher cost on Monday.
If that can be determined, then the manager may be able to intervene to
reduce the excess cost or may understand that the cost, although larger,
is warranted because it is attributable to desirable features of the
operation of the organization, e.g., more services are provided on
Mondays or more serious problems are dealt with on Mondays.

Making correct causal attributions is often far from straightforward. At
the heart of the matter is that a causal inference requires dealing with
*the counterfactual.* That is, making a causal inference requires at
least an implicit answer to the question \"What would have happened if
the causal event had not occurred?\" That may be a relatively simple
matter if a causal connection is direct and well understood. If more
services are offered, then costs are almost certain to increase.
Moreover, it is likely to seem pretty clear that if services are cut,
costs will go down. If, however, it is observed that a group of new
workers given three days extra training make \"only\" two errors per
shift in the month following training, it may not be so easy to be sure
how they would have performed had they not had the training. One cannot
both given the training and not give the training. An inference is
required.

Descriptive research may be quite useful when it helps to understand
some process, particularly if that process is amenable to intervention.
It may be important to know, for example, whether supervisors actually
read documents that cross their desks for signatures in order to plan
for streamlining of the flow of paper through an organization. Such a
study might be limited to enquiry by questionnaire, but it might involve
interviews or even observations.

**Necessity for comparison observations**

When one thinks about it, no observation is meaningful in and of itself.
Observations acquire meaning by the opportunity to compare them to some
expected or observed value. Stars are dim only because some other stars
are bright. An object is likely to be seen as blue only if it might well
have been another color. An observation (or set of observations) made in
an organization is interpretable only in relation to some prior
expectations. Those expectations might have been derived in any number
of ways: logic, common sense, theory, or other observations.

We attribute causality to some intervention in relation to some event
because we are capable of imagining that the outcome might have been
otherwise, i.e., we invoke the counterfactual possibility. If an
assistant tells a manager, "We are having fewer staff complaints from
the long-term care unit," and the manager replies, "That's because of
the new procedures we instituted last quarter," the manager is
implicitly accepting the proposition that had *the procedures not been
instituted* the complaints would not have decreased. What might be the
justification for that proposition?

*Logic? Common sense?* It may seem to the manager and the assistant
simply logical if things are not going well and a new set of procedures
to improve matters is introduced, improvement should occur and that any
that does occur should be attributed to the intervention. Things are
rarely so simple. Almost always some plausible rival explanations exist,
and, if one of them is correct, it may change the decision about what to
do in an important way. For example, some studies had shown that the
testing of prospective employees for honesty resulted in reductions in
episodes of theft or inventory loss. More definitive studies, however,
seemed to indicate that the primary effect was the result simply of the
demonstration to employees that management cared about theft and was
resolved to do something about it. Testing employees for honesty can be
expensive and hard on employee morale; by comparison communication of
managers\' concerns for theft may be inexpensive and accepted as a
legitimate managerial function. Complaints from a unit might be reduced
for similar reasons, i.e., simply because a manager expressed concern
for them and a determination to reduce them.

*Theory?* Similar to logic but more explicitly reasoned is a theoretical
justification for a conclusion. A manager might rely at least to some
extent on a theory of management that would predict that a particular
type of intervention would have the effect intended and result in an
outcome reflected in a decrease in complaints. The legitimacy of the
conclusion would depend heavily on the resemblances between the specific
intervention and the type represented by the theory and between the
outcome supposed by the theory and that realized in the empirical study.
A characteristic of a theoretical explanation is that the mechanism for
producing the change is explicit. For example, a particular intervention
might operate to reduce complaints by improving conditions leading to
complaints, by helping staff to be more accepting of deficient
conditions, or by improving staff morale so as to decrease the
likelihood of complaints.

*Other observations?* By contrast, a manager might have access to other
observations that would support the conclusion that the change could be
attributed to the intervention. The manager might note, for example,
that complaints had been at a consistently high level for quite some
period of time, decreasing only when the intervention was implemented.
Or the manager might know that on one or more other units in which no
intervention was tried, no decrease in complains was found. The manager
might even be satisfied to know that after the intervention, complaints
were no more frequent than reported in other similar organizations in
his or her geographic area.

*Comparison data.*

Interpretation of any observations is, obviously, much enhanced by
having available data from other sources with which to compare
observations at hand. Essentially, the reason is that the comparison
data make it possible to make a judgment in relation to the
counterfactual notion of what the data would have looked like under
other circumstances, e.g., if the intervention had not occurred. If
complaints seem high and an intervention is tried, followed by a
decrease in complaints, the counterfactual asks what the complain level
would have been without the intervention. Answering that question is not
always easy, and the answer is not only obvious. Complaints might have
been going down anyway; a change in patient mix might have made fewer
complaints more likely; new personnel in the unit, independent of the
intervention, might have elicited fewer complaints.

Comparison data could be helpful in varying degrees, depending on just
what kinds of comparisons were available. Inspection of
rate-of-complaint data prior to the intervention might show that the
idea that complaints were going down anyway was unlikely.

Those data would not necessarily be helpful in determining whether the
change might have been due to changes in personnel or patient mix.

*True (randomized) experiments.*

The best comparison data would result from circumstances in which
everything was identical to the conditions in the group exposed to the
intervention *except* the occurrence of the intervention itself. The
problem is that so often it is difficult, sometimes impossible, to be
sure that everything was the same except for the intervention. Two units
might, however apparently similar, differ in subtle ways in patient mix,
or in personnel, or in some other variables that might be related to the
occurrence of complaints. No matter how carefully units were selected
and how similar they seemed to be, the possibility exists that the
patients in them would differ in some unknown, but important way. By
"important" is meant that the difference would be related in some way to
an outcome of interest, say the tendency to complain.

The best way to maximize the likelihood that patients on two units will
be the same even on unobserved variables is to assign them *randomly* to
the two units. By such an assignment process, if a patient with a low
tendency toward complaining were assigned to the unit to receive the
intervention, probability would guarantee that a similarly low patient
would be likely (but not certainly) to be assigned to the other unit.
Over a series of observations, chance generally evens things out. The
logic of scientific inquiry then is that if one can assume that two
groups (units, in this case) were equivalent to begin with, then any
later difference between them can be attributed to differences in the
way they were treated, i.e., the intervention. The equivalent comparison
group answers the question posed by the counterfactual, what things
would have been like in the absence of the intervention.

That, in a nutshell, is the essence of the randomized clinical trial,
almost universally regarded as the strongest basis for an inference
about a causal relationship between two variables. Any other comparison
data are considered inherently weaker as a basis for causal inference
and, therefore, more likely to be misleading.

*Threats to validity of the inference.*

It is common practice to refer to problems with various comparison data
series in terms of "threats to validity," meaning threats to the
validity of a causal inference. It will help to review some of the more
likely threats and to indicate how they might crop up in specific
comparisons.

*Selection.* If groups, say intervention and comparison groups are
assembled in any way other than by random selection, there is at least a
possibility, often a near certainty, that differential selection into
the groups will have made them different from the beginning, thus making
any final difference difficult to interpret. For example, if groups are
assembled by asking for volunteers for the intervention, that group may
be more enthusiastic, higher in risk-taking, more eager to please, of
more of almost anything else than the left-over group of non-volunteers.
Thus, if the intervention group turns out better than the comparison
group, that difference cannot unequivocally be attributed to the
intervention itself. Even if one takes intact groups, e.g., patients in
two general medical units of a hospital, one cannot be certain that they
do not differ in important ways. Medical studies, for example, have run
afoul of the problem that patients selected for one treatment over
another tend to be different from patients no so selected. In one
well-known study, patients who received surgical treatment had better
outcomes than patients not subjected to surgery, but that ultimately was
shown to be because patients not selected for surgery were too sick to
endure it.

# Maturation. Naturally occurring processes may be mistaken for intervention effects under some circumstances. If one unit has very new leadership and another is well established, changes associated with the maturing of the leadership could be mistaken for effects of an intervention. If costs of some process are increasing over time, an intervention might appear to result in increased costs even thought it were cost neutral. 

*History.* Sometimes in the middle of a research study, external events
will occur that have an impact on the phenomenon being studied. A
researcher unaware of a change in accounting procedures in an
organization might mistakenly conclude that an intervention decreased
costs when, in fact, the change was illusory. Events outside the system
may, similarly, have effects that could be taken for the effects of an
intervention. It is entirely likely, for example, that the events of
Sept. 11 and subsequently may have affected all sorts of responses made
within work and treatment facilities. Productivity probably dropped for
a while, various complaints probably decreased, and absenteeism
increased. Because of the enormous salience of Sept. 11, few
investigators would be likely to miss its effects on most data being
collected during that time. Similar effects might well be missed,
however, if they were less obvious. History is a special threat to
studies involving comparisons of data for the same group over time, but
if history has more impact on one group than another, it can affect any
comparison.

*Regression artifact.* Observed values of any variable must be
considered to be in some part in error. If an observation is toward one
extreme or another of a distribution, the probability is that error is
in some degree involved in the location of the observation. For example,
if on some occasion it is observed that complaints from a treatment unit
are much higher than average, the probability is great that the number
is high in part because of the odd confluence of factors not likely to
persist for long, e.g., a temporary staff shortage, a malcontented
patient, an equipment breakdown. Then, with no intervention at all, the
number of complaints would fall at the next occasion of measurement
(called regression toward the mean). Because interventions are likely to
be initiated exactly when complaints are high, the natural effects of
regression can easily be mistaken for an intervention effect.
Incidentally, and conversely, an intervention that happens to come along
when things are noted to be especially good may appear to have a bad
effect because regression works in both directions.

## Research Designs

As noted earlier, the randomized experiment is generally regarded as the
gold standard for research, particularly for supporting causal
inferences. So why are randomized experiments not used exclusively? In
the first place, they simply cannot be done for many problems because
the variable of interest cannot be experimentally controlled, whether
for practical or for ethical reasons. We are interested in the effects
of education on people's lives, but we cannot "give" education to people
who do not want it. Similarly, we are interested in the effects of
sanctions on criminal behavior, but we cannot, ethically---or
legally---punish some people and not punish others on an experimental
basis. Second, randomization may be unacceptable to some persons or
groups, i.e., they may be unwilling to be randomized to experimental
conditions. If randomization is possible, it is usually preferable as a
research option, but often alternatives must be sought.

A wide range of options for research is available, each with specific
advantages and deficiencies dependent on the nature of the problem.
These options are frequently referred to a quasi-experimental designs.
All research designs are, however, simply ways of systematizing
observations in such a way as to maximize information concerning the
counterfactual proposition. Those research designs likely to be of
greatest usefulness are the following:

*Nonequivalent comparison group design.*

The most widely used quasi-experimental design is, undoubtedly, the
nonequivalent comparison group design, by which is meant a research
scheme that calls for collecting data from an intervention group and a
comparison group that do not involve random assignments. Consequently,
the groups must be *regarded* as nonequivalent, as potentially having
been *selected* in such a way that they are unequal to begin with. Two
hospitals differing in management types may be compared, and they may
actually be quite equivalent save for management, but there is no way of
knowing that, so they must be regarded as nonequivalent. Obviously, the
more reason with which one can argue that the groups or organizations
are equivalent, the better the case for interpreting any outcome
differences between them. That is why when, as is inevitable, plans must
be made to compare interventions between two groups or organizations, it
is important to select the two (or more) with as much care as possible.
One would not opt deliberately to compare two organizations of widely
disparate size or from very different communities unless those
variables, size or geography, were the exact conditions of interest.

Initial differences between groups or organizations may be dealt with
statistically or by developing other, supportive comparison data sets.
Statistical allowances for differences is often, but not always,
straightforward, depending on the nature and size of the initial
differences. Statistical corrections for initial differences also depend
for their persuasiveness on the quality and relevance of the data
available. If initial measures are available on the variables to be
measured as outcomes, allowance for differences is not likely to be
terribly controversial, at least if differences are not large and other
assumptions are tenable. When, however, it is necessary to use proxy
variables, controversy is inevitable. For example, if one has to use a
symptom-count as a proxy for severity of illness, the correction for
potential initial differences in severity may be questionable.

*Interrupted time series.*

Often data are assembled systematically over time, usually for
administrative or clinical rather than for research purposes. If, in the
middle of a data series, some event occurs relevant to the processes
involved in producing the series, it may be possible to estimate the
effect of the event by comparing the values in the series prior to the
interruption to those obtained afterwards; hence the label "interrupted
time series." Consider, for example, that on a particular date, say the
first day of July, the procedures involved in providing aftercare for
patients is changed and later that year a manager wonders whether the
change may have affected subsequent follow-up visits to an outpatient
department. The manager discovers that records of such visits are
available by week for the six-month period prior to the change and
later. The manager might then plan to continue assembling the data until
six months of records are available subsequent to the change. It might
then be possible to compare the data series before the change with the
data afterwards and reach some conclusions about the effect of the
change. Basically, what the manager wants to do is estimate what the
data would have looked like without the change (the counterfactual) and
compare that estimate with the actual data. The manager's statistician
would look for a change in level (intercept) of the two series at the
point of the change and for differences in slopes of the series between
the two time periods, e.g., whether a general trend in an upward
direction prior to the change might reverse to a downward trend after.

The interrupted time series can be a very useful and persuasive research
strategy under the right circumstances. Those circumstances begin with
the availability of records if the time frame is long (managers will not
want to wait for very long to assemble the necessary data) and will
include the stability of the series (widely fluctuating values may
frustrate attempts to detect changes) and the abruptness of the change
(effects of phased-in interventions are difficult to detect). The
research strategy does lend itself very readily to many administrative
interventions if quick answers are not required or if a long-term
perspective is needed.

*Observational data with statistical corrections.*

A very common research strategy, but scarcely a design in the usual
sense, is to assemble observational data and attempt to "rule out" rival
explanations by logic and statistical means. A recent example is a study
based on a large data set including extensive questionnaire responses of
78,000 nurses. The investigators were interested in factors determining
the occurrence of breast cancer and discovered that nurses who had
worked on night shifts for an extended period of time had an increased
risk of breast cancer. That is, they compared breast cancer rates in
nurses who did and did not work night shifts and related breast cancer
rates to number of years working night shifts. Since many other
variables could have been related both to night shift work and to breast
cancer risk, they "adjusted" the data for those variables by statistical
means. The data for the women who did not work night shifts, when
adjusted for "confounding" variables, presumably showed what the data
for the other nurses would have looked like if they had not worked night
shifts.

An advantage of observational data is that they are usually fairly easy
to obtain and are often already available in existing data files. On the
other hand, the investigator must often make do with whatever data are
available in those files. In the nurses data file, night shift work was
defined only as at least three nights per month; a more refined measure
would have been desirable. Even if observational data are being
collected de novo, the investigator must rely on whatever values for
variables happen to show up in the data. If a particular condition of
interest is rare, then it simply will not show up very often, although
when it does, the investigator can try to make sure that the case gets
included in the data set. If a manager is interested in the
effectiveness of bilingual supervisors, he or she will be limited to
observing the effectiveness of those supervisors actually on the job.

Moreover, observational data are often not likely to be very persuasive
when sample sizes are small. Statistical corrections for confounding
groups usually require a fairly large number of cases.

Observational data are particularly subject to biases resulting from
"data dredging," looking through large quantities of data until
something interesting appears to turn up.

## Measurement

Good measurement is of absolutely critical importance to good research.
It is unfortunate that problems in measurement so often go unrecognized
and, if recognized, are treated so lightly. It is impossible to
demonstrate effectiveness of any intervention without reliable
measurement. Think, for example, of trying to "influence" the a variable
with values created by throwing dice. That is exactly what is involved
when measures of outcomes are unreliable. Measures that have only modest
reliability can be expected to show at best only modest effects of
interventions. Poor (unreliable) measures will almost always result in
an underestimate of the effects of any intervention.

Prescriptions for good measurement are easily made but not so easily
followed. Good measurement begins with good definitions of just what it
is to be measured, good definitions of constructs. Multiple measures are
highly desirable, especially when measures differ substantially in their
structure and likely sources of bias. Measures should, ideally, be
nonreactive, i.e., they should not be readily susceptible to
self-serving biases and other sources of distortion. They should also be
closely related to the phenomena of interest so that values on the
measure map clearly onto the underlying variables of true concern. Such
prescriptions are obvious; realizing them is difficult.

One measurement problem that is very often overlooked ins the assessment
of the intervention or, the *independent variable*, as it is often
called in the jargon of research methodology. Just as we cannot expect
to show much effect on a variable that is poorly measured, so we cannot
show much effect from a variable that is only weakly implemented.
Researchers too often take the independent variable for granted, and
only infrequently do they attempt to quantify it. If an accounting
system is to be evaluated, then it is important to know to what extent
and how well the system is implemented. If those using it are poorly
trained, if they do not like it and, hence, do not use it, or if it
involves technical difficulties that result in frequent down-time, a
fair evaluation of the system cannot be obtained. Just as it is
important in evaluating medications to know how much of a dose was
actually received by patients, so it is important to know how much of an
intervention was actually achieved in a managerial setting.

*Methodological consultation.*

The design of a good, persuasive research project nearly always requires
technical expertise for the many decisions to be made. Managers who wish
to become involved in research should be quick to seek
methodological---and statistical---consultation. Consideration may also
need to be given to the need for consultation on measurement.

## Statistical analysis

Data must be subjected to statistical analyses before they are
interpretable. Summary statistics such as means, standard deviations,
and correlations are easily calculated; in fact, they can be done on
simple spread-sheet programs. Their interpretations are not always
straightforward, but they usually pose no great problems. Analyses aimed
at inferring causes, however, are usually more complex and often are
highly complex, requiring specialized knowledge and software. Research
outside laboratories, and most of that in laboratories today, is
necessarily a multidisciplinary enterprise. A managerial team must
include specialists in administration, purchasing, accounting,
maintenance, and so on, and a managerial research team must, similarly,
include an appropriate mix of specialists.

Research will in every way be improved if statistical expertise is
enlisted from the very beginning of a project. Few projects can be
brought to a fully satisfactory conclusion by turning things over to a
statistician to make sense of after all the other work is done.

## Generalizability of research 

An issue of persistent concern in the interpretation and use of
information derived from research is the extent to which the results of
the research may be generalized to other settings, i.e. to settings
similar, but not identical, to that in which the research was done.
Concerns about generalization often relate to fairly obvious
characteristics of settings, e.g., the identity of the organizations or
the people associated with them, but legitimate bases for concerns may
lie at a deeper level. Superficially, a manufacturing facility and a
health care facility seem quite dissimilar, but with respect to the
fundamental organizational processes by which they operate, they may be
quite similar: leadership requirements, information needs,
accountability provisions, quality controls, etc. Researchers can
facilitate generalization greatly by being explicit in their assumptions
about the settings in which they are working and the nature of the
variables underlying the processes they are studying.

To some extent every research study has to be regarded as constituting a
"special case of real life." That is, real life is complicated, messy,
dynamic; research studies, of necessity, either impose order on or
derive order from the situation and the resulting data. Research

Requires simplification along many dimensions, and that simplification
may restrict the generalizability of the interpretations of findings. A
great strength of experiments is that they are designed in such a way as
to keep everything as simple as possible and as nearly identical as
possible between groups. Those restrictions may mean, however, that
conditions in the experiment are considerably different from those that
obtain in "real life," thus making direct applicability of the findings
questionable. For example, in an experiment to test a new accounting
system, everyone involved would be likely to be very carefully trained
and supervised, well beyond what would achievable in any ordinary,
functioning organization. Research requirements often cause
investigators to focus on specific outcome measures that may not capture
all the consequences of interest, The important issues for
generalizability often have to do with specific research arrangements
related to the intervention, the circumstances of its implementation,
and the measurement of outcomes. With respect to generalizability, those
issues may be more important than more obvious things such as the
identity of the organization of the types of people working in it or
served by it.

In biomedical research a distinction is often made between *efficacy*
and *effectiveness* of interventions, the former referring generally to
effects of interventions implemented under carefully controlled and
sometimes artificial conditions, and the latter to effects obtainable
under more-or-less real life conditions. In biomedical research it is
regularly necessary to carry out studies related to the transition from
the "ideal" conditions to the laboratory to the conditions of the every
day world. Undoubtedly, management research may sometimes require the
same kinds of transitional studies.

Observational studies offer the advantage that they usually occur in
real world settings, and they may not require as many constraints as are
necessary to carry out experiments. (It is the case, however, that
observational studies may at least require simplifications involved in
quantifying inputs and outcomes.) A major shortcoming of observational
studies, however, aside from their general messiness (because nothing is
really controlled) is that the legitimacy of any causal inferences may
be very much in doubt.

In the first place, the direction of causality may be more apparent than
real. In a study of the effects of shift work on family relationships,
it appeared that shift-workers had poorer family relationships than
workers with only daytime jobs. Further investigation suggested,
however, the probability that a good many workers choose shift work in
order to get away from unpleasant family relationships. When that
possibility was taken into account, the effect disappeared. Causal
inferences must be developed with great care in observational studies.

Secondly, however, observational studies may be questionable as a basis
for causal inferences because of the necessity for assuming that
naturally occurring, observed correlations would hold for deliberate
interventions. For example, an investigator might determine that a
relationship existed between amount of training and productivity of
workers. That finding might lead to the supposition that giving more
training to workers would increase productivity. That inference would be
a big leap, however. It is not only likely, but probable, that workers
who have, on their own, acquired more training differ systematically
from workers without such training. It cannot at all be assumed that
"giving" workers training would have the same effect as their having
gotten it.

Thus, observational studies, too, have limitations on their
generalizability. All the foregoing leads to the generalization that it
is unusual that any important uncertainties can ever be resolved by
single studies, whatever their nature, and even multiple studies should
have only modest effects on our confidence that we know what we are
doing.

Finally, it is important to remember that generalizations can be invoked
at different levels: findings, principles, and theory. A particular
research finding, let us say that implementing a particular form of
feedback to workers about their performance, may generalize to other
settings so that those that implement the same feedback get pretty much
the same results. At a more general level, it might be that the
principal that feedback improves performance is supported, so that a
range of feedback arrangements could be counted on. And at a still more
general level, the results of a feedback study could be regarded as
generally supportive of a "theory of participative management,"
strengthening it in some small way as a basis from which to derive a
wide range of ideas about improving management and performance.