+------------+---------------------------------------------------------+
| **\<       | **Stevens, Susan M., Dornburg, Courtney C.,             |
| digital    | Hendrickson, Stacey M. L., Forsythe, J. C., Bauer, T.   |
| picture\>\ | L. & Davidson, G. S.**                                  |
| \< IF      |                                                         |
| AVA        | Sandia National Laboratories / P.O. Box 5800/           |
| ILABLE\>** | Albuquerque / NM 87185 USA[^1]\                         |
|            | E-mail: <smsteve@sandia.gov>                            |
|            |                                                         |
|            | Phone Number: (505)844-0704[^2]                         |
+------------+---------------------------------------------------------+
|            |                                                         |
+------------+---------------------------------------------------------+

ABSTRACT

An experiment was conducted comparing the effectiveness of individual
versus group electronic brainstorming in addressing real-world "wickedly
difficult" challenges. Previous laboratory research has engaged small
groups of students in answering questions irrelevant to an industrial
setting. The current experiment extended this research to larger,
real-world employee groups engaged in addressing organization-relevant
challenges. Within the present experiment, the data demonstrated that
individuals performed at least as well as groups in terms of number of
ideas produced and significantly (p\<.02) outperformed groups in terms
of the quality of those ideas (as measured along the dimensions of
originality, feasibility, and effectiveness).

Key words: brainstorming, electronic media, groups vs. individuals,
"wickedly difficult" problems.

INTRODUCTION

In today's highly dynamic and competitive world, it is essential that
organizations generate novel ideas of high quality to develop or
maintain their competitive advantages. Historically, one method of idea
generation has been verbal brainstorming, a process where groups of
individuals, typically in the same room, work to create and exchange
ideas. Popular opinion holds that verbal brainstorming yields more (and
better) ideas than the same number of individuals working alone would
produce \[5, 7, 10\]. However, verbal brainstorming has been found to
result in certain undesirable consequences (such as evaluation
apprehension, production blocking and social loafing) when compared to
individual, or nominal, brainstorming \[9\]. To address some of these
limitations, electronic brainstorming (EBS) has been proposed as an
alternative. An EBS session consists of individuals interacting and
exchanging ideas via a computer. While EBS has been shown to be superior
to verbal brainstorming, the research comparing EBS to electronic
nominal brainstorming has produced rather mixed results with some
research finding EBS superior to electronic nominal groups \[11\], some
researchers finding electronic nominal groups superior to EBS \[1\] and
others finding no difference between the two groups \[4\].

To date most of the research in this area has been performed in
laboratory settings with college students electronically brainstorming
about industrially irrelevant topics, leaving generalizability to
industrial applications unclear. There are several key differences that
must be addressed in order to apply the existing research to an
industrial setting. First, most research has studied three- to
four-person student groups, rather than larger work teams that leverage
diverse skill and knowledge bases. Second, it is unclear how results
from short, one-time brainstorming sessions generalize to real-world
situations where groups of individuals brainstorm over a period of time.
Third, students are often presented with irrelevant topics whereas
groups in typical industrial settings grapple with "wickedly difficult"
problems (i.e., problems characterized by a lack of agreement about the
very nature of the problem itself; \[14\]). The current literature's
brainstorming topics are not as related to students as a "wicked"
problem might be to a vested employee. Fourth, while assessing the
quantity of ideas is often the focus in research conducted in college
settings, the *quality* of ideas may be a more important metric when
considering "wickedly difficult" issues.

An experiment to investigate these issues was conducted at Sandia
National Laboratories in the summer of 2007. Specifically, this
experiment explored the effectiveness of EBS within the industrial
setting of a modern, national research laboratory. To date, most of the
research performed in this area has involved college students in an
experimental setting; thus, this study extends the research to a more
applied setting with practical applications. Over the course of four
days, employees and contractors at Sandia voluntarily enrolled and
contributed ideas in the web-based brainstorming experiment. The
participants were randomly assigned to either group or nominal
electronic brainstorming conditions and were asked to work on a "wicked"
problem proposed by Sandia President Tom Hunter. Both the quantity and
quality of ideas were assessed.

METHOD

Participants

Participants were volunteers, including 147 employees and contractors at
Sandia. Of those who enrolled in the experiment, only 69 employees who
actually contributed ideas were included in analyses.

Materials

The experiment took place on a website that was created and managed by
the experimenters. The participants electronically signed the informed
consent and electronically completed the demographic and satisfaction
questionnaires.

Procedure

Participants were primarily recruited through an advertisement in the
*Sandia Daily News* (an internal news source emailed daily to Sandia
employees). In addition, the experimenters sent personal recruitment
emails and placed a link to information regarding the study on the
Sandia intranet Homepage. The recruitment messages informed the
employees that a brainstorming experiment was being conducted and would
consist of electronic brainstorming (either alone or in a group) via a
website for four days. Once the employees expressed interest in
participating in the study, they were directed to a website which
described the study in detail and informed the participants of what the
experiment would entail. Once they had (electronically) signed the
informed consent, the participants created a userid and password for use
during the experiment and completed a demographic questionnaire.
Finally, the participants were directed to a website in which they could
submit ideas to the question.

The participants were randomly divided into one of two groups: EBS or
electronic nominal brainstorming. All of the participants brainstormed
ideas regarding the question provided by Lab President Tom Hunter.

When the participants logged onto the website, they were shown the
question and were asked to submit their ideas. Those in the electronic
nominal condition worked alone and did not see the ideas of other
participants. Those in the EBS condition brainstormed with others and
were able to see and build off of the ideas of the other members in the
group. The responses in both conditions were anonymous, being tied
solely to the participant-generated userid. There were two reasons for
this anonymity. First, anonymity in group brainstorming sessions has
been reported to reduce evaluation apprehension \[2\]. Evaluation
apprehension is the tendency for people to hold back ideas for fear that
others will negatively evaluate them \[11\]. However, when anonymity is
assured this fear of negative evaluation may be reduced. Second, by
presenting the participants with their peers' performance (because the
participants were able to view which userid came up with each idea), the
participants might be less likely to engage in social loafing \[8, 12\].

Participants in the EBS conditions were asked to adhere to the rules of
brainstorming per Osborn \[10\]. In addition, the participants were told
that abusive language and name calling would not be tolerated, and those
who did not follow the rules would be locked out of the experiment.

At the end of the experiment, the participants were asked to complete a
Satisfaction Questionnaire \[3\]. The questionnaire asked several
questions regarding satisfaction with the experiment, along with
motivation and interest in the task.

RESULTS

Quantity Analysis

In order to measure the effectiveness of the brainstorming sessions, we
assessed the quantity and quality of participant responses. The number
of unique ideas generated within the nominal and group conditions was
counted. A repeated measures analysis of variance (ANOVA) was performed
on the number of ideas expressed on each of the days (Day 1, Day 2, Day
3, Day 4) by research group membership (nominal or group). There was a
significant effect for the number of ideas expressed on each day (Wilks'
lambda, *F* (3, 65) = 2.784, *p* = .048, η*~p~^2^* = .114) in which
there was a larger number of ideas put forward on day one compared to
the following three days. However, there was no significant difference
in the number of ideas between the group and nominal conditions. In
fact, absolute values favored the nominal condition (mean = 6.26, *SD* =
12.85) over the group condition (mean = 4.66, *SD* = 9.21).

Quality Analysis

In addition to quantity measures, responses were also examined for
quality. In order to perform this analysis, responses were summarized
into general concepts representing each participant's answer to the
Hunter question. Responses unrelated to the question, like those
addressing the website design, were not considered. Following the
example of Barki and Pinsonneault \[1\], the quality of each idea was
scored according to the three separate dimensions of originality,
feasibility, and effectiveness. In this scoring scheme, *originality*
referred to the extent to which the idea was novel, *feasibility*
referred to the extent to which the idea was precise and the ease with
which it could be implemented and *effectiveness* referred to the extent
to which the idea helped to solve the given problem. Two raters were
chosen due to their background and experience in operations management
and industrial/organizational psychology. The raters independently
scored the ideas on these qualities using a seven point Likert Scale
with one corresponding to low evidence for the component and seven
corresponding to high evidence for the component. The ratings were then
averaged for each idea. Because the most meaningful ideas were of the
most interest, the maximum ratings rather than average ratings were
evaluated. Thus, if a participant received ratings of 3, 4, and 5 for a
particular day, the maximum rating of 5 was used as that participant's
dependent value. Independent samples t-tests comparing the two
conditions (nominal and group) on each of the three quality dimensions
were conducted to independently evaluate originality, feasibility, and
effectiveness. Significant differences were found for all t-tests such
that the nominal condition outperformed the group condition for
originality (*t* = 3.69, *p* \< .001), feasibility (*t* = 2.39, *p* =
.02) and effectiveness (*t* = 2.65, *p* = .01).

Satisfaction Analyses

The groups' responses to questions on the Satisfaction Questionnaire
were also compared using a t-test. While we did not obtain a significant
difference between the two groups (likely due to the small number of
respondents, n = 9), our data suggest that the participants in the EBS
group were generally more satisfied, motivated and interested in the
brainstorming task than were those in the nominal condition which is
consistent with what has been established in the literature \[3, 6,
13\].

DISCUSSION

Our primary empirical finding demonstrates that within the given
experimental design electronic nominal brainstorming is at the very
least as effective as electronic group brainstorming. Although there was
no difference in the number of ideas generated by the groups, the
nominal condition outperformed the group condition in terms of the
quality of ideas. The quality of the ideas in the nominal condition was
significantly better across all three quality ratings, including
originality, feasibility, and effectiveness. These results are not due
to differences in satisfaction, motivation or interest in the task; in
fact, the Satisfaction Questionnaire data tend to favor the group
condition over the nominal condition. Although these results are
preliminary, they demonstrate that employees may effectively use
electronic nominal brainstorming as a cost effective means to work on
wickedly difficult problems.

The finding that individuals are more successful than groups in
electronic brainstorming suggests a time- and cost-savings potential for
companies. Generally, when electronic group brainstorming is compared to
verbal brainstorming, it is touted as having the advantages of shorter
meetings, increased participation by remote team members, better
documentation via electronic recording, improved access to the meeting
records and, importantly, cash savings \[5\]. When there is no longer
the mandate that these electronic communications occur concurrently,
these advantages would seem to be even greater. One might assume that
participants in an electronic nominal condition would require less time
to contribute ideas as compared to those in a group condition where they
would (ideally) read the other postings before giving their ideas. On
the other hand, at least some of the submissions suggested that they
were prepared offline and pasted into the web site forms. Thus an
evaluation of the time savings in this experiment is not addressed.
However, nominal brainstorming does allow for increased participation
due to greater scheduling flexibility.

In sum, our data demonstrate that within the current industrial setting
and electronic interface, nominal brainstorming was at least as
effective as group brainstorming. This study is to our knowledge one of
the first to empirically examine brainstorming within an industrial
setting. Additionally, the current experiment is the first to extend
brainstorming groups beyond the typical 3- or 4-person groups
(occasionally 12-person) to large, 30-person groups. It is also the
first to examine how a longer duration of 4 days affects results.

While our results demonstrate that nominal brainstorming is as effective
as group brainstorming, more research is necessary in order to fully
circumscribe the generalizability of this finding to other questions,
interfaces, and industrial settings. Future research may compare
different computer-mediated technologies, interfaces, and experimental
manipulations. For example, a more wiki-like interface might allow users
to build on other people's ideas more easily than the interface used for
the current experiment and, thus, outperform a nominal group. In
addition, other "wickedly difficult" questions could be addressed as
well as electronic brainstorming performance at alternative industrial
settings.

REFERENCES

\[1\] Barki, H. & Pinsonneault, A. (2001). Small group brainstorming and
idea quality: Is electronic brainstorming the most effective approach?
Small Group Research, 32(2), 158-205.

\[2\] Cooper, W. H., Gallupe, R. B., Pollard, S. & Cadsby, J. (1998).
Some liberating effects of anonymous electronic brainstorming. Small
Group Research, 29(2), 147-178.

\[3\] Dennis, A. R. & Valacich, J. S. (1993). Computer brainstorms: More
heads are better than one. Journal of Applied Psychology, 78(4),
531-537.

\[4\] DeRosa, D. M., Smith, C. L. & Hantula, D. A. (2007). The medium
matters: Mining the long-promised merit of group interaction in creative
idea generation tasks in a meta-analysis of the electronic group
brainstorming literature. *Computers in Human Behavior, 23,* 1549-1581.

\[5\] Furnham, A. (2000). The brainstorming myth. Business Strategy
Review, 11(4), 21-28.

\[6\] Gallupe, R. B., Bastianutti, L. M. & Cooper, W. H. (1991).
Unblocking brainstorms. Journal of Applied Psychology, 76(1), 137-142.

\[7\] Guerin, B. (1986). Mere presence effects in humans: A review.
Journal of Experimental Social Psychology, 22, 38-77.

\[8\] Karau, S. J. & Williams, K. (1993). Social loafing: A
meta-analytic review and theoretical integration. Journal of Personality
and Social Psychology, 65(4), 681-706.

\[9\] Kerr, N. L. & Tindale, R. S. (2004). Group performance and
decision making. Annual Review of Psychology, 55, 623-655.

\[10\] Osborn, A. F. (1957). Applied Imagination. (Rev. Ed.) New York:
Scribner.

\[11\] Paulus, P. B. & Yang, H-C. (2000). Idea generation in groups: A
basis for creativity in organizations. *Organizational Behavior and
Human Decision Processes, 82(1),* 76-87.

\[12\] Roy, M. C., Gauvin, S. & Limayem, M. (1996). Electronic group
brainstorming: The role of feedback on productivity. Small Group
Research, 27(2), 215-247.

\[13\] Valacich, J. S., Dennis, A. R. & Connolly, T. (1994). Idea
generation in computer-based groups: A new ending to an old story.
Organizational Behavior and Human Decision Processes, 57, 448-467.

\[14\] "Wicked problems". Retrieved August 21, 2007 from Wikipedia: The
Free Encyclopedia Online on the World Wide Web:
http://en.wikipedia.org/wiki/Wicked_problem.

[^1]: Sandia is a multiprogram laboratory operated by Sandia
    Corporation, a Lockheed Martin Company, for the United States
    Department of Energy's National Nuclear Security Administration
    under contract DE-AC04-94AL85000.

[^2]: Additional contact information: Email: <ccdornb@sandia.gov>,
    <smhendr@sandia.gov>, <jcforsy@sandia.gov>, <tlbauer@sandia.gov>,
    <gsdavid@sandia.gov>; Phone Number: (505)284-1432, (505)284-2665,
    (505)844-5720, (505)284-8723, (505)844-7902.