A Manufacturing Collaboratory

Case Study — Companion

Document

Michelle P. Steves

Amy J. Knutilla

Elizabeth E. Wierba

NISTIR 6809
ii

NISTIR 6809

A Manufacturing Collaboratory

Case Study — Companion

Document

Michelle P. Steves

Manufacturing Systems Integration Division

Manufacturing Engineering Laboratory

Amy J. Knutilla

Knutilla Technologies

Elizabeth E. Wierba

Collaboratory for Research on Electronic Work

School of Information

University of Michigan

July 2001

U.S. Department of Commerce

Donald L. Evans, Secretary

National Institute of Standards and Technology

Karen H. Brown, Acting Director
i

Table of Contents

Table of Contents.................................................................................................................i

Abstract ..............................................................................................................................iii

Keywords .......................................................................................................................iii

Executive Summary ...........................................................................................................iv

Assessment techniques, challenges and observations..................................................... v

Concluding remarks .......................................................................................................vi

1.0 Introduction ............................................................................................................. 1

1.1 Background ............................................................................................................... 1

1.2 Research summary [16 (Sec. 2)] ............................................................................... 1

2.0 Research environment................................................................................................... 2

3.0 User-centered Design: Surveys and Interviews (Time 1) ............................................. 3

3.1 Interviews [16 (Sec. 2.2.2)]....................................................................................... 3

3.2 Pre-intervention Survey [16 (Sec. 2.5)] .................................................................... 4

4.0 Collaboration Requirements [16 (Sec. 2.4, 3.1)]........................................................... 4

5.0 Collaborative tools recommendation, deployment, and use [16 (Sec. 3.2)] ................. 6

6.0 Post-intervention survey and follow-up interviews [16 (Sec. 2.6)] .............................. 7

6.1 Items eliminated [16 (Sec. 2.6.1)]............................................................................. 7

6.2 Items altered [16 (Sec. 2.6.2)]................................................................................... 7

6.3 New items [16 (Sec. 2.6.3)]....................................................................................... 7

7.0 Results........................................................................................................................... 8

7.1 Time 1 and Time 2 Survey Findings [16 (Sec. 3.3)]................................................. 8

Communication Patterns-Time 2 [16 (Sec. 3.3.4 & Figures 2.1, 2.2)] ....................... 8

Technology Adoption-Time 1 [16 (Sec. 3.3.5 & Figure 1.5)] .................................... 9

Technology Adoption-Time 2 [16 (Sec. 3.3.6 & Figure 2.6)] .................................... 9

Workflow-Time 1 [16 (Sec. 3.3.7 & Figure 1.6)]..................................................... 10

Workflow-Time 2 [16 (Sec. 3.3.8 & Figure 2.8)]..................................................... 10

Working Relationships-Time 1 [16 (Sec. 3.3.9 & Figures 1.7, 1.8, 1.9)]................. 10

Working Relationships-Time 2 [16 (Sec. 3.3.10 & Figures 2.9, 2.10)].................... 11

Coordination, performance and organizational citizenship-Time 1 [16 (Sec. 3.3.11 &

Figures 1.10, 1.11, 1.13)] .......................................................................................... 11

Coordination, performance and organizational citizenship-Time 2 [16 (Sec. 3.3.12 &

Figures 2.11, 2.12)] ................................................................................................... 11

7.2 Comparing Means at Time 1 and Time 2 [16 (Sec. 3.4 & Appendix I – Tables 8,

9)] .................................................................................................................................. 12

7.3 Relationships among variables [16 (Sec. 3.5 & Appendices G, H)]....................... 12

Impact of collaboration tools [16 (Sec. 3.5.1)] ......................................................... 12

Performance [16 (Sec. 3.5.3)] ................................................................................... 13

Trust [16 (Sec. 3.5.4)] ................................................................................................... 14

8.0 Discussion [16 (Sec. 4)].............................................................................................. 15

8.1 Adoption and use of collaborative tools [16 (Sec. 4.1.1)]....................................... 15

8.2 Impact of collaborative tools [16 (Sec. 4.1.2)]........................................................ 16

8.3 Assessment techniques, challenges and observations [16 (Sec. 4.2)]..................... 17

8.4 Practical recommendations [16 (Sec. 4.3)] ............................................................. 19
ii

8.4.1 Advice to managers and engineers [16 (Sec. 4.3.2)]............................................ 19

8.4.2 Advice to researchers [16 (Sec. 4.3.3)]................................................................ 21

9.0 Concluding remarks .................................................................................................... 21

References......................................................................................................................... 21
iii

Abstract

This report summarizes a case study of a collaborative tool intervention within a

geographically-distributed, product-development team of an automotive parts supplier.

This report serves as a companion document to a larger, more detailed volume, prepared

as the final report to the National Institute of Standards and Technology (NIST) by the

University of Michigan (UM), A Manufacturing Collaboratory Case Study [16]. In this

research, an application sharing tool, NetMeeting, was deployed after an in-depth

analysis of the team’s collaboration requirements. Team members, who were perceived

as early technology adopters, were trained in the tool use. Using pre- and postintervention surveys and interviews, UM researchers examined levels of adoption, use of

the collaborative tool, and the relationship between tool use and changes in team

performance. Results indicated that a small proportion (1/3) of the team used

NetMeeting, and that NetMeeting users described a positive impact of this tool on their

collaborative work. Lower than expected adoption rates may have been due to key

features of NetMeeting, such as synchronous application sharing, not being as valuable as

expected for collaborations that spanned time zones. These features were more widely

used for local collaborations. Furthermore, workarounds already in use within the team,

such as transmitting engineering drawings as email attachments, may have been sufficient

to inhibit exploration and adoption of alternative collaboration tools – even when the

newly introduced tools had desirable features as described by the users. Results are

discussed in terms of the adoption, use, impact of collaboration tools, techniques for

assessment and evaluation of collaboration tools, and practical recommendations for

users. While the data indicated team performance was mixed, there is an abundance of

information useful to researchers and users of tools for collaboration. This report

provides the interested reader with an overview of the research, as well as, pointers into

the full report [16] for complete detail.

Keywords

NetMeeting; CSCW; collaboration technologies; collaborative tool adoption
iv

Executive Summary

The National Institute of Standards and Technology (NIST) and the University of

Michigan (UM) embarked on a research project to examine the deployment of

collaboration technology in a manufacturing setting. The primary research goals were

assessment and better understanding of a) the use of collaborative tools in a

manufacturing (field) setting and b) the impact of this use on team performance and

effectiveness, based on a comparison of pre- and post-intervention measures. An

additional goal was to provide a collection of practical recommendations to potential

manufacturing users and analysts, who may be considering using these tools in today’s

growing global economy.

The research setting and approach

In this research, the University of Michigan (UM) worked with an industrial partner,

Auto 11

, to deploy and assess the performance and impact of network-based tools for a

geographically-distributed, engineering team, CAR. UM conducted interviews and

administered a voluntary, web-based, pre-intervention survey to identify collaboration

needs and to determine baseline, collaboration conditions. A suite of collaboration tools

was recommended to Auto 1. The CAR team management agreed to the deployment and

assessment of a tool for synchronous viewing of engineering drawings and documents,

Microsoft (MS) NetMeeting2 (NM). In conjunction with the Auto 1 Information

Technology (IT) group, UM trained key CAR team personnel upon NetMeeting

deployment. After 6 months of use, a second survey was administered. UM used the two

surveys to analyze the adoption and impact of network-based, collaborative tools on the

team’s performance. Using this analysis, previous research, and follow-up questions to

Auto 1 management, UM addressed the meaning of the results in terms of the adoption,

use, and impact of collaboration tools, techniques for assessment and evaluation of

collaboration tool use, and practical recommendations for users.

Tool use and impact

Although the analysis of the pre- and post-intervention surveys showed that a small

proportion (1/3, n=34) of the team used NetMeeting, NM users described a positive

impact of this tool on their collaborative work. NetMeeting use reduced some of the

difficulties indicated in the requirements analysis, such as interruptions in midconversation to send email attachments of drawings. With respect to the low adoption

rate, results suggested a key feature of NetMeeting, specifically synchronous application

sharing, was not as valuable as expected for collaborations that spanned time zones; but,

it was more widely used for local collaboration. Another factor was that workarounds

already in use within the team were sufficiently successful to inhibit exploration and

 

1

 Pseudonyms are used for the corporation, Auto 1, and its elements, e.g., CAR.

2

 Any commercial product identified in this document is for the purpose of describing a collaborative

software environment. This identification does not imply any recommendation or endorsement by NIST.
v

adoption of alternative collaboration tools, even when these tools had desirable features

as identified in the collaboration requirements analysis.

Assessment techniques, challenges and observations

Over the course of one year at Auto 1, UM researchers used several techniques and

approaches to assess and evaluate the use and impact of collaborative tools. They made

numerous visits to CAR team sites, including European locations to conduct face-to-face

interviews and training sessions. In collaboration with the Auto 1 IT group, UM

conducted one-on-one training of NM and tracked deployment consequences for six

months. Finally, UM developed, administered, and analyzed two surveys.

These efforts were conducted within a dynamic environment that threatened support for

this research at Auto 1. There was continual uncertainty with respect to the ultimate

success of the CAR team’s product. There was also uncertainty regarding the legal

relationship between Auto 1 and the University of Michigan concerning the overall

research project with respect to intellectual property rights. Additionally, halfway

through the research period, the Auto 1 vice president, who had invited and nurtured the

research project, left the company. Even though the deployment and evaluation phases of

the project were ultimately completed, these experiences reinforced the known

difficulties in conducting this type of research in a business environment [6].

Observations from these efforts included:

• Conducting this type of research in a corporate environment, within actual

engineering teams, required enormous cooperation and compromise to accommodate

competing priorities.

• In the Auto 1 corporate environment, where IT resources were strained, the research

team expended a great deal of effort attempting to bring a critical mass of users to a

minimum level of competency with the target collaborative tools. With additional

support from the Auto 1’s IT group, more effort could have been expended directly

on observation of the tools in use or enhancing practices with the tools.

• The use of surveys had the advantage of gathering data broadly and at relatively low

cost. However, low response rates reduced the capacity to draw useful conclusions.

• Overcoming inertia in organizational settings is very difficult. For instance, if a

solution to a collaboration issue already existed, there was little incentive for

participants to change to another solution, even with anticipated improvements.

• A high-level champion in the field is vital for success of the implementation [6] and

of the research. The “champion” at Auto 1 played an essential role in negotiation

across the two divisions from which CAR team members originated. Momentum on

this project slowed considerably when that high-level champion left Auto 1.

• UM’s status as outsiders allowed them to probe the CAR team members to reveal

practices and procedures that the engineers themselves, and Auto 1 personnel, did not

have the time or inclination to explore. However, as outsiders, researchers confronted

a very steep learning curve to understand the content of the engineering work. The

effort expended on mastering the elementary aspects of the engineers’ tasks could

have been allocated to a more detailed examination of performance of the tasks and

the relationship between these tasks and collaborative tools. In this way, inclusion of
vi

a veteran automotive engineer, specifically one from Auto 1, on the research team

might have accelerated understanding of what the CAR team was trying to do. Yet,

in the time- and resource-starved environment where CAR team members worked, it

was unlikely that CAR team management or Auto 1 would have consented to an

engineer/designer acting in a formal liaison role.

• It is important to anticipate competing demands in this type of research. Specifically,

adoption of novel tools with accompanying novel practices is not something that

unfolds on a time scale consistent with most projects. Therefore, researchers will

confront the need to perform some level of “pump priming” – that is, there has to be

some level of collaboration-tool adoption and use to produce behavior and outcomes

that can be used to evaluate the impact of collaborative tools. Realistic determination

of resources required to jump-start use, particularly where none currently exists, is

difficult.

Practical recommendations for managers and engineers include:

• Selected collaborative tools must meet a specific need to merit the effort of

deployment, adoption, and subsequent use. For example, the requirements gathering

effort conducted early in this project, successfully highlighted difficulties experienced

by the CAR team members when attempting to do cross-site work. At least for some

of the CAR engineers, as indicated by adoption and use rates, the set of introduced

collaborative tools met the identified needs. Yet, for many, the collaborative tool still

imposed too great a burden to learn and master, relative to the perceived benefits.

• Successful deployment of collaborative tools requires a coordinated approach that

involves the IT support organization at an early stage.

• Targeted changes in work practices, such as the adoption and use of collaborative

tools, need adequate support to ensure success. In this study of NM adoption and

impact, engineers were asked to master both the tool itself and also the choreography

of working with a distant colleague via the tool. The significant investment of one-onone training surrounding the introduction of NetMeeting was successful for individual

users. However, it was not conducted on a broad enough scale to create a critical

mass of use for the team in the timeframe of this study.

Concluding remarks

The primary goals of this research were an assessment and better understanding of the

adoption, use, impact, and evaluation of collaborative tools in a manufacturing setting.

Because the collaborative tool intervention was specifically targeted to needs identified in

the requirements gathering phase, a primary research expectation was to find a positive

impact of tool use. While there was some indication of improved team performance with

use of the tools, the overall adoption did not meet expectations. Although results were

mixed, assessment of collaborative tool intervention is often difficult, even at the very

coarse granularity of “success” and “failure” [1]. Therefore, one important outcome of

this research is the well-documented case study [16], so that over time results can be

interpreted more fully, and future research and collaborative tool deployments can draw

from it.
1

1.0 Introduction

This report summarizes a case study of a collaborative tool intervention within a

geographically-distributed, product-development team of an automotive parts supplier.

This report serves as a companion document and overview to the 170+ page report of that

study, prepared as the final report to the National Institute of Standards and Technology

(NIST) by the University of Michigan (UM), A Manufacturing Collaboratory Case Study

[16]. Additionally, it is our intention that this document provide a portal to the abundant

detail found in the final (full) report. Therefore, section references from the full report

[16] are given throughout this document for the interested reader.

The primary research goals of the study were assessment and better understanding of a)

the use of collaborative tools in a manufacturing (field) setting and b) the impact of this

use on team performance and effectiveness, based on a comparison of pre- and postintervention measures. An additional goal was to provide a collection of practical

recommendations to potential manufacturing users and analysts, who may be considering

using these tools in today’s growing global economy.

1.1 Background

Members of NIST’s Manufacturing Engineering Laboratory and UM’s Collaboratory for

Research on Electronic Work3 (CREW) embarked on a research project to examine the

deployment of collaboration technology in a manufacturing setting. This research

complements the work that NIST’s Manufacturing Engineering Laboratory is doing in

collaboration technologies and extends earlier work by UM in collaboratories. UM

provided expertise in collaboratories, having deployed existing or prototype applications

in real field settings, group behavior, and principles and methods of human-centered

design, [16 (Sec. 1.2)]. NIST provided expertise in manufacturing, engineering, and

collaboration tools for manufacturing.

Both organizations recognized the need for manufacturers to understand how to best use

collaboration technologies. Field studies are an excellent way to generate this knowledge.

However, reports on field studies typically provide anecdotal conclusions that can be

difficult from which to learn. While we knew that conclusive results would be difficult to

obtain when working in an area with numerous variables, we expected much useful

information from a well-documented, case study. The UM final report, “A Manufacturing

Collaboratory Case Study” [16], is that well-documented, case study.

1.2 Research summary [16 (Sec. 2)]

In this research, UM worked with an industrial partner, Auto 14

, to implement and assess

the performance and impact of network-based tools for a geographically-distributed,

engineering team, CAR. UM conducted interviews and administered a voluntary, web-

 

3

 Part of the School of Information, University of Michigan

4

 Pseudonyms are used for the corporation and its elements, e.g., Auto 1 and CAR.
2

based, pre-intervention5

 survey to identify collaboration needs and to determine baseline

collaboration conditions. A suite of collaboration tools was recommended to Auto 1. The

CAR team management agreed to the deployment and assessment of a tool for

synchronous viewing of engineering drawings and documents, Microsoft (MS)

NetMeeting (NM). UM also recommended deployment of a shared-calendar system and a

presence-awareness system – neither of which were approved by CAR management for

UM deployment, but had some spontaneous adoption that is discussed. UM trained key

CAR team personnel upon NetMeeting deployment. After 6 months of use, a second,

post-intervention6

 survey was administered. UM used the two surveys to analyze the

adoption and impact of network-based, collaborative tools on the team’s performance.

Using this analysis, follow-up questions to Auto 1 management, and previous research,

UM addressed the meaning of the results in terms of the adoption, use, and impact of

collaboration tools, techniques for assessment and evaluation of collaboration tool use,

and practical recommendations for users.

To place this research in context, there are only two published studies on NetMeeting in

the literature [11 & 7]. In the case of Mark et al. [11], the study focused on room-toroom use of NetMeeting in an engineering-design setting within an aerospace

manufacturing organization. Its use was primarily as an adjunct to audio conferences.

Data were gathered on four teams over a period of three months. In the case of Finholt et

al. [7], the study focused on targeted dyads of remote users performing software

engineering within a telecommunications organization, again, covering a period of three

months. The UM/NIST research effort with their industrial partner, Auto 1, extends this

body of knowledge by focusing on a geographically-distributed, product-development

team over a one-year period and generating a well-documented case study. [16 (Sec. 4.1)]

2.0 Research environment

The motivation for conducting this study was to understand the impact of network-based

collaborative tools on the performance of a geographically-distributed work team. The

significance of this research area is highlighted by the growing globalization of work,

which requires an increase in coordinated activity across dispersed sites and employees,

e.g., [4, 13]. Specifically, in the manufacturing sector it is becoming common practice for

engineers from different backgrounds and at different locations to combine their efforts to

produce novel products. The UM/NIST strategy to assess the effects of modern

communication technologies for collaboration was a detailed examination of a

geographically-dispersed, engineering team.

With the cooperation of the industrial partner, Auto 1, UM identified a collaborative

product-development group [16 (Sec. 2.1)], the CAR team. The team spanned four

countries, three continents, and two divisions of the company. The CAR team had

common characteristics of multi-site engineering projects:

 

5

 The “pre-intervention” identifier is synonymous with “Time 1.” These identifiers refer to the period of

time prior to the deployment of NM.

6

 The “post-intervention” identifier is synonymous with “Time 2.” These identifiers refer to the assessment

period approximately six months after the initial deployment of NM.
3

• The engineers were located all over the world (Europe, the United States, and

Australia) and regularly confronted barriers of distance and time.

• The members of the team had to cope with cultural, linguistic, procedural, and

technological differences – often with little allowance or guidance from the

organization.

• The infrastructure to support the work of the team was dynamic, including the

underlying network technologies and data formats that required ongoing adjustment

and accommodation by the engineers.

• The team brought together engineers with diverse technical backgrounds. This

required that problems and solutions be expressed via mutually understandable terms

and concepts.

• The team was under extreme pressure to meet scheduled deliverables. This pressure

was compounded by fact that key milestones and product characteristics were under

constant negotiation with prospective customers.

• The success of the team had strategic importance for the company, since the potential

market for the product in development was estimated at over $1 billion.

• The team was newly assembled to produce the target automotive subsystem and did

not have a prior history of working together.

• The team was the initial effort by the larger organization to join expertise across

corporate divisions to develop an innovative product.

Against this backdrop, UM identified and proposed a collaboration technology

deployment strategy based on rigorous analysis of the team’s requirements. Then UM

instrumented the deployment to assess the impact of collaboration technology use on the

team’s effectiveness. [16 (Sec. 2.1, 2.3)]

3.0 User-centered Design: Surveys and Interviews (Time 1)

To understand the collaboration needs of the CAR team and to establish a baseline of

collaboration conditions, UM developed and administered a web-based survey to each of

the 50 CAR team members and interviewed a subset (24). The pre-intervention survey

and interviews were designed to capture an initial data set of team conditions against

which future data (from the post-intervention survey) could be compared.

3.1 Interviews [16 (Sec. 2.2.2)]

The selected subjects for the interviews were individuals whose point of view or insight

might be particularly helpful because of their position or skill set. The interviews with the

team members and management addressed:

• Work roles and background

• Identification of local and remote collaborators

• Current means of communicating with remote collaborators

• Opportunities for remote collaboration

Following user-centered, design principles, e.g., [5], UM personnel addressed these

topics to gain an understanding of the CAR team members’ current work practices.
4

Questions were aimed also at uncovering established norms and potential barriers to

future collaborative tool introduction. The interviews were semi-structured in format.

This means the interviewers allowed for conversation with the informants, so that

informants could expand on topics most relevant to themselves and open the possibility

for gaining additional, unforeseen information.

3.2 Pre-intervention Survey [16 (Sec. 2.5)]

All 50 members of the team were invited by their management via email to participate in

the study and complete the web survey. No incentives for participating in the study were

offered. Thirty-four out of 50 (68%) employees completed the pre-intervention survey

from 6 geographically-distributed sites, in 3 countries. Most respondents (65%) came

from European sites (18 from Germany, 4 from France, 12 from the United States) [16,

Appendix F].

The web survey consisted of 48 questions covering:

• Background items [16 (Sec. 2.5.1)]

• Demographic information

• Job satisfaction

• English speaking skills

• Location

• Workstation

• Independent variables [16 (Sec. 2.5.2)]

• Technology adoption motivation

• Work-related communication patterns

• Non-work related communication patterns

• Workflow

• Outcome measures [16 (Sec. 2.5.3)]

• Coordination and performance

• Organizational citizenship

• Trust

• Group identification

4.0 Collaboration Requirements [16 (Sec. 2.4, 3.1)]

This section describes the themes, related work-practices, and barriers that were reported

by the CAR product-development team during the requirements gathering phase of the

study.

These themes were derived by qualitative analysis of the interviews to determine the

most frequently mentioned issues in distributed work within the CAR team. This was

accomplished using standard practices for qualitative data analysis [8, 9, 12]. This

involved constructing inductive code categories first by reading through the background

interviews and creating an extensive list of all the reported issues, then clustering these

issues into categories of related statements. Descriptive statistics from the pre-
5

intervention survey supported three themes: reliance on email, coordination issues, and

maximizing synchronous time.

Heavy reliance on email. The following quote from one respondent was representative of

attitudes towards email as a collaboration tool: “The most valuable tool is email...

because it’s something that’s always there.”

To overcome the coordination issues inherent in remote collaboration, CAR team

members had established norms to help them expedite their work. While these norms

were resourceful adjustments to dispersed teamwork, they were often cumbersome

solutions for collaborative work problems. Team members described using email

foremost as a tool for contacting remote team members, as well as, the primary means for

exchanging task requests, data, reports, designs, and sketches. As needed, these items

were exchanged as attached files.

Yet, these practices fell short of respondents’ needs. Traveling team members

experienced difficulties accessing their email accounts remotely. File transfers were

sometimes slow and caused delays in work. File transfers were often slowed due to large

file sizes, generally Computer-Aided Design (CAD) drawings. Additionally, encryption

standard incompatibilities led to cumbersome security procedures. Finally, email did not

offer a rich medium for information exchange. For instance, email did not easily allow

users to identify or describe visual details, which are common in engineering design

work.

Coordination issues. Difficulties in remote collaboration are often due to inherent

coordination problems of geographically-distributed team members [14]. For example,

typical team coordination activities, such as scheduling meetings, are often slowed by

large time zone differences.

In this study, respondents expressed frustration due to: 1) difficulty in coordinating with

team members located in other time zones, 2) limited synchronous time they shared with

co-workers located in other time zones, and 3) the resulting work delays. On average,

team members reported 4 delays each month, with 90% encountering delays of a day or

more when working with remote collaborators. Furthermore, team members described

difficulty in scheduling common meeting times with both local and remote co-workers.

Often, problems in scheduling stemmed from difficulties locating local co-workers who

traveled frequently. Yet, problems in locating and scheduling were more pronounced

with remote rather than local co-workers. Coordination issues were correlated

statistically with both delays in work and difficulties establishing trust with remote team

members.

Maximizing the effectiveness of synchronous time. One of the challenges facing this

geographically-distributed team was the scarcity of common work times. CAR team

members indicated experiencing difficulty scheduling common work time with remote

team members due to limited overlapping work hours.
6

This theme is somewhat interwoven with the ‘coordination’ and ‘reliance on email’

themes, in that, often team members coordinated synchronous meetings via email. For

example, one participant mentioned that one “may say (over email) ‘call me this

afternoon, I have to talk to you about’ such and such.”

Participant team members most commonly used the telephone for synchronous work,

while a few team members used shared application tools, and fewer still used video

conferencing. Typically, team members conducted meetings via the telephone in order to:

discuss coordination issues, conduct design review, acquire further information or details,

and clarify issues mentioned in email. Team members explained that they would like to

use visual aids to enhance such meetings. Some team members sent faxes to exchange

sketches or other materials simply and quickly. In addition, team members who discussed

sketches, designs, or visual materials, expressed the desire to have tools to improve

mutual understanding of the material, such as being able to point to areas in a drawing

under discussion.

Additionally, team members traveled between North America and Europe in order to

facilitate working together. Language differences caused some communication problems

that were best handled with face-to-face meetings. Email and telephone interactions were

described as working better after there had been an initial, face-to-face meeting. Finally,

Americans often used overtime to synchronize work with European counterparts.

However, this practice had limited utility as the labor laws of one of the European

countries involved in the study constrained their citizens to an 8-hour workday.

5.0 Collaborative tools recommendation, deployment, and use [16 (Sec. 3.2)]

The requirements analysis resulted in the recommendation of three technology

interventions: a tool for synchronous viewing of engineering drawings and documents

(e.g., Microsoft NetMeeting); a tool for sharing calendar information (e.g., Microsoft

Outlook); and a tool providing presence-awareness information (e.g., ICQ7

). Only one

direct intervention was pursued due to resource constraints within Auto 1 and legal

concerns about the export of personal data – such as schedule information – outside the

European Union. That intervention was a supported introduction of NetMeeting.

In conjunction with the CAR team management, UM researchers identified a subgroup of

engineers for early adoption of NetMeeting and arranged a schedule of collaborative tool

implementation. Members of the subgroup were selected because their management

believed they had an immediate need to collaborate with remote colleagues. Fifteen team

members in two U.S. and two European locations were trained, involving a 15 minute

introduction followed by a 30 minute guided use of NetMeeting in a domain-relevant,

document-sharing session. Simultaneous training sessions across sites were conducted so

that the team members could engage in remote collaboration in the course of their

training. As a follow-up to the NetMeeting training, UM researchers sent unsolicited

weekly hints to subgroup members on the use of NetMeeting and requested feedback via

email on the CAR team’s weekly use of the collaborative tools.

 

7

 http://www.icq.com/
7

Although the CAR management team elected not to adopt a presence-awareness or

calendar tool as part of this research effort, there was some ad hoc adoption among CAR

team members. The adoption of these tools is addressed in [16 (Sec. 4.1.2)].

6.0 Post-intervention survey and follow-up interviews [16 (Sec. 2.6)]

The post-intervention survey consisted of 42 questions in 5 sections: technology

adoption, communication patterns, coordination, performance and trust. 34 participants

(68% of the team) responded to the post-intervention survey. Approximately a third of

these respondents had also completed the pre-intervention survey. Of these, 70% were

located in Europe and 30% were located in offices in the United States.

While the basic format and sections of the second survey were the same as the first

survey, several changes were made. These changes were made to avoid redundancy and

to reduce the amount of time required to complete the second survey. Feedback about the

first survey indicated that many respondents required more than the promised 30 minutes

to complete the survey. Additionally, a significant number of respondents skipped

questions at the end of the survey.

6.1 Items eliminated [16 (Sec. 2.6.1)]

Questions regarding the participants’ background and workstation, non-work

relationships and friendships, and group identification were eliminated. Background and

workstation information had remained essentially unchanged over the course of the study.

Questions regarding team members’ non-work communication patterns were eliminated

due to the excessive time required to complete these questions and to the low response

rate from the first survey. The group identification questions were removed due to low

response and the fact that responses to these questions did not vary. That is, these

questions were uniformly answered in positive terms, leading analysts to question the

validity of these items for measuring group members’ actual sentiments.

6.2 Items altered [16 (Sec. 2.6.2)]

The questions about work communication patterns were changed so that they would no

longer require participants to repeatedly answer the same questions regarding up to 15 of

their co-workers, as in the first survey. Instead, respondents were to indicate the number

of co-workers with whom they communicated on a daily basis.

6.3 New items [16 (Sec. 2.6.3)]

The first survey asked individuals to discuss their motivation to adopt technology. In

contrast, the second survey asked about actual tool use. In the second survey, questions

were added to assess the impact of the collaboration tools that had been implemented

during the study. Specifically, new questions included:

• “Did the use of any of these collaborative tools change the manufacturing design

process (i.e., the way you and your team went about working on the CAR
8

project)?” If participants answered “Yes”, they were asked to respond to the

following questions:

• “The use of these collaborative tools improved the quality of the product”

• “The use of these collaborative tools improved the efficiency of the design

process”

• “The use of these collaborative tools improved the speed of the design

process”

7.0 Results

To better understand the changes that occurred in the CAR team during the research

period, UM analyzed the two surveys in three ways. First, the individual items in the

surveys were analyzed and compared. This gave a flavor for changes occurring between

the pre- and post-intervention surveys. The findings are given in section 7.1. Second, the

difference in mean scores on outcome variables was analyzed using a paired-sample t-test

[3]. This test compared identical measures in both surveys. The findings are presented in

section 7.2. Third, correlation analysis was used to determine whether variables were

associated with one another. These findings are discussed in section 7.3.

7.1 Time 1 and Time 2 Survey Findings [16 (Sec. 3.3)]

Communication Patterns-Time 1 [16 (Sec. 3.3.3 & Figures 1.1, 1.2)]

With respect to communication, team members were asked about their work contact

patterns, frequency of communication with other team members across various media,

and travel to other sites.

• On average, participants reported communicating often with 5.8 (s.d. = 5.39) coworkers.

• The most frequently used modes of communication were face-to-face (83%

reported high use), email (35% reported high use), and telephone (29% reported

high use).

• European team members reported higher levels of telephone and email than their

U.S. counterparts.

• Across all modes of communication, respondents contacted the German product

development site most often. This was the site where the design work was

occurring.

Communication Patterns-Time 2 [16 (Sec. 3.3.4 & Figures 2.1, 2.2)]

• Patterns of media use were similar to those found at Time 1, except that the

percentage of participants reporting high frequency of face-to-face

communication was lower at Time 2.

• Differences in media use between U.S. and European Union (EU) persisted: EU

used the telephone and collaboration tools more often, whereas U.S. used more

face-to-face communication.

• Collaboration-tool use had spread to more than half the team.
9

• Like Time 1, respondents contacted the German product-development site most

often. However, unlike Time 1, respondents traveled to the German

manufacturing site most often. This change from the first survey likely reflects the

downstream production process occurring at Time 2.

• Respondents reported traveling half as often as in Time 1.

In terms of collaboration-tool use, American respondents did not use collaboration tools

at all, compared with 42% of European respondents. Half of the European respondents

reported using collaboration tools regularly, between 1 and 4 times per week. 8% used

these tools very frequently. It is possible that the collaboration-tool use in Europe

reduced the need for face-to-face communication for European respondents, as compared

to Americans. In fact, a post-training interview suggested that European team members

often used NetMeeting to hold meetings between office buildings at the German productdevelopment site.

Technology Adoption-Time 1 [16 (Sec. 3.3.5 & Figure 1.5)]

The overall level of interest in using new technology for collaboration was high:

• 100% of U.S. respondents reported a strong desire to adopt a shared, mark-up

tool, compared with 55% of European respondents.

• 83% of U.S. respondents reported a strong desire to adopt availability and

presence tools, compared with 59% of European respondents.

• 83% of U.S. respondents reported a strong desire to adopt an electronic-calendar

tool, compared with 55% of European respondents.

Technology Adoption-Time 2 [16 (Sec. 3.3.6 & Figure 2.6)]

Whereas the technology adoption items in the Time 1 survey assessed the participants’

inclination to adopt new tools, the items at Time 2 assessed their actual use of these tools,

and the impact it had on their work.

• High calendar use, low application-sharing and presence-awareness tool use.

• 50% (n=32) of all respondents agreed that NetMeeting had an impact on their

work, with most agreeing that it improved the efficiency and speed of the design

process.

The overall use of new technology for collaboration was high for some tools and not

others. For instance, at the time of the second survey, all (100%, n=34) participants were

using the electronic calendar. However, only 38% used a presence-awareness tool or an

application-sharing tool. Sixty-two percent of respondents indicated that they did not use

NetMeeting at all in the last 6 months. Regular use of the shared-application tool was

positively correlated with agreement that collaborative tools had changed the design
10

process. Adoption of the calendar and presence-awareness tools was not significantly

correlated with this variable.

To further interpret the effect of collaboration-tool use, participants were asked whether

the use of these tools changed the manufacturing design process, and how the tools had

an impact.

• 50% of respondents (n=32) reported that the collaboration tools they used had

changed the design process.

• 59% agreed that the speed of the design process had improved.

• 82% agreed that the efficiency of the design process had improved.

• 47% agreed that that the quality of the design process had improved.

Thus, of those who perceived a change in the design process, a majority agreed that it

was a positive change. Additionally, the tools mostly affected the efficiency and speed of

the design process, with a lesser effect on the quality of the work produced.

Workflow-Time 1 [16 (Sec. 3.3.7 & Figure 1.6)]

CAR team members indicated the percentage of their work that fit into each of the

following types of workflow:

• Independent: co-workers perform work separately.

• Sequential: work flows between co-workers in one direction.

• Reciprocal: work flows between co-workers in a “back and forth” manner over

time.

• Team: co-workers diagnose, problem-solve and collaborate as a group at the

same time.

On average, the greatest mean percentage of the respondents’ work was conducted

independently (n=34, mean = 30%, s.d. = 22.49).

Workflow-Time 2 [16 (Sec. 3.3.8 & Figure 2.8)]

• On average, the greatest mean percentage (29%, s.d. = 23.42) of the respondents’

work was conducted as team work, in contrast to Time 1, where independent

work had the highest mean percentage.

• In general, team members seemed to be spending more time in reciprocal and

teamwork than independent at the conclusion of the study. The mean percentage

of sequential work did not change.

Working Relationships-Time 1 [16 (Sec. 3.3.9 & Figures 1.7, 1.8, 1.9)]

The surveys addressed two types of issues regarding relationships at work: group

identification and trust.
11

• In general, group identification was high. For example, a larger percentage of

respondents (96%) believed that criticism of the project was taken as an insult by

local, rather than by remote team members (37%).

• Overall levels of trust in both local and remote team members were high.

However, respondents reported consistently lower levels of trust for remote team

members than for local team members.

Working Relationships-Time 2 [16 (Sec. 3.3.10 & Figures 2.9, 2.10)]

• Group identification was not measured at Time 2 [16 (Sec. 2.6.1].

• Overall levels of trust seemed to have remained at the same high level from the

beginning of the study.

• Although not statistically significant, the authors of the full report reported an

increased disparity at Time 2 between trust in local and remote co-workers.

• Trust was lower for remote team members than for local team members.

Coordination, performance and organizational citizenship-Time 1 [16 (Sec. 3.3.11 &

Figures 1.10, 1.11, 1.13)]

• Respondents reported greater difficulty in coordinating their work with remote

rather than local co-workers.

• There was a positive correlation between receiving timely information about

changes in current plans from remote co-workers and reported trust in remote coworkers.

• Team members indicated experiencing delays in work involving both local and

remote co-workers.

• A greater number of respondents (50%) reported a high frequency of delays (four

or more delays in the previous month) involving local co-workers than with

remote co-workers (42%).

• Most respondents (90%) reported a high average length of delay (one to several

days delay) in work involving remote co-workers. In contrast, less than half of the

respondents (48%) reported such lengthy delays with local co-workers.

• Difficulty scheduling common meeting times with remote co-workers was

positively correlated with frequency and average length delays involving remote

co-workers.

• In general, respondents reported high levels of organizational citizenship.8

Coordination, performance and organizational citizenship-Time 2 [16 (Sec. 3.3.12 &

Figures 2.11, 2.12)]

• As at Time 1, participants continued to have greater scheduling difficulties with

remote rather than local co-workers.

 

8

 Citizenship behaviors are those behaviors relating to cooperation with co-workers, helping team members,

creating a favorable environment, and so on [2, 10, 15].
12

• A high frequency of delays was more common with local co-workers, as at Time

1.

• The number of high frequency delays decreased at Time 2.

• High length of delay was more common with remote than local co-workers, as at

Time 1.

• Mean length of delay decreased.

• Time 2 organizational citizenship levels were slightly higher than at Time 1.

7.2 Comparing Means at Time 1 and Time 2 [16 (Sec. 3.4 & Appendix I – Tables 8,

9)]

Using a paired-sample t-test to compare before and after measures on the same sample

yielded only one significant difference. The mean percentage of time spent in

independent work changed significantly (p < .01), from 32.5% at Time 1 (standard

deviation (s.d.) = 22.49) to 21.25% (s.d. = 20.71) at Time 2. Some other variables

displayed relatively large differences in means, however, these differences were not

statistically significant.

One reason these findings may not be completely accurate in describing changes from

Time 1 to Time 2 is the nature of the statistical test. The paired-sample t-test eliminates

data from participants who did not respond in both Time 1 and Time 2 for each variable

in question. Because not all survey participants completed both surveys or answered all

the questions of both surveys, the sample size for the paired-sample t-test was often

reduced to 1/3 of the total participants who completed the surveys. The smaller the

sample size, the greater an effect must be in order to be determined as statistically

significant. Thus, differences in means that may seem large (like 17.92 total number of

days spent visiting other sites at Time 1 compared with 12.25 at Time 2), are not

statistically significant. Furthermore, the means used for the paired-sample t-test

represent only that data collected from the subgroup of participants who completed the

item both times. Thus, the entire Time 1 survey mean for total number of days spent

visiting other sites was actually 17.71 at Time 1 (approximately the same as the mean of

17.92 used for the statistical test) and 8.19 at Time 2 (much lower than the subgroup’s

mean of 12.25).

7.3 Relationships among variables [16 (Sec. 3.5 & Appendices G, H)]

UM used correlation analysis to determine whether variables were associated with one

another. All the correlation results listed in the following section are significant at p <

.05 unless stated otherwise.

Impact of collaboration tools [16 (Sec. 3.5.1)]

• Quality and efficiency improvements due to the collaborative tool intervention

were positively correlated with one another, as were efficiency and speed

improvements. Thus, the impact of NetMeeting may be described as a general
13

improvement. And, that impact can be characterized more specifically with

respect to coordination, performance, and trust among co-workers (see below).

• Application-sharing tool adoption was positively correlated with agreement that

NetMeeting had changed the design process.

• There was a positive correlation between the number of local and remote coworkers that participants communicated with daily, and perceiving a change in the

design work as a result of the implementation of NetMeeting. Thus, the greater

the number of team members with whom one communicated with on a daily

basis, the more likely one is to have perceived an impact on design work as a

result of tool implementation.

Coordination [16 (Sec. 3.5.2)]

• Difficulties with one type of coordination were associated with difficulties in

other coordination activities. Local and remote scheduling difficulties were

positively correlated. Receiving timely information about changes in plans from

local and remote co-workers was correlated positively. And, difficulty finding

local and remote co-workers was correlated positively. Furthermore, individuals

who perceived scheduling difficulties with local co-workers also perceived

greater difficulty in receiving information from remote co-workers.

• There was a positive correlation between the number of local co-workers with

whom one communicated and receiving timely information from local coworkers. It is possible that one’s position in the organizational hierarchy may

confound this association. For instance, senior team members, particularly

managers, are often likely to communicate with a larger number of employees

than do junior engineers. Additionally, they typically command faster response

rates.

• As a result of the NetMeeting intervention, perceptions of efficiency and speed

improvements in the design process, were positively correlated with the

perception that remote co-workers provided timely information regarding changes

in plans. This finding supports the notion that NetMeeting provided value in

working with remote co-workers.

• Coordination difficulties were associated with decreased individual citizenship

behavior and perceptions of other’s citizenship behavior. For instance, difficulty

in scheduling meetings with local co-workers was positively correlated with

difficulty in passing information to remote co-workers. It was also positively

correlated with difficulty in receiving information from local and remote coworkers. Therefore, as individuals perceived fewer barriers to coordinating with

local co-workers, they tended to perceive greater organizational citizenship from

their team members.

Performance [16 (Sec. 3.5.3)]

• Length and frequency of delay were significantly correlated to one another. For

instance, certain team members may be more likely to experience frequent, long
14

delays with both local and remote co-workers, while others experience few

delays, which are of shorter duration.

• There was a positive correlation between frequency of local, face-to-face workrelated communication and average length of delay due to remote co-workers.

This means that a greater incidence of face-to-face work with local co-workers

was associated with shorter delays in work caused by remote co-workers.

• Coordination difficulties related to performance issues. For instance:

• The greater ease one had in locating co-workers, the shorter the perceived,

average length of delay.

• Increased local coordination was associated with shorter delays in obtaining

work input from local team members.

• The more participants rated local co-workers as giving them timely

information, the shorter the average delays they experienced as a result of

local co-workers. And, similarly, for remote workers.

Therefore, it is possible that alleviating coordination problems may contribute to

performance improvements. The next two items examine the specific relationships

between collaboration-tool use and coordination, and collaboration-tool use and

performance.

• Coordination and NetMeeting use. There was only partial support for a

relationship between collaboration-tool use and improvements in coordination.

Regular use of NetMeeting was not significantly correlated with coordination.

However, of the team members who perceived that the design process had been

changed by NetMeeting use, coordination improvement was associated with

perceptions of efficiency and speed improvements in the design process as a result

of NetMeeting. These findings suggest that the relationship between NetMeeting

use and coordination improvements is unclear, and deserves further investigation.

• Performance and NetMeeting use. There were mixed results supporting a

relationship between the perceived impact of NetMeeting on the design process

and performance. Perceptions of improved efficiency due to the NetMeeting were

associated with a lower number of delays caused by remote co-workers. Yet,

indications that NetMeeting had affected the design process was correlated with a

higher frequency of delays due to remote co-workers. Furthermore, agreement

that NetMeeting improved the quality of the design process was associated with

increased length of delay due to local co-workers. These findings indicate that the

relationship between collaboration tool use and performance is not clear, because

the indicators of that relationship are mixed, both positive and negative.

Trust [16 (Sec. 3.5.4)]

• Higher levels of trust in local co-workers were associated with higher levels of

trust in remote co-workers.

• There was no specific relationship between NetMeeting use and trust. However,

perceptions that the use of NetMeeting improved the speed of the design process
15

were positively correlated with increased trust in remote co-workers and increased

trust overall.

• Ease of coordination was related to increase in trust. For instance, greater ease in

locating distant co-workers was associated with greater trust in local and remote

team members. Also, receiving timely information about changes in plans from

local and remote co-workers was positively correlated with increased trust in local

and remote co-workers.

• There was a positive relationship between performance and trust. For example,

lower average length of delays was associated with increased levels of trust in

local co-workers.

• Increased organizational citizenship related to an increase in trust. For instance,

receiving useful information from local and remote co-workers was positively

correlated with local, remote and overall trust.

These findings support the study’s original assumptions about the importance of being

able to coordinate with and rely on team members to build and maintain trust in the team.

Thus, understanding how collaboration tools play a role in affecting the coordinationperformance-trust triangle remains a question worthy of investigation.

It is more difficult to assess the exact relationship of collaboration-tool use and trust.

While there was no specific relationship between NetMeeting use and trust, the belief that

collaboration tools improved the speed of common work was related to increased trust.

8.0 Discussion [16 (Sec. 4)]

Through a variety of mechanisms, the CAR team did experience a significant increase in

use of collaborative tools over the period of the study. Although adoption of NetMeeting

was lower than anticipated, it did occur. CAR managers launched an effort independent

of the research study to encourage shared-calendar use. Additionally, there was some

spontaneous adoption of presence-awareness tools. The analysis of this use led to a

number of critical insights. They can be categorized in terms of three contributions to

understanding:

• The general role of collaborative tools in geographically-distributed teams and the

specific impact on geographically-distributed manufacturing engineering teams

• Techniques and methods for assessment and evaluation of technology-related

changes, particularly related to the introduction and adoption of collaborative

tools

• Practical recommendations to organizational leaders in the manufacturing sector,

and more broadly, about how and when to introduce collaborative tools to ensure

maximum success

8.1 Adoption and use of collaborative tools [16 (Sec. 4.1.1)]

To summarize, fifteen members of the CAR team, who were judged by their management

to have the greatest need for NetMeeting features, were trained on NM use. This strategy

succeeded in exposing NetMeeting to a significant fraction of the CAR engineers. For

example, from a baseline level of no use, 36% of the CAR team reported some level of

NetMeeting use by the end of the study period. However, all reported users were from
16

European sites. [Note: Researchers were aware of some regular use at U.S. sites, but

these engineers did not respond to the post-intervention survey.] A small number of team

members, 6%, reported regular use of NetMeeting. No formal deployment effort was

involved for the other two recommended collaborative tools, shared-calendar and

presence-awareness applications. By the end of the study, 97% of the CAR team

reported some level of shared-calendar use. And, a large number of team members, 82%,

reported regular, shared-calendar use. Adoption of presence-awareness tools was much

lower, with some reported use by 36% of respondents by the end of the study.

An unresolved question is why adoption of the calendar tool was so much broader than

the adoption of NetMeeting and presence-awareness tools. Prompted by the requirements

analysis, the lead managers within the CAR team decided to recommend use of a sharedcalendar tool. However, NetMeeting was also endorsed strongly by management. One

factor in the differential adoption rates may have been the types of capabilities provided

by the two tools. For the CAR team, there was a smaller number of engineers who

benefited from the relatively specialized capabilities of NetMeeting versus those who

benefited from the relatively generic capabilities of a shared-calendar tool. That is,

scheduling is a more universal need. By contrast, a tool for application sharing appeals

mainly to those workers who must collaborate with distant colleagues and must do so in

real-time, as when viewing a common drawing or document. Finally, the manager of the

one division observed that most of their work with EU colleagues was concentrated on

the French manufacturing site – and not the German product development site. At the

time of the study, the French site had very poor network connectivity. As a result, low

NetMeeting use might not have reflected the team members’ desire to use the tool, but

rather reflected the low performance of the network links between the U.S. and the

French location.

8.2 Impact of collaborative tools [16 (Sec. 4.1.2)]

When team members were asked about the collective impact of collaboration-tool use on

changes in the manufacturing design process, the overall response was that 50% felt that

changes had occurred. However, focusing on those team members who used the various

tools, the numbers were different. For example, among NetMeeting users 66% reported

change compared to 37% among non-users – while only 46% of calendar and 45% of

presence-awareness users reported change. We conclude that NetMeeting use produced a

more profound perception of change among users than either shared-calendar or

presence-awareness use.

Of those team members who reported changes, 56% of NetMeeting users felt that

efficiency and speed had improved, although only 22% felt that quality had improved.

Among the shared-calendar users who reported changes, 83% felt that efficiency had

improved, 75% felt that speed had improved, and 50% felt that quality had improved.

Finally, among the presence-awareness users who reported changes, 80% felt that

efficiency had improved, 60% felt that speed had improved, and 60% felt that quality had

improved. Therefore, it appears that the primary impact across all collaboration-tool use

was on efficiency and speed, with a smaller impact on quality.
17

Perceptions of efficiency improvements and perceptions of speed improvements were

positively correlated with the perception that distant co-workers provided timely

information (r=0.72 and r=0.66, respectively; both p values <0.05). This relationship

suggests that use of collaborative tools did reduce some of the difficulties indicated in the

initial round of interviews, such as interruptions in mid-conversation to send email

attachments of drawings.

The analysis of critical performance outcomes, such as the frequency and duration of

delays with respect to the collaboration tool indicated no specific relationships. Further,

team members were asked about critical process issues, like frequency of travel. While

frequency of travel did decline according to responses at the second survey

administration, there was no significant association of this decline with use of the

introduced collaboration tools. A division manager, when questioned about the reduced

travel, attributed the reduction to the natural ebb and flow of travel corresponding with

different periods of the manufacturing design cycle. That is, in his opinion, higher travel

at the time of the first survey administration probably reflected a more intensive overall

level of interaction between the European and U.S. sites than at the time of the second

survey.

Finally, team members responded to questions about critical social and psychological

factors, such as trust. Research expectations were that trust might be improved as

distributed team members increased opportunities to establish common ground and

overcome spatial and cultural barriers by using collaborative tools. However, there was

no specific relationship between the use of collaborative tools and trust. There was a

factor associated with collaboration-tool use, perception of improved speed, that was

positively related to trust in remote co-workers, and overall increased trust, but not in

local co-workers alone. This suggests that greater responsiveness plays a role in

assessments of trustworthiness, and to the extent that collaborative tools improve

responsiveness, they may be expected to influence perceptions of trustworthiness.

8.3 Assessment techniques, challenges and observations [16 (Sec. 4.2)]

The main focus of this research was characterization of the use and impact of

collaborative tools within a distributed, engineering team. However, an important,

secondary goal was the demonstration of techniques for assessment and evaluation of

collaboration-tool interventions that could be generalized to other field settings.

Over the course of one year at Auto 1 UM researchers performed the following tasks:

• Made dozens of visits to CAR team sites, including two trips to European locations

• Organized three “all hands” meetings involving Auto 1 upper management, CAR

team management, the Auto 1 IT organizations, and UM and NIST researchers

• Developed, administered, and analyzed two surveys

• Developed and implemented a complete technology deployment, including one-onone training, and tracked the consequences of the deployment for six months

These efforts were conducted against a corporate backdrop at Auto 1 that included:

• Uncertainty with respect to the ultimate success of the CAR team’s product
18

• Uncertainty concerning the legal relationship between Auto 1 and the University of

Michigan concerning the overall research project, particularly with respect to

intellectual property

• The departure, halfway through the research period, of the Auto 1 vice-president who

invited and nurtured the research project

These difficulties in conducting the research reflect the first crucial lesson in terms of the

assessment and evaluation strategy. Doing research in the field, within actual working

teams, requires enormous cooperation and compromise. For example, UM wanted to

introduce three collaborative tools. Because of Auto 1 legal and operational concerns,

they were only allowed to introduce one tool.

Second, having negotiated access to Auto 1 on the basis of minimizing demands placed

on the Auto 1 IT organizations, UM researchers were placed in the position of conducting

the bulk of training and also debugging problems with applications, such as NetMeeting

server failures. This meant that tremendous effort was expended to bring users to a

minimal level of competency with the target collaborative tools and to address tool

failures. Under other circumstances, that effort might have gone directly into observation

of the tools in use or to enhancing practices with the tools.

Third, the surveys had the advantage of gathering data broadly at relatively low cost.

However, low response rates and the small number of respondents for both surveys,

reduced the capacity to draw useful conclusions. For example, the post-intervention

survey showed no US-based NetMeeting users. However, interview and site visit

experience suggested that there were regular NetMeeting users at U.S. locations.

Fourth, overcoming inertia in organizational settings is very difficult. For example, if a

solution to a collaboration issue already existed, there was little incentive for participants

to change to another solution. This was true for NetMeeting use. Frequent users of email

attachments found NM to be a benefit to them, but those who used attachments

infrequently did not use NetMeeting. Similarly, another subgroup used an applicationsharing tool other than NetMeeting. Since this solution was adequate for them, they did

not find it necessary to change applications – this was communicated in a follow-up

interview. This was true even though the lead engineer of this subgroup was perceived

by management to be technology-savvy and an early adopter of technology. Learning the

new tool was not a problem, but there was little incentive for him to change.

Fifth, UM’s entrée to Auto 1 and early development of the project was greatly facilitated

by the enthusiastic support of the Vice-President (VP) for Human Resources in one of the

divisions. This VP played an essential role in helping negotiate across the two divisions

involved with the CAR project. The significance of this influence reinforces the

importance of a high-level champion within the field organization [2]. Momentum on

this project slowed considerably when this high-level champion left Auto 1.

Finally, UM’s status as outsiders allowed them to probe the CAR team members to reveal

practices and procedures that Auto 1 personnel did not have the time or inclination to
19

explore. However, as outsiders, they confronted a very steep learning curve to

understand the content of the engineering work. Again, effort expended on mastering the

elementary aspects of the engineers’ tasks could have been allocated to more detailed

examination of performance of the tasks – and the relationship between these tasks and

collaborative tools. In this way, inclusion of a veteran automotive engineer, especially

one from Auto 1, on the research team might have accelerated basic understanding of

CAR team tasks. Yet, in the time and resource starved environment where CAR team

members worked, it was unlikely that CAR team management or Auto 1 would have

consented to an engineer/designer acting in a formal, liaison role.

8.4 Practical recommendations [16 (Sec. 4.3)]

While the CAR team’s experiences could be considered representative of the universal

difficulties and challenges confronted by multi-site engineering organizations, it was a

single team within one organization. Therefore, any generalizations should be made with

appropriate caution. Specifically, unique features of this research setting included the

history of very autonomous divisions within Auto 1. This autonomy may have

contributed to greater than normal difficulties in cross-division communication and

cooperation. Also, the strategic importance of the CAR project, and the corresponding

high visibility of the project within Auto 1, may have increased the team’s motivation to

perform. It is conceivable that the team performed to the point where the addition of

collaborative tools had less impact than would be the case for a team not in the spot light.

Finally, because of the cross-cultural interaction within the CAR team, multi-site

collaboration issues may have been confounded with cultural and language issues.

Although, most team members spoke and understood English very well.

With these caveats in mind, there are a number of practical recommendations that can be

made based on the CAR team’s experiences. These recommendations fall into two

categories: a) advice to managers and engineers about adoption and use of collaborative

tools; and b) advice for future researchers studying the impact of collaborative tools in

engineering settings.

8.4.1 Advice to managers and engineers [16 (Sec. 4.3.2)]

The foremost conclusion of this study is that collaborative tools must meet a specific

need to merit the effort of deployment, adoption, and subsequent use. For example, the

requirements gathering effort conducted early in this project successfully highlighted

difficulties, or constraints, experienced by the CAR team members while attempting to

conduct cross-site work. At least for some of the CAR engineers, the set of introduced

collaborative tools met identified needs. Yet, for many, the available tools still imposed

too great a burden to learn and master, relative to the perceived benefits. That is, for

those who interacted heavily with colleagues at distant sites, the collaborative tools that

were introduced were well received. For others, existing solutions were sufficiently

satisfactory to discourage investigation of alternatives or the infrequent interaction with

distant colleagues did not prompt use of the new tools.

There were also a number of engineers who wanted to use the collaborative tools, but

couldn’t due to infrastructure deficiencies or deficiencies in the tools themselves. For
20

instance, there was extremely poor network connectivity between France and the U.S.

Additionally, the application-sharing tool did not support poster- or wall-sized drawings.

Finally, it may be the case that collaboration technologies supporting both synchronous

and asynchronous modes might have better supported the CAR engineers. NetMeeting

supports synchronous collaboration only. The introduction of asynchronous collaboration

technology might have been a useful complement to the synchronous applications,

particularly given the small number of overlapping work hours between Europe and the

U.S. However, adoption and use of other tools could also have been vulnerable to

deficiencies in the available network infrastructure and user inertia.

A second conclusion is that targeted changes in work practices, such as the adoption and

use of collaborative tools, need adequate support to ensure success. The required support

level varies with the complexity and novelty of the proposed tool. For example, a factor

in the widespread adoption of shared, calendar tools within the CAR team was certainly a

familiarity with calendars generally, and the seamless transference of knowledge and

practices with paper calendars to online, shared calendars. By contrast, for most CAR

engineers, NetMeeting represented a completely new tool, with no analogs from past

practice. Engineers were asked to master both the tool itself, and the choreography of

working with a distant colleague via the tool. The significant investment of one-on-one

training surrounding the introduction of NetMeeting was successful for individual users.

However, it was not conducted on a broad enough scale to create a critical mass of use

for the team in the timeframe of this study.

Less successful was the approach of finding high visibility users to model NetMeeting

use, and by example, stimulate wider adoption of the tool. At least within the CAR team,

engineers had much to do in performing the normal aspects of their jobs. It was

unrealistic to expect that engineers would devote time to instruct colleagues in the use of

a sophisticated and complex tool. In addition, some of the users identified as the most

influential adopters of new technology had already taken the initiative of finding

solutions to their collaboration needs by using an alternative tool. This alternative

solution detracted from the effort to broaden the base of NetMeeting users. Although the

approach was not as successful as was hoped, at the conclusion of the study, there was

evidence within one German site of spontaneous training among engineers beyond the

initial, targeted users and diffusion of NetMeeting-use.

A final conclusion is that successful deployment of collaborative tools requires a

coordinated approach that involves the IT support organization at an early stage. This

project had early endorsement from the IT managers within the CAR team. Also, within

Auto 1, the IT managers reported to the human resources organization and the key

internal sponsorship came from a human resources vice president. The early

endorsement meant that within the constraints of their limited resources, the Auto 1 IT

managers were able to smooth many potential barriers to the project. For example, in the

course of conducting other business at the European sites, IT personnel checked on the

installation of NetMeeting on engineering workstations. IT personnel also installed and

maintained an internal NetMeeting server.
21

8.4.2 Advice to researchers [16 (Sec. 4.3.3)]

The foremost practical recommendation for future research on collaborative tools in field

settings would be to anticipate competing demands in this type of research. Specifically,

adoption of novel tools with accompanying novel practices is not something that unfolds

on a time scale consistent with most projects. Therefore, researchers will confront the

need to perform some level of “pump priming.” This means there has to be some level of

collaboration-tool adoption and use to produce behavior and outcomes that can be used to

evaluate the impact of collaborative tools. Realistic determination of resources required

to produce use, particularly where none currently exists, is difficult. In this study, for

example, UM researchers robustly documented the enthusiastic need for features

contained in tools like NetMeeting. Yet, for various reasons, stimulating sufficient levels

of use remained a great labor. This labor, in the form of one-on-one training, follow-up

visits, and on-going encouragement of use came at the expense of more detailed

documentation of what was going on within the CAR team in general, and more

specifically, what people were doing with collaborative tools. The compromise solution,

including the two survey administrations, was one way to balance the competing

demands of tool deployment and training and data gathering. However, as noted earlier,

this solution had certain liabilities, such as a poor ability to tease out details of

engineering practices and the failure to capture data from all team members.

9.0 Concluding remarks

The primary goals of this research were an assessment and better understanding of the

adoption, use, impact, and evaluation of collaborative tools in a manufacturing setting.

Because the collaborative tool intervention was specifically targeted to needs identified in

the requirements gathering phase, a primary research expectation was to find a positive

impact of tool use. While there was some indication of improved team performance with

use of the tools, the overall adoption did not meet expectations. Although results were

mixed, assessment of collaborative tool intervention is often difficult, even at the very

coarse granularity of “success” and “failure” [1]. Therefore, one important outcome of

this research is the well-documented case study [16], so that over time results can be

interpreted more fully, and future research and collaborative tool deployments draw from

it.

References

[1] Blythin, S., J. Hughes, S. Kristoffersen, T. Rodden, and M. Rouncefield. Recognising

‘Success’ and ‘Failure’: Evaluating Groupware in a Commercial Context. Proceedings of

GROUP’97 (Phoenix, AZ, November 1997), ACM Press, p. 39-46.

[2] Bolon, D. Organizational Citizenship Behavior Among Hospital Employees: A

Multidimensional Analysis Job Satisfaction and Organizational Commitment. Journal of

Healthcare Management, 42(2), 1997, p. 221-241.
22

[3] Box, G., W. Hunter, and J. Hunter. Statistics for Experimenters. John Wiley & Sons,

Inc, New York, 1978.

[4] DeSanctis G. and B. Jackson. Coordination of Information Technology: Team-based

Structures and Computer-based Communication Systems. Journal of Management

Information Systems, 10, 1994, p. 85-110.

[5] Dix, A., J. Finlay, G. Abowd, and R. Beale, Human Computer Interaction, 2nd Ed.,

Prentice Hall, Europe, 1998.

[6] Ehrlich, S. Strategies for Encouraging Successful Adoption of Office Communication

Systems. ACM Transactions on Information System, 5, 4, 1987, p. 340 – 357.

[7] Finholt, T., E. Rocco, D. Bree, N. Jain, and J.D. Herbsleb. NotMeeting: A Field Trial

of NetMeeting in a Geographically-distributed Organization. SIGGROUP Bulletin, 20(1),

1998, p. 66-69.

[8] Glaser, B. and A. Strauss. The Discovery of Grounded Theory: Strategies for

Qualitative Research. Aldine, Chicago, 1967.

[9] Glaser, B. and A. Strauss. (1970) Discovery of Substantive Theory: A Basic Strategy

Underlying Qualitative Research. In Qualitative Methodology, W. Filstead (Ed.), Rand

McNally, Chicago, 1970, p. 288-297.

[10] Katz, D. The Motivational Basis of Organizational Behavior. Behavior Science 9

(2), 1964, p. 131-33.

[11] Mark, G., J. Grudin, and S. Poltrock. Meeting at the Desktop: An Empirical Study

of Virtually Collocated Teams. Proceedings of ECSCW’99, Copenhagen, Denmark,

1999, p. 159-178.

[12] Miles, M. and A. Huberman. Qualitative Data Analysis: A Sourcebook of New

Methods. Sage Publications, Newbury Park, CA, 1984.

[13] Mohrman, S., J. Galbraith, E. Lawler and Associates. Tomorrow’s Organization:

Crafting Winning Capabilities in a Dynamic World. Jossey-Bass, San Francisco,

CA,1998.

[14] Olson, G., and J. Olson. Distance Matters. Human-Computer Interaction. 15 (2-3),

2000, p.139-178.

[15] Organ, D. Organizational Citizenship Behavior: The Good Soldier Syndrome.

Kexington Books, Lexington, MA, 1988.
23

[16] Wierba, E. and T. Finholt. A Manufacturing Collaboratory Case Study. National

Institute of Standards and Technology. GCR 01-811; 2001 May. (Available at:

http://www.nist.gov/msid/collaboratory/doc/gcr-811.pdf)