U.S. Army Research Institute

for the Behavioral and Social Sciences

Research Report 1862

Assessing Army Professional Forums - Metrics for

Effectiveness and Impact

Anna T. Cianciolo

Global Information Systems Technology, Inc.

Charles G. Heiden

Human Resources Research Organization

Michael I. Prevou

U.S. Command & General Staff College

October 2006

Approved for public release; distribution is unlimited.

20070402125
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate Of the Department of the Army

Deputy Chief of Staff, G1

Authorized and approved for distribution:

STANLEY M. HALPIN MICHELLE SAMS

Acting Technical Director Acting Director

Research accomplished under contract

for the Department of the Army

Global information Systems Technology, Inc.

Technical review by

Kimberly Metcalf, U.S. Army Research Institute

Joseph Psotka, U.S. Army Research Institute

Kathleen A. Quinkert, U.S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS,

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from... to)

October 2006 Final February 2005 - September 2005

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

W74V8H04D0044

Assessing Army Professional Forums - Metrics for 5b. PROGRAM ELEMENT NUMBER

Effectiveness and Impact 665803

6. AUTHOR(S) 5c. PROJECT NUMBER

Anna T. Cianciolo (Global Information Systems Technology, D730

Inc); Charles G. Heiden (Human Resources Research 5d. TASK NUMBER

Organization); Michael I. Prevou (U.S. Command & General 269

Staff College)

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

Global Information Systems Technology, Inc.

100 Trade Center Drive -Suite 301

Champaign, Illinois 61820

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

ARI-LDRU

U. S. Army Research Institute for the Behavioral & Social

Sciences 11. MONITOR REPORT NUMBER

2511 Jefferson Davis Highway Research Report 1862

Arlington, VA 22202-3926

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public distribution; distribution unlimited

13. SUPPLEMENTARY NOTES

Contracting Officer's Representative and Subject Matter POC: Joseph Psotka

14. ABSTRACT (Maximum 200 words):

The rate of change in the operational environment outpaces the development of doctrine and

schoolhouse instruction, leaders must direct their own development in order to adaptively and

professionally meet the challenges brought on by Army transformation. Army professional forums (APFs),

powered by advances in collaborative toolsets and multimedia presentation software, provide a means for

leader self-development and professional growth. The research was conducted as an initial exploration of

this area.

The early stages of this initiative present the most valuable opportunity to establish checks that

ensure the initiative is functioning effectively and meeting organizational goals. Establishing such checks

enhances the organizational impact of the initiative. The APF assessment framework developed in the

present research enables the assessment of APFs during the early stages of the Army knowledgemanagement initiative.

15. SUBJECT TERMS

Leadership, Knowledge Sharing, Army Leaders, Organizational Effectiveness, Army Professional Forums,

Assessments, Leader Competence

SECURITY CLASSIFICATION OF 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

_ .... ABSTRACT OF PAGES

16. REPORT 17. ABSTRACT 18. THIS PAGE Ellen Kinzer

Unclassified Unclassified Unclassified Unlimited 120 Technical Publication Specialist

703-602-8047
ii
Research Report 1862

Assessing Army Professional Forums - Metrics for

Effectiveness and Impact

Anna T. Cianciolo

Global Information Systems Technology, Inc.

Charles G. Heiden

Human Resources Research Organization

Michael I. Prevou

U.S. Command & General Staff College

Leader Development Research Unit

Stanley M. Halpin, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

October 2006

Army Project Number Personnel and Training

665803D730 Analysis Activities

Approved for public release; distribution is unlimited.

iii
ACKNOWLEDGEMENT

We gratefully acknowledge Mr. Stephen McComb of Triple-I Corporation for his tireless

assistance of our effort. We are also grateful for the thoughtful checks on our thinking and

valuable discussion provided by Ms. Michele Costanza of Triple-I Corporation and Majors Brad

Hilton and Tony Burgess. Dr. Joseph Psotka provided valuable guidance. We are indebted to the

generosity of our interviewees who volunteered their time to ensure our effort was on track.

iv
ASSESSING ARMY PROFESSIONAL FORUMS - METRICS FOR EFFECTIVENESS AND

IMPACT

EXECUTIVE SUMMARY

Research Requirement:

As the rate of change in the operational environment outpaces the development of

doctrine and schoolhouse instruction, leaders must direct their own development in order to

adaptively and professionally meet the challenges brought on by Army transformation. Army

professional forums (APFs), powered by advances in collaborative toolsets and multimedia

presentation software, provide a means for leader self-development and professional growth.

The Army senior leadership recognizes the importance of agility and professionalism in

its leaders and is supporting four active APFs with nearly twenty more in development as part of

the broader Army knowledge-management initiative. The ability to assess the functioning and

impact of an APF on individual, unit, and organizational performance is key to ensuring that

APFs provide return on investment to the Army and meet the developmental needs of their

members. Yet, tools and methods for assessing the functioning and impact of APFs remain

largely unexplored. The purpose of the present research was to conduct an initial exploration of

this area.

Procedure:

In this research, we proposed a general framework for evaluating APFs. The framework

posits a causal chain linking the structure and characteristics of the APF ultimately to enhanced

Army effectiveness through activity in the APF, changes in individual member capability, and

improvements in unit effectiveness. We used the framework to assess S3-XO.net, an APF geared

towards battalion and brigade operations officers (S3s) and executive officers (XOs).

Findings:

The proposed APF assessment framework is an efficient means for determining whether

APF structure and characteristics and APF activity are adequate for achieving impact. In

addition, the framework provides a starting point for evaluating individual-, unit-, and

organization-level impact through linking activity in the forum to key challenges jointly faced by

APF members and the organization.

When the framework was used to assess S3-XO.net, we found that although the APF has

a sound technological foundation it currently lacks the member activity necessary to achieve

significant impact. Generating a higher level of activity in the forum requires the establishment

of a core group of active participants combined with more active facilitation of conversation. A

deeper understanding of the needs of the forum's target audience would also aid in effective

facilitation and generation of activity.

v
Utilization and Dissemination of Findings:

The early stages of an initiative present the most valuable opportunity to establish checks

that ensure the initiative is functioning effectively and meeting organizational goals. Establishing

such checks enhances the organizational impact of the initiative. The APF assessment framework

developed in the present research enables the assessment of APFs during the early stages of the

Army knowledge-management initiative. The framework also serves a cost-effective prerequisite

to comprehensive individual, unit, and organizational effectiveness assessment as the initiative

reaches more advanced stages.

vi
EVALUATING ARMY PROFESSIONAL FORUMS - METRICS FOR EFFECTIVENESS

AND IMPACT

CONTENTS

Page

IN T R O D U C T IO N ......................................................................................................................... 1

KNOWLEDGE SHARING, THE ARMY LEADER, AND ORGANIZATIONAL

E X C E L L E N C E .............................................................................................................................. 2

The APF - Serving the Passionate Professional ........................................................................ 2

Turning Passion Into Im pact ...................................................................................................... 3

Demonstrating Impact to High-Level Decision Makers ........................................................... 4

A Proposed Causal Framework Linking APF Activity to Organizational Effectiveness .......... 4

APF Structure and Characteristics ................................................................................ 4

Social and Intellectual C apital ...................................................................................... 5

Leader Competence and Professionalism ...................................................................... 6

Organizational Effectiveness - Unit ............................................................................. 6

Organizational Effectiveness - The Army .................................................................... 6

ASSESSING APF STRUCTURE AND CHARACTERISTICS ............................................... 7

Structure and Characteristics Necessary for Facilitating APF Activity .................................... 7

T h e P lan ............................................................................................................................. 7

P erso n n el ........................................................................................................................... 9

Functional R equirem ents ............................................................................................. 10

Technical R equirem ents ............................................................................................... 12

Structure and Characteristics - A Proposed Framework for Assessment ................................ 12

ASSESSING SOCIAL AND INTELLECTUAL CAPITAL ................................................. 16

Social and Intellectual Capital Present in a Healthy Professional Forum ............................... 16

C o n n ectio n s ....................................................................................................................... 17

C o n tex t .............................................................................................................................. 1 8

C o n ten t .............................................................................................................................. 18

C on v ersatio n ...................................................................................................................... 2 0

Social and Intellectual Capital - A Proposed Framework for Assessment ............................. 21

ASSESSING LEADER COMPETENCE AND PROFESSIONALISM ................................. 26

ASSESSING ORGANIZATIONAL EFFECTIVENESS - UNIT ........................................... 27

ASSESSING ORGANIZATIONAL EFFECTIVENESS - THE ARMY ................................ 27

ILLUSTRATION - APPLYING THE METRICS TO EVALUATING S3-XO.NET ............ 28

CON CLU D IN G REM A RK S .................................................................................................... 28

R E F E R E N C E S ............................................................................................................................. 3 1

APPENDIX A - LIST OF INTERVIEWEES ........................................................................ 37

APPENDIX B - ASSESSMENT OF S3-XO.NET .................................................................. 39

APPENDIX C - CONDUCTING ASSESSMENT OF THE ARMY PROFESSIONAL FORUM

..... ............................. . ................................................ .. ................................... ........ . . . . 1 01

APPENDIX D - FORUM MEMBER SURVEY ....................................................................... 107

A PPEN D IX E - A CR ON Y M S .................................................................................................. 111

vii
CONTENTS (continued)

LIST OF TABLES

Page

Table 1. Assessment Framework for the APF Plan ................................................................. 14

Table 2. Assessment Framework for APF Personnel .............................................................. 15

Table 3. Assessment Framework for Functional Requirements ............................................... 16

Table 4. Assessment Framework for Connections ................................................................... 22

Table 5. Assessment Framework for Context .......................................................................... 23

Table 6. Assessment Framework for Content .......................................................................... 24

Table 7. Assessment Framework for Conversation ................................................................. 25

T able 8. S3-X O .net Plan Status ................................................................................................. 41

Table 9. S3-X O .net Personnel Status ....................................................................................... 44

Table 10. S3-XO.net Functional Requirements Status ............................................................. 46

Table 11. S3-XO.net Connection Characteristics Status .......................................................... 53

Table 12. Conversation Posts by S3-XO.net Member Role ...................................................... 55

Table 13. Knowledge Posts by S3-XO.net Member Role ........................................................ 56

Table 14. S3-XO.net Context Characteristics Status ............................................................... 58

Table 15. S3-XO.net Content Characteristics Status ............................................................... 59

Table 16. Know ledge Post Locations ..................................................................................... 60

Table 17. Founding Officer Contributions to Knowledge Posts ............................................. 62

Table 18. S3-XO.net - Conversation Characteristics Status ................................................... 64

Table 19. Type of Initiating Posts for One-Post and Multi-Post Discussion Threads ............. 66

Table 20. A Proposed Framework for Assessing Potential Impact on Officer Values, Personal

Characteristics, and Professionalism (Be) .............................................................................. 82

Table 21. A Proposed Framework for Assessing Potential Impact on Officer Tactical, Technical,

Conceptual, and Interpersonal Competence (Know) .............................................................. 83

Table 22. A Proposed Framework for Assessing Potential Impact on Officer Influencing,

Operating, and Improving Competence (Do) ......................................................................... 84

Table 23. A Proposed Framework for Assessing Potential Impact on Unit Effectiveness ..... 92

Table 24. A Proposed Framework for Assessing Potential Impact on Army Effectiveness ........ 93

LIST OF FIGURES

Page

Figure 1. Proposed Framework for Linking APF Activity to Organizational Effectiveness ......... 5

Figure 2. S3-X O .net M ain Page ................................................................................................ 40

viii
INTRODUCTION

It would be a profound understatement to claim that Army leaders must be agile in order

to meet the demands of the current operational environment. At a time when the Army's

organizational structure is undergoing the most significant change since the 1960's (Smith,

2005), leaders must command a wider range of missions than ever before, often within the

domain of a highly adaptive, asymmetric threat. Moreover, the increased involvement of U.S.

forces in full-spectrum operations is changing the fundamental role of the Army from

warfighting to more broadly supporting the political ends of the federal government through

peacekeeping, stabilization, counterinsurgency, and conventional warfighting, among others. As

the rate of change in the operational environment outpaces the development of doctrine and

schoolhouse instruction, leaders must direct their own development in order to adaptively and

professionally meet the challenges brought on by Army transformation.

Army Professional forums (APFs), powered by advances in collaborative toolsets and

multimedia presentation software, provide a means for leader self-development and professional

growth. As members of an APF, geographically distributed leaders can directly share practical

solutions, generate new ideas, and exchange lessons learned from operational experiences faster

than the institutional knowledge-dissemination process currently allows (Baum, 2005). Through

discussion and the development of social networks, APFs also enable the growth of a sense of

professional community and mutual obligation among their members, many of whom may never

meet face-to-face. The Army senior leadership recognizes the importance of agility and

professionalism in its leaders and currently supports four active APFs with nearly twenty more in

development as part of the broader Army knowledge-management initiative comprised of the

Battle Command Knowledge System (BCKS) and the Army Knowledge Online (AKO) portal.

The ability to assess the functioning and impact of an APF on individual, unit, and organizational

performance is key to ensuring that APFs provide return on investment to the Army and meet the

developmental needs of their members. Until the present research was conducted, however, this

topic was unexplored.

The purpose of the present research was to conduct an initial effort addressing this gap in

understanding of the effectiveness and impact of APFs. This report documents the assessment

framework we designed for evaluating the functioning and impact of APFs and the results we

achieved when applying our assessment framework to an established APF called S3-XO.net.

However, our goal was to establish impact-assessment metrics applicable across APFs. This

report therefore provides guidance for applying the metrics to APFs other than S3-XO.net. In

addition, we summarize the future research directions necessary to ensure the generalizability of

the proposed metrics and to understand the impact of knowledge sharing on the Army more

generally.
KNOWLEDGE SHARING, THE ARMY LEADER, AND ORGANIZATIONAL

EXCELLENCE

The APF - Serving the Passionate Professional

Knowledge-sharing programs have been adopted by several commercial organizations

whose leadership has recognized the importance of employee agility and initiative to competitive

advantage and organizational success. Serving as the foundation for best practice in knowledge

management is the community of practice (CoP; Crager & Lemons, 2003). Wenger, McDermott,

and Snyder (2002) define CoPs as: "groups of people that share a concern, a set of problems, or a

passion about a topic and who deepen their knowledge and expertise in this area by interacting

on an ongoing basis" (p. 4). Online CoPs provide a virtual roundtable where people with similar

concerns can exchange knowledge and work together toward innovative practical solutions

(Bobrow & Whalen, 2002). Online CoPs have generated significant return on investment through

increased productivity and decreased redundancy in commercial organizations that have

supported their development and implemented them effectively (Crager & Lemons, 2003).

The APF represents an extension of the online CoP because it goes beyond building

practical solutions to building the profession itself (Woodie, 2005). APFs differ subtly from

online CoPs. Whereas APFs share the design principles of online CoPs, the primary purpose of

APFs is to support professional conversation among forum members through moderated and

facilitated discussion. APF members do not share only a single common problem or class of

problems (e.g., maintenance troubleshooting; Bobrow & Whalen, 2002) as CoP members do.

Often, multiple CoPs are situated inside a single APF. Also, membership in these forums--unlike

in CoPs--consists of present, future and past practitioners spanning organizational boundaries.

Through voluntary participation, APF members provide a positive solution-oriented voice

within the Army profession. APFs such as companycommand.mil (CC.mil) and NCOTeam.org

reflect the passion for the profession and the devotion to learning and personal growth typical of

U.S. Army leaders. Consider the following quotes from members of these APFs:

"I was on this site whenever I logged onto a computer in Iraq. ... I'm definitely a

stronger and more well rounded NCO [noncommissioned officer] because of

NCOTeam.org.... When I did have a pretty big personal/professional issue, I addressed

my concerns to my peers on this site and received nothing but support and guidance.

An Army of One really doesn't mean an Army of individuals; we truly are one team

here."

- Staff Sergeant JL, Iraq

"I look forward to ... see what great information you all are going to put out. I can't wait

to share it with the other platoon leaders and the Company XO [executive officer]."

- Paul

"I really appreciate this web-site, and the newsletter. I arrived in Iraq in April and have

done a lot of growing up. CC[.mil] was a great help through some tough experiences.

Thanks." - Captain K, from IZ

2
"This resource has been, and continues to be, an invaluable tool for professionals to

interact and benefit from the collective knowledge of others in our profession of arms...

This forum is simply a tool that every NCO, from infancy to seasoned veteran, can be

assured that they are following in the footsteps of excellence..."

- Ist Sergeant JMD, Assistant Commandant, U.S. Army NCO Academy

CC.mil and NCOTeam.org have been heralded far and wide in the media, featured in best

practice reports (Snyder & Briggs, 2003), books (Dixon, Allen, Burgess, Kilner, & Schweitzer,

2005), news articles [Baum, 2005; 13th Corps Support Command, 2003], and even discussed on

the radio (National Public Radio, January 25, 2005). They are widely recognized for the strong

interpersonal bonding and active knowledge sharing among their members. Long-standing and

well-established APFs such as these represent the gold standard when evaluating the

effectiveness of efforts to cultivate and sustain APF activity.

Turning Passion Into Impact

Although it may seem common sense that active APFs have an impact on organizational

performance, an understanding of how APF activity generates organizational effectiveness is still

in its infancy. Put another way, it remains to be explained how passionate activity in an APF

results in individual, interpersonal, team, community, and unit growth and how this growth

ultimately promotes organizational excellence. Such an understanding is necessary to

demonstrate organizational impact, and is required to assure high-level decision makers that

money invested in staffing and running APFs is money not better invested in other leaderdevelopment endeavors. In addition, understanding the process by which APFs generate

organizational excellence helps knowledge-management administrators and designers determine

how best to facilitate the effectiveness of APFs by addressing weaknesses and enhancing

strengths. Understanding the impact process requires that a causal framework be developed to

link activity in an APF ultimately to organizational excellence (Crager & Lemons, 2003; Lesser

& Storck, 2001; Snyder & Briggs, 2003). Case studies from commercial organizations provide

the outline of such a causal framework and provide the first evidence that efforts to support

problem-focused knowledge sharing (i.e., in CoPs) can have a tremendous return on investment

(e.g., Crager & Lemons, 2003).

Developing a causal framework for understanding Army effectiveness is an exceedingly

difficult task for multiple reasons. First, Army effectiveness is not easily quantified (i.e., it is not

reflected in units of production or revenue dollars) or even conceptualized (e.g., the Army is

successful both when it wins conflicts and when it prevents conflicts through stabilization and

reconstruction). Second, when attempting to determine the impact of a particular APF, there is

the problem that nearly every aspect of Army effectiveness is determined by the collective

activity of members of that APF and other individuals. In business, by contrast, members of a

CoP are more likely to be focused on a circumscribed problem set and to represent a department

that makes a discrete contribution to productivity (e.g., repair service, see Bobrow & Whalen,

2002). Therefore, the impact of knowledge sharing on departmental effectiveness can be more

easily identified and departmental impact on overall organizational performance can be

estimated. Third, through BCKS and AKO, leaders have a very broad range of resources

available to them for knowledge acquisition and sharing. The adaptive cross-membership of

individual leaders in multiple communities makes isolating the impact of a particular APF on

3
organizational effectiveness difficult. Finally, adaptive cross-membership in multiple

communities (including those available on secure Internet connections and those on insecure

government and public connections) has implications for where leaders will go to meet their

needs given that they have limited time to spend. Understanding how activity in an APF affects

unit and organizational effectiveness therefore requires understanding how the environment in

which leaders are operating (e.g., wartime vs. peacetime, known vs. unknown enemy, and so on)

influences the needs of forum members and the ability to meet these needs in the forum.

Demonstrating Impact to High-Level Decision Makers

The difficulty of developing a causal framework does not mean that an attempt should

not be made to understand how APF activity affects organizational performance. If the present

analysis stopped at simply describing APF activity (though this is an important aspect of

determining impact), then the purpose of our research would not be fulfilled. High-level decision

makers would not have the information they require, and a bridge would not be built between

APF enthusiasts and those who must make strategic decisions about how to allocate limited

resources. Moreover, valuable information that could be used to enhance the effectiveness of

current APFs or guide the development of future APFs would not be gathered. What the

challenge in both developing a meaningful causal framework and demonstrating impact means is

that impact must be clearly understood within the context of the APF of interest and its members.

There is useful guidance for developing such an understanding in best-practice reports such as

Crager and Lemons (2003) and Lesser and Storck (2001). The key is to align metrics of APF

activity with organizational objectives and mission using as a shared reference point the tasks the

APF members must perform to do their jobs effectively. A rational analysis of this alignment is a

widely recognized critical first step not only in evaluating the impact of knowledge management

[see, e.g., Hanley, 2003; Crager & Lemons, 2003; Lesser & Storck, 2001; Department of the

U.S. Navy, 2001] but also for designing strategies that result in the successful creation,

sustainment, and growth of a knowledge forum itself (Snyder & Briggs, 2003).

A Proposed Causal Framework Linking APF Activity to Organizational Effectiveness

Figure 1 shows our proposed causal framework for linking APF activity to organizational

effectiveness. Each component of the framework is explained in the following five sections. The

assessment criteria and metrics following the presentation of the framework were determined

through a combination of literature review and user and stakeholder interviews (see Appendix

A).

APF Structure and Characteristics

The structure and characteristics of the APF serve as the foundation of our proposed

causal framework. Structure and characteristics represent the features of the APF that support

knowledge retrieval and the development of social connections that lead to discussion and the

development of new knowledge. These features are the determinants of APF activity and impact.

APF structure and characteristics may include interface design features/usability (e.g.,

organization of knowledge) and community features (e.g., member bios), as well as the

administrative backbone of the APF, including the composition of the core support team, the

4
organizational support for the APF, and the alignment of the APF goals with the organizational

mission, among other considerations.

SPF Structure and Characteristics

I

Social and Intellectual Capital

Individual Technical, Tactical,

SConceptual Competency

Interpersonal Competency/ Professional Self-Identity

Leadership Effectiveness 7 V

Sense of Professional Community

IT

Organizational Effectiveness -

The Unit I

Organizational Effectiveness -

The Army

Figure 1. Proposed Framework for Linking APF Activity to Organizational Effectiveness

Social and Intellectual Capital

Enabled by APF structure and characteristics is the development of social and intellectual

capital. If the APF is designed and administered effectively, members will be able to retrieve

actionable (i.e., relevant and timely) knowledge from the community. Effective APF design and

facilitation will also enable members to participate actively in developing a network of

individuals with whom to discuss novel problems and collaboratively generate new knowledge.

Discussion and the development of social capital are key characteristics of APFs that

differentiate them from knowledge repositories or portals (Kendall & McHale, 2003; Schweitzer,

2003). Forum members actively develop social and intellectual capital for there to be a

demonstrable impact of the APF on individual, unit, and organizational performance.

5
Leader Competence and Professionalism

The development of social and intellectual capital leads to improved leader competence

and sense of professionalism. Specifically, the acquisition and development of actionable

knowledge enhances individual technical, tactical, and conceptual competence. Leaders are

better equipped to handle such challenges as applying the military decision making process

(MDMP) or leveraging technology, for example, because they have been exposed to the effective

methods their peers have used to develop planning standard operating procedures or to work with

emerging digital systems.

Professional self-identity also develops through active knowledge sharing and

development because it is through this activity that the leader actively participates in gaining a

concrete sense of his role in his current duty position and the associated tasks, duties, and

responsibilities he must perform. Participating in the APF also fosters the skills and attitudes

necessary for continuous professional self-development.

Discussion with peers promotes a sense of professional community, a sense that one

belongs to a group of people who share similar duties and responsibilities, who play a similar

role in the larger organization, who face similar problems and challenges, and to whom one is

responsible for being respectful and helpful.

Interpersonal competency is enhanced through storytelling in which leaders share how

they handled novel and sensitive interpersonal issues. Leadership effectiveness also is enhanced

through storytelling in which leaders share how they handled authentic leadership issues. It is

through this storytelling that the tacit knowledge critical to leadership effectiveness is acquired

(Cianciolo, Antonakis, & Sternberg, 2004).

Organizational Effectiveness - Unit

Organizational effectiveness at the unit level is a result of the development of leader

competence and professionalism. The expert and mature leader is more technically competent, as

well as a more effective role model and team player, which is critical in supporting the day-today operations, morale, professionalism, and combat effectiveness of the unit.

Organizational Effectiveness - The Army

Organizational effectiveness at the Army level results in large part from the enhanced

functioning of its tactical warfighting units. In short, these units must see first, understand first,

act first, and finish decisively in full-spectrum operations. The organizational effectiveness of the

Army also is enhanced by achieving better results with fewer resources and retaining more highquality personnel. This result is achieved, in part, through increased professional selfdevelopment on the part of Army personnel and increased satisfaction with Army life.

Each of these sections of the causal framework, together with implications for metrics

and their associated measures will be discussed in detail in the following sections.

6
ASSESSING APF STRUCTURE AND CHARACTERISTICS

As described previously, an APF's structure and characteristics determine the level of

activity in the APF and the impact it can be expected to have on performance. If the APF does

not have a solid foundation, the forum will not be active and cannot be expected to generate

impact. Similarly, if an APF is found to be inactive and incapable of generating impact, the root

of the problem may be revealed by evaluating the effectiveness of the APF's structure and

characteristics. In order to determine the effectiveness of APF structure and characteristics, the

following three broad aspects of the APF foundation should be assessed in detail: (1) the plan on

which the development of the forum is based; (2) the personnel structure that supports forum

growth and activity; and (3) the degree to which the forum meets the general functional

requirements for online knowledge sharing and community building. An explanation of each of

these aspects of APF structure and characteristics is presented below, followed by proposed

metrics for assessing these aspects.

Structure and Characteristics Necessary for Facilitating APF Activity

The Plan

Several sources converge on the importance of starting the APF development process

with an initial plan for the forum (e.g., BCKS, 2005; Dixon et al., 2005; Gerber, 2003; Glennie &

Hickok, 2003; DON, 2001; Wenger et al., 2002). The plan ensures that the design of the forum is

aligned with member needs, which stimulates activity, and with the mission and goals of the

parent organization, which translates activity into impact. The key features of the plan include:

" A statement of the purpose of the forum

V The envisioned impact of the forum and the criteria to be used to assess this impact

v/ A description of the target audience, including the needs of this audience

V/ A statement of the cultural norms, core values, or conventions to be honored

V A specification of the roles and responsibilities of the supporting personnel

/ A functional and technical specification of the online meeting space

According to Dixon et al. (2005), defining a forum's purpose provides a framework for

the administrative team to use in determining the scope of their efforts -- to determine

membership criteria, the nature of content and its organization, and the functional requirements

of the forum. The forum type, or general purpose, may be (a) helping, where the purpose is to

provide a place where members help each other solve practical problems; (b) best practice,

where the purpose is to develop and disseminate best practice information; (c) knowledge

stewarding, where the purpose is to develop tools, insights, and approaches needed by forum

members in the field; or (d) innovating, where the purpose is to develop highly innovative

solutions and ideas (Wenger et al., 2002). Wenger et al. recommend that a forum be focused on

one of these general purposes in order to avoid overwhelming forum members. In addition, the

forum's general purpose should be determined by the overall strategic intent of developing the

forum in the first place.

7
The envisioned impact of the forum directly ties the forum's purpose to achieving the

parent organization's mission and goals. The statement of envisioned impact should specify how,

if the forum evolves according to its purpose, it will influence the attitudes and behavior of its

members in a way that benefits the organization (e.g., fosters organizational commitment,

shortens learning time, etc., see Lesser & Storck, 2001). This specification should result in

specific, concrete criteria to be used for later evaluating the impact of the forum.

Description of the target audience and its needs is critical. Wenger et al. (2002)

recommend that the scope of the target audience be wide enough to encompass new people and

ideas, but not so wide that members frequently are not interested in the content posted or find it

difficult to retrieve the information they seek (see also Gerber, 2003; Stuckey & Smith, 2004).

Ultimately, the target audience should include people who know and people who need to know

(Dixon et al., 2005). An important aspect of analyzing the needs of the target audience is

determining how essential the planned forum is to meeting these needs (BCKS, 2005). It should

be determined that the needs of the target audience (a) can actually be met by the planned forum;

and (b) cannot be better met by some other means. In addition, once the target audience's needs

are identified, those needs that are particularly important to the organization and those needs

about which forum members are especially passionate should be selected for initial focus in the

forum.

The plan should include a statement of the cultural norms, core values, or conventions to

be honored because such a statement sets the tone for the forum. Setting the correct tone for the

forum will build members' trust that knowledge shared or generated in the forum will benefit

their development and that participating in the forum will not have negative implications for their

career. Such trust is critical to the health of the forum (Gerber, 2003). Moreover, trust in the

forum, built through shared values, will help keep members engaged in the forum (Prevou &

Burgess, 2003).

Another key component of the plan is the specification of the roles and responsibilities of

the supporting personnel. Numerous sources converge on the following general categories of

supporting personnel: leader, core group, facilitators, and technical support (Dixon et al., 2005;

DON, 2001; Snyder & Briggs, 2003; Wenger et al., 2002). The plan should designate the specific

tasks assigned to each category of personnel, who will perform these tasks, and how much time

personnel are expected to spend on these tasks relative to other job duties. The communication

channels between categories of personnel should also be identified in order to facilitate timely

and effective communication.

The final component of the forum plan is the functional and technical specification of the

online meeting space. This specification should include a description of what activities the online

meeting space will support (e.g., discussion, solution sharing, social networking, etc.), the

interface features or functionalities the online meeting space must have to support these activities

(search capability, conversation posting and archiving, etc.), and the hardware/software and

security requirements necessary to successfully implement the online meeting space. Because the

same software platform (Tomoye Simplify) will be used to support all APFs, this component of

the forum plan is of greater importance for identifying future functional and technological

8
requirements for APFs as a whole than for identifying the same requirements for an individual

forum.

Personnel

As described above, there are four general categories of personnel required to initiate and

sustain a healthy APF. These categories and their requirements for serving the healthy APF are

described in more detail below.1

The leader, or community coordinator, of the forum serves the critical leadership

functions of providing an overall vision for the forum and helping supporting personnel to

achieve this vision through a well developed professional network and his/her role within the

formal organization (Wenger et al., 2002). The leader must be knowledgeable about the domain

(i.e., the general topic area of interest to forum members) addressed in the forum, though not

necessarily a subject matter expert, and must have a passion for the professional development of

the forum members (Wenger et al., 2002). Forum leaders must also take the initiative in fostering

the forum's long-term health even in the absence of subject matter expertise (Wenger et al.,

2002). Most importantly, the leader must be supported and willing to commit between 20 and 50

percent of his/her time to leading the forum. A common pitfall in forum leadership is to spend

more time cultivating activity in the online space of the forum (a role of the core group and

facilitators, see below) than cultivating offline enthusiasm for the forum through active

marketing and networking with community members (Wenger et al., 2002).

The core group is the foundation of the APF. Building a core group is among the critical

first steps in establishing an active community because it is through this group of highly active

members that the forum establishes its identity and sets the conditions for sustained growth

(Dixon et al., 2005; Wenger et al., 2002). The core group should comprise knowledgeable and

experienced members of the forum target audience because they must use their subject matter

expertise to make judgments about the importance and relevance of knowledge in the forum, and

to analyze and synthesize the knowledge (Dixon et al., 2005; Department of the U.S. Navy,

2001). In addition, the members of the core group must have a strong, wide social network in

order to have thorough visibility on the needs and interests of the target audience, to connect

members of the target audience to willing experts who can help, to garner support for the forum

in its initial stages of growth, and to be an advocate for the forum during institutionalization

(Dixon et al., 2005). Moreover, because the core group is the heart of forum facilitation, its

members should have subject matter expertise in online professional forums (both the technical

and behavioral aspects) or at least facilitating collaborative activity online. Finally, and perhaps

most importantly, the core group must have a passion for developing the profession, particularly

as it relates to the professional growth of the forum's target audience. This passion is critical to

attracting new members to the site, as well as expert contributors and financial supporters, and to

maintaining the level of involvement necessary to sustain the forum (Stuckey & Smith, 2004).

SIt should be noted that with the possible exception of forum leaders, individuals' duties may fall under multiple

categories such that, for example, a member of the core group also facilitates conversation or a facilitator provides

technical support. It is more important that the functions of the four personnel categories be served, than that each

category is comprised of different individuals.

9
Because professional forums do not maintain themselves, the activity of the facilitators is

absolutely critical to the health of the forum (Stuckey & Smith, 2004). Facilitators play the key

roles of setting the tone in the forum and responding to member needs for knowledge and

discussion in particular topic areas. To set the tone, facilitators must monitor discussion threads

to keep conversation moving in a productive direction and to ensure that the core values of the

forum are being honored. Facilitators respond to member needs in a variety of ways, including

(a) seeking out and organizing relevant knowledge that can be posted in the forum for easy

viewing, retrieval and discussion; (b) identifying subject matter experts who can contribute to the

discussion of particular topics; and (c) linking members with questions to members with answers

(Gerber, 2003; Stuckey & Smith, 2004).

In the initial stage of forum development, the leader and core group work together to

identify and send out personal invitations to representatives of the target audience ("pointmen")

to participate in the forum and spread the word that the forum is up and running (Dixon et al.,

2005). These individuals also identify "command contacts," subject matter experts who are

willing to support the professional development of the forum members (Dixon et al., 2005).

Facilitators also participate in identifying and contacting pointmen and command contacts,

especially as the forum begins to grow. The leader, core group, and facilitators must form a

cohesive, mutually supportive, and passionate team through open communication, camaraderie,

and desire to grow the profession (Dixon et al., 2005).

The technical support staff translates functional requirements into technical requirements

and interface design features. The technical support staff is therefore responsible for designing

and implementing the collaborative environment, to include software, hardware, Internet, and

security requirements. Moreover, technical support staff members must maintain the

collaborative environment, ensuring that (a) knowledge and individuals within the forum are

easy to locate; (b) interface features (e.g., hotlinks, forms) work as intended; (c) additional

features needed by forum members or support personnel (e.g., content rating forms, survey

forms, synchronous chat) are implemented; (d) database functions (e.g., address book, activity

metrics, knowledge search) work properly; and (e) facilitators and other support personnel are

trained to use the software. Technical support staff members should be experts in both

knowledge management and information technology.

Functional Requirements

Functional requirements are those requirements the online meeting space must meet in

order to provide an environment suitable for an online community. Fundamentally, this

collaborative environment must support APF members' ability to share and discuss content and

to develop social connections (Nahapiet & Ghoshal, 1998; Stuckey & Smith, 2004). The online

meeting space must therefore provide a place where APF members can engage in discussion,

especially asynchronous discussion (see Schweitzer, 2003), about the topics that interest them.

Members must be able to locate content of interest and identify other members with whom it will

be beneficial to discuss that content. Because the typical APF user is likely to be limited in his

free time, he must be able to locate content and other members extremely quickly (i.e., "three

clicks" or less), else he will find some other means to meet his needs or simply not meet them at

all.

10
The first step in meeting these functional requirements is that the APF itself be easy to

locate and easy to join. This requires that prospective members either be able to locate the APF

through BCKS, AKO, The Center for Army Lessons Learned (CALL), Google, installation

homepages, or some other readily available means or that they have heard of the APF through its

members, the APF leadership and/or core group, instructors in the schoolhouse, commanding

officers, mentors, or other colleagues (e.g., observer/controllers at the combat training centers).

Prospective members must be able to join the forum with minimal effort (e.g., application forms

are simple to fill out) and there must be a very short time lag between applying for membership

and being granted membership (Prevou & Burgess, 2003), Importantly, potential APF members

must feel that membership restrictions are logical and fair given the mission of the forum, else

they will feel overly managed from the top down and buy-in to the forum will be diminished.

Members should also be personally welcomed to the forum when they join in order to set the

tone that the community is there for them and recognizes the importance of their membership

(Prevou & Burgess, 2003). Additional welcoming gestures that may set the tone and foster a

sense of community include a frequently asked questions list, a hands-free tutorial of the online

meeting space, an APF mission statement and history, and other background information (Prevou

& Burgess, 2003). In addition, a statement of the core values and conventions may be useful, as

well as an area for user comments that fosters a sense of belonging to the online community.

A critical functionality of the online meeting space is the support of vibrant conversation

among forum members. It is this functionality that distinguishes the discussion forum from a

knowledge repository, a distinction that determines the generation of new knowledge and the

growth of the community. Asynchronous discussion is the backbone of successful discussion

forums, with synchronous discussion less frequently found (Schweitzer, 2003). The online

meeting space must therefore enable the storage and meaningful organization of conversation

threads and allow forum members to easily view conversations and post their own contributions.

Features of the online meeting space should also help members evaluate the value of a

conversation through indicating (a) the general topic of the conversation; (b) the number of posts

involved in the conversation; (c) the date of the initial and most recent conversation posts; (d) the

contributors of each post in the conversation; and (e) member ratings of the value of the

conversation. It must be remembered, however, that vibrant conversation ultimately springs from

a dedicated core group and facilitators who focus, protect, and foster the conversation (see above

section, also Dixon et al., 2005; Stuckey & Smith, 2004), no matter how easy to use the

discussion functionality of the online meeting space is.

Once in the online meeting space, quickly locating and retrieving content (including

discussion threads) requires that it be organized into topic and sub-topic areas of greatest

relevance to forum members, and that the most important and recent topics be placed

prominently on the very front page of the forum (Dixon et al., 2005). In addition, there must be a

robust search capability that supports searching the entire forum or selected communities in the

forum as well as searching for content using a variety of user inputs (e.g., title, author, key

words, content type, and so on). Awareness of content on the site can also be accomplished

through "featured content" and "recent discussions" areas within each topic, "flags" indicating

which topic and sub-topic has new content, direct links to content outside the forum, links

between related content inside the forum, and regular newsletters featuring additions to the

11
forum. Members should be able to identify and indicate which topics are the most popular and

which contributions are the most useful (e.g. through metrics to indicate number of posts to a

particular topic and by using rating functions).

Finding people is as important to the health of the APF as finding content. Forum

members must be able to identify other members, their background, and how to contact them.

Member bios must include information that will help members to determine whether a particular

person should be targeted for discussion or to establish the credibility of a person who has posted

content. This information should include branch, current duty position, assignment history,

combat experience, and personal areas of expertise, among others. Each member bio should

include a list of how many posts the individual has made with links to each of these posts. As

with content, there should be a robust search capability for locating people. The expertise of

particular forum members may be highlighted through a "featured people" area or a regular

newsletter. To facilitate a sense of community, an area indicating who is currently logged in to

the forum is claimed to be beneficial (Dixon et al., 2005). Such a feature allows members to

characterize the type of members who typically visit the forum. In an APF that supports

synchronous chat, such a feature allows members to determine whether there is an expert

currently online with whom a chat might be advantageous.

Technical Requirements

The speed, accessibility, and security of the network in which the online meeting space is

situated are clearly important considerations for the leadership and administrative support of the

professional forum to consider. The cost, flexibility, and organizational fit of the software used

to generate the online meeting space are additional important factors. Decisions about

networking and software that take these factors into consideration serve organizational needs

(e.g., cost savings, political posturing, and information security) as much or more than the needs

of the forum members, and for this reason are considered beyond the scope of the present

analysis.

Structure and Characteristics - A Proposed Framework for Assessment

To assess APF structure and characteristics, a red-amber-green rating system can be used

in order to translate qualitative information into "analytic" metrics (i.e., metrics that will allow

rough comparisons between aspects of structure and characteristics within or across APFs). This

system, which allows a combined red/amber and amber/green rating, is essentially an alternative

format to a five-point rating scale. The red-amber-green system is preferable to a numerical

rating scale so that (a) status ratings will communicate the predicted impact of the rated element

on the health of the forum; and (b) the use of numbers does not mislead the reader to think that

the rating system has quantitative properties (e.g., an interval or ratio scale) that it does not. The

assignment of a status rating to any element is a subjective decision, based on an understanding

of best practice in knowledge sharing, and ratings can, at best, be thought to fall on an ordinal

scale. The assessment framework presented in this section features the structure and

characteristics elements for which status should be assessed, and a description of the rating

criteria ("red," "amber," and "green") for each of these elements. Overall Status can be

12
determined by taking the average of the component status ratings featured in each table if the

red-amber-green scale were treated like its analog, the 1-5 ordinal rating scale.

13
-ý CC30

o -0 coc . M C)

) D' a.) 28 

0 70 to ~* ~ *-

t4:-~ 0 )WC"

4ý - = . - *: o ,W

04 0 cn 

0) j 

0 U'0 U)0 o ET -ý j

L) ) -'..U _

"CC0 ID 0 o) 

0 ) N 0) rnc

u L) 0

Cd)9

0) C2 2 r o c U) 

0 2.

S ~ c E. C.)O~-

0 C

"o 0) 0 0C a.M 0

M) w- " PQ)

CZ U .0.0

00 0 O 00 

'D o u -0 c ;:"r

0)o

." 0r0 0 05.

> E

- 0

- :1 0CC, o~

0.0

0 0) 0

0) 0)0)toP

0.) 0. ca
I0 0

S G0 o. -w

r.G~ 0 Q) o '

CA~0* w00C

o 0.) 0

o. E 0.l j"

- 0 

C.w .800 ) WN)

7 to

E 0 0 

02c E E".) 0.o 

0 'o)0 0) - t - 0)0 0r

A. >z 0) 'D 5) u ) 0)0

C O O0) 0 E E0. .5 o D0 00

0 0 0) ca0c

00 M)Ou X

Q)

2 . 0 B r.2

I. C! I. -

e 0 0to

71- 0

I E- ý J L O c cn0
+ tI-E

-- ju 0'

E.o - 0 -= 

0 E~ t

Cý T g0 a" u-0RJ, Nu o 00

,,o cc - m E

b.4

r -3

o ~ .5 C13'

0 0 0 0 0

-a u p 4ý s-, M.- -

goa 80o0- 0

0 a

00Z z- u - t5 E E> m Et;r

M .2 2 4 r

UC 0 0 -a 0 r_

-c u 0 u 8040

M ~ .I
ASSESSING SOCIAL AND INTELLECTUAL CAPITAL

Social and Intellectual Capital Present in a Healthy Professional Forum

Social capital, as defined by Nahapiet & Ghoshal (1998), is comprised of the physical

connections among people, the shared language and perspective among these people, and the

social-psychological relations that these people share with one another. This definition of social

capital roughly corresponds to Dixon et al.'s (2005) construct of "connections" among the

members of an APF, but also recognizes the role of shared context, trust, and mutual obligations

among forum members in facilitating interpersonal growth and the development of intellectual

capital. Intellectual capital comprises the knowledge that is exchanged and constructed among

the members of an APF. This definition of intellectual capital corresponds roughly to Dixon et

al.'s (2005) construct of "content." A further description of each aspect of social and intellectual

capital is provided below.

Connections

Connections are the overall structure of who is reached and how (Nahapiet & Ghoshal,

1998). Dixon et al (2005) point out that in an APF, those who need to know must be connected

to those who know. These individuals comprise the target membership of the community, whose

scope must be carefully defined in order to keep member interaction relevant and engaging

(Gerber, 2003; Stuckey & Smith, 2004). Similarly, a balance of people who know and who need

to know among the forum's actual members is important for facilitating productive discussion.

The connections between individuals assembled in this way support the exchange and integration

of knowledge necessary for the development of intellectual and social capital to occur (Nahapiet

& Ghoshal, 1998).

Connections among members of an APF may take the form of discussion threads inside

the forum or emails, phone calls, or face-to-face conversations outside the forum. Dixon et al.

(2005) describe the important role that APF facilitators play in helping forum members establish

these connections. When individuals post a question in the forum, facilitators should respond

immediately to the post, indicating to the individual that their question has been acknowledged.

Facilitators then use their own connections or contact members of the forum to identify the best

person to answer the question in a timely fashion. The swiftness with which networks can be

established via the forum is a function, therefore, of the depth and breadth of the facilitators'

social networks and how easily the background and expertise of forum members can be

determined. Sufficient background information provided in member bios also allows forum

members to be proactive in seeking connections with experts. Because responding thoughtfully

to questions about the profession takes time and effort, the experts contacted to make a

contribution must be motivated to serve the individual who contacted them, the forum of which

the individual is a member, or the professional community more broadly. The degree to which

forum members fill out their member bios may reflect one's self-perception as a contributor to

the community or willingness to be identified as an expert. For example, very high ranking

officers who wish to learn from the community may not fill out biographical information out of

concern that their recognized presence in the forum may stifle conversation and activity.

17
Motivation to participate in knowledge exchange requires (a) trust that the exchange will

not lead to undesirable professional consequences (Gerber, 2003); (b) expectation that the

exchange will be conducted in a professional manner according to shared values; (c) a sense of

obligation to serving the community present in the forum; and (d) personal identification as a

member of the forum (Nahapiet & Goshal, 1998). For this reason, the APF must not only

facilitate connections between people on the basis of the knowledge they can impart but also on

the basis of shared norms, values, and expectations. The norms and values established by tone of

the forum and maintained through monitoring and facilitation help to develop a stable, giving

relationship between members and the community.

Context

Nahapiet & Ghoshal (1998) identified two aspects of shared context: shared narratives

and shared codes and language. These theorists argue that shared context facilitates the

development of intellectual capital by (a) providing cognitive access to individuals with

knowledge to share; (b) fostering the expectation that knowledge exchange will be useful; and

(c) enabling the conceptual overlap that makes knowledge exchange possible. Shared context

may be facilitated through restrictions on membership in the APF and/or it may grow out of

activity in the forum.

Dixon et al. (2005) claim that a sense of shared context, particularly shared narratives,

among APF members can be fostered through participation in shared experiences enabled by the

forum. These shared experiences can take several forms, such as collaborative book reviews,

synchronous group discussions online or face-to-face, regular membership seminars or meetings,

or collaborative exercises online. Importantly, participation itself in the forum also creates shared

experience. Forum activity must foster a sense of cohesion among members, a sense of ease with

intellectual and social exchange, in order for relationships to develop and continued knowledge

sharing to occur. Cohesion fostered in the APF could play an important role in facilitating

integration among its members outside the forum, for example if their duty position calls them to

work together (Sanders, 2002).

Content

Intellectual capital can take four forms (Spender, 1996): conscious, automatic,

objectified, and collective. Conscious knowledge is an individual's explicit knowledge,

exemplified by his or her tools or explicit procedures used to solve practical problems. In the

context of an APF, such tools or procedures may include policy letter templates, orders formats,

unit SOPs, and training management matrices, among other baseline products (Kilner, 2002, also

Bobrow & Whalen, 2002, and Gerber, 2003). A person's conscious knowledge at times may be

difficult to distinguish from his or her automatic, or tacit knowledge (Spender, 1996). Automatic

knowledge often takes the form of stories--stories of success or failure from which the discerning

learner can extract the link between particular situational contexts and the appropriate actions to

take in these contexts (Cianciolo et al., 2004; Spender, 1996). Stories may provide important

illustrative context for effectively applying another person's conscious knowledge.

18
Objectified knowledge is an organization's explicit knowledge, which, in the case of the

Army, would include doctrine, lessons learned, tactics, techniques, and procedures, and

administrative procedures, among others. Objectified knowledge has been validated according to

institutional standards (Spender, 1996) and serves multiple functions, such as providing a frame

of reference for decision making and taking action, providing a means for holding individuals

accountable for their behavior in the organization, and providing a foundation on which further

knowledge is built. Collective knowledge is an organization's tacit knowledge, implicit rules--

socially generated and socially observed--for the behavior of organization members. Collective

knowledge shapes individuals' understanding of organizational expectations for their behavior,

expectations that may or may not be made explicit in objectified knowledge. In the case of the

Army, collective knowledge may include shared understanding of what behaviors reflect Army

values, how individuals in particular positions should interact with one another, and what actions

are likely to lead to success within the organization.

Spender (1996) claimed that sharing explicit (i.e., conscious and objectified) knowledge

is akin to a library or database access and that only the development of tacit (i.e., automatic and

collective) knowledge reflects true learning. This point is arguable, depending on whether the

perception of new affordances enabled by explicit knowledge is considered a form of learning

(Gibson & Pick, 2000) or whether accomplishing new understanding is considered necessary for

applying new explicit knowledge. In any case, the APF must support the exchange and

application of explicit knowledge as well as the sharing and generation of tacit knowledge, as

both are critical determinants of performance.

The question of what knowledge should be present in an APF and how to judge its

quality may not be quite so difficult as it seems initially. Simply put, the applicability of

knowledge indicates its value. Spender (1996) strongly resisted the "objectification" of

knowledge, noting that knowledge must be interpreted in terms of its utility in supporting

purposeful action. Importantly, objectification and assessment of forum content on the basis of a

criterion other than relevance may create the perception that external interests are driving the

forum. The interviews conducted as part of this project universally acknowledge that APFs

driven by the top of the organization, and not by the members of the APFs themselves, will fail.

Driving the APF from the top not only may erode the trust of the forum members but also may

steer the content and conversation away from their needs (Gerber, 2003). Assessment of APF

content will result in inadvertent driving from the top if assessment criteria are based on the

needs of those at the higher levels of the organization, rather than the members of the

community.

Dixon et al. (2005) also emphasized the importance of relevance when evaluating the

quality of knowledge. Emphasis on relevance implies that trained professionals are, to varying

degrees, experts in their own right, capable of making sound judgments regarding the quality of

knowledge. The amount of range restriction in intellectual ability that is repeatedly shown in

professional populations supports the assumption that most Army officers will exhibit relatively

sound judgment about when to apply knowledge acquired in an APF. Setting the tone in the APF

for a professional exchange serves as a partial barrier to making ineffectual and/or inappropriate

posts (including posts that do not meet information assurance requirements). Community

policing by both facilitators and forum members aids in the rapid removal of inappropriate posts.

19
Along these same lines, a second implication is that abstract criteria are insufficient for

judging the quality of knowledge. Rather, the quality of knowledge is reflected in the degree to

which it is aligned with the needs of the forum member seeking it. Knowledge will be maximally

relevant, and thus of highest quality, if it is developed by the target members of the forum, rather

than by other interested parties 2. To the degree that the information needs of forum members are

aligned with key determinants of organizational effectiveness, knowledge sharing and

development will have a powerful impact on the organization.

Dixon et al. (2005) pointed out that the relevance of knowledge is determined in part by

the degree to which it can be directly applied as opposed to modified before application.

Although this criterion is clearly sensible, nearly all knowledge shared or developed in an APF

will require some degree of intelligent translation for application to a specific context. Forum

members who make contributions should provide contextual information that will aid in the

interpretation of their posts. In addition, facilitators should be actively engaged in ensuring that

context is provided, for example through establishing connections among posts such that the

same knowledge discussed in multiple contexts can be revealed and understood. Such

connections and synthesis expedite knowledge sharing and generation.

A final, additional aspect of relevance is timeliness (Dixon et al., 2005). Content in the

forum must be continually updated in response to the needs of its members, with new additions

made readily apparent. To some extent, the active forum serves as its own engine, although

facilitators play a critical role in keeping discussion active by sustaining social connections in the

forum, maintaining meaningful boundaries around the community, and continually drawing

useful information from the environment (Stuckey & Smith, 2004). Facilitators help ensure that

questions posted are answered as quickly as possible and that stale content is moved from the

active forum to an archive.

Conversation

Conversations are the means by which a sense of trust and community develop and by

which new knowledge is generated (Dixon et al., 2005). Therefore, the APF is one means by

which social and intellectual capital co-evolve (Nahapiet & Goshal, 1998). Initial intellectual

capital fosters the development of social capital, as conversation around sharing knowledge

builds trust and a sense of community. Without conversation, an APF would simply be a

searchable database of information. It is therefore critical that conversation be part of an APF in

order for it to be considered a community and a source for the development of new knowledge.

Conversation may serve different purposes in the forum, including clarification or concretization

of an abstract concept, exploration or brainstorming of new ideas, challenge or debate about

existing ideas, or validation of or agreement on existing ideas. The sharing of tacit knowledge is

another important purpose of conversation in the APF often achieved through storytelling.

As discussed previously with regard to content, conversations must be timely, applicable,

and professional in order to be useful, and they must meet information assurance requirements.

2 However, facilitators do play an important role in establishing the social connections that lead to the sharing and

development of relevant knowledge.

20
They must be on-topic and develop at a pace that suits the needs of the person who started the

conversation. Importantly, conversations must concretize abstract ideas by providing context that

will aid in the application of the knowledge they contain. Discussion threads should build

meaningfully on one another, resulting in synthesized ideas or lessons learned. Ideally,

discussion threads are driven by the target members of the community themselves so as to

prevent "discussion stray," which reduces the utility of the conversation for the forum members

(Major Tony Burgess, personal communication). Because conversations are a shared experience

among forum members, they must build trust and a sense of community through adherence to the

values of the forum and the standards of professionalism. Conversations that build trust are

respectful and constructive and do not digress into strings of complaints or personal attacks.

Conversations that lead to mutual understanding and knowledge generation build a sense of trust

that participating in the community will be helpful and boost motivation to actively participate.

Informal, face-to-face communities have long provided opportunities to generate

knowledge through discussion. These groups are more limited in their size and reach than the

APF because members must come from a relatively small geographical area. Use of phone and

email increases the scope of these groups, but not to the extent made possible by the online

forum. However, building an online meeting space is no more the same as building a healthy

forum than outfitting a classroom is the same as conducting an effective class. Activity in the

APF is as much a function of having the right place to gather as it is having the right topics to

discuss and the right people with whom to have conversations. Review of the structure of APF

content and membership provides insight into how active that forum can expect to be. Analysis

of the quality of APF member interaction and of the products of this interaction provides insight

into how much of an impact that forum can be expected to have.

Social and Intellectual Capital - A Proposed Framework for Assessment

As with evaluating APF structure and characteristics, a red-amber-green rating system

can be used in order to translate qualitative information about social and intellectual capital into

"analytic" metrics. The assessment framework presented in this section features the social and

intellectual capital elements for which status should be assessed, and a description of the rating

criteria ("red," "amber," and "green") for each of these elements. Overall Status can be

determined by taking the average of the component status ratings featured in each table if the

red-amber-green scale were treated like its analogue, a 1-5 ordinal rating scale.

21
.!: 0' )

El~ --S cl

0 P.

0~\

E) m .0 ý ýg.ua+

co 0

- ~ a~ Ln 00 1 >V ~a cc) r.a 0 ~ Q) ~ u

o u. 0) 0 1OO0 U

0 0> 0 ýo :sn u 'o u oz r. c

0 X) u 0 CC a )co

E C, 00 

0o 0 0 0 Q 0 

0, C\P E~a o~- 0-a0 wp 't,- pe

W) C)

0,o o 0. 

00 0 ~) ~ C,

a) X > co

o(
CA)

00

-a) Go

04j~a z- 0 X

co t

Cd~a

Xa

0 00

cu " rý C a).E

(A 0 V 0)

Q)L

M "0 0

0~a

0~

> W) 00

0co

hl)

0~

x u

oco

a)
0 + -

0 D 0o 0

0o ~-0 

C) 0 ..

0 41t)( ) 0'0 )

cc 0 c C0 -~ 0 0 0 o tz

6.)

oŽ~1 -0 c)- 

-- -M :ý 1 4,0

000

+ + 0

0 0 a) m 6.) ~

0 V-

-U 6) 6

0 ) 0 >0~ 

0 C) 00 0 '. o C

v~ -M~ U)

.~ ~ ( 6 C) >i 

" Z~ .= Ca 

rnN

6) 0 'n t)

0j , I : ;- 66 - p0 0 $ 0) vl 0)7

C) C)6b

6o 10 0o; k U U0 0
0 ~0\

kr -c!~ Cl

a.) 00

L) 0j

0 ~ m 0 0 j

00

o E - E

00

00

o c, u ý"

.0~

0A 0 (U 0

cq ~ o ~ o00 ~ 0 u k

"~ 0 C00 

00k

4O '0-oW .4 CC0

o 0 c 0 COC) co C 0 0z 0

knJ~ $l 0 u 2c

E 0

00 C\

Qo 0

lTO 0 0Q crs '4
ASSESSING LEADER COMPETENCE AND PROFESSIONALISM

Assessment of the impact of APF activity on individual leader competence and

professionalism requires identifying the general-level competencies that give rise to effective

performance in the jobs of the forum members. It also involves identifying how the Army

defines professionalism as it relates to the target members of the forum. Army values and certain

other personal characteristics specified in doctrine represent a significant component of what it

means to be an Army professional and are applicable across APFs. The Army's overarching

framework of"Be, Know, Do" (FM 22-100 -Leadership 3) captures what it means to be an

Army professional in terms of personal characteristics, competence, and action.

In order to manage the scope of the impact assessment, one should target those personal

characteristics, competencies, and actions for which common or persistent problems have been

identified in the target membership. These are the areas where one can expect to see

improvement due to an intervention because performance is not already at ceiling. These are also

the areas where performance improvement is most badly needed, so impact in these areas is most

likely to be aligned with organizational objectives.

There are multiple ways to assess whether APF activity has had an impact on the

personal characteristics, competencies, and actions selected. Ideally, one would develop and

administer in a quasi-experimental setting valid, reliable special-purpose assessments to capture

leader competence and professionalism both before and after becoming a member of an APF.

Growth in these areas could be compared to the growth occurring in these same areas for APF

non-members with otherwise similar characteristics (e.g., years in the Army, participation in

institutional instruction, etc.).

Given that leaders do not typically become members of an APF en masse, and that the

construction and validation of the required special-purpose measures would likely necessitate a

nontrivial research effort of its own, the feasibility of the quasi-experimental approach is highly

questionable. A reasonable proxy is to assess whether or not the content and discussion in the

APF addresses the key problem areas in leader competence and professionalism identified in the

initial analysis. Using the social and intellectual capital assessment criteria described in the

previous section of this report, APF functional effectiveness in the key areas of interest could be

assessed. For assessing aspects of community development and professionalism, it would be

feasible to survey APF members and non-members, asking them to provide self-reports of

community and professional identity. Comparing classroom performance data and/or officer

evaluation report data between APF members and non-members may serve as a substitute for

special-purpose measures and surveys, provided there is not significant range restriction in

grades and evaluations. If comparisons between members and non-members are used, it is

important to distinguish between more and less active members so that average levels of impact

can be understood in terms of time spent in the forum.

3 FM 22-100 is currently under revision and will be reissued as FM 6-22. Many of the concepts and the underlying

structure presented in FM 22-100 will carry forward to FM 6-22, so much of the framework described in this report

will remain intact after the revised doctrine is issued.

26
ASSESSING ORGANIZATIONAL EFFECTIVENESS - UNIT

To identify targets for impact assessment at the unit level, "unit" must be defined in terms

of the APF members. For example, if the APF is geared toward company commanders, the unit

should be defined as a company with special emphasis on the interaction between the company

commander and his or her platoon leaders. If the APF is geared towards staff executive officers,

the unit should be defined as the staff. Defining "unit" in this way ensures that unit performance

falls largely within the sphere of influence of the forum member rather than external factors,

which is required for a fair assessment of impact.

As with assessing leader competence and professionalism, key problem areas in unit

performance (once "unit" has been defined) should be identified in order to manage the scope of

the impact evaluation. Similarly, evaluating organizational effectiveness at the unit level should

also focus assessment on these key problem areas.

Assessment of unit effectiveness ideally involves observation of the unit performance of

APF members and non-members. Unfortunately, as with assessing leader competence and

professionalism, assessing unit performance would require the development and validation of

unit performance measures, not to mention significant logistical and administrative challenges. A

reasonable proxy would be to compare the unit performance of APF members and non-members

as assessed by observer/controllers when the units rotate through combat training centers or as

represented by operational data from units recently returned from deployment. As stated

previously, if comparisons between members and non-members are used, it is important to

distinguish between more and less active members so that average levels of impact can be

understood in terms of time spent in the forum. An alternative proxy is to assess whether or not

the content and discussion in the APF addresses the key problem areas in unit performance

identified in the initial analysis. Here too, one could use the social and intellectual capital

assessment criteria to assess the APF functional effectiveness in the key areas of interest.

ASSESSING ORGANIZATIONAL EFFECTIVENESS - THE ARMY

In one sense, identifying targets for impact assessment at the organizational level is

simple; the Army seeks to reduce costs while at the same time improve capability. These broad

goals may be accomplished, for example, by improving retention, a matter of critical importance

to today's Army, streamlining the transformation effort, and maintaining high levels of

professionalism in a time of great flux in roles and responsibilities.

The difficulty in assessing impact at the organizational level lies in the fact that one APF

can only reach a small fraction of the Army's leaders. Moreover, it may take several years to see

impact as junior members of an APF work toward positions of authority in the organization. It

reasonably may be expected that multiple APFs are required to make a demonstrable difference

at the organizational level in the broadest sense.

That said, it may be possible to capture the potential organizational impact of a single

APF by identifying the implications of organizational initiatives, such as transformation, for the

jobs of the APF members. Job demands placed on APF members by organizational initiatives

should be addressed in the forum in order for the forum to have an impact on the performance of

27
the organization as a whole. In addition, knowledge sharing and discussion in the forum should

support continual development in support of organizational goals, including increased sense of

commitment to the organization and enhanced capability to make learning a lifelong endeavor.

ILLUSTRATION - APPLYING THE METRICS TO EVALUATING S3-XO.NET

In order to test the applicability and feasibility of our metrics and their associated

measures (e.g., Tables 1 through 7), we used them to assess S3-XO.net, an APF geared towards

battalion and brigade operations officers (S3s) and executive officers (XOs). Specifically, we

assessed the strength of S3-XO.net's structure and characteristics, the effectiveness of the social

and intellectual capital developed in S3-XO. net, and the potential for individual, unit, and

organizational impact of S3-XO. net content. The complete analysis is presented in Appendix B,

with only the general findings summarized here.

The causal framework was applied successfully to evaluating S3-XO.net in a costeffective and diagnostic manner. At the time of our analysis, we found that although current and

future S3s and XOs were desirous of a means to connect and share knowledge with experts,

member activity in S3-XO.net was occurring at lower levels than was expected given the age of

the forum. Our investigation revealed that greater emphasis was devoted to the technical

specification and maintenance of the forum, rather than to the means by which activity in the

forum would be generated. Moreover, S3-XO. net lacked the personnel structure and support

necessary to improve the functioning of the forum. For these reasons, metrics of leader

competence and professionalism and organizational effectiveness were explored but the attempt

was not made to measure them.

CONCLUDING REMARKS

The present research represents an initial effort to explore and develop a general

framework for assessing the effectiveness and impact of APFs. We have based the framework on

an overview of the scholarly and technical literature on knowledge sharing and its assessment.

The framework and its application are also based on interviews with a range of people involved

in APFs--from organizational thinkers to forum members to forum developers. The result of our

initial work is a generalizable framework that can be used as a foundation for assessing the

effectiveness and impact of APFs at both the beginning and advanced stages of their

development. Appendix C provides some guidance as to when assessment should be conducted

to maximize its utility.

Future research can take several directions to advance this effort. First, the validity of the

APF assessment framework when applied to multiple APFs should be explored. The exploration

should feature APFs in differing phases of development, from very early stages to advanced

stages. Such an exploration would further illuminate the link between APF structure and

characteristics and APF activity. Second, individual-, unit-, and organization-level impact should

be analyzed using more active APFs, such as CC.mil. This analysis would illuminate the link

between APF activity and impact, as well as identify appropriate means for directly assessing

impact through special purpose measures (e.g., surveys, tactical games, etc.), observation of unit

performance, and cohort tracking on organizational variables of interest (e.g., retention). Finally,

28
the validity of the red-amber-green assessment criteria should be examined closely by

systematically investigating the link between different levels of activity in APFs and different

levels of individual- and unit-level impact as directly assessed.

As a parting comment, we advise that careful attention be paid to ensuring that APF

assessment reflects the delicate balance between the requirements of forum members to grow

organically in an informal, consequence-free environment and the requirements of the

organization to justify its investment in knowledge sharing to stakeholders. The consensus in the

literature indicates that too much organizational involvement, perceived or otherwise, will reduce

the trust of forum members and ultimately kill the informal knowledge sharing that provides

organizational advantage (e.g., Gerber, 2003; Wenger & Snyder, 2000). However, knowledgesharing forums depend on organizational cultivation to thrive and live up to their promise

(Wenger & Snyder, 2000). Successful effectiveness and impact assessment therefore must serve

to direct cultivation rather than impose requirements or change.

Consider the following analogy, extended from Wenger and Snyder (2000). A farmer one

day discovers that wild onions are growing in one of his plots. He shares a couple of these onions

with some close friends, and everyone agrees they are quite unique and flavorful. He believes

that cultivating and selling the onions will give his farm a competitive advantage. However, the

onions are easily pushed out by other plants and susceptible to dry spells. Motivated by the

payoff he thinks cultivating the onions will provide, he invests time and money to clear a plot for

them (reducing the plot size of some other vegetable) and to set up a rudimentary irrigation

system. The onions thrive. The farmer is pleased, but he knows the real test of how wisely his

resources were spent is whether the onions sell at the farmer's market. He uses his onion sales at

the farmer's market as an assessment of his judgment. Naturally, the onion sales fluctuate as

market demand changes. When sales are up, he continues his current cultivation practices. He

uses quiet sales periods as evidence that he needs to change something in his cultivation

practices. He changes techniques to help bring out the best the onions have to offer, such as

sweet flavor one year and size another year. The onions thrive, the market responds, and the

farmer is pleased.

In the present research, it was our intention to lay the groundwork for an assessment

framework that directs APF cultivation. The application of our framework to assessing the

effectiveness and impact of S3-XO.net demonstrates how assessment can lead to

recommendations for better serving the needs of the forum members and, by extension, the needs

of the Army. Future research must follow this lead such that APF assessment stimulates growth,

rather than stifles it.

29
30
REFERENCES

2 Officers punished in 2003 for mistreatment of detainees. (2005, May 18). New York Times.

13th Corps Support Command (131h COSCOM). (2003). CSM Daniel Elders wins Army

Knowledge Pioneer Award. 13 COSCOM News, October. Retrieved from

http://www.hood.armp.mil/13coscom/.

Battle Command Knowledge System (BCKS). (2005). Leader network collaborative

environment handbook for building a professionalforum. (Available from the Battle

Command Knowledge System, Fort Leavenworth, KS).

Barnett, J. S., Meliza, L. L., & McCluskey, M. R. (2001). Defining digital proficiency

measurement targets for U.S. Army units (Technical Report No. 1117). Alexandria, VA:

U.S. Army Research Institute for the Behavioral and Social Sciences.

Baum, D. (2005, January 17). Battle lessons: What the generals don't know. The New Yorker.

Bilafer, G. M. (2001). Where have all the warriors gone? Strategy Research Report. Carlisle

Barracks, PA: U.S. Army War College.

Bobrow, D. G., & Whalen, J. (2002). Community knowledge sharing in practice: The Eureka

story. Journal of the Society for Organizational Learning, 4(2).

Bonin, J. A., & Crisco, T. E., Jr. (2004). The modular Army. Military Review, March-April, 21-

27.

Brown, R. B., & Dedrich, C. E. (2003). Developing agile, adaptive soldiers. Military Review,

May-June, 33-41.

Britt, T. W. (1998). Responsibility, commitment, and morale. Military Review, JanuaryFebruary, 77-82.

Center for Army Lessons Learned (CALL). (1998). NTC trends compendium: 1 QFY98 through

2QFY98. Fort Leavenworth, KS.

Center for Army Lessons Learned (CALL). (2003). CTC Trends - National Training Center

(NTC): 1QFY03 through 2QFY03. Fort Leavenworth, KS.

Cianciolo, A. T., Antonakis, J., & Sternberg, R. J. (2004). Practical intelligence and leadership:

Using experience as a "mentor". In D. V. Day, S. Zaccaro, & S. Halpin (Eds.), Leader

development for transforming organizations - Growing leaders for tomorrow (pp. 211-

236). Mahwah, NJ: Lawrence Erlbaum Associates.

31
Cianciolo, A. T., & DeWillie, G. (2004). A framework for visualization, planning, and analysis:

Proposed TTPs. Unpublished manuscript.

Cianciolo, A. T., & Sanders, W. R. (To appear). A task analysis of U.S. Army war-gaming:

Implications for assessing the performance of combined arms task force battle staffs.

Proceedings of the Human Factors and Ergonomics Society 47 th Annual Meeting.

Cotter, P. A. (2002). The role of retrans in the IBCT. Military Review, May-June, 50-51.

Crager, J., & Lemons, D. (2003). Consortium Learning Forum best-practice report: Measuring

the impact of knowledge management. Houston, TX: American Productivity & Quality

Center.

Delaney, R. G. (2004). Army transformation: The human condition of soldiering. Military

Review, May-June, 37-39.

Department of the U.S. Navy. (2001). NA VSEA Community ofpractice practitioner's guide.

Washington, DC.

Dixon, N. M., Allen, N., Burgess, T., Kilner, P., & Schweitzer, S. (2005). Company Command:

Unleashing the power of the Army profession. West Point, NY: Center for the

Advancement of Leader Development and Organizational Learning.

Fischer, S. C., Spiker, V. A., & Riedel, S. L. (2000). Application of a theory of critical thinking

to army command and control. Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences.

Gerber, S. (2003). Over the top S3-XO.net business plan. Unpublished Master's thesis.

University of Kansas.

Glennie, W. P., & Hickok, J. (2003). Meeting critical defense needs with CoPs. Knowledge

Management Review, 6(3), 16-19.

Glenister, C. A. (2002). Information operations in the IBCT. Military Review, May-June, 59-62.

Gordon, D. (Host). (2005). Training day (The Connection, National Public Radio). Available at

http://www.theconnection.org/shows/2005/01/20050125 a main.asp.

Gray, D. R. (2001). New age military progressives: US. Army officer professionalism in the

information age. Strategy Research Project. Carlisle Barracks, PA: U.S. Army War

College.

Gunawardena, C. N., Lowe, C. A., & Anderson, T. (1998). Transcript analysis of computermediated conferences as a tool for testing constructivist and social-constructivist learning

theories. Proceedings of the 14 th Annual Conference on Distance Teaching and Learning,

Madison, WI, 139-145.

32
Hanley, S. (2003). Show me the money: A practical framework for KM metrics. Presented at KM

World & intranets Conference & Exposition, Santa Clara, CA.

Horey, J., Fallesen, J. J., Morath, R., Cronin, B. et al. (2004). Competency based future

leadership requirements (Technical Report No. 1148). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences.

Kasales, M. C. (2002). The reconnaissance squadron and ISR operations. Military Review, MayJune, 52-58.

Kasales, M. C., & Gray, M. E. (2003). Leveraging technology: The Stryker brigade combat

team. Armor Magazine, January-February, 7-24.

Kendall, R. E., & McHale, K. J. (2003). Evolution: Advancing communities ofpractice in naval

intelligence. Unpublished Master's thesis. Naval Postgraduate School, Monterey, CA.

Kilner, P. (2002). Transforming Army learning through communities of practice. Military

Review, 82(3), 21-17.

Langley, J. A. G. (2004). Network-centric warfare: An exchange officer's perspective. Military

Review, November-December, 47-52.

Leibrecht, B. C., Lockaby, K. J., Perrault, A. M., & Meliza, L. L. (2004). Measuring digital staff

proficiency in current and future forces (Research Report No. 1825). Alexandria, VA:

U.S. Army Research Institute for the Behavioral and Social Sciences.

Lesser, E. L., & Storck, J. (2001). Communities of practice and organizational performance. IBM

Systems Journal, 40(4), 831-841.

Lussier, J. W., & Shadrick, S. (2004). How to train deployed soldiers: New advances in

interactive multimedia instruction. Proceedings of the Interservice/Industry Training,

Simulation, and Education Conference (I/ITSEC), Orlando, FL.

Lussier, J. W., Shadrick, S., & Prevou, M. I. (2003). Think like a commander prototype:

Instructor's guide to adaptive thinking (Research Product No. 2003-02). Alexandria, VA:

U.S. Army Research Institute for the Behavioral and Social Sciences.

Marlin, J. (1997). Cynicism and careerism: Threats to Army ethics. Army Magazine, May.

Nahapiet, J., & Ghoshal, S. (1998). Social capital, intellectual capital, and the organizational

advantage. Academy of Management Review, 23(2), 242-266.

Olmstead, J. A. (1992). Battle staff integration (IDA Paper P-2560). Alexandria, VA: Institute

for Defense Analysis.

33
Operation Iraqi Freedom Mental Health Advisory Team (OIF MHAT). (2003). Report.

Washington, DC: Department of the Army.

Patterson, M. S., & Phipps, J. E. (2002). Ethics: Redirecting the Army 's moral compass. Senior

Service College Fellowship Research Paper. Carlisle Barracks, PA: U.S. Army War

College.

Pleban, R. J., Thompson, T. J., & Valentine, P. J. (1993). The commander's battle staff

handbook.: An introduction to stafffunctional area duties for new battalion staff officers.

Alexandria, VA: U.S. Army Research Institute for the Behavioral & Social Sciences.

Prevou, M. I., & Burgess, T. (Eds.). (2003). S3-XO.net team reflections. Unpublished

manuscript.

Sanders, W. R. (2002). Collective staff training in a virtual learning environment (Research

Report No. 1788). Washington, DC: U.S. Army Research Institute of the Behavioral &

Social Sciences.

Schweitzer, S. J. (2003). Discussion forums: The core of online communities ofpractice.

Retrieved from the World Wide Web on March 17, 2004. Available at:

http://iava.cs.vt.edu/public/classes/communities/uploads/schweitzer597 project.pdf.

Smith, R. (2005). The Army Modular Force. Retrieved from the World Wide Web on April 25,

2005. Available at http://www.army.mil/professionalvideo/movies/modular.html.

Snider, D. M., Nagl, J. A., & Pfaff, T. (1999). Army professionalism, the military ethic, and

officership in the 21"s Century. Carlisle Barracks, PA: Strategic Studies Institute, U.S.

Army War College.

Snyder, W. M., & Briggs, X. (2003). Communities ofpractice: A new toolfor government

managers. IBM Center for the Business of Government.

Spender, J. C. (1996). Organizational knowledge, learning, and memory: Three concepts in

search of a theory. Journal of Organizational Change Management, 9(1), 63-78.

Stuckey, B., & Smith, J. D. (2004). Sustaining communities ofpractice. Paper presented at the

International Association for the Development of the Information Society (IADIS)

International Conference: Web-Based Communities, Lisbon, Portugal.

Toomey, C. J. (2003). C4ISR in the Stryker brigade combat teams. Military Review, May-June,

42-46.

U.S. Army Infantry School. (1937). Mailing list (Vol. XIV).

34
U.S. Army Training and Leader Development Panel (ATLDP). (2003). The Army Training and

Leader Development Panel officer study report to the Army. Unpublished manuscript.

The Pentagon, Washington, DC: U.S. Department of the Army.

U.S. Department of the Army. (1999). FM 22-100: Army Leadership. Washington, DC.

U.S. Department of the Army. (2002). FM 5-0: Army planning and orders production.

Washington, DC.

U.S. Department of the Army. (2003). FM 6-0. Mission command: Command and control of

army forces. Washington, DC.

U.S. Department of the Army. (2005). FMI 3-09.42: HBCTfires and effects operations.

Washington, DC.

U.S. Department of the Army. (2005). FMI 3-09.61: Brigade troops battalion operations.

Washington, DC.

Wagner, R. K., & Sternberg, R. J. (1985). Practical intelligence in real-world pursuits: The role

of tacit knowledge. Journal of Personality & Social Psychology, 49(2), 436-458.

Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities ofpractice.

Boston, MA: Harvard Business School Press.

Wenger, E., & Snyder, W. M. (2000). Communities of practice: The organizational frontier.

Harvard Business Review, January-February, 139-145.

White, D. (2001). The air defense commander and CCIR: Rather than critical information

requirements analysis to anchor their decisions, commanders are treated to a fire hose of

data. ADA Magazine.

Wilson, J. S., & Snow, M. W. (2003). Will the real logistics integrator please stand up?

Reflections for the brigade XO. Armor Magazine, January-February, 48-49.

Wong, L. (2000). Generations apart: Xers and Boomers in the officer corps. Carlisle Barracks,

PA: Strategic Studies Institute, U.S. Army War College.

Wong, L. (2002). Stifling innovation: Developing tomorrow's leaders today. Carlisle Barracks,

PA: Strategic Studies Institute, U.S. Army War College.

Wong, L. (2004). Developing adaptive leaders: The crucible of experience of Operation Iraqi

Freedom. Carlisle Barracks, PA: Strategic Studies Institute, U.S. Army War College.

Woodie, T. E. (2005). Learning together: The role of the online community in Army professional

education. Monograph of the School of Advanced Military Studies, U.S. Army

Command and General Staff College, Fort Leavenworth, KS.

35
36
APPENDIX A - LIST OF INTERVIEWEES

CC.mil Core Group Members (3)

NCOTeam.org Facilitators (2)

Instructor.net Developer (1)

S3-XO.net Founding Officers (3)

S3-XO.net Facilitator/Administrators (5)

NTC Observer/Controllers (8)

Future S3s/XOs (7)

Current or Past S3s/XOs (2)

Instructors (3)

Doctrine Writers (2)

Organizational Thinkers (5)

37
38
APPENDIX B - ASSESSMENT OF S3-XO.NET

In the present research, the metrics and assessment frameworks described in the main

body of this report were applied to evaluating the Army Professional Forum (APF) S3-XO.net.

This section presents the results of the assessment and recommendations for enhancing the

forum's activity and potential impact.

A Brief History of S3-XO.net

S3-XO.net was designed to: "Create and foster knowledge and experience sharing among

future, serving, and prior operations [officers] and executive officers [S3s and XOs, at the

battalion and brigade level] in order to build combat ready units (S3-XO.Net In-Process Review

#2, 29, as cited in Gerber, 2003)." S3-XO.net was developed during the winter and spring of

2003 by a group of Command and General Staff College (CGSC) students under the tutelage of

Major Tony Burgess (member of the CC.mil core group) and Lieutenant Colonel Mike Prevou

(CGSC professor), and with the support of the then head of the Combined Arms Center.

S3-XO. net was developed as part of an Army response to the success of CC. mil, an APF

launched two years earlier (Gerber, 2003). The Battle Command Knowledge System (BCKS) is

currently under development as the main component of this response. The purpose of BCKS is to

harness knowledge sharing in the service of speeding up the combat decision cycle and of

enhancing leader professional development. S3-XO.net is currently situated as one of several

APFs under the umbrella of the Leader Network component of BCKS. Whereas other

components of BCKS, such as Warrior Knowledge Base and Unit Network, are geared

specifically to enhancing battle command, Leader Network is intended to have its primary

impact on the development of professional competence.

The founding officers of S3-XO.net developed a plan for the forum, designed the online

meeting space, and generated initial activity in the forum after it was launched. Within months of

launching S3-XO.net, in May 2003, its founders, including Major Burgess, graduated from

CGSC and were reassigned (Gerber, 2003). Activity in S3-XO.net therefore dropped significantly

immediately after its launch as the founding officers moved on. Moreover, Lieutenant Colonel

Prevou was deployed to Afghanistan for one year, so the leadership of the APF by necessity was

handed off in August/September of 2003. Two majors at Fort Leavenworth assumed leadership

of the forum at this time. In the spring of 2004, as activity on S3-XO.net was regaining

momentum, S3-XO.net was shut down for approximately two months due to concerns about the

security of the information discussed on the site. After placing login restrictions [i.e., use of

Army Knowledge Online (AKO) username and password] and acquiring permission to restart,

the S3-XO.net leadership brought the site back up in May, 2004. In December of 2004, due to

difficulties re-establishing momentum4 on the now-secure site, the community was re-launched

in an effort to get the word out that S3-XO. net was indeed back up and running.

4 This difficulty re-establishing momentum may have been due to several causes including lack of awareness that the

site had been re-launched and difficulty with the new login procedures, but it is also likely due to the reassignment

of all but one of the founding officers without a succeeding core group evolving out of the peripheral members of

the forum.

39
The leadership of S3-XO.net was handed off a second time in January 2005, in an effort

to institutionalize and stabilize the forum's foundation. The head of Fort Leavenworth's Center

of Army Tactics has been assigned to oversee the community's progress, with a CGSC instructor

serving as his action officer. CGSC instructors currently serve as part-time facilitators and one

on-site contractor provides full-time technical and editorial support. Through a course

requirement, CGSC students also play a facilitation role by participating in discussion threads.

S3-XO.net in particular, and knowledge management in general, has the full support of the

commander of the Combined Arms Center.

As of 2003, there were approximately 23,545 personnel in the S3-XO. net Regular Army

target audience-- approximately 16,205 majors and lieutenant colonels and 7,340 captains on the

career path that involves S3/XO positions (Gerber, 2003). Including Marine Corps or Reserve

component personnel in the S3-XO.net target audience would approximately double its size

(Gerber, 2003). Figure 2 depicts a screen capture of S3-XO.net's main page as it currently looks.

23-50 Net Homepane posted 530OPD NET CALLI A pwle1dge ob j. ctsf on oh sigte O d ote cont nso wcag vatr What was

... k nowltcedgebs for ouroser

~ ~ disconncted frompthir.pers....Oet i you antdoteto tspoble. W

l Biggest -11-/ )

p achallengereo a no n h

adto daysyo pd dp m iserving aa:

,~~~~~~~~~.~~h e h aet Call eso u otl X,, Sun J05t 

14)

Welcome aboard new members. We have grown! We currently boast roughly

2500 future, current, and pastS3s and XOsa Members have posted 5300

knowledge objects on our site. Our content continues to grow creating a vast knowledge-base for our users.

53s and XOs are experts that are distributed across the Army and often

disconnected from their peers. S3-XONet is your antidote to this problem. We

•tbest promote practices a new across method boundaries of collaboration, of information exchange, and sharing of time and distance, You will be able to

• contribute to the generation of new ideas and make valuable contacts. In

addition, you will solve daily problems and foster individual competencies, • making a significant contribution to overall Army learning. Check out Net

Cal June 2005 to get the latest news on our portal.

••.! This site is authorized to car.ry only unclassified FOUO informatio .... d belo .... ly, (FOUO) Tactics, Techniqu......nd

• ••:!• .. ,•. Procedu.... that relate to Operation Iraqi Freedom, Operation Enduring Freedom an..d Operation Noble Eagle (GWOT)

• IU•L . D1 ..... I.,uto MDMP fr th,l 01,16oav -it (S.CT, 4111. 11l 2005-06"07 9;07

Figure 2. S3-XO. net Main Page

40
S3-XO.net Structure and Characteristics

The Plan

On 5 February, 2003, a draft list of requirements for S3-XO.net was prepared by one of

the S3-XO.net founding officers. This list of requirements has been slightly modified for use as a

list of requirements for all of the APFs in the BCKS Leader Network. In addition, a BCKS

document (2005) detailing how to establish a forum on the Leader Network provides additional

administrative requirements (particularly personnel requirements) that apply to S3-XO.net. The

status of each component of the S3-XO.net plan is indicated in Table 8, with a justification for

each status rating provided below.

Table 8. S3-XO.net Plan Status

Status Plan Component Red Amber Green

Statement of Purpose

Envisioned Impact

Description of Target Audience

Statement of Cultural Norms, Core Values, or Conventions

Specification of Roles and Responsibilities

Functional and Technical Specification

Overall Rating

Statement of Purpose

S3-XO.net's statement of purpose states that S3-XO.net will: "Facilitate the effectiveness

of battalion/brigade executive and operations officers..." and that it will capitalize "on officer's

untapped stores of experience and knowledge helping leaders turn tacit knowledge into explicit

knowledge and increase the learning curve of all those participating in the community."

Although not stated explicitly, it appears in this statement of purpose that S3-XO.net is intended

to be a helping forum because of its focus on disseminating tacit knowledge, an important

determinant of practical problem solving (Wagner & Sternberg, 1985).

The implied target audience is Army officers who are either on the track to become a

battalion (BN)/brigade (BDE) XO or S3, or are currently a BN/BDE XO or S35. This may

include captains, majors, and lieutenant colonels in the Active Component [approximately

23,545 personnel (Gerber, 2003), see above] and potentially these same officers in the Reserve

Component (Gerber, 2003). The statement of purpose also implies that the content featured in the

site should take the form of both discussion threads and practical products in order to facilitate

tacit-knowledge transfer and articulation as explicit knowledge (this is described explicitly in

5 It may be argued that previous BN/BDE S3s or XOs, instructors, training developers, and doctrine writers are also

members of the target audience because participation in S3-XO.net may support these individuals' efforts to enhance

S3/XO effectiveness. However, to the degree that the forum is designed to address the needs of S3s/XOs, including

these other people in the target audience (as opposed to playing supporting roles) may result in applying top-down

structure to the forum or in defining the target audience too broadly, both of which will stifle activity (Gerber, 2003;

Stuckey & Smith, 2004; Wenger et al., 2002).

41
other parts of the requirements document). Moreover, the general functional requirements of the

system must include a way for forum members to download and upload files, to find each other,

and to carry on conversation.

The statement of purpose was assigned a combined amber/green rating because although

the statement has clear implications for target audience, it is less explicit about what type of

forum S3-XO.net is intended to be or what audience it is intended to serve (see the next section

for additional justification). This characterization is a critical one to make (Wenger et al., 2002).

Envisioned Impact

The S3-XO.net requirements document makes the following statements about envisioned

impact: "... [S3-XO.net] will become a powerful tool in developing adaptive leaders, peer-topeer knowledge transfer, relevant doctrine, and officers who are life long learners. There is also

evidence to show that participation ... [will] increase commitment to the organization and Army

service ethic." A list of potential areas of impact is also provided, and is reproduced below:

* "Foster innovation by encouraging the free flow of ideas and challenge status quo thinking

* Encourage life long leader learning

* Improve time management by streamlining research and development of manuals regulations,

SOP, etc.

* Streamline operations and reduce costs by eliminating redundant or unnecessary processes

* Provide a sense of community, worldwide, among leaders stationed in the U.S. and deployed or

stationed overseas

* Receive real-time feedback between institutional Army and field units

* Sharing of expert experience and wisdom (tacit knowledge)

* Converting expert tacit knowledge and experiences (wisdom) to explicit knowledge

* Capturing of "actionable" knowledge from lessons learned, operational experiences, projects, etc.

that can be easily retrieved by interested parties

* Improving the quality of instructors by directly impacting the experience curve

* Providing a mechanism to increase peer to peer mentoring

* Creating a platform for sharing knowledge that starts in the classroom and can be extended into

field operations"

This statement of envisioned impact provides a clear description of how the effectiveness

of the target audience is to be achieved through participating in the forum. However, the

statement does not describe where the impact will be (e.g., effectiveness in terms of the

competencies and duties/responsibilities of the S3/XO are not defined) and therefore does not

provide criteria for assessing this impact. Moreover, the impact statement also suggests that S3-

XO.net is intended to be multiple types of forum at once, including helping, innovating, and

knowledge stewarding, and is intended to serve multiple stakeholders. This implies that previous

BN/BDE S3s or XOs, instructors, training developers, and doctrine writers may be considered

(implicitly or explicitly) as additional members of the target audience rather than as playing a

supporting role in the forum or indirectly benefiting from the forum's activity. Potentially

including these individuals in the target audience imposes a top-down structure to the content

and discussion in the forum, which stifles forum activity (Gerber, 2003; Stuckey & Smith, 2004;

Wenger et al., 2002). For these reasons, the envisioned impact was given a combined red/amber

rating.

42
Description of the Target Audience

Neither the S3-XO.net requirements document nor the BCKS (2005) document includes a

description of the needs of the target audience. Description of the target audience was therefore

assigned a red rating.

Statement of Cultural Norms, Core Values, and Conventions

The S3-XO.net requirements document does not include a description of the cultural

norms, core values, or conventions to be honored while participating in S3-XO.net. However, the

BCKS (2005) document describes some conventions, including presenting only For Official Use

Only and above (i.e., less restricted) information in the forum, not using the forum for promoting

personal agendas or marketing commercial items, and keeping conversation professional (e.g.,

avoiding personal attacks and other negative behavior). While conventions are discussed, the

values to be upheld in the forum are not, nor are the means for enforcing adherence to

conventions and values. For this reason, the characterization of cultural norms, core values, and

conventions has been assigned a combined amber/green rating.

Specification of Roles and Responsibilities

The initial S3-XO.net requirements document contains a brief, general specification of the

roles and responsibilities of the forum support personnel. However, BCKS (2005) does detail

roles and responsibilities. These include forum members, the leadership team, the chief editor,

topic leads and point leads (i.e., facilitators), the administrative support team, and the sponsor.

Specific duties/responsibilities are outlined for each role, as well as some guidance for how the

roles interact/overlap. One shortfall in the otherwise exemplary specification of the roles and

responsibilities is the apparent lack of a role for technical leadership and implementation. The

Chief Editor is assigned some degree of responsibility over technical maintenance, and a

software administrator is mentioned, but additional detail as to the interplay between technical

implementation and forum evolution is not provided. In addition, the role of a core group is not

specified. That is, although the roles of the leadership team described in BCKS (2005) seem

similar to the roles of the core group, the importance of a passionate, dedicated, closely knit team

to support the forum through informal leadership does not seem to be recognized. Because of the

importance of both technical support and a passionate core group to forum success, the

specification of roles and responsibilities has been given a combined amber/green rating. It

should be noted that although this specification of roles and responsibilities is rather thorough, it

has been drafted nearly two years after the release of S3-XO.net, so its impact on the current

status of S3-XO.net may be somewhat limited.

Functional and Technical Specification

The S3-XO.net requirements document has a very detailed functional and technical

specification that has also been recently updated. This specification includes conceptual as well

as technical requirements, to include the knowledge-sharing behaviors that must be supported

(e.g., carry on asynchronous and synchronous conversations, add content, etc.), network

43
requirements, front-end and back-end features of the site, and security requirements. The

exemplary specification was assigned a green rating.

Personnel

Table 9. S3-XO.net Personnel Status

e C r Status Personnel Category Red Amber Green

Leadership

Core Group

Facilitators

Technical Support

Overall Status

Leadership

Financial support for S3-XO.net is channeled through funding designated for BCKS as a

whole. Initial funding for BCKS came in the summer of 2004 from the Commanding General of

I Corps in Iraq. The CG of the Combined Arms Center (CAC) at Fort Leavenworth has been

assigned sponsorship of BCKS (and therefore, by extension, S3-XO. net), and is responsible for

procuring additional funding in the upcoming fiscal years. The CG of CAC is supportive of

knowledge management and of S3-XO.net in particular. From its inception, S3-XO.net was

supported by the CAC CG, and continues to enjoy this support even after a change in general

officers holding this position. Competition for additional funding appears to come from AKO,

which is the institutionally recognized Army knowledge management solution.

S3-XO.net's formal leadership team is comprised of three individuals. Although all three

of these people are supportive of knowledge management and horizontal knowledge sharing,

none of them have assumed the leadership position voluntarily. These three leaders oversee the

forum's development, address problem areas as they arise, provide general-level vision for the

forum, and foster communication between the informal forum leaders and the BCKS sponsor.

Leadership of S3-XO.net must compete with the numerous other responsibilities of these highranking individuals. Of the informal leaders, one is among the officers who founded the forum.

Both of the informal leaders are passionate about the professional development of S3s and XOs,

and work hard to bring S3-XO.net to the attention of the target audience, to develop the argument

for securing support for the forum, and to provide administrative leadership to support personnel.

However, these two dedicated leaders must also juggle multiple priorities, and have limited time

to devote to leading S3-XO.net. The leadership team (formal and informal) will experience

turnover of at least three of its members within a year as officers retire or are reassigned.

The sponsorship/leadership of S3-XO.net has been assigned an amber rating. The

sponsorship and leadership of the forum is dedicated and supportive of the forum's mission, but

currently lack the time and internal compass required to foster activity and growth in the forum.

44
Core Group

Currently, S3-XO.net lacks a core group. At no point in the forum's history did S3-XO.net

have a group of core members who saw themselves as such and who took responsibility for

ensuring the growth of the forum. S3-XO.net's founding officers consisted largely of CGSC

students or, in terms of the target audience, future S3s and XOs. Therefore these officers, while

enthusiastic and skillful, were not fully representative of the target audience and ultimately saw

the forum as a professional development tool for people other than themselves. With the help of

their instructor, who was previously an S3 and an XO, they could articulate in general terms

what an S3 or XO could hope to gain from membership in S3-XO.net, but they did not have firsthand experience with facing the day-to-day challenges of the S3/XO. They also did not have the

benefit of having gained a large social network of S3s/XOs to work with in fully identifying

target-audience needs and interests. Lacking this network, they did not have the means to rapidly

post content in the forum that would generate discussion among current S3s/XOs or to link

members to one another. The founding officers (with one important exception, who is currently

on the leadership team), having left CGSC, are no longer actively involved in S3-XO.net and a

core group of target audience members has not evolved to take their place. For these reasons, the

core group for S3-XO.net has been assigned a red rating.

Facilitators

Responsibility for S3-XO.net facilitation is not clearly assigned to particular individuals.

One on-site contractor is assigned full-time to the forum, but must split his time between

facilitation and technical support, with the majority of his efforts devoted to technical support.

Three members of the leadership team play some role in facilitation, as do CGSC instructors and

a handful of on-site contractors. These people volunteer their time to post content on the forum

that will facilitate discussion and to guide discussion into productive directions. While passionate

about the professional development of the S3/XO, these people report having very limited time

to devote to facilitation. Such duties as performing leadership activities or preparing course

materials must take precedence over monitoring discussion threads, synthesizing knowledge on

the forum, and actively linking forum members. There also appears to be some confusion as to

whose initiative is key to the success of S3-XO.net -- the leadership, the facilitators, or the

members themselves. Representatives of each group seem to believe a different group is

primarily responsible for stimulating forum activity. For these reasons, S3-XO.net facilitation has

been assigned a red rating.

Technical Support

Technical implementation of S3-XO. net is the primary responsibility of one on-site

contractor who is assigned to the forum full-time both as technical support and as a facilitator.

This person ensures that the online meeting space (including access rights) is working properly,

makes changes to the look and feel of the forum, and resolves technical difficulties. However,

S3-XO.net does not appear to have a formal role for technical leadership. This leadership role is

necessary to provide vision for adapting the structure and appearance of the forum to meet the

changing needs of the forum membership. Moreover, a representative of the on-site contractor

and a member of the forum leadership have both reported that additional technical staff (e.g., a

45
Web designer) is required and that lack of funding and personnel has prevented them from doing

several development activities they wanted to do. For these reasons, S3-XO.net technical support

has been assigned an amber rating.

Functional Requirements

Table 10. S3-XO.net Functional Requirements Status

Status ______

Functional Requirement Category Red Amber Green

Locating the Forum

Joining the Forum

Tone in the Forum

Conversation Support

Locating Content

Locating People

Overall Status

Locating the Forum

S3-XO.net can be located through BCKS, via Google search, and through AKO. When

using Google search, the search terms "S3-XO.net," "S3 XO forum," or "S3 XO knowledge"

will return the S3-XO.net homepage as the number one hit. Using the search terms "S3 net," "XO

net," or "XO knowledge" will return the S3-XO.net homepage within the first page of hits. Using

the search terms "XO.com," "XO forum," "S3 forum," or "S3 knowledge" will not return the S3-

XO.net homepage in the first seven pages of hits6 . Using the search term "S3.com" only returns

one hit, and it is not the S3-XO.net homepage. Importantly, these results suggest that the Googleusing officer seeking to join S3-XO.net should have some idea of what the forum is called in

order to find it quickly. On the S3-XO.net splash page available via Google search, potential

users can easily click on a link to begin the registration process.

To locate S3-XO.net via AKO, the potential member must first select the "Knowledge

Networks" tab, then select "Battle Command Knowledge System," then select "S3-XO.net" from

two possible links. S3-XO.net cannot be found by entering "S3-XO.net" or related terms in the

AKO search functionality. Here too, the implication is that the potential member must already

have some idea of what they are looking for before they conduct their search. In the case of

AKO, the potential member must also know that S3-XO.net is part of the BCKS, which is part of

the Army Knowledge Network. Given our interviews with current and potential forum users, it is

highly unlikely that the potential S3-XO.net user will have this knowledge unless someone from

the forum has personally invited them to join7. The S3-XO. net splash page available via Google

6 The use of "seven pages" as a rough metric reflects the assumption that most people will have stopped searching

and decided to use a new search term by the time they go through five or more pages of hits without finding what

they were looking for. "7 Most of the officers we interviewed had not heard of the terms "community of practice" or "professional forum."

All of them, however, knew what CC.mil was. This suggests that "front-end" terminology (i.e., the forum name)

should be used to aid potential members in locating the forum, rather than "back-end" terminology (e.g., the Army's

terms or theoretical terms).

46
search directs current users to login to the forum via AKO, but does not provide direction as to

how to find the forum once inside AKO. In fact, the splash page says that logging in to AKO will

allow the member to "proceed directly to the site." Clicking on the link to begin the registration

process will allow current members to login, but this is not evident from the instructions

provided on the splash page.

An important barrier to word-of-mouth advertising for S3-XO.net is that there are

relatively few S3s and XOs in a particular unit (e.g., approx 3 S3s and 3 XOs in a BDE, as

opposed to approximately 9 company commanders and 27 platoon leaders), who may or may not

communicate or may see themselves as in competition with one another. The S3/XO population

is therefore widely distributed and does not necessarily have the impetus to communicate

amongst its members. It is especially important, then, for the S3-XO.net leadership to be active in

getting the word out about the forum. In August 2005, the S3-XO.net staff plans to hold a

membership drive geared towards students at CGSC. In addition, an invitation to join S3-XO.net

will be sent to all Army majors. These efforts represent the first large-scale initiative to recruit

new members. For these reasons, Locating the Forum is assigned an amber status.

Joining the Forum

To join S3-XO.net, prospective members must fill out a short online application and wait

for approval. The on-site contractor assigned to the forum checks for membership applications

on a daily basis and grants approval within 24 hours.

Four survey questions (complete survey is located in Appendix D) asked S3-XO.net

members to indicate their impressions of joining the forum. These questions addressed the

appropriateness of membership restrictions, the usability of the membership application, the time

it took for membership to be granted, and the tone of the welcome letter sent out to new

members. Seven members completed the survey questions out of 19 members to whom the

survey was administered, comprising a response rate of 37%8. On average (i.e., across the four

survey questions), the number of respondents giving the process of joining the forum a moderate

rating was approximately 3. Three of seven people on average gave the process a high-moderate

rating and approximately 1 gave it a high rating. For this reason, Joining the Forum is assigned

an amber-green rating.

Where respondents expressed an opinion about membership restrictions, it was that the

restrictions on membership are reasonable, but may require some modification. All respondents

It is unknown how representative of the S3-XO.net member population this small sample is. Only a small sample

was taken in order to evaluate economically the feasibility of surveying members for their impressions of the forum.

However, an attempt was made to make the sample surveyed as representative of the target membership as possible

by selecting the following: 4 current S3s or XOs who started a conversation thread that contained multiple posts (1

person responded), 2 current S3s or XOs who started a conversation thread that contained only the single initiating

post (1 person responded), 7 current S3s or XOs who had not made a conversation post but had visited the site

within a month of the survey (4 people responded), 6 current S3s or XOs who had not made a conversation post but

had become a new member and visited the site within a month of the survey (1 person responded). Nearly all of the

members who belonged to these categories were administered a survey. This sample is the same survey sample used

throughout this report. For the purpose of formally evaluating an APF, a much larger sample should be surveyed. A

larger sample will also permit an analysis of the types of members whose needs are being met/not met by the forum.

47
agreed that the application form conformed to their expectations of an online form. In general,

respondents indicated that the time it took for membership to be granted was 24 hours or less.

The majority of respondents felt that the welcome letter was a nice gesture.

Tone in the Forum

New members who join the forum are greeted with an auto-generated welcome letter

addressed from the Chief Editor. The letter is addressed to "Fellow Professionals" and indicates

how the new member can contribute to the betterment of "hundreds of units." The letter also

indicates that the Chief Editor is there for the new member if he has an idea or feedback, and will

do all he can to help. Although the welcome letter is important for building a sense of

community in the forum, there may be some mismatch between the tone of the welcome letter

and the tone that may be expected by the new member. That is, our interviews with potential

members indicated that they would first join a community to meet a need and only later, if the

need were consistently satisfied (and they had time), would they feel an obligation to contribute.

In addition, Gerber (2003) noted that three key determinants of whether members of the target

audience would use S3-XO. net were the utility of information, ease of use, and ability to

network. It may be more effective for the welcome letter to place greater emphasis on how the

forum will support the member than how the member can support the forum. It would also be

preferable if the welcome letter were addressed to the new member individually or to "Fellow

Professional" in the singular. This may make the form letter approach seem slightly more

personalized. Finally, in the very first paragraph of the welcome letter it is indicated that the

password provided during registration is incorrect and should be ignored. This does not set the

tone that the forum support staff is on top of things technologically, and may make the user

skeptical that the online meeting space will be usable. Of the 19 S3-XO.net members surveyed,

the majority of respondents indicated that they felt the welcome letter was a nice gesture, but

none indicated that the welcome letter made them feel like a valued new member of the forum.

At the top of the S3-XO.net homepage the overall function of the forum "Your

connection for building better units" is clearly stated (e.g., compare to CC. mil: "Building

combat-ready teams!"). The main page also presents a short description of the forum's purpose,

stated as follows: "A professional forum where professionals can come together to discuss

current issues." This statement sets up well the expectation that forum members are expected to

maintain a positive, professional bearing. However, although this description is accurate, it is

also somewhat vague. Moreover, as our interviews indicated, S3s/XOs initially entering the site

are not particularly interested in having a conversation, but would like to find time-saving

information quickly. The introductory text on the main page, containing update information, may

be a suitable place to set a positive tone in a manner that is more engaging or directly relevant to

the typical user. For example, this text includes a description of what the forum has to offer its

members as well as a statement describing the security restrictions on what information can be

shared in the forum. There is no Frequently Asked Questions section available in the forum, nor

is there a user comments area, an area for press releases or other information, or a brief tutorial

or introduction to the online meeting space.

There are two additional characteristics of the software used to build the online meeting

space that may generate a tone not intended by the forum support personal, namely that members

48
are not truly free to use their judgment to make posts. The first characteristic is that discussion

posts made by a member who has been purged from the forum remain in the forum, but the

author designation changes from the author's name to "purged user." Members are purged from

the forum for reasons other than malfeasance (e.g., the forum grows two large, and members of

subgroups split off into different forums), but this is unknown to the current members. In one

case in S3-XO.net, a purged user happened to have authored a discussion post that is critical of

the logistical information management system used in Iraq. Because this is the only discussion

post in the forum that takes a noticeably critical tone, readers of this post may assume that the

user was purged for being critical of the Army. The second characteristic is that discussion posts

that have been modified by forum support personnel (e.g., to correct spelling or make other

minor changes) are appended with a statement that they have been modified and who did the

modifying, but no indication of why modifications were made. This may create the impression

that members are not in control of how their posts appear to others. Efforts to set the tone in S3-

XO.net is here rated a combined red/amber for the above reasons.

Conversation Support

S3-XO.net is built on Tomoye Simplify, software specifically designed to support online

social interaction. Tomoye allows members to build discussions around "knowledge objects"

posted to the forum (e.g., documents, video clips, presentations, etc.) and/or to generate new

conversations on related topics that are not specifically associated with an upload to the site.

Powering discussion in this way, the use of Tomoye solves the problem of where to "locate"

discussion in the forum because discussion is enabled everywhere (see Schweitzer, 2003).

Because of its flexibility and ease of use, Tomoye is also the preferred software solution for

supporting discussion forums within the Army and in other branches of the service (e.g., Navy;

Kendall & McHale, 2003) and is recommended for the commercial sector (Schweitzer, 2003).

Discussion thread titles in S3-XO.net are authored by the initiator of the conversation, and

generally reflect the conversation topic (e.g., "Request for a division TACSOP [tactical standing

operating procedure]" and "Single COA [course of action] or Multiple?"). However, one or two

threads have titles that are not related at all and suggest that the author was uncertain how to

make a discussion post. For example "how can I ask for information or start a discussion" is the

title for a conversation initiated by a request for information about improvised explosive devices.

One or two initial conversation posts are duplicated in the forum, suggesting that the first time

they made their posts these users were uncertain whether they had successfully uploaded them to

the forum. Despite the occasional glitch, the conversation capability of Tomoye appears to work

well, and includes helpful information for evaluating each post, such as its author and the date

and time of the post. Currently, there is no capability for members to rate the utility of a

conversation thread or individual discussion post. Because the current version of Tomoye is the

backbone of a highly active APF (CC.mil), Conversation Support was rated green.

Locating Content

S3-XO.net has three methods available for locating content: a general search capability

located at the upper right of the main page, a knowledge search capability located in the lower

left, and a conversation search capability also located in the lower left of the main page. The

49
general search capability searches the entire forum, including conversations, documents, bios,

and even topics, using keywords entered by the user. The knowledge search allows users to

specify using keywords for the type of knowledge they would like to find. Searchable knowledge

types include: Joint Planning; Chief of Staff of the Army Task Force Updates; References; Field

Interview; Tactics, Techniques and Procedures, Lesson Learned; Topic; Dog Tag (i.e., member

bio sketches); Institution; Multimedia File; Tool/Document; Website Link; Discussion; News

Article; Event; Announcement; and Community. Once a knowledge type is selected, the user can

indicate what fields he would like displayed and how he would like the results arranged by

pressing a "filter" button and selecting items from a pull-down menu9. Unfortunately, the

knowledge search capability does not have a "find" or "go" button, making it potentially robust

but currently not functional because there is no way to initiate the search. The conversation

search allows the user to search for keywords present in discussion threads. All of these search

functions allow the user to sort search results in ascending or descending order, but it is unclear

what is ascending or descending (e.g., number of posts or number of views or some other metric,

such as date posted).

Each topic and sub-topic in the forum has a link that allows members to view the

conversation and knowledge shared on that topic. Topics with recent posts are flagged with a

flashing callout to indicate that a new conversation has taken place. In addition, a number

featured next to the topic or sub-topic name indicates the activity associated with the topic.

Activity associated with individual posts to the forum (i.e., the number of times a post has been

viewed) is indicated by numbers that appear near the post's title. However, the metrics used to

track the number of views is currently inaccurate, the topic activity metric is not well understood,

and the algorithm behind the flags indicating recent posts is not reliable. There is a "Rate this

Page" functionality, which allows the user to rate a particular page on a scale of 1 (worst) to 5

(best). This rating functionality currently appears to exist only on the main page, however, and

also does not apply to individual content contributions on the page.

Members of S3-XO.net receive an announcement via email that there is a monthly

newsletter posted in the forum. The newsletter is accessed by clicking a hotlink on the main page

called "Net Call (Month) (Year)." There currently is no content analysis or synthesis conducted

in the forum. However, the current and future S3s/XOs we interviewed indicated that content

that is analyzed and synthesized will be of maximal use to them. Links between content within

the forum are possible, and can be made either by the facilitators or by forum members. Links to

content outside the forum [e.g., at the Center for Army Lessons Learned (CALL)] may also be

made or content may be brought in and made downloadable (e.g., field manuals). The forum

provides links to CALL, BCKS, AKO, the Joint Electronic Library, and the Army Library

Program.

Four survey questions (complete survey is located in Appendix D) asked S3-XO.net

members (see footnote #8 in Joining the Forum for a description of the survey sample) to

indicate their impressions of finding content in the forum. These questions addressed the ease of

navigating content topics, using the search capability, finding new content additions, and

understanding the arrangement of content. On average, approximately 3 respondents gave

9 There are different pull-down menu options, depending on what knowledge type is selected. Pressing the "filter"

button links the knowledge type selected by the user to its associated data fields.

50
finding content in the forum a high-moderate rating. Two people on average gave finding content

a high rating, 2 gave it a moderate rating, and approximately 1 gave it a low-moderate rating.

In general, respondents indicated that navigating content topics was easy or fairly easy.

Most respondents indicated that the search capability worked fairly well with only occasional

problems or that they had not noticed anything special about the search capability one way or the

other. Respondents generally indicated that they had no trouble finding the new content additions

to the forum. Respondents also agreed that the way content is arranged in the forum generally

makes sense or that they had not noticed anything particular about how the content was arranged,

good or bad. For all of the above reasons, Locating Content was assigned an amber rating.

Locating People

Through our interviews, we discovered that by the time officers are ready to assume the

S3/XO duty position, they have already established many relationships within the organization,

and have made decisions about who to trust. They must be able to locate people in the forum

they already know and respect in order to feel that the forum can be of use to them. The ability to

establish the credibility of strangers is critical in their determination of the validity (i.e., veracity

and relevance) of posted content. The future S3s/XOs we interviewed also indicated that it would

be highly desirable to be able to locate the current S3s/XOs in the units they are preparing to

join. They are unable to locate this information easily through AKO or through installation

homepages.

S3-XO.net has a member directory that allows members to alphabetically search for

someone they think may be a member of the forum or to search for members depending on

certain biographical criteria (e.g., duty status, assignment history, branch, etc.). Biographical

sketches must include full name and email address. Sketches may also include home and duty

phone, duty status, branch, current duty position, current duty station, deployments, S3/XO

experience, topic areas of expertise, and links to the member's most recent posts (knowledge or

conversation). Clicking on the name of a contributor to the forum allows members to access that

contributor's biographical data. The main page of S3-XO.net features a list of the members who

are currently logged in to the forum. These names can also be clicked on to retrieve biographical

data. There currently is not a "featured person" area in the forum. However, S3-XO.net does

enable members to indicate that they are an "interested party" in a particular topic (e.g., fire

support, leader development, etc.). Experts in the topics for which there is an interested party

may then direct their assistance to these members.

Locating People was assigned a green status, though in the specific case of S3-XO.net, it

would be extremely beneficial to feature an additional directory that provides the contact

information of members and non-members holding BN/BDE S3/XO duty positions in units

Army-wide.

51
Structure and Characteristics - Recommendations

The Plan

Several sources have suggested that one can build a forum but that a separate, even more

challenging effort is to get people to come to that forum (e.g., Dixon et al., 2005; Kendall &

McHale, 2003; Wenger et al., 2002). Potential users must feel comfortable with the forum

culture and confident that the forum will meet their needs. An in-depth understanding of the

target audience is required to bring the forum to the attention of potential members and to ensure

that initial activity in the forum is directed to member sensibilities and needs (Gerber, 2003;

Stuckey & Smith, 2004; Wenger et al., 2002).

The overall rating of S3-XO.net's plan is amber, with highest ratings for functional and

technical specification and lowest ratings for identification and understanding of the target

audience. The strengths and weaknesses of the current plan suggest that emphasis has been

placed on the implementation of the forum at the cost of the purpose and goals of the forum.

Particularly because S3-XO.net's founding members were not current S3s and XOs and because

the rapid institutionalization of the forum involves civilian contractors in the facilitation process,

it is essential that the S3-XO.net support staff aggressively pursue an understanding of the target

audience's needs. This could include visits to U.S. Army Forces Command (FORSCOM)

installations to talk with S3s/XOs and their commanders, discussions with battlestaff

observer/controllers at the combat training centers, discussions with officers in the pre-command

courses who have recently left S3/XO positions, and interviews with S3s and XOs recently

returned from deployment. Although future S3s/XOs make up part of both the group of founding

officers and the target audience, these officers do not always know what their knowledge needs

are and will benefit from the discussion of current S3s/XOs as they hash out their challenges in

the forum.

In the end, the forum leadership and the other personnel supporting the forum should be

able to articulate not only how they think participation in S3-XO.net will enhance unit

effectiveness, but also where specifically they expect to see impact. Although a complete frontend analysis of the S3-XO.net target audience is beyond the scope of the present project, later

sections of this report will hopefully begin to address this difficult but critical issue.

Personnel

Even with a solid understanding of the target audience, a great deal of activity on the part

of a core group, facilitators, and leadership is required to generate self-sustaining APF activity

(Dixon et al., 2005; Kendall & McHale, 2003). The S3-XO.net leadership and support personnel

are extremely competent and passionate about officer professional development but

unfortunately must juggle the needs of the forum with many other job duties. The forum requires

several personnel dedicated full-time to accomplish its facilitation and marketing needs. These

individuals ideally would be Army officers who have just completed S3/XO duty positions, who

have elected the role of shepherding the forum, and who are reflective about their careers and

their profession. If coupled with the ideal support personnel described above, additional full-time

civilian contractors would also be a valuable aid in facilitation. However, clarification as to who

52
has primary responsibility for locating individuals, connecting members, and placing content in

the forum must be clearly determined before additional personnel are dedicated to the forum.

Functional Requirements

The overall status of S3-XO. net's functional requirements is amber/green, reflecting, as

did the technical and functional specification of the forum plan, the strength behind S3-XO.net's

online meeting space. Tomoye Simplify supports robust social interaction, and pending updates

to the software will enhance its potentially robust search capabilities. Where S3-XO.net's

functional requirements need improvement--Finding the Forum and Tone in the Forum-recent

initiatives by the forum leadership and support staff suggest a promising future direction in

recruiting and retaining the interest of new members.

S3-XO.net Social and Intellectual Capital

Connections

Table 11. S3-XO.net Connection Characteristics Status

Connection Characteristics Status

Red Amber Green

Membership Composition

Participating Members

Identifying Experts

Facilitation of Connections

Connection Quality

Overall Status

Membership Composition

S3-XO.net membership data were downloaded from the forum on 8 April 2005. To

prepare these data for analysis, duplicate accounts (33), guest accounts (1), invalid accounts (3),

and purged user accounts (113) were deleted. In addition, incorrect data verifiable through AKO,

such as title and duty status, were corrected'°. Of the 2215 members retained for analysis, 697

accounts were missing significant portions of biographical data (i.e., all optional fields were

blank). Because the S3/XO experience of the members lacking biographical data could not be

determined using AKO, these 697 members are not included in the below analysis of

membership composition. This creates a sample size of 1518 for S3-XO. net membership,

approximately 69% of the total population of S3-XO.net members as of 8 April 2005.

Among the 1518 members in the S3-XO.net membership sample, 221 reported being

current S3s/XOs and 285 reported being future S3s/XOs. Therefore, 33% of the total

10 This was done infrequently on a case-by-case basis where inconsistencies in the biographical data suggested an

error. For example, a DA civilian or retired officer may list their status as "active duty" instead of "other" and

"retired," respectively. Some reserve officers listed themselves as "Mr." instead of using their military title.

53
membership sample is a member of the target audience". Forty-five percent of the target

membership is a current S3/XO and 55% is a future S3/XO. Assuming the 697 accounts lacking

biographical data are representative of the 1518 accounts including biographical data, we could

conclude that as of 8 April 2005 there are 731 target members in the total membership

population, with 329 current S3s/XOs and 402 future S3s/XOs. According to Gerber (2003), the

approximate size of the target audience for S3-XO.net is 23,545. With 731 current or future

S3s/XOs, S3-XO.net would have approximately 3% of the target audience in its membership.

It is not certain that those members with significant portions of biographical data missing

are representative of the sample of members whose biographical data are filled out. It may be the

case that biographical data for some members were missing intentionally, for example for very

high-ranking officers or non-commissioned officers. Other members who did not provide

biographical data may not make have taken the time to do so because they do not belong to the

target membership and therefore do not see themselves as active participants in the forum. In any

event, the difference between 731 members of the target audience (when using the entire

membership population) and 506 members (when using the membership sample) is trivial

relative the size of the estimated target audience--a difference of one percentage point.

Because S3-XO.net members represent only 2-3% of the target audience, and only 33%

of the of forum membership is a member of the target audience, but there is a nearly 50-50

balance among current and future S3s/XOs, the assigned status for Membership Composition is

red/amber. Note that since 8 April 2005, there have been, on average, 18 new people added to

the S3-XO.net membership each week. Between 8 April and 23 June 2005, 202 new members

have been added, 45% of which are target members. Although this rate of new membership is

somewhat sluggish, it may reflect a promising trend of increased percentage of target members

in the overall membership. The balance among current and future S3s/XOs in the recently added

target membership also represents promising movement in a positive direction (54-46 compared

to 43-57).

Participating Members

Conversation. To examine conversation composition, the principal investigator reviewed

the 344 discussion posts present under "Conversations" headings in S3-XO.net as of 12 May

2005. Forum members who did not indicate their S3/XO experience were removed from the

analysis, reducing the number of posts to 327. As shown in the table below, 34% of the

conversation posts on S3-XO. net were made by target members. This percentage of contributions

may be expected given that an estimated 33% of the total membership is a target member.

Although the criteria for evaluating target member participation in conversation ["Green"

(85%+)] are arbitrary in the sense that no particular percentage of target member participation in

conversation has ever been linked to community health or impact, common sense suggests that a

majority of the members participating in the conversation should be from the target membership

in order for conversation to be relevant and to have a potential impact in the target community

11 See Section I. The remaining 67% of the sample membership is composed of 175 former S3s/XOs, 834 members

with "N/A" indicated for their S3/XO experience, and 3 missing entries. Eighty-seven percent of the 834 members

with "N/A" indicated for their S3/XO experience were CGSC students, evidently not on track to become an S3 or

XO. The remaining 13% had various duty positions, including either military or civilian positions.

54
(see also Gerber, 2003; Stuckey & Smith, 2004). Thirty-four percent falls short of the most

liberal definition of majority (51% or more).

If the founding officers are considered target members, the percentage of conversation

posts made by target members rises to just below 46% (this percentage does not include the

contributions of one founding officer with multiple postings who is a past S3/XO). Of the

original 34% of conversation posts made by target members, 24% of these posts was made by

current S3s or XOs (compare to the fact that an estimated 45% of the target members are current

S3s or XOs). If the founding officers are included, approximately 17% of the conversation posts

were made by current S3s or XOs.

Table 12. Conversation Posts by S3-XO.net Member Role

Member Role # of Conversation Posts Percentage of Total Conversation Posts

Target Member 111 34%

Founding Officer 50 15%

S3-XO.net Staff 62 19%

Former S3/XO 53 16%

N/A 51 16%

TOTAL 327 100%

Note that the discussion in the forum may not represent the total amount of discussion

that has occurred due to the forum. Forum members may initiate or continue discussion outside

of the forum through email, telephone, or face-to-face contact. Contact outside of the forum may

increase the impact of forum activity on officer development. Moreover, it is more likely that

target members will initiate and maintain discussion outside of the forum than non-target

members. Discussion and knowledge sharing inside the forum, however, must be relevant to

target member needs in order to stimulate discussion and knowledge sharing outside the forum.

Relevance may be at risk if relatively few target members are participating.

Knowledge. To examine knowledge contributions, a sample of 161 knowledge objects

posted to the site as of 12 May 2005 were reviewed. This sample size reflects slightly more than

12 half of the total number of knowledge posts2. Four posts were removed from the below analyses

because the S3/XO experience of the member having made the post could not be determined.

Knowledge post types reviewed included tools/documents, website links, announcements, books,

multimedia files, and ideas/tactics, techniques, and procedures/lessons learned. These type

categories mirror those used in S3-XO. net to organize information. As shown in the table below,

17% of the knowledge posts on S3-XO.net were made by target members. If the founding

officers are considered target members, this percentage increases to a majority contribution of

60%. Of the original 17% of knowledge posts contributed by target members, 26% was made by

current S3s/XOs (compare to the fact that an estimated 45% of the target members are current

S3s or XOs). When the founding officers are considered target members, this percentage

12 Our estimate of approximately 283 knowledge posts differs significantly from the estimate of 5300 knowledge

posts featured on the S3-XO. net homepage. Our estimate does not include conversation posts (approximately 341),

member "dog tags" or bio sketches (approximately 2215), documents on S3-XO.net itself, located in the archives

(approximately 24), topics (unknown number), institution descriptions (unknown number), or communities

(unknown number).

55
decreases to 8% (this percentage does not include the contribution of one founding officer who is

a past S3/XO).

In the case of both conversation posts and knowledge posts, current S3s and XOs were

contributing at a lower rate than they are represented in the target membership. However, current

S3s and XOs do not appear to be visiting the site at a lower rate than they are represented in the

target membership. In 12 samples of S3-XO.net's "members currently online" feature between 18

May and 10 June 2005, the proportion of current S3s/XOs online (relative to future S3s/XOs)

roughly matched the proportion of current S3s/XOs (relative to future S3s/XOs) in the target

membership. Whereas current S3s/XOs make up 45% of the target membership (as of 8 April),

the average percentage of the target membership online (between 18 May and 10 June) that was

a current S3 or XO was 49%. It is possible to conclude from this brief sampling of members

online that current S3s/XOs are less likely than future S3s/XOs to post when they are online, not

that they are online less frequently. In any case, because target members are making a minority

of the contributions both in terms of conversation and knowledge unless the founding officers are

considered, and the founding officers no longer actively participate in the forum, Participating

Members was assigned a red status rating.

Table 13. Knowledge Posts by S3-XO.net Member Role

Member Role # of Knowledge Posts Percentage of Total Knowledge Posts

Target Member 27 17%

Founding Officer 67 43%

S3-XO.net Staff 31 20%

Former S3/XO 17 11%

N/A 15 10%

TOTAL 157 100%

Identifying Experts

Sixty-nine percent (1518/2215) of forum members provided complete or nearly complete

biographical data, which other members could use to locate experts. However, only 17%

provided information regarding areas of expertise. Of the current S3s/XOs, 44% provided

expertise data; future S3s/XOs, 46%; and previous S3s/XOs, 50%. In comparison, only 4% of

S3-XO.net members who selected "N/A" for S3/XO experience provided information regarding

areas of expertise. Given that the large majority (87%) of the members who selected "N/A" for

S3/XO experience did not expect to become an S3 or XO and may not see themselves as

contributors to the site, it is reasonable to expect low level of expertise data from these people. In

contrast, the key members of the forum (target members and experts) provided much higher and

roughly equivalent levels of expertise data.

Three survey questions (see Appendix D for the complete survey) asked S3-XO.net

members (see footnote #8 in Joining the Forum for a description of the survey sample) to

indicate their impressions of finding experts in the forum. These questions addressed the ease of

finding experts in the forum, the ease of finding experts external to the forum, and the confidence

respondents had in expert members' willingness to help them. On average, 3 of the 7 respondents

56
gave finding experts a high-moderate rating. Approximately 2 people, on average, gave finding

experts a moderate rating, and approximately 2 gave it a high rating.

Interestingly, for finding experts either in the forum or because of the forum (external

experts), the modal response was that respondents had not given thought to using the forum to

locate experts. Of those who did consider using the forum for finding experts, roughly equal

proportions indicated that they were either confident or certain that the forum is a useful resource

for locating experts. Respondents generally agreed that they were somewhat confident a forum

member would help them if asked, with some respondents indicating that they were completely

confident. For these reasons, finding experts was assigned an amber rating.

Facilitation of Connections

As mentioned previously, S3-XO.net enables members to indicate that they are an

"interested party" in a particular topic, and experts in these topics may then direct their assistance

to these members. However, it may be unlikely that an expert will contact an interested party for

an open-ended discussion if a specific question has not been asked, so it is unclear how effective

this feature is in facilitating connections. The sole person assigned to full-time support of S3-

XO. net reported not having specific connections with experts in any of the content areas in the

forum, though he did indicate that he was able to call on personnel stationed at Fort Leavenworth

to help with answering some member questions.

Two survey questions (see Appendix D for the complete survey) asked S3-XO.net

members (see footnote #8 in Joining the Forum for a description of the survey sample) to

indicate their impressions of posting questions in the forum. These questions addressed how

question posts were acknowledged and confidence in the facilitators' ability to ensure that

questions were adequately answered. Of the seven respondents to the survey, only four had

posted a question. On average, five respondents gave posting questions a moderate rating.

Approximately one person, on average, gave posting questions a high-moderate or high rating.

One person on average gave posting questions a low-moderate rating.

Of the four respondents who posted a question to the forum, the majority could not

remember if they had received an acknowledgement of their posting, but one member recalled

never receiving an acknowledgement. The majority of respondents also indicated that they had

not considered how far they thought forum facilitators would go to help them answer a question.

Of those respondents who had given it some consideration, all were confident that the facilitators

would make an effort to help them answer their question, with equal proportions indicating they

thought the facilitators would go "all out" versus "only so far." For these reasons, Facilitation of

Connections was assigned a red rating.

Connection Quality

Three survey questions (see Appendix D) asked S3-XO. net members (see footnote #8 in

Joining the Forum for a description of the survey sample) to indicate their impressions of making

connections in the forum. These questions addressed the ease of making connections in the

forum, the utility of these connections, and the likelihood of re-using connections to get

57
additional help. On average, four people gave making connections a high-moderate rating.

Approximately two respondents on average gave making connections a moderate rating, and

zero a high rating.

Interestingly, the majority of respondents indicated having no opinion about how

effective it was to seek help in the forum. Of those who expressed an opinion, there was

agreement that help is generally received by those who request it. The majority of those surveyed

also indicated that experts in the forum seemed to be a valuable resource but that they had not

personally benefited yet from these experts. Finally, the majority of respondents indicated that,

depending on the person, they would feel comfortable asking for help multiple times from an

expert whom they had met in forum. Because the members we surveyed seemed positive about

the quality of connections in the forum, but had not been actively seeking or had benefited from

a connection, Connection Quality was assigned an amber rating.

Context

Table 14. S3-XO.net Context Characteristics Status

Context Characteristics Status Red Amber Green

Shared Narratives

Shared Codes and Language

Overall Status

Shared Narratives

S3-XO. net provides opportunities to participate in shared experiences via book reviews.

"Book Reviews" is a feature inside the "Professional Development" area on the site. One book,

American Soldier, had been reviewed in a discussion thread featuring six conversation posts

among two future S3s/XOs and three forum members who selected "N/A" for their experience as

an S3/XO. Upon examination of the discussion thread around the book, however, it is not clear

whether it should be classified as a shared experience. That is, after the second posting in the

thread, the conversation moved away from the book to other topics, and it is not clear from their

contributions that members joining the conversation late in the process actually read the book.

The book evidently was as effective in generating conversation as other knowledge resources

posted to the site, but the conversation may not have come out of a shared experienceper se.

There were no other formally designated opportunities in or through S3-XO.net to

participate in shared experiences. For example, the electronic calendar listing community events

did not have any events listed and there were no other types of collaborative exercises.

Two survey questions (see Appendix D) asked S3-XO.net members (see footnote #8 in

Joining the Forum for a description of the survey sample) to indicate their impressions of shared

experience via the forum. These questions addressed the sense of professional community gained

from the forum and the sense of shared experience with forum members. On average, five

respondents gave sharing experience a high-moderate rating. Approximately I person gave

sharing experience a moderate rating, and one gave it a high rating.

58
The majority of respondents indicated that they could envision a strong sense of

professional community and shared experience developing in the forum, but did not report that

such a sense of shared community or shared experience had already developed. Roughly equal

proportions of those surveyed reported that they had no opinion regarding a sense of professional

community or shared experience or that they had already developed a sense of professional

community or shared experience. For these reasons, Shared Experience was assigned an amber

rating.

Shared Codes and Language

One survey question (complete survey is located in Appendix D) asked S3-XO.net

members (see footnote #8 in Joining the Forum for a description of the survey sample) to

indicate their impressions of the nature of the content in the forum. The modal number of

respondents (3) indicated that sometimes content is out of their area of interest (e.g., branch), but

overall it is not a problem. Roughly equal proportions of the remaining respondents indicated

that they felt the content generally spoke well to their area of interest, that they had not noticed

one way or the other whether content spoke to their area of interest, or that they have had to work

to find content speaking their area of interest. Therefore, shared Codes and Language was

assigned an amber rating.

Content

Table 15. S3-XO.net Content Characteristics Status

Content Characteristics Status Red Amber Green

Contributing Members

Content Synthesis

Content Organization Content Activity

Content Quality

Professional Links

Overall Status

Con tributing Members

As shown in the table below, of the 157 knowledge posts examined, the majority was

located in either in the tools/documents area (39%) in each topic or situated within conversations

(41%). The significant majority of knowledge posts found in tools/documents was made by the

founding officers and facilitators (82%). Contributions of knowledge posts during conversations

were more evenly distributed among founding officers and facilitators (34%), target members (3 1%), former S3s/XOs (17%), and members who selected "N/A" for their S3/XO experience

(18%). Target members made nearly all of their knowledge posts (96%) in conversation, rather

than in the tools/documents area. That the vast majority of target member posts occurred in the

context of a conversation suggests that target members currently are more likely to take the effort

to respond .to a specific request for knowledge sharing, rather than to respond to an abstract,

59
perceived need in the larger community. This is in contrast to the posting activity of facilitators

and the founding officers, who made 30% of their posts in the context of conversations.

As shown previously in Table 13, 17% of the S3-XO.net knowledge posts sampled was

made by target members, falling short of a 51% majority. If the founding officers are considered

target members, this percentage increases to a majority contribution of 60%. Of the original 17%

of knowledge posts contributed by target members, 26% was made by current S3s/XOs (compare

to the fact that an estimated 45% of the target members are current S3s or XOs). When founding

officers are considered target members, this percentage decreases to 8%. Because target

members are making a minority of knowledge posts unless founding officers are considered, and

the founding officers are no longer actively participating in the forum, Contributing Members

was assigned a red status rating.

Table 16. Knowledge Post Locations

Location # of Posts Percentage of Posts

Conversations 65 41%

Tool s/Documents 62 39%

Website Links 22 14%

Multimedia Files 3 2%

Announcements 3 2%

Books 1 1%

Ideas/TTPs/Lessons Learned 1 1%

Total 157 100%

Content Synthesis

Two of the 324 conversation posts in the forum featured a link to other conversations in

the forum. Both of these posts were made by the same member, a future S3/XO who was

attempting to draw attention to the discussion thread he had started in another topic area. In one

case, the conversation in which the link was featured was related to the target conversation of the

link. In the other case it was unrelated. One of the 102 conversations started in the forum had

some degree of summarization made by a facilitator who was also a previous S3/XO. Six of the

102 conversations featured comments on the discussion itself (e.g., whether it was on track),

which were made by facilitators, a future S3/XO, and members selecting "N/A" for their S3/XO

experience. Of the 157 knowledge posts reviewed, two or three were posted in multiple related

topics. One post, a video-captured interview, was broken into several components and the

components distributed to relevant topics.

Although some attempt has been made to cross-reference posts in the forum, much of the

related content and conversation remain distinct from one another. In each new topic page (e.g.,

Warfighting-4 Operations), frequently there are separate areas for conversation, tools/documents,

website links, etc., but there currently is no cross-linking of related posts in these separate areas.

For example, in Admin/Log-- Maintenance Meetings, there are four separate but highly related

conversations about maintenance meetings that are not cross-linked or consolidated, nor are they

cross-linked with maintenance meeting related documents in the Tools/Documents area on the

same page. This is the case across topic pages as well as within topic pages. For example, in

60
Warfighting-) Open Discussion, there is a discussion thread on how combat training centers have

changed given the new warfighting environment. This discussion is not referenced, however, in

Training-> Combat Training Centers. As another example, there are several discussions and

knowledge posts related to the military decision-making process in several different topic pages,

but only one of them is linked to another. For these reasons, and because discussion

summarizations have not yet been made, Content Synthesis was assigned a red rating.

Content Organization

Content in S3-XO.net is organized according to the fundamental duties of S3s and XOs--

warfighting, training, leadership, administration/logistics, and transformation--and includes other

top-level topic categories addressing the profession-First 90 Days, Tools and References, and

Professional Development. Each top-level topic category features its own content, but also has

several sub-topic categories constructed around more specific topics. Some sub-topics, in turn,

have their own sub-topics. In the absence of a formal examination of the knowledge structures of

S3s and XOs, there are no rigorous criteria for evaluating content organization, but the current

organization seems reasonable, given what is explicitly known about the professional lives of

S3s and XOs. Furthermore, an informal review of the content in S3-XO.net does not raise serious

questions as to why particular knowledge objects or discussion threads are located where they

are.

Recall from previous discussion that of a small sample of S3-XO.net members surveyed,

respondents (86%) generally indicated that navigating content topics was easy or fairly easy.

Respondents (86%) also generally indicated that they had no trouble finding the new content

additions to the forum. Respondents (86%) agreed that the way content is arranged in the forum

generally makes sense or that they had not noticed anything particular about how the content was

arranged, good or bad.

The only concern raised by the content organization was that it appeared to be driven

from the top down, rather than from the bottom up. A formal assessment of S3/XO knowledge

structures (to include tacit as well as explicit) has not been conducted. Moreover, there were

intermediate-level sub-topics that have very little content or no content at all, suggesting that the

topic headers were placed before the content accumulated to justify such headers. Content

distributed across inorganic categories could be consolidated until organic categories arise from

forum activity. The dispersion of content was not a grave concern, save that it may have had

implications for usability and member engagement. The software supporting the online meeting

space was rather slow, and the time required to move between multiple topic pages may have

discouraged browsing. For these reasons, Content Organization was assigned a green rating.

Content Activity

Content activity was difficult to track because knowledge posts do not have upload dates

associated with them and download metrics evidently were not tracked by Tomoye. Approximate

measures of content activity, as presented here, must therefore suffice. The table below shows

the percentage of each type of knowledge post made by founding officers, all but one of whom

has been largely inactive in the forum since the June of 2003 (the active founding officer's posts

61
are excluded from the table). It may be assumed, then, that the knowledge posts made by these

individuals are approximately two years old. The remainin3 posts are therefore two years old or

younger. By dividing the remaining number of posts by 96 we obtain an estimate of the

average weekly rate at which new knowledge posts in the sample have been added to the forum.

The rate projected for all knowledge posts, because the sample examined here is approximately

half of the total knowledge posts on the site, would be approximately double the one shown in

Table 17. This gives an average rate of .27 posts a week. In contrast, one post every 12 hours

would equal a rate of 14 posts per week, per topic. There are eight high-level topics in S3-

XO.net, so a lower-bound active rate for content update is 112 posts per week. As shown in

Table 17, the number of posts per week in S3-XO.net was well below the lower-bound active rate

for content update. For this reason, Content Activity is assigned a red status rating.

Table 17. Founding Officer Contributions to Knowledge Posts

# of Posts by # Remaining # Posts Founding Officers Posts per Week

Conversations 7 11% 58 .60

Tools/Documents 37 60% 25 .26

Website Links 18 81% 4 .04

Multimedia Files 1 33% 2 .02

Announcements 2 67% 1 .01

Books 0 0% 1 .01

Ideas/TTPs/Lessons Learned 1 100% 0 0

Content Quality

The 161 knowledge posts that were sampled from S3-XO.net were analyzed for the

degree to which their associated context could assist the user in determining whether the post

was relevant to them and adequate for their purposes14. The rubric for evaluating the

contextualization of website links was as follows:

Context was defined as the information associated with the website link regarding

(1) what information can be found at the link; (2) the quality of that information or

the conditions under which the information can be of use; and (3) the means for

accessing that information if the link does not work. Website Link posts were

assigned a numeric rating of their contextualization, with 0 indicating no

contextualization and 3 indicating sufficient contextualization. A rating of 3 was

earned if all three aspects of context were addressed, a 2 if two aspects were

addressed, a 1 if one aspect was addressed, and 0 if none of the aspects was

addressed. Context was determined not only from the information provided by the

contributor, but also by (a) surrounding discussion posts, including comments

associated with a knowledge post not made in the context of a conversation; (b)

information gleaned from the link name; and (c) inferences that could be made

13 This number is derived by multiplying 52 times 2 and subtracting 8, which is the approximate number of weeks

the forum was shutdown and moved behind AKO. 14 Note context ratings should be independently assigned by a subject matter expert and compared to the context

ratings presented here. This step is necessary to validate the present rubric.

62
about the quality or the nature of the information at the link depending on its

location (e.g., CALL vs. USA Today).

The rubric for evaluating the contextualization of non-website link knowledge posts (i.e.,

tools/documents, multimedia files, and ideas/tactics, techniques, and procedures/lessons learned)

is as follows:

Context was defined as the information associated with the knowledge upload

regarding (1) the purpose/use of the upload; (2) the source of the upload; and (3)

the quality of that upload or the conditions under which the upload can be of use.

All knowledge uploads were assigned a numeric rating of their contextualization

in a fashion analogous to rating website link posts. Similarly, context was

determined by factors in addition to the information provided by the contributor of

the upload, including (a) surrounding discussion posts; (b) information gleaned

from the name of the file; and (c) inferences that could be made about the quality

or nature of the upload depending on its source or author.

Context ratings for Website Link posts sampled (N= 43) ranged from 0-3, with an

average of 1.72 and a mode of 2. For all other knowledge uploads sampled (N = 118), ratings

also ranged from 0-3, with an average of 1.98 and a mode of 3. Website Link posts earning

lower scores often did so because they lacked instruction for accessing the information provided

at the link if the link did not work. In our exploration of Website Link posts, we tested 19 such

posts and found that 42% did not work. For this reason, such instruction is critical for Website

Link posts to be useful. Other knowledge uploads earning lower scores often did so because they

lacked information regarding what their purpose/use was. This occurred more frequently when

knowledge uploads did not appear in the context of a conversation, but were posted

independently of a specific request or discussion. Knowledge uploads in the context of

conversation earned an average context rating of 2.10, whereas knowledge uploads external to

conversation earned a context rating of 1.67.

Obviously, this context assessment is subjective, conducted by an examiner who is quite

external to the target membership. Contextualization information should be provided by the

members of S3-XO.net themselves through ratings of individual knowledge posts and feedback

on their ease/difficulty of application. Evidence of such assessment is present in S3-XO.net

where members asked clarification questions about knowledge posts and where they indicated

that a knowledge post was of particular use. However, there currently is no easy, systematic

means for them to share their views on individual knowledge posts.

One survey question (see Appendix D for the complete survey) asked S3-XO.net

members (see footnote #8 in Joining the Forum for a description of the survey sample) to

indicate their impressions of the utility of content downloads in the forum. The majority of

respondents (4) indicated that they had found at least three downloads in the forum that have

been very helpful. Equal proportions (1) of the remaining respondents indicated that they have

found one useful download and are confident there will be more, that they have not given much

thought to how useful the downloads are, or that they have not yet found a useful download. For

all of the above reasons, Content Quality was assigned an amber status.

63
Professional Links

S3-XO.net currently does not provide access to leader-development opportunities

that are either available outside the forum [e.g., the Think Like a Commander adaptivethinking training program, Lussier, Shadrick, & Prevou (2003), or simulations or other

materials used in instruction at CGSC] or organic to the forum (e.g., situational judgment

vignettes, collaborative exercises, etc.). For this reason, Professional Links is assigned a

red rating.

Conversation

Table 18. S3-XO.net - Conversation Characteristics Status

Conversation Characteristics Status

RO r Amber Green

Participating Members

Conversation Activity

Conversation Quality

Overall Status

Participating Members

As shown previously in Table 12, 34% of the conversation posts on S3-XO.net have been

made by target members. Even when the founding officers are considered target members, the

percentage of conversation posts (46%) falls short of the most liberal definition of majority (51%

or more). Twenty-four percent of the conversation posts was made by current S3s or XOs, when

founding officers are not considered target members, and 17% of the posts were made by current

S3s or XOs when founding officers are considered to be target members. For this reason,

Participating Members is assigned a red status rating. Again, however, it should be noted that

discussion in the forum might not represent the total amount of discussion that has occurred due

to the forum.

Conversation Activity

As of 12 May 2005, there were 107 unique discussion threads initiated in S3-XO.net.

Forty-eight of these threads (45%) are one-post threads. Of these 48 one-post threads, four

clearly belong to multiple-post threads, but have somehow become separated, resulting in 103

unique discussion threads. The proportion of post type in the remaining 44 one-post threads is

compared to theproportion of post type in the initiating discussion posts for multi-post threads in

the table below'. The rubric for assigning a type to a discussion post is as follows:

Invitation to Build "New" Knowledge - Invitations to build "new" knowledge are

general statements or open-ended questions that address a novel, distal, or

15 This rubric was developed through an iterative process in which two raters assigned a post type to a sample of the

103 initiating posts, collaborative refined the rubric, assigned a post type to another sample of the initiating posts,

and so on. Inter-rater agreement reached using the above rubric was 80%. Where disagreement occurred, a final

post-type assignment was given on the basis of discussion between the two raters.

64
somewhat theoretical topic. In general, these invitations address organizational

matters, such as policy, structure, future directions, etc. The inference is that the

person initiating the discussion is not seeking one "right" answer as a basis for

action, but wishes to muse collaboratively with the professional community about

a broad topic. Asking the question does not assume that other community

members have personal experience with the topic but that they are likely to have

an educated, professional opinion.

Request for Input - Requests for input are questions that seek to gather a range of

opinions about or experiences with a particular topic, generally a proximal matter

of technical or tactical procedure. The inference is that the person initiating the

discussion assumes that others in the community have personal experience with

the topic, that there may be more than one right way of addressing the question,

and/or that sampling multiple perspectives is necessary to construct an effective

answer to the question.

Directed Question - Directed questions seek a "definitive," even "yes/no" answer

to a concrete question. Rather than seeking the collaborative opinion or experience

of the professional community, the directed question is an attempt to validate or

augment information or receive clarification using the expertise of someone in the

professional community with relevant experience/knowledge. The inference is

that the person asking a directed question assumes that somebody already has a

"correct" answer and seeks to engage that person.

Request for Knowledge Resources - Requests for knowledge resources ask

members to provide tools/documents or other knowledge products that can be

applied to unit functioning, such as tactical SOPs and training schedules. These

requests may be accompanied by a request for input or directed question, but are

assigned as requests for knowledge resources when the apparent emphasis is on

receiving tools, rather than opinions, experiences, or instruction.

Request for Expert - Requests for expert ask members to refer someone who has

particular knowledge in a specific area. The inference is that the person making

the request assumes that someone in the forum is or knows an expert in a

particular area. If a request for expert is accompanied by a request for input, a

directed question, or a request for knowledge resources, emphasis is placed on

these other modes of inquiry rather than on the request for expert.

Information Post - Information posts do not ask a question, but provide

information about a topic. This information may be a summary of some kind,

and/or may include either a link to a website, a tool/document to download, or a

description of an external knowledge resource. Generally, the Information Post

contextualizes the knowledge resources to which it refers. However, if an

information post is accompanied by a request for input, a directed question, or an

invitation to build "new" knowledge, emphasis is placed on these other modes of

inquiry rather than on the fact that information was posted.

65
Unknown - Unknown initiating posts appear to be an answer to someone else's

question or comment, but they cannot be clearly linked to any of the extant

conversation threads. These posts may address by name the unknown inquirer to

whom they are responding or may indicate agreement or disagreement with an

unknown conversation partner.

Table 19. Type of Initiating Posts for One-Post and Multi-Post Discussion Threads # Initiating # Initiating % of Total

Post Type 1-Post Threads l Multi-Post Threads

Invitation to Build "New"

Knowledge 1 2% 6 10%

Request for Input 6 14% 17 29%

Directed Question 7 16% 10 17%

Request for Knowledge 7 16% 9 15%

Resources

Request for Expert 0 0% 2 3%

Information Post 20 45% 14 24%

Unknown 3 7% 1 2%

Total 44 100% 59 100%

Fifty-seven percent of the 103 initial discussion posts were followed by a response. Sixtyeight percent of the initial discussion posts that were explicit requests for conversation (i.e.,

invitations to build "new" knowledge, requests for input, directed questions, requests for

knowledge resources, and requests for expert) were followed by a response. This moderate rate

of response to questions posed in the forum combined with the fact that only 28% of all posts

was followed by another post within 24 hours (and none of these was an initial post), resulted in

a red-amber status rating for Conversation Activity.

Note, however, that there are already almost as many discussion posts between 1 January

2005 and 12 May 2005 as there were in the entire year of 2003. This increase in posts bodes well

for the Conversation Activity in S3-XO. net. However, only 46% of the initial discussion posts

made in this time have been followed by a response, and only 24% of these posts were followed

by another post within 24 hours (none of the posts followed by another post within 24 hours was

an initial post). None of the promises to post additional information, replies, etc., made by

discussion participants were fulfilled through the forum, nor have any clarification questions

been answered in the forum. This reflects a decrease in the conversation activity from the

previous two years (compare 46% response rate in 2005 with 57% in 2003). Responding to

explicit requests for conversation also decreased from 68 to 58%.

Conversation Quality

In general, high standards of professionalism have been maintained in S3-XO.net. Of the

341 discussion posts reviewed, only one post was found to have an arguable level of

professionalism. This post was a referral to a debate going on in another conversation thread.

This referral would qualify as a helpful link to related conversation, except that the conversation

66
designated by the referral was completely unrelated to the conversation in which the referral was

situated. This gave the referral the feel of a pop-up ad.

As indicated in Table 19 above, 59 of the conversations started in the forum as of 12 May

2005 have resulted in conversation threads featuring two or more posts. These 59 conversations

were analyzed for the type of posts involved and for the flow of conversation succeeding the

initial post. The one multiple-post conversation thread that began with an "Unknown" initiating

post type was not analyzed because it was unclear how the conversation should flow from such a

post. Additionally, a conversation initiated by an information post was not analyzed because the

only other post involved was an advertisement for an unrelated thread (see the above paragraph).

The remaining 57 conversations were analyzed in detail. In the absence of definitive tools for

analyzing discussion content in knowledge-sharing forums, we sampled from the scholarly

literature where appropriate.

Invitations to Build "New" Knowledge. Six posts initiating multiple-post threads were

invitations to build "new" knowledge. Quality conversation following an invitation to build

" new" knowledge involves collectively constructing understanding or perspective that has not

previously been achieved. The product of the conversation is collectively "owned." That is,

discussants should come away from the conversation with the feeling that they have participated

in creating something new--knowledge that was not possessed by any of the discussants prior to

the interaction. This requires that discussants share and discuss perspectives on information,

identify and explore areas of disagreement, participate in the negotiation of new meaning, test

and modify new meaning, and come to consensus on new meaning (Gunawardena, Lowe, &

Anderson, 1998).

Five of the six invitations received more than one response. In two of the conversations,

discussants contributed knowledge products to aid the discussion. Importantly, four of the

conversations involved discussants interacting with each other, commenting on or building from

each other's contributions. In addition, two of the six conversations featured a comment on the

direction of the conversation by one of the discussants. However, none of the conversations

featured a post that closed the loop, indicating that the discussants believed the knowledgecreation process had been completed or that something had been learned from the process. Four

of the six conversations involved individual discussants making multiple contributions, but only

in one case did more than one person make multiple posts to the conversation. It appears that

invitations to build "new" knowledge can be active (average thread length was 6 posts), but also

that they seem to stop before new knowledge is constructed.

Requests for Input. Seventeen of the initiating posts were requests for input. Quality

conversation following a request for input involves the sharing of multiple perspectives and

achieving a more complete, or representative, understanding of a complex phenomenon. This

requires that the discussion have multiple participants constructively sharing their perspectives

and a synthesis of these perspectives that illuminates the trends in the information or attitudes

shared.

Twelve requests for input received multiple responses (between two and nine), and nine

spawned 17 related questions. Four of the related questions were answered, and two received

67
comments. None of the three clarification questions raised were answered. Five of the

conversations featured comments on aspects of the discussion, two of these conversations

featuring multiple comments. As with invitations to build "new" knowledge, requests for input

engaged active responding (7 posts on average). However, as was the case previously, the

synthesis and dialogue critical to building new understanding generally did not take place.

Directed Questions. Ten initiating posts were directed questions. Quality conversation

following a directed question involves answering the question posed. The asker should come

away from the discussion with actionable knowledge that he did not previously have. This

requires articulating one's question to elicit the required answer and engaging discussants to

provide information until a satisfactory answer is reached.

All ten questions were answered either by a direct answer (6) or by reference to

knowledge resources (4). Two questions received multiple answers and two featured comments

on aspects of the discussion. One of these comments was on the discussion itself and one

involved a summarization of what was said in the preceding posts. Two clarification questions

were asked, but none was answered. Three conversations spawned four related questions, two of

which were answered. Two of the directed questions received offers of assistance. Here too,

members appeared responsive to other members' needs, although the dialogue necessary to

ensure the answer is maximally relevant (e.g., answering clarification questions) did not occur.

None of the ten conversations featured a post that closed the loop, indicating that the question

initially asked had been answered.

Requests for Knowledge Resources. Nine of the initiating posts were requests for

knowledge resources. Quality conversation following a request for knowledge resources involves

referring the requestor to the knowledge resources he seeks and ensuring the requestor can access

and apply the resources referred to him. This requires delivery of or direction to resources and

providing context for applying these resources.

All of the nine requests were responded to either by one or more references to knowledge

resources (5 requests), offers of assistance (2), or some combination of the two (2). Six requests

received between two and five references to knowledge resources. Three conversations featured

comments on the requests or the knowledge resources referred. Requests for knowledge

resources appeared to have engaged productive responding. However, none of the nine

conversations featured a post that closed the loop, indicating that the initial need had been met.

Requests for Experts. Two of the posts initiating multiple-post threads were requests for

experts. Quality conversation following a request for an expert involves providing the requestor

with a referral. In both conversations, a referral was made, and one request received an offer of

assistance. As with requests for knowledge resources, requests for experts appeared to result in

productive interaction with S3-XO.net members and facilitators. Here too, however, none of the

requests for experts featured a close-the-loop post indicating that the requester made the

connection he needed.

Information Posts. Thirteen of the posts initiating multiple-post discussion threads were

information posts. Quality conversation following an information post involves questioning and

68
commenting that contextualizes the information provided, gauging its quality and applicability to

potential users.

Eight of the information posts received comments from S3-XO.net members on the post

itself or on the topic of the post more generally. Two information posts received multiple

comments. One highly decontextualized information post received a clarification question, but it

was not answered. Eight of the information posts spawned nine related questions, three of which

were answered. Overall, information posts did not generate a great deal of discussion. The

average length of discussion threads initiated by information posts was 3 posts (including the

initiating post). It is possible that conversation was not necessary to sufficiently contextualize the

information posts, but attempts by S3-XO.net members to use the conversation for learning (i.e.,

through related questions) generally did not engage other members.

One survey question (see Appendix D for the complete survey) asked S3-XO.net

members (see footnote #8 in Joining the Forum for a description of the survey sample) to

indicate their impressions of the utility of conversations in the forum. The modal number of

respondents (4) indicated that they had read one valuable conversation in the forum and that they

are confident there would be more. Roughly equal proportions of the remaining respondents

indicated that they have read at least three valuable conversations (2) or that they had not yet

read a valuable discussion (1). For these reasons, Conversation Quality was assigned a red-amber

status.

Social and Intellectual Capital - Recommendations

Connections

Membership composition is key to generating productive activity in the forum (Stuckey

& Smith, 2004; Wenger et al., 2002), and S3-XO.net requires a significant increase in target

members. Rather than focusing recruitment efforts on CGSC students, greater emphasis should

be placed on recruiting current S3s and XOs through, for example, advertisements of the site on

installation homepages and on other discussion forums (including the informal chat forums in

AKO), exchange with National Training Center observer/controllers, and visits to installations.

The upcoming membership drive planned by the S3-XO.net support personnel is a step in the

right direction, but more active recruiting of current S3s/XOs may be necessary. Once members

have been recruited, it may be helpful to make it a requirement to supply biographical

information, especially information about assignment history. Ideally, new members will see

themselves as contributors to the forum and voluntarily provide this information, but currently

approximately 48% of new target members do so. Yet, according to our interviews with current

and potential members of S3-XO.net, this information is critical in establishing the credibility of

post authors. Our interviews also indicated that the target audience of the forum is highly

desirous of being able to locate the S3s/XOs in the units they are assigned to for their next

S3/XO duty position. Locating these officers, keeping their contact information current in the

forum, and ideally, recruiting these officers to participate in the forum would represent a difficult

but very important step toward facilitating the development of connections.

69
Context

The opportunities to develop shared narratives in S3-XO.net seem relatively infrequent

and sparsely attended. Alternative methods to book reviews for creating shared experiences may

prove to be more engaging if they take less time and are more directly related to the jobs of the

S3/XO. These methods might include product critiques (e.g., tactical operations center setups,

training plans, etc.) or professional article critiques (e.g., publications in Armor magazine,

Military Review, or Parameters). If they are carefully created, command challenges such as those

found on CC.mil may also generate discussion in a direction more closely related to matters of

leadership, interpersonal relationships, and ethics. Facilitators should use their own social capital

to arrange for multiple members to participate in a shared experience at one time in order to

generate activity and interest. Simply posting the opportunity may not be enough, as the

relatively sparse discussion surrounding other knowledge or information posts indicates.

Content

Content activity is sluggish, and mostly driven by non-target members of the forum.

Moreover, four percent (approximately 7) of the 157 contributions sampled were made by a

current S3 or XO. There are multiple possible explanations for why there are relatively few

knowledge postings to the site, especially by current S3s/XOs, including (a) S3s/XOs with

limited time only post knowledge when it has been directly requested; (b) S3s/XOs feel reluctant

to share their knowledge either out of a sense of competition, anti-intellectualism, or concerns

about criticism; or (c) current S3s/XOs do not see themselves as contributors to the forum.

Contact and encouragement by facilitators may help increase posting by target members, if these

members are not posting for the above reasons (Dixon et al., 2005).

In contrast, content organization and contextualization in S3-XO.net appear to be

effective. However, some minor changes could increase the ease by which content is accessed

and reviewed for relevance. First, the front page of the forum should be changed frequently by

featuring new content, new members, or new points of interest for the S3/XO. Recent changes to

the front page of the forum suggest a hopeful step in the right direction provided that follow-on

changes do not take four months to occur. Second, related content should be consolidated such

that discussions or knowledge objects on the same subject are not distributed throughout the

forum unless they are distributed as a whole. Third, duplicate posts and duplicate headers should

be removed (e.g., there are two Rear Detachment Operations folders inside Admin/Log;

Leadership has an Ethics and Morals Folder and a Morals and Ethics folder). Topic folders

without posts should be removed. Finally, unless metrics and counters provided by Tomoye can

be explained or repaired, they should be removed. A couple of changes could enhance the

contextualization of knowledge objects, including reducing the number of clicks required to get

to comments on knowledge objects, featuring "date/time posted" information for all knowledge

objects, and adding an enhanced rating capability such that individual objects can be rated and

their ratings. Ratings and date/time posted information should be viewable before the object is

clicked on.

Conversation

The implication of several interviews we conducted is that the role of facilitators is

especially critical in a discussion forum that is geared towards S3s/XOs. The current S3/XO who

would login to S3-XO.net is several years older than, for example, the current company

70
commander logging into CC.mil. S3s/XOs are more experienced with the Army as an

organization, and therefore have more confidence in their place in the organization and in their

ability to solve problems. S3s/XOs who are in the National Guard have the additional experience

of having held down a successful career outside of the Army. For this reason, they may feel less

need for affiliation/validation than company commanders. Among S3s/XOs in the active Army,

there is a strong sense of competition for coveted command positions. S3s/XOs also deal with

problems much larger in scope than company commanders. They must delegate projects, rather

than tasks, and must deal with combined arms tactics and training. This makes them busier than

ever (working 12-hour days and weekends) at a time when they are more likely to have families

who place demands on their time outside of work. The greatest incentive for an S3/XO to

participate in the forum, then, is confidence that the forum will get him what he needs when he

needs it and obligation to give back what he has received. Facilitators must provide this incentive

by ensuring that responses to member questions are timely and adequate.

Conversation in S3-XO.net is sluggish and it is unclear how useful the productive

conversation is to members. Shorter times between posts not only would reflect greater activity,

but also will generate greater activity. To increase the perception of activity in the conversation

threads, discussions beginning and ending in 2003 or 2004 should be archived and made easy to

reference. In addition, requests or questions for which there are multiple posts should feature

some kind of follow up that closes the loop indicating how the request or question was addressed

(see Dixon et al., 2005). Fewer conversations starting with references to knowledge resources or

information posts may generate more discussion unless the forum staff can arrange to have

multiple members generate and sustain conversation activity around such posts. Facilitators must

actively link members with questions to members or other people with answers through

exploration of the membership or making use of their own social capital (Dixon et al., 2005).

Facilitators must use these connections to ensure that every question has some level of response

(not just acknowledgement) within 24 hours.

Overall

The software supporting the S3-XO.net online meeting space, Tomoye Simplify, is

generally recognized as the platform of choice for online communities of practice, at least in the

military (Glennie & Hickok, 2003; Kendall & McHale, 2003). Tomoye is preferred by the

military because it is more cost-effective than other software solutions available (including a

solution developed in-house) and relatively easy to use both by facilitators and forum members.

Importantly, it provides several of the functionalities key to effective collaboration, including

asynchronous chat, search capability, and member databases.

Unfortunately, one limitation of Tomoye is that it is slow and it is often difficult to wait

for new pages to appear. Effort should be made to reduce the number of steps members must go

through to find particular items or simply to browse. Otherwise, members may lose patience and

disengage from the forum. Some relatively straightforward changes that could address this

limitation are as follows:

1) Provide information in the post initiating a conversation that indicates whether the

conversation contains knowledge objects and when the most recent post to the conversation was

made.

71
2) Show all of the sub-topics associated with a particular top-level topic even when the

page opened is in a sub-topic. This way, members do not have to return to the top-level topic to

view another sub-topic.

3) Add a "browse knowledge" capability so that members can see a list of all of the

knowledge objects at one time, organized by topic/subtopic and with links to related discussion

posts, including comments.

S3-XO.net Impact on Member Competence and Professionalism

Recall from earlier in this report that in order to manage the scope of individual-level

impact assessment one should target those personal characteristics, competencies, and actions for

which common or persistent problems have been identified in the target membership. In the case

of S3-XO.net, the target membership is comprised of current and future S3s and XOs. Individuallevel development should therefore prepare future S3s/XOs to assume their next duty position

and should support current S3s/XOs as they perform their job duties.

Recall also that the feasibility of the quasi-experimental approach to assess individuallevel impact is highly questionable. A reasonable proxy is to assess whether or not the content

and discussion in S3-XO.net addresses the key problem areas in S3/XO competence and

professionalism. The social and intellectual capital assessment criteria applied above could be

used to assess the functional effectiveness of the forum content in the key areas of interest.

For these reasons, the following section presents an overview of the personal

characteristics, competencies, and actions expected of S3s/XOs, highlighting general areas where

difficulty has been identified. We used a combination of literature review and interviews with

subject matter experts to understand organizational expectations of the S3/XO and to identify

areas of difficulty. A list of the types of subject matter experts we interviewed is provided in

Appendix A. If S3-XO.net is to have an impact on officer competence and professionalism, and,

by extension, unit and organizational effectiveness, these areas should represent a significant

proportion of the productive discussion and knowledge sharing occurring in the forum.

Requirements of the S3/XO

The S3/XO duty position is a critical point in an Army officer's career because the

position that he assumes after being an S3/XO will set the tone for the rest of his career. That is,.

his performance as an S3/XO is a critical determinant of the Army's decision to select him for

promotion and command. At this point in his career, the Army officer generally has decided to

remain in the Army until retirement. In addition, the Army has invested in him, having chosen

him to go to CGSC as preparation to become an S3/XO and then choosing him for a position that

is a critical stepping-stone to command. Although the S3/XO does not have to juggle the

demands of command and administration, as does the company commander, he must work

extremely hard to compete with the other highly qualified S3s/XOs for limited command

positions and continued upward mobility in the Army. The future S3s/XOs we interviewed

72
expressed that an S3/XO community that could help them save time and increase productivity,

develop camaraderie, and seek advice or advise others would be highly desirable.

As field grade officers and leaders in the U.S. Army, S3s and XOs are expected to

embody certain values and personal characteristics, demonstrate particular competencies, and

execute specific actions. These values, competencies, and actions correspond to the Army's Be,

Know, Do conceptualization of leadership [Field Manual (FM) 22-100; U.S. Department of the

Army (DOA), 1999], and serve as the foundation for effective units and, ultimately, an effective

organization. Participation in S3-XO.net has the potential to foster officer attitudes, capabilities,

and skills that meet or exceed the expectations of unit commanders and the organizational Army

not only through knowledge sharing but also through relationship building that fosters a sense of

mutual trust, identification, and obligation within the professional community.

Values -Being

The Army values described in FM 22-100 are loyalty, duty, respect, selfless service,

honor, integrity, and personal courage. Soldiers are expected to be loyal to the U.S. Constitution,

to the nation's civilian leadership, to the Army, to the unit they serve, and to other Soldiers. They

are to put the welfare of the nation, of the Army, and of other Soldiers ahead of their own. They

are expected to fulfill their obligations to the best of their ability and to treat people in the

manner in which they wish to be treated. They must live up to the Army values, making legal,

moral, and ethical choices even in the face of fear, danger, or adversity. Adherence to these

values constitutes an important aspect of Army professionalism--the conduct, aims, or qualities

that characterize the professional leader.

Several environmental conditions currently threaten the level of professionalism achieved

in the Army after the Vietnam War. One condition is the tension between post-modem relativism

predominant in civilian culture and the objective professional ethic expected by the Army

(Snider, Nagl, & Pfaff, 1999). A second condition is the challenge to the Army's and Soldiers'

warfighting identity introduced by full-spectrum operations and the increased prevalence of

unconventional attacks against national interests (e.g., computer hacking) that places civilian

experts in roles of active national defense (Gray, 2001). Challenges to the unique purpose and

legitimacy of the Army as the nation's warfighters, which serve as the basis of professionalism

(Snider et al., 1999), threaten adherence to Army values, especially in a time when the objective

professional ethic has come under question. Moreover, doctrine describes Army values, but

knowing what these values are is not sufficient to make decisions that adhere to these values,

particularly when a decision requires giving one value precedence over another (Patterson &

Phipps, 2002). Increased ethnic and cultural diversity in the Army presents further challenges to

maintaining an objective professional ethic and arriving at a unified approach to moral decisionmaking on the basis of an overview description of values.

Some recent reports suggest that adherence to Army values is problematic and has

negative implications for retention and the quality of the retained force. For example, Bilafer

(2001) claims that senior officers in the Army are failing to adhere to Army values and suggests

that this failure to maintain rigorous standards for behavior is having negative effects on the

morale of junior and intermediate officers, with troubling implications for retention, cynicism,

73
and careerism (see also Marlin, 1997; Wong, 2000). For each Army value, Bilafer (2001)

presented stories illustrating the importance that some senior commanders have placed on selfpromotion over good conduct. These stories feature absence of top-down loyalty, lack of care for

Soldiers, including inadequate mentorship [see also Army Training and Leader Development

Panel (ATLDP), 2003], inconsiderate behavior toward subordinates, including harassment and

intimidation, and misrepresentation of unit readiness. Although this report is now more than four

years old, it serves as an alert to the effects that threats to professionalism can have on leader

behavior.

Another recent study (Patterson & Phipps, 2002) reports that the number of

commissioned officers separated from the Army for ethical misconduct (as opposed to other

causes) increased by 11% between 1992 and 2000, from 1.5% to 3.9% of all separations. This

study likely underestimates the actual occurrence of ethical misconduct by Army officers

because many instances of ethical misconduct may never come to light. In addition, the

aftermath of recent (Abu Ghraib, Bagram) and historical (My Lai) ethical scandals suggests that

ethical misconduct conducted by officers or approved by officers that does come to light may not

necessarily result in separation of officers from the Army (though see "2 Officers Punished in

2003," 2005).

In contrast to these findings, Wong (2000) and the Operation Iraqi Freedom Mental

Health Advisory Team (OIF MHAT, 2003) have both reported high levels of professional

behavior or feelings of professionalism among officers even in the midst of grave job

dissatisfaction and low morale. It is difficult to know without comprehensive study how

pervasive unethical or unprofessional behavior in the Army is, let alone the impact that this

behavior has on unit and organizational effectiveness. Assessing the "true" status of Army

professionalism is difficult in part because Army professionalism is not necessarily well

understood (Gray, 2001) or measured, particularly now as the Army's fundamental role is in the

midst of change.

In any case, threats to organizational identity and individual responsibility--especially

full-spectrum operations--are real and affect not only commissioned officers but the Soldiers

they lead (Britt, 1998). The morale of Army Soldiers, particularly those serving in Operation

Iraqi Freedom is quite low, with suicide rates among Soldiers higher than in any other recent

conflict, including Vietnam (OIF MHAT, 2003). Moreover, it may be argued that in an

organization with an objective professional ethic just one person who strays from that ethic just

one time represents a significant threat to the moral fiber of the organization as a whole. Field

grade officers therefore must be constantly mindful of their adherence to Army values and must

also ensure that the Soldiers they lead are supported in their efforts to do the same.

Participation in an APF may help field grade officers defend against threats to

professionalism through exposure to and bonding with colleagues in a professional setting. Such

activity could expose officers to positive or even inspiring examples of professionalism, allowing

social pressures to shape moral and ethical behavior. Similarly, such exposure may buttress the

professionalism of those officers who experience a lack of professionalism in their unit and are

concerned that it is typical of the Army as a whole. Sharing knowledge in the professional forum

may also provide an opportunity for officers to collaboratively define the unique purpose and

74
legitimacy of their own roles in supporting their unit, especially where transformation and fullspectrum operations have required role re-definition. In order for participation in an APF to have

this kind of benefit, discussion must address questions of professional identity, of values, and of

ethical and moral leadership. Discussion must include stories and personal reflections, which are

vehicles for sharing tacit knowledge necessary for successful leadership (Cianciolo, Antonakis,

& Sternberg, 2004). To support the S3 and XO in specific, discussion must also address the

personal characteristics staff officers are expected to embody.

FM 6-0 (DOA, 2003) identifies nine personal characteristics continually developed by

effective staff officers: competent, proactive, creative, flexible, self-confident, loyal, teamoriented, effective manager, and effective communicator. Some of these personal characteristics

overlap in obvious ways with Army values, but others are specified more in terms of the

requirements of staff officers in particular. These are areas of special challenge for most staff

officers, including the S3 and XO.

For example, the competent staff officer as described in FM 6-0 is familiar with his own

and other staff officers' functional duties. He is also a team player who advises, consults and

cooperates with other staff officers. However, staff officers at the battalion and brigade level

have consistently shown difficulties with horizontally and vertically sharing information and

integrating their efforts (CALL, 1998, 2003). The successful staff officer manages time and

resources effectively, avoiding duplication of effort, but reinventing the wheel is generally

recognized to be a common problem in the Army. Finally, the effective staff officer must be

concise in his communications, writing concise orders and plans, staff studies, staff summaries,

and reports. Staff officers have particular difficulty in this area, often failing to analyze

information before briefing the commander. Challenges to efficient and effective communication

will only increase as digital systems provide increased volumes of information to sift through

(Langley, 2004; White, 2001).

Difficulties in these areas appear to stem from the perceived lack of time available for

staff officers to perform immediate tasks while simultaneously establishing a foundation for the

effective performance of future tasks. Moreover, the organizing principles that staff officers

could use to increase efficiency (e.g., commander's intent, understanding of each other's roles;

Cianciolo & Sanders, to appear; Olmstead, 1992) are often lacking. For the S3 and XO to be

supported by community building and knowledge sharing, discussion must provide a means to

save time but must also enable the S3/XO to (a) better conceptualize his role in the staff, in the

command and control (C2) team, and in the unit; (b) increase his awareness of the interpersonal

as well as technological means available to him to manage information effectively; (c)

internalize his role as a leader and role model for his peers and subordinates, and (d) envision all

situations as potential leverage for collective and personal success.

Competencies - Knowing

FM 22-100 (DOA, 1999) outlines four areas of leader competence--tactical, technical,

conceptual, and interpersonal--which broadly characterize what a leader must know to lead

successfully. Tactical competencies are the skills necessary to conduct operations and are

enhanced by effective technical, conceptual, and interpersonal competence. Technical

75
competencies relate to the basic tasks and functions the officer must perform on the job.

Conceptual competencies enable sound analysis, judgment, and decision-making. Interpersonal

competencies are those competencies that enable the leader to exert his influence and ensure his

will is carried out.

Tactical Competence. Tactical competence comprises warfighting capability and the

synchronization/orchestration of friendly military and non-military assets to achieve decisive

results on the battlefield. As staff officers, neither the S3 nor the XO has the command authority

to employ units in combat under normal circumstances. Therefore, to demonstrate tactical

competence staff officers must be proficient at synchronizing mission plans and operations.

Synchronization is defined as "arranging activities in time, space, and purpose to mass maximum

relative combat power at a decisive place and time" (FM 3-0, DOA, 2001). Aiding the

commander in planning and executing missions by achieving synchronization is the fundamental

tactical role of staff officers. It is the responsibility of the XO and S3 as leaders of the staff to

ensure that the staff is trained and integrated such that they can synchronize their areas of interest

effectively.

Synchronization involves knowing the capabilities and limitations of each battlefield

operating system (BOS) at a particular time given a particular location, integrating the efforts of

each BOS, and leveraging BOS capabilities to create a desired effect on the enemy or the

environment. Fundamentally, synchronization is difficult because it requires staff officers to

anticipate one another's needs and communicate freely in a time-constrained environment with

substantial personnel turnover, diverse experience levels, and considerable complexity (e.g.,

digital C2 systems, unpredictable enemy activity, transforming tactical roles). Moreover, the

rapid everyday operational tempo experienced by staff officers can preclude the development of

technical, conceptual, and interpersonal competence necessary for conducting collective tactical

tasks as staff officers struggle just to keep up with day-to-day managerial demands. As the chief

of staff, the XO must step back from immediate day-to-day activities and assess the staff's

collective capability, identify areas of strength and weakness, and train the staff to strengthen

weak areas and maintain areas of strength. In addition, the XO must put standing operating

procedures (SOPs) into place for how the staff as a collective will conduct mission planning and

support the commander during execution, which will aid in team communication and

information management. As the operations officer, the S3 must ensure that his day-to-day

activity, such as conducting training and evaluating unit readiness, involves a systems approach

in which unit readiness is conceptualized as the operational effectiveness of the combined assets

available to the unit. The S3 also assists the commander in the execution of current and planning

of future operations.

To support the development of tactical competence in the XO, then, knowledge sharing

and discussion must provide a means for XOs to rapidly prepare staff readiness assessment and

training plans and to quickly develop SOPs for mission planning and staff coordination during

execution. Knowledge sharing and discussion must provide a means for XOs and S3ss to more

easily conceptualize operations as combined arms missions and to translate the conceptualization

of combined arms missions into effective training plans. Both S3s and XOs may demonstrate

increased tactical competence if knowledge sharing and discussion enabled them to understand

and address the implications of diverse levels of experience in the staff, to rapidly form trusting

76
relationships with new staff officers, and to manage more effectively the demands of the

complex operational environment.

Technical Competence. Technical competence comprises awareness and use of doctrine,

technological exploitation, understanding of how diverse conditions affect unit readiness, and

information management. Technical competencies particularly important to the S3/XO include

leveraging digital C2 technology to aid in situational awareness and understanding and

information dominance (Leibrecht, Lockaby, Perrault, & Meliza, 2004). Lacking technical

competence in these areas has direct and indirect implications for tactical competence. Directly,

technical competence enables the S3/XO to resource, allocate, exploit, and apply available

digital C2 equipment for enhancing mission planning and tracking the progress of operations.

Indirectly, technical competence enables the S3/XO to train the staff and subordinate officers on

the digital skills required to act decisively in the area of operations.

Individual technical competence is particularly challenging to develop for multiple

reasons, not the least of which are limited availability of systems on which to train, limited

access to alternative digital training techniques, rapid rate of change in the capability of digital

C2 systems, and assignment patterns that remove officers from operational units for several years

at a time. Collective technical competence as developed by the S3/XO must address such factors

as bandwidth constraints on what information can be shared when, stovepiped digital systems for

each BOS (where access to digital systems is equivalent across BOSs, which is not always the

case), differing versions of the same digital systems at different echelons, and asynchronous

system upgrades across BOSs (Leibrecht et al., 2004).

Supporting the S3/XO in developing individual technical competence involves sharing

knowledge as to what training is available for using the digital C2 systems that each staff officer

must use to conduct planning and track operations, what tactics, techniques, and procedures

(TTPs) have been used to leverage the capability of these systems, and what resources are

available for better understanding the latest status of these systems and how they can be used to

achieve information dominance. Supporting the S3/XO in developing collective technical

competence in their units involves sharing digital SOPs, TTPs for addressing bandwidth and

access constraints, and digital skills training techniques used in unit drills, among others.

Conceptual Competence. Conceptual competence is comprised of establishing with

others one's intent, taking a systems perspective to understanding situations, reasoning critically,

thinking creatively, thinking reflectively, adapting to changing circumstances, and predicting the

second and third order effects of one's own actions on the environment. By the time officers

become majors, they are expected, as a matter of course, to take initiative to identify and solve

problems and learn from their experiences. They must develop novel solutions to difficult

problems in an operational environment where the enemy is asymmetric and constantly modifies

his tactics. They must respond quickly and adaptively to environmental changes, anticipating

change where they can and thinking flexibly in unanticipated circumstances. Finally, they must

empower their fellow officers to take their own initiative by making their intentions clear and

developing conceptual competence in others. Conceptual competence enables S3s/XOs to

oversee effectively the administrative and logistical support and training of their unit at a time

when resources are extremely tight and the structure of the unit itself is changing. Conceptual

77
competence, through increased adaptability and innovation, also enables S3s/XOs to better

support mission planning and staff integration at a time when many aspects of tactics are

uncertain and the exact role of specific staff officers in supporting the planning process is

uncertain.

The increased demand in the Army for such competencies as critical reasoning, creative

thinking, and adaptive ability has not yet been accompanied by the cultural change necessary to

allow junior officers the opportunity to develop these competencies (Wong, 2002). Emphasis has

been placed in the schoolhouse on developing adaptive, innovative leaders and critical thinkers

(e.g., Fischer, Spiker, & Reidel, 2000; Lussier & Shadrick, 2004), but lacking the opportunity to

exercise them in the field, officers cannot retain new competencies developed in the

schoolhouse. Although the war in Iraq has shifted tactical emphasis down to the company and

platoon levels, providing junior officers unprecedented experience with innovative command

decision-making, most current S3/XOs have assumed their duty position without having had this

experience (Wong, 2004). As a result, many S3s/XOs must use their immediate experiences to

gain conceptual competence.

Supporting the S3/XO in rapidly gaining conceptual competence from his immediate

experiences must involve promoting the S3/XO's inclination to recognize and use his

experiences and those of others as learning opportunities. This requires providing an

environment in which the S3/XO (a) is encouraged to make explicit the situations in which he

has worked, the problems he has identified, the solutions he has created, and their corresponding

outcomes; (b) can safely discuss a problem he is facing with others who have faced similar

problems in order to clarify his definition of the problem, identify a range of possible solutions,

or seek feedback on his proposed solution; and (c) can rapidly access best practice in addressing

problems similar to that which he is facing. Such an environment would facilitate

resourcefulness, situated reflection, and learning orientation, which serve as a basis for

conceptual competence (Cianciolo et al., 2004).

Interpersonal Competence. According to Horey, Fallesen, Morath, Cronin, et al. (2004),

eight interpersonal competencies characterize the future Army leader. Three that have not

already been addressed in previous sections are (1) vitalizing a positive climate; (2) arming self

to lead; and (3) extending personal influence. Vitalizing a positive climate involves setting the

conditions for positive attitudes and effective work behaviors. Arming self to lead is that activity

the leader engages in to ensure he is prepared for challenges, aware of himself and his

environment, and engaged in lifelong learning. Extending influence involves ensuring that one's

influence goes beyond the direct lines of authority and chains of command. For the S3/XO,

interpersonal competence means creating a climate in which staff actions are cohesive and

integrated (Olmstead, 1992), where each staff officer knows and is empowered to perform his

role on the team, where personality differences do not affect the functioning of the staff, and

where each staff officer's effort is focused on collectively supporting the commander.

Interpersonal competence also means the S3/XO must take the initiative to build a constructive

relationship with the commander, fellow staff officers (including those at higher or lower

echelons), the command sergeant major, and subordinates. These relationships must enable him

to determine what is expected of him to lead the staff effectively, to identify his strengths and

78
weaknesses as they relate to leading in his current situation, and to understand the particular

challenges to the effectiveness of his staff and unit.

The S3/XO's development or display of interpersonal competence can be difficult in

units where the command climate is negative or apathetic, where the previous S3/XO has not left

behind information useful for functioning within the unit, where there is a great deal of turnover

of staff officers, where there are personnel shortages that reduce time available or staff

capability, and where staff officers are inexperienced and unlikely to take initiative. Some

combination of these conditions characterizes most units, especially high turnover rates,

personnel shortages, and inexperienced staff officers.

Supporting the S3/XO in developing and/or demonstrating interpersonal competence

requires that S3s/XOs have access to the officers who preceded them--to be able to discuss the

peculiarities of the unit to which they have been assigned and to share methods for working

effectively in the unit. S3s/XOs must also be exposed to opportunities to learn general methods

for dealing with turnover (e.g., SOPs dealing with collective staff tasks, such as planning) and

inexperienced staff officers (e.g., exercises or other techniques to foster initiative). Finally, the

S3/XO must be aware of what worked or didn't work for others who dealt with difficult

interpersonal situations, such as low morale, poor command climate, or times of high stress

and/or workload. Storytelling is often an effective vehicle for sharing tacit knowledge about how

to handle challenging interpersonal issues (Cianciolo, Matthew, Wagner, & Sternberg, in press).

Actions - Doing

FM 22-100 (DOA, 1999) outlines three broad categories of action in which successful

field grade officers must demonstrate competence-influencing, operating, and improving.

Influencing involves communicating effectively, leading by example, and moving beyond

reactive activity to engage in anticipatory decision making. Operating involves effective mission

planning, preparation, and execution, as well as objective information assessment. Improving

involves engaging in self-development and lifelong learning, building successful teams, and

promoting self-awareness. Several aspects of these competencies, such as synchronization,

building staff integration and cohesion, and preparing oneself to lead, have been addressed

previously and will not be addressed here. Only unique elements are discussed.

Influencing. Influencing involves exemplifying Army values and "warrior ethos,"

communicating effectively, and making effective decisions, all of which have been addressed in

the preceding discussion.

Operating. In addition to synchronization during mission planning and execution,

operating involves the gathering and analysis of information in order to understand its reliability,

validity, and sources. It is important that information assessment be as unbiased and objective as

possible. Both the S3 and XO must be able to make actionable recommendations to the

commander on the basis of sound information about the threat and the environment.

Recommendations must also be made on the basis of an accurate understanding of the friendly

situation.

79
The assessment of information on the threat and the environment is particularly

challenging in today's operational environment because there is a great deal more information

being gathered by a greater diversity in sensory capability. This capability includes advanced

sensing technology as well as the use of Soldiers and informants in the civilian populace as

human intelligence gatherers. The primary threats to the reliability and validity of incoming

information about the threat and the environment include (a) lack of information integration

necessary to determine the true meaning and implications of intelligence data; and (b)

uncertainty regarding the credibility of civilian informants. The primary threats to the reliability

and validity of incoming information about the friendly situation include (a) incompleteness of

information due to inadequate awareness of those reporting the information; and (b)

misrepresentation of information (e.g., troop readiness status) due to lack of awareness or

reporting of future projections rather than current status.

Discussion and knowledge sharing that supports the S3/XO in effectively assessing

information should cover such topics as methods for integrating information coming from

diverse sources, techniques for determining the credibility of civilian informants, and methods

for ensuring that staff estimates and other information regarding friendly status are complete

and/or accurate.

Improving. Improving involves developing a lifelong learning orientation, empowering

subordinates to take on increased responsibility and initiative, and building cohesive teams, all of

which have been addressed in the preceding discussion.

Member Competence and Professionalism - A Proposed Framework for Assessment

To assess the impact that S3-XO.net could potentially have on member competence and

professionalism, a red-amber-green rating system analogous to the one used earlier in this report

was used. The red-amber-green criteria for each aspect of member competence and

professionalism are presented in the tables below. It is important to note that the below

assessment criteria are recommended as a starting point and that several of the assessment

criteria presented here must change as current issues evolve. In addition, these criteria may not

be exhaustive or reflect the areas of greatest joint interest to the Army and S3-XO.net members.

Finally, it should be noted that the proposed framework is for evaluating potential impact of the

forum, rather than assessing impact directly. The assumption behind this approach is that (a) it is

more feasible and more tractable to assess the potential for impact rather than impact directly;

and (b) if direct impact is of particular interest, the proposed approach is useful for determining

whether taking on such an effort would provide useful additional information.

Impact on Member Competence and Professionalism - Assessment of S3-XO.net

The impact of S3-XO. net on the competence and professionalism of its members was not

formally assessed in the present investigation because status ratings of conversation, content, and

connections were all red-amber. More specifically, conversation activity and quality both were

assigned a red-amber status, participating members and facilitating connections among members

were assigned a red status, and identifying experts was assigned an amber status. This suggests

that what conversation was occurring was not of good quality and may likely not have been

80
among individuals of the target membership. Given these status ratings, it is unlikely that S3-

XO.net could have a potential impact status greater than red or red-amber.

Formal analysis of impact can be expected to provide information above and beyond

analysis of social and intellectual capital when status ratings of conversation, content, and

connections are amber-green or green. Impact analysis at that point addresses whether the social

and intellectual capital actively developed in the forum is in line with joint organizational/forum

member needs. Direct assessment of impact addresses whether relevant social and intellectual

capital development can overcome the cultural, logistical, administrative, or other challenges to

effective individual performance.

Informal assessment of S3-XO.net indicates that several of the topics identified as

relevant to having an impact on officer competence and professionalism are present in the forum,

including, for example, leadership challenges in transformation, combined arms training in a unit

of action, the military decision-making process in a collaborative unit, and so on. However,

conversation tended to be quite sparse and stories were infrequently used to share tacit

knowledge.

Impact on Member Competence and Professionalism - Recommendations

Our general-level recommendation for ensuring that conversation in the forum is relevant

to joint Army and S3-XO.net member needs is to survey the available literature addressing issues

currently challenging the organization, particularly as it relates to S3s/XOs, and to interview

individuals who have keen insight into the issues facing S3s/XOs and the Army as a whole.

Ideally facilitators in the forum are members of the professional community and have an organic

sense of the issues, but in the case that they do not, it is essential to know what or who to target

to generate useful information in the forum. Non-expert facilitators should have an expert

advisor who helps them identify, anticipate, and address member needs.

Examples of literature or other documentation that may be helpful in identifying S3/XO

needs include professional articles found in Military Review, which is targeted toward field grade

officers, and CALL reports featuring combat training center trends, particularly as they relate to

the battle staff. Many of the National Training Center assessment teams (e.g., Cobras, Broncos,

etc.) have forums in AKO that post trends. These may be helpful resources. Occasionally,

branch-specific professional publications, such as Armor magazine or Field Artillery magazine,

will have articles addressing battalion staff issues. Up to date information about organizational

initiatives released in the Army News or posted on organizational websites may provide useful

information for anticipating S3/XO needs and prioritizing them for addressing in the forum

(Captain Robert Thornton, personal communication). Parameters and documents produced by

the Strategic Studies Institute at the U.S. Army War College provide insight into issues

challenging the Army as an organization.

Fundamentally, the most important people to interview about what conversation and

content on the forum will be most helpful to current and future S3s/XOs are current and future

S3s/XOs. It can be quite difficult to win time from these individuals, especially current S3s/XOs,

but of the people we interviewed, nearly all of them were enthusiastic about horizontal

81
CA 0 0 0 r 04).- oia ~ ~ coi ~ ~ *~0

0 0)+4) Z .4 1M 4)30 ))4~4 0 

0 0 z CZ 4 ) 0 4 " ,z ~ ) )C C ) 4 ~ ~ Cl) 9 a a 4 (

CO* 0 i -l - -C 0 1 4) C) a)0 0 c ) )U 0q q

-C -1 4) ) CJ>ý 0. o - 01ý r

0 0 > 0

~ OO. - - I- .rn

o R o 0ý 

U) tý ." J 1 ) C Cd ~ ~~~ 0 ý . 0 >

0) -0 0) o 0 . 0

C' 4 I0- on

0- V

coý as t ) V r) 0 mr) E 0 0

on m cdf zy

o~~c 0

-0 0' 0 02~. 0 o

'o -00 U ;a

0 S r,

0~ ~~~ -- ~ )~C~* r~ CZ)0l 

04 ~ ) ) 0 Rt 0 ~ )~ )~ r'-~ 9

tr ) 1 0 En 0Mch Q)U \, > UC

C) En C) s,~ v)o I g4 g Q.) 'A 0 (

0 4) 4) C) (4 C

0 l) 020. .;. 0 0 0 E4 50ý

0-0 4)

o 0 p > r .m C
00o

0O 0

og -:i -m- -

or cc ~4 u m ). ;3 +

P~ -SE I

0 ~~~~~ N rj4 N -t Z or 0

o i 0E

00 -f 0_ u___0____C__ _ _ _ _C

0 -0 00

'o .) o - -- M . .' r 00 

0 CA) 0 Z r.

- , 4 I- -

H0 00C)

CD~~C. 0 ' w A4 0 

0 0 r. 0 C o C0 

0. 0) CD oON

c)) C. . o)C 

-0 00

Cd 0 60) 00
co

*)4

Cdl

o o ~ C' o ) o 0-v0 

4ý -0 0 - 1

0 0 (41

-4 0- 0

o~C >~~ L)r-0

oc

0

E.C))r. c 0)

a.)

C's 00)CI

0 -O)0 V

to. r -' 0 2 ,'. -

M 00

> CP 0

0 0 0

-e9 0)0 <

0n coc0r

uv)02

CIS c)c 0 S 50

0 0

;z 0 o

14 o

u , a 
knowledge sharing with peers. They were generous with their time and provided very helpful

insights. Additional people to approach about identifying needs include the authors of relevant

articles found in the professional literature. Through their writing, these individuals have

demonstrated their expertise and interest in particular topics, and willingness to spend time to

bring these topics to the attention to others. These authors typically have a wealth of experience

to draw on and easily can be located through AKO. Other insightful people include the

observer/controllers at the Combat Training Centers and senior members of the community.

Locating individuals to discuss the needs of S3s/XOs simultaneously helps facilitators

develop social capital, people they can rely on to answer a related question that comes up in the

forum or to recommend individuals who have special insight into that question. The development

of this social capital is especially important when facilitators are themselves not current or future

S3s/XOs. In addition, if interaction between facilitators and interviewees is constructive, positive

word about the forum is spread even though the activity was not part of a formal advertising

campaign.

S3-XO. net Impact on Unit Effectiveness

As with managing the scope of individual-level impact assessment, it is important to

manage the scope of unit-level impact assessment by targeting those aspects of unit effectiveness

for which common or persistent problems have been identified. In addition, for a fair assessment

of impact, "unit" must be defined in such a way that unit effectiveness can be directly influenced

by the target membership of the APF whose impact is being assessed. Because the target

membership of S3-XO.net is comprised of current and future S3s and XOs at the BN and BDE

level, "unit" is here defined as the BN or BDE staff. Improvements in unit-level effectiveness

should therefore address common or persistent problems identified in BN or BDE staff

performance.

The same feasibility issues surrounding the assessment of direct impact on individuallevel competence and professionalism also apply to using the quasi-experimental approach to

assess directly unit-level impact. Here too, a reasonable proxy is to assess whether or not the

content and discussion in S3-XO.net addresses the key problem areas in BN/BDE staff

effectiveness. The social and intellectual capital assessment criteria as applied in the proposed

framework for evaluating potential impact on individual-level effectiveness could be used to

assess the potential impact of S3-XO. net on unit-level effectiveness in the targeted areas of

interest.

The following section presents an overview of the role of the S3/XO on the BN/BDE

staff, highlighting general areas where difficulty has been identified' 6. We used a combination of

literature review and interviews with subject matter experts to understand the role of the S3/XO

16 The present overview is focused primarily on problems occurring during mission planning and execution. This

focus reflects that of the available literature on staff performance, which comes from combat training center

assessments and descriptions of combat lessons learned. A survey of the non-mission challenges faced by S3s/XOs

and their implications for unit effectiveness would provide valuable information for evaluating the impact of S3-

XO.net. However, the difficulty obtaining an understanding of non-mission challenges suggests that such challenges

may be of less interest to organizational decision makers than mission-related challenges.

85
and to identify areas of difficulty. A list of the subject matter experts we interviewed is provided

in Appendix A. If S3-XO.net is to have an impact on unit effectiveness and, by extension,

organizational effectiveness, these areas should represent a significant proportion of the

productive discussion and knowledge sharing occurring in the forum.

Requirements of the BN/BDE Staff

Ultimately, the staff must serve as an extension of the commander by being his eyes, ears,

and voice in all matters of command, whether in garrison or theater [U.S. Army Infantry School

(USAIS), 1937]. The purpose of the staff is to make the commander's authority known down the

chain of command and to inform the commander's decision making through the supply of

timely, relevant information (USAIS, 1937). Whereas the commander leads the unit, the staff

manages the day-to-day activity of the unit, taking advantage of every opportunity to increase the

efficiency of staff functioning (FM 6-0; DOA, 2003).

The XO is the leader of the staff, normally delegated executive management authority by

the commander. He directs staff tasks, oversees staff coordination, and ensures efficient and

prompt staff actions (FM 6-0). All coordinating staff officers (e.g., intelligence officer,

operations officer, and logistics officer), special staff officers (e.g., engineering, fire support,

etc.), and attachments are directly responsible to the XO (FM 6-0). He is responsible for staff

training and subordinate unit readiness, and ensures that the staff functions as a collective and

that their efforts are maximally coordinated with the commander's intent. He also maintains

information flow between the commander and the rest of the staff. The XO must establish a close

relationship with the commander, understanding how the commander's personal characteristics

and command style influence his intent (FM 6-0).

The S3 is responsible for coordinating and overseeing unit training, operations and plans,

and force development and modernization (FM 6-0). He must ensure that operations are

conducted according to the commander's intent, which includes troop preparation, mission

planning, and coordination of execution, among other responsibilities. Together with the

commander he develops the mission essential task list, then he identifies the associated training

requirements and conducts training to ensure that subordinate units meet these requirements.

During operations, the S3 must work closely with the other coordinating and special staff

officers in order to ensure that tactical operations are adequately resourced and synchronized. He

fosters the development and modernization of the unit by recommending and maintaining a force

structure that optimally balances resource allocation with mission requirements.

Key Challenges - The Entire Staff

Advise and Inform the Commander. In order to effectively advise and inform the

commander, the staff must "provide relevant information to inform the commander's situational

awareness/understanding and decision making" (FM 5-0; DOA, 2002). Staffs fairly reliably have

difficulty providing an effective analysis of incoming information such that the implications or

the "so what?" of the information for decision-making are clear (e.g., White, 2001). This

difficulty arises particularly during staff planning (e.g., mission analysis) but also during mission

execution. For example, recent observations from the National Training Center indicate that

86
staffs often fail to create a decision support template (DST) or decision support matrix (DSM)

during mission planning. The DST/DSM specifies the "so what?" of information by linking the

commander's critical information requirements (CCIR) to points in time and space on the

battlefield where intelligence must be gathered and a decision must be made. Moreover, staffs

who do create a DST/DSM often fail to refer to it in order to filter incoming information and

inform the commander in a timely manner that a decision is at hand.

Difficulty analyzing incoming data for their implications may stem from a variety of

causes. Among the most common of these causes are ineffective battle rhythm in the tactical

operations center (TOC) leading to fatigue and error, absent or ignored TOC SOPs for managing

information, and failure to understand the commander's intent. Shortfalls in these areas are

particularly problematic in asymmetric warfare, where information dominance is a critical source

of combat advantage (Kasales, 2002). The S3 and XO are both directly responsible for ensuring

that information management during planning and execution runs smoothly and supports the

commander's decision making.

Prepare Plans and Orders. The preparation of plans and orders involves executing the

military decision-making process (MDMP). Staffs have had difficulty executing the MDMP for

decades. These difficulties include managing time during the planning process, conducting

effective mission analysis (e.g., identifying enemy decision points), fully developing a course(s)

of action, conducting an effective war-game, and producing a synchronized plan (e.g., CALL,

1996, 1997, 1998, 2003). The challenge to effectively apply the MDMP has only increased as

units face an asymmetric enemy. Specifically, the latest trends from the National Training Center

indicate that staffs treat new methods for targeting operational efforts (e.g., effects-based

operations) as separate from the MDMP and therefore discard the MDMP. Limitations in

understanding how the MDMP applies in conventional warfare are preventing the application of

the MDMP to planning counterinsurgency operations and stability and support operations. In

addition, staffs often take too much time to plan, failing to allow subordinate units enough time

to plan their own missions.

Some causes of difficulty in applying the MDMP include ineffective battle rhythm in the

TOC leading to fatigue and error, inadequate staff training, unfamiliarity with doctrine, poor

integration or cohesion within the staff, absent or ineffective planning SOPs, and failure to

understand the commander's intent (and to seek guidance when his intent is vague). The XO is in

charge of training and coordinating the staff (FM 6-0; DOA, 2003). He must set a planning

timeline and ensure that the timeline is adhered to (Pleban, Thompson, & Valentine, 1993). He

must also oversee the planning process, ensuring that his planning SOPs are put into action and

that staff efforts are integrated and aligned with the commander's intent. This requires a close

relationship with the commander and proactive personnel management, with particular attention

to the tendency for certain staff officers (e.g., the S4, forward support battalion support

operations officer) to be excluded from the planning process (Wilson & Snow, 2003). The S3

must take a systems approach to planning and execution, visualizing and understanding the

importance of the contribution of each BOS to achieving decisive effects.

Manage Information Within Own Field of Interest. Information management is an

especially difficult task for battle staffs, particularly as it relates to understanding the CCIR and

87
their implications for developing the intelligence collection plan, analyzing incoming

information to determine its relevance to the CCIR and the commander's decision making, and

disseminating information in order to answer the CCIR. This task becomes even more

challenging during stability and support operations conducted within the province of an

asymmetric enemy because intelligence collection operations involve everyone in the unit (i.e.,

as opposed to involving only a scout platoon or brigade military intelligence assets). This

increase in number of Soldiers collecting information, combined with digital situational

awareness technology, dramatically increases the amount of information entering the TOC. In

order to sift through large amounts of incoming information, staff officers must know and

communicate what they are looking for, who they expect to provide the information, and the

time by which they must have the information they seek.

Some possible causes for difficulty with managing information include failure to

understand the commander's intent, lack of planning and/or TOC SOPs, lack of integration

among the S3 and the intelligence officer, and fatigue or lack of time available in the TOC (due

to ineffective time management or inadequate battle rhythm). As stated previously, the S3 and

XO are both directly responsible for ensuring that information management during planning and

execution runs smoothly and supports the commander's decision making. The XO must lead

information-management by making clear the commander's intent and initial information needs,

ensuring that the staff thoroughly identifies the commander's additional information needs, and

establishing/implementing the SOPs for managing information in the TOC during mission

execution. The S3 aids in the information analysis effort by effectively representing the

commander's concept of operations and coordinating with the intelligence officer during mission

planning.

Perform Staff Coordination. Staff integration has eluded battle staffs for decades (e.g.,

Grossman, 1994; Olmstead, 1992). That is, staff officers generally do not share information with

one another during the planning process, do not understand and anticipate each others' needs,

and do not coordinate with one another when developing components of the mission plan. The

lack of integration among staff officers has deleterious effects on the quality of planning

products--not only the plan itself, but also such products as the synchronization matrix and

decision support matrix. A plan that is not synchronized will likely be ineffective, but could also

be dangerous, for example through increased risk of fratricide. In addition, difficulty with staff

integration not only occurs within the staff of a particular unit, but also between the staffs of

superior and subordinate (e.g., BDE and BN) units.

There are several potential causes of inadequate staff integration, including a rapid rate of

turnover within the staff team, lack of clarity regarding the commander's intent for individual

staff roles (e.g., XO vs. S3), lack of awareness regarding the doctrinal roles and information

requirements of each staff officer (including oneself, Cianciolo & Sanders, to appear),

underestimation of the importance of each staff officer, and interpersonal conflict.

Challenges - XO

The following duties/responsibilities (FM 6-0; DOA, 2003) are especially challenging for

the XO:

88
"* Integrate and synchronize plans and orders

"• Supervise management of the CCIR

"* Establish, manage, and enforce the staff planning timeline

"* Conducting staff training

"* Ensuring staff work conforms to the mission, commander's guidance, and time available

"* Ensuring the staff integrates and coordinates its activities internally and with higher,

subordinate, supporting, supported, and adjacent commands

"* Ensuring all staff sections participate in and provide functional expertise to intelligence

preparation of the battlefield

"* Directing and supervising staff planning

"* Supervising intelligence, surveillance, and reconnaissance (ISR) integration

There are several factors that create difficulty in accomplishing the above listed

duties/responsibilities, including turnover within the staff, inexperienced staff officers who lack

initiative, ineffective time management by the next higher staff, distractions during the planning

process (e.g., the XO is pulled away from the planning process to handle a logistics problem),

tactical ambiguity arising from significant unit reorganization, interpersonal conflict within the

staff, and environmental and cultural pressures to stovepipe staff officer knowledge and activity.

Through effective leadership and the use of such tools as SOPs, graphical timelines, and staff

training TTPs, the XO is expected to deal with these factors to run an effective staff. Because the

XO is the leader of the staff, his effectiveness translates into the effectiveness of the staff as a

whole. Shortfalls in accomplishing the duties/responsibilities listed above readily can be seen in

the challenges faced by the staff as a whole described in the previous section.

Challenges - S3

The following duties/responsibilities (FM 6-0, DOA, 2003) are especially challenging for

the S3:

Training

0 Conducting training within the command

e Preparing training guidance for the commander's approval

0 Helping the commander develop the unit's mission essential task list (METL)

0 Identifying training requirements, based on the unit METL and training status

* Determining requirements for and allocation of training resources

Plans and Operations

0 Synchronizing tactical operations with all staff sections

0 Reviewing plans and orders for synchronization and completeness

• Ensuring necessary combat support requirements are provided when and where required

* Integrating ISR into the concept of operations

* Integrating and managing the ISR effort through an integrated staff process and

procedure

* Developing the ISR plan (with the rest of the staff)

89
"* Synchronizing ISR with the overall operation throughout the operations process (with the

rest of the staff).

"• Integrating fire support into operations

"* Determining combat service support resource requirements (with the personnel and

logistics officer).

"* Participating in course of action and decision support template development (with the

intelligence officer and fire support coordinator).

Force Development and Modernization

"• Processing procedures for unit activation, inactivation, establishment, discontinuance,

and reorganization (force accounting)

"* Evaluating the organizational structure, functions, and workload of military and civilian

personnel to ensure their proper use and requirements (manpower utilization and

requirements)

"* Developing and revising unit force data for documenting any changes to the modification

table of organization and equipment (MTOE) and modification table of distribution and

allowances (MTDA)

"* Conducting formal, on-site manpower and equipment surveys

"* Ensuring MTOE and MTDA documents reflect the minimum essential and most

economical equipment needed for the assigned mission

"* Determining qualitative and quantitative personnel requirements for new equipment and

systems

Unit training is a major challenge for the S3, due to equipment shortages to conduct

training, shortages of NCOs to assist in training, and lack of doctrinal guidance to deal with the

training implications of force reorganization and digitization. The major challenge to conducting

plans and operations for the S3 is coordinating with the other staff officers in order to develop

and execute a unified plan. The importance of combat service and combat service support to

mission success tends to be underestimated, and as a result several coordinating and special staff

officers are left out of the planning process. During mission execution, when the main command

post or TOC and the combat trains command post are geographically distributed, staff integration

becomes much more difficult. Dramatic and rapid changes in the structure and organization of

the Army present significant challenges to the S3 to handle force development and

modernization. Doctrine has not yet been developed to support decision-making regarding

balancing personnel and resource allocation with mission requirements. In addition, the scale of

the changes in unit structure (i.e., installation-wide implications) makes oversight of force

accounting difficult because a great deal of change is occurring at one time. Finally, the new BN

S3 is relatively inexperienced, having not learned much about the day-to-day demands of his job

from previous duty positions or in the schoolhouse. The S3, however, usually outranks the other

officers of the staff, except the XO, and is expected to bring leadership to his job duties.

Unit Effectiveness - A Proposed Framework for Assessment

To assess the impact that S3-XO.net could potentially have on unit effectiveness, a redamber-green rating system analogous to the one used earlier in this report was used. The redamber-green criteria for each aspect of unit effectiveness are presented in the table below. It is

important to note that the assessment criteria presented below are recommended as a starting

90
point and that several of the assessment criteria presented here must change as current issues

evolve. In addition, these criteria may not be exhaustive or reflect the areas of greatest joint

interest to the Army and S3-XO.net members. Finally, it should be noted that the proposed

framework is for evaluating potential impact of the forum, rather than assessing impact directly.

The assumption behind this approach is that (a) it is more feasible and more tractable to assess

the potential for impact rather than impact directly; and (b) if direct impact is of particular

interest, the proposed approach is useful for determining whether taking on such an effort would

provide useful additional information.

Unit Effectiveness - Assessment of S3-XO.net

The impact of S3-XO.net on BN/BDE staff effectiveness was not formally assessed in the

present investigation in part because status ratings of conversation, content, and connections

were all red-amber. Given these status ratings, it was unlikely that S3-XO.net could have a

potential impact status greater than red or red-amber. In addition, in the previous section, it was

determined that S3-XO.net likely has not yet achieved significant individual-level impact.

Individual-level impact is necessary for unit-level impact to be achieved. Informal assessment of

S3-XO.net indicated that some of the topics identified as relevant to having an impact on unit

effectiveness were present in the forum, including, for example, S3 battle rhythm, TOC

operations, MDMP training, and so on. However, conversation on these topics tended to be quite

sparse.

Formal analysis of impact can be expected to provide information above and beyond

analysis of social and intellectual capital when status ratings of conversation, content, and

connections are amber-green or green and substantial impact on the individual-level has been

determined. Impact analysis at that point addresses whether social and intellectual capital

actively developed in the forum can address recognized shortfalls in unit effectiveness and

whether improvements in forum member competence and professionalism generate

improvements in unit effectiveness. Direct assessment of impact addresses whether relevant

social and intellectual capital development and the enhancement of individual competence can

overcome the cultural, logistical, administrative, or other challenges to effective unit

performance.

Unit Effectiveness - Recommendations

Our recommendations for generating discussion and knowledge sharing that will have a

greater impact on unit effectiveness are the same as the recommendations for enhancing impact

on member competence and professionalism. If the facilitators are not themselves current

members of the target membership, then they must otherwise become intimately familiar with

the needs and interests of the members and organization they are serving.

S3-XO.net Impact on Army Effectiveness

Managing the scope of organization-level impact assessment requires targeting those

aspects of organizational effectiveness that present common or current challenges to the target

members of the APF to be assessed. As a counterexample, one issue the Army currently faces is

91
M) 0 W) 4) C) ~C4, 0 m~

~ a0 o a X

1 .0 0 = )u Mu' )4

L) 'D 0aa " x 4

Ca) 0 ' '0 0 x+ 0 ýo ýL0C"0C

Q) > a) 4- Q.)) 0~ ~ ' o 0 > Ca)~ 0 4) a) w Q))0~ 4 a (

=T t~) - co a) c 0 -C 4 - L)) 4- ca 0 )3

E) + -- a+v

o0 r0n

)00 ~ C a)b~ ~~4 0 :5oa~ -

Q)a W V) 

E 0 0.

cnO 0

~ -co ý 0c -S1 .E0 

0" a) a) EA r

V~ ~ *-- - t M: -C 0T cl 0 0 4, - . CO

CollCa 0 0 0 ' t Q) 0

o) 0 v ~~ >< 0" E! -

-e ~ 0 a 4, '1)

) -a 4- c d

cip a) C: 0 O. 0 ~ o0 a)C a) a

0 r C)

0 - ) 0 0- C) C, 0 T 0 0 Iv 'I )G

4- ~ a ~ ~ ~ 4

a)0- - 0 M 0 ) a)l Z3 4

a)a 0) Q. o ; a)

a ) 00C. C)C.

0D o 11.()0. a) a)

a)) a)n0)1

.) 4-4- 00 g , L)L (). L s U v
+~~ w- I o

oc- "I oý; u 

-' r- g 0 1E

o to

o 0 o b r.

0 + o 4) co0 - 0, t.) o 4.

t:.~ 0- 0 0

3-.(- -- 0o r.O

0.E -0 -t t 0 o u 0

(N ~ ~ I-. 0' * 0 0 0 -

0o 4 + 2 L 0 0

.- .0,- 0

0 42 E -5.

Cd al~ u
meeting recruiting quotas. Although, in theory, participation in an APF may address recruiting

challenges (e.g., recruiters could use such an APF to exchange techniques for "getting to yes" or

for locating candidates likely to enlist), participation in an APF geared towards BN/BDE

S3s/XOs likely will not. Because the target membership of S3-XO.net is comprised of current

and future S3s and XOs of BN/BDE staffs, improvements in organization-level effectiveness

should therefore address organizational issues that relate to management and performance at the

BN/BDE echelons.

Feasibility issues make using the quasi-experimental approach undesirable for assessing

organization-level impact unless there is definitive evidence suggesting that such an assessment

would provide additional information beyond a more feasible approach. It is sensible to use a

review of the content and discussion in the forum as a proxy for direct impact assessment before

moving on to more complicated and costly assessment. The following section presents a brief

overview of selected Army-wide initiatives and their implications for the jobs of the BN/BDE S3

and XO. We highlight areas of particular challenge using our findings from a combination of

literature review and interviews with subject matter experts (see Appendix A for a list of the

subject matter experts we interviewed). If S3-XO.net is to have an impact on organizational

effectiveness these areas should represent a significant proportion of the productive discussion

and knowledge sharing occurring in the forum.

The Battalion, The Brigade, and Organizational Initiatives

Identifying the organization-wide impact of activity in a single APF is exceedingly

difficult for the reasons described at the beginning of this report. It should be possible, however,

to articulate what affect on organizational excellence that more widespread knowledge

management (KM) practices can be expected to have. It is one thing to say that KM will make

the Army more effective and another to articulate how this is to occur and what "effective"

means. An ongoing analysis of BCKS 7 , conducted by researchers at the Institute for Defense

Analysis, has listed four ways in which the Army KM initiative is expected to influence

organizational excellence. These are the operational goals of BCKS. Three of these goals will be

discussed briefly below because of their applicability to APFs in particular.

The first means by which KM is expected to improve organizational effectiveness is

through enhanced learning. Through KM, leader development will be enhanced by increased

access to learning content, greater opportunity to repeat the practice of critical skills and

competencies, and more opportunities to receive performance feedback. Second, KM is expected

to result in increased leader ability to make intuitive decisions. Increased opportunity to practice

skills and competencies with expert feedback using a wider variety of environmental conditions

is expected to help leaders improve their pattern recognition and responses to complex situations.

Third, KM is expected to result in the development of more adaptive leaders. Leaders' ability to

recognize novel situations and respond effectively will be increased through access to knowledge

and discussion that will aid in understanding the current situation and its implications.

The above-described operational goals for BCKS represent high-level goals for S3-

XO.net. The key challenges faced by S3s/XOs discussed previously may be addressed by

17 Recall that S3-XO.net is a part of Leader Network, a component of BCKS.

94
meeting the goals of enhanced learning and more adaptive decision-making, among the

development of other skills, competencies, and attitudes. Further articulation of the expected

organizational impact of S3-XO. net requires identifying the organizational initiatives that

enhanced S3/XO functioning may facilitate. The tremendous amount of flux in the Army as it

transforms to the Future Force has direct implications for the leadership and management of

BDEs and BNs because these units are significantly affected by the transformation effort. Below,

two key aspects of Army transformation and their implications for the S3/XO are briefly

described. The general means by which enhanced S3/XO functioning may facilitate

transformation are also presented.

Modularity

Modularity is the process of transforming current Army structure into a modular force.

The overarching goal of modularity is to allow combatant commanders more flexible use of land

power to address a full spectrum of operations through the adaptive selection of brigade "modules" or brigade combat teams (BCTs) to meet mission requirements (Smith, 2005). Simply

put, modularity is changing the structure of the Army brigade such that it has organic combined

arms capability, to include maneuver, intelligence, logistics, and signal, among other functions.

BCTs, for example the Stryker brigade combat team (SBCT), will be standardized in their

structure and in their SOPs such that the units are interchangeable and rapidly deployable.

Additional multi-functional support brigade "modules" (e.g., aviation, fires, and sustainment),

with standardized component "packages," will also be available to the combatant commander in

order to task organize BCTs to meet mission requirements.

Modularity reduces the need for combatant commanders to task organize brigades with

combined arms assets. It also introduces new types of units, such as the maneuver enhancement

brigade and the brigade troops battalion, which integrate forces that were once separate. In the

modular force, two types of Unit of Employment (UE), UEx and UEy, will replace the division

and corps echelons. The UE echelons will not have a set structure, but will be assembled flexibly

on the basis of mission requirements and unit readiness.

The major implications of modularity for the S3 and XO include significant changes in

personnel procedures, equipment, and employment tactics and associated leader and staff

training requirements (Delaney, 2004). Moreover, because of the additional organic assets in the

BCT, the structure and duties of the BCT staff (which is analogous to a division staff; Glenister,

2002) have increased in scope at the same time that tactics are also changing due to new types of

units or modified unit structures. Modularity represents the greatest shift in force structure since

the 1960's (Smith, 2005), and is in some form of progress at installations nationwide. Some more

specific implications of changes to the current force for the job of the S3 and XO are described

below. The discussion below is not complete because such a discussion is beyond the scope of

this short-term project, but it is meant to illustrate the nature of challenges that transformation

presents to the S3/XO and how improving S3/XO capability facilitates this Army-wide initiative.

ISR in the BCT. Two types of BCT, the SBCT and the infantry BCT, feature a

reconnaissance, surveillance, and target acquisition (RSTA) squadron (Smith, 2005). This

organic ISR capability in one sense eases ISR planning by the BCT staff because such planning

95
is now handled by the RSTA squadron commander and staff (Kasales & Gray, 2003). However,

close coordination must occur between the BCT S3 and RSTA squadron staff such that the ISR

plan is maximally integrated with the brigade maneuver effort. Moreover, the RSTA squadron is

not the only contributor to the BCT ISR effort, but is complemented by the BCT intelligence

officer, a military intelligence (MI) company, and infantry battalion MI assets (Toomey, 2003).

The BCT S3 must know how to coordinate these resources such that the ISR plan supports the

BCT maneuver effort and that the information collected is timely and relevant (Kasales & Gray,

2003) -- a task that was already difficult with reduced complexity. In the absence of well-defined

tactics for employment, the RSTA squadron S3 must fully understand the capabilities of the

range of assets available to the squadron commander (i.e., human intelligence, tactical unmanned

aerial vehicles, ground surveillance radar, reconnaissance squadrons, radio intercept, Fox

nuclear, biological, and chemical vehicles) and how they are best integrated to meet the squadron

and BCT mission requirements. Integration of staff efforts across echelons is an additional

training challenge for both the BCT and RSTA squadron XO.

Fires, Targeting, and Effects. Asymmetric warfare in an urban environment has two

broad implications for the use of fires and targeting. First, the built-up nature of the urban

environment places limitations on what fires assets can be used such that unnecessary casualties

and destruction to infrastructure are minimized. Second, the "battlefield" on which the

asymmetric enemy conducts warfare is socio-political, where the trust and cooperation of the

local populace (and other key stakeholders) serves as "key terrain;" decisive victory requires not

annihilation of the enemy but obviation of the enemy through the effective targeting and denial

of this key terrain to the enemy (Cianciolo & DeWillie, 2004).

The Army has responded to the need for effects-based targeting in part by designing the

fires and effects coordination cell (FECC), which is responsible for coordinating efforts to

achieve lethal and non-lethal effects on the enemy (Glenister, 2002). The FECC is situated

within the headquarters of the BCT, serving as the engine that drives lethal and non-lethal

targeting for the unit, planning and executing fires and effects tasks in collaboration with the

other BOSs (FMI 3-09.42; DOA, 2005). It is an organic, standing organization with information

operations capabilities, which differentiates it from the traditional brigade fire support element

(FMI 3-09.42; Glenister, 2002). The field artillery battalion organic to the BCT supplies the

effects coordinator who directs the FECC. The information operations section of the FECC is

comprised of personnel from information operations, civil affairs, electronic warfare, and

psychological operations. The FECC provides direction to the field artillery battalion's efforts to

support the BCT.

The introduction of the FECC requires that field artillery battalion commanders and staffs

address the task of fires and effects task coordination (Glenister, 2002). The field artillery

battalion S3 must coordinate closely with the BCT FECC and the field artillery brigade

headquarters such that the battalion does not become overtaxed by close support, shaping, and

counterstrike responsibilities (FMI 3-09.42). This S3 must also ensure that his unit is trained to

effectively execute coordinated fires and effects tasks as the FECC and battalion commander

direct. The BCT S3 must have a solid understanding of how coordinated fires and effects tasks

will support the maneuver effort. The persistent challenge to S3s to coordinate maneuver with

fire support becomes more difficult as "overlapping fires" and "clearing effects" take on new

96
meanings when attempting to achieve social or political effects on the civilian populace

(Hamilton & Gist, 2004). The increased complexity of the staff amplifies the challenge to the

BCT XO to train and synchronize the staff effectively. S3s and XOs are not fully supported by

doctrine in their effort to adapt to and exploit the FECC. Interim doctrine exists [FMI (Field

Manual Interim) 3-09.42; DOA, 2005], but it is not fully consistent with or applicable to the

tactics, techniques, and procedures adopted in practice and described in the professional

literature (Glenister, 2002; Hamilton & Gist, 2004). In addition, the interim doctrine appears to

confound achieving effects with performing tasks, a confound that is counterproductive and

widely adopted.

New Units - The Brigade Troops Battalion. The brigade troops battalion (BTB) is

organic to certain types of BCT (e.g., the heavy BCT). Its C2 cell (commander and staff)

provides a centralized means for coordinating and supporting what were once separate

companies and detachments typically attached to a traditional brigade (FMI 3-90.61; DOA,

2005). The BTB is comprised of four organic units [BTB headquarters and headquarters

company (HHC), BCT HHC, a military intelligence company, and a signal network support

company], and may have one or more attached elements, including civil affairs teams,

psychological operations, public affairs, and specialized engineer teams, among others (FMI 3-

90.61). Its general mission is to support the BCT staff during planning, to carry out and support

BCT-designated tasks, to support the BCT's command posts and subordinate units, and to

provide rear area and base security (FM 3-90.61).

Over the next two to three years, the commanders, XOs, and S3s of the BTB will come

from engineering battalions that have been disbanded. The challenge to the BTB XO to train and

coordinate a staff representing multiple functional areas is not a new one, though effective

coordination of the particular specialties present in the BTB requires learning of unfamiliar

functional areas. The S3 must also learn about these unfamiliar functional areas in order to

effectively (a) identify the training needs of each type of unit and of the BTB overall; (b) provide

this training; and (c) plan and support the coordinated efforts of these units. That the combat

engineering battalion is being largely removed from the force structure is a sore point for several

Army engineers. Some engineering S3s and XOs assuming a duty position in a BTB may also

have to strive to succeed despite low morale or frustration with changes beyond their control.

Digitization

Digitization is the process of enabling battlefield visualization, collaborative mission

planning, and information sharing via networked computer systems (Barnett, Meliza, &

McCluskey, 2001). Digitized tactical operations centers feature digital situation awareness

displays in place of maps and overlays. The digitized staff plans missions from distributed

locations on the battlefield in parallel with higher and lower echelon staffs. They create and

disseminate orders electronically. As is the case with all technological developments, digitization

addresses some past challenges and also creates some new challenges, including new equipment

that must be understood, staffed, and maintained, new skills that must be learned, new

coordination that must be established, new doctrine and tactics that must be developed and

learned, and so on (Barnett et al., 2001). All of these challenges have implications for the S3 and

XO. Some of these implications are summarized below.

97
Maintaining Connectivity. Maintaining connectivity throughout an engagement is a

difficult task made more difficult by digitization. The BN and BDE staff must plan the location

of retransmission vehicles very carefully such that they are positioned to support

communications among and between echelons that use radio systems with different planning

ranges and with different roles during the engagement (Cotter, 2002). Digital transmission via

Near-Term Digital Radio is required to keep digital BN and BDE TOCs from reverting back to

analog. However, Frequency Modulation and Enhanced Position Location Reporting System

radios are required for companies to communicate with each other and with their parent BNs.

The need to support collaborative planning, battle tracking, and information dissemination at the

BN and BDE level may conflict with the need for companies to communicate with their parent

BNs using FM radio during the close fight, especially when retransmission resources are limited

or reduced by enemy action (Cotter, 2002). The BN and BDE XO must ensure that their signal

officer is integrated into the planning process such that retransmission needs are resourced and

retransmission vehicles are effectively positioned in time and space during an engagement while

also secured from enemy interference or attack.

Digital Skills Training and Maintenance. In addition to maintaining connectivity and

performing basic manipulations of the interface, users of digital C2 equipment must exploit the

capabilities of the technology such that using it is not tantamount to doing the same thing with a

different method. The use of digital C2 equipment should result in fundamentally doing things

differently. It requires expertise to diagnose and correct the instability of the tactical Internet,

expertise that is not resident in most Soldiers. For these reasons, it is critical that the S3 and XO

identify and train individual and collective digital skills, but they must do so in the relative

absence of doctrinal or other support for performance assessment and feedback (though see

Leibrecht et al., 2004). Moreover, digital skills are highly perishable, so S3s/XOs must be

creative in determining means by which these skills can be practiced, such as through integrated

use of the Army Battle Command System (ABCS) to do garrison tasks (e.g., to monitor garrison

activities such as training (Brown & Dedrich, 2003; Kasales & Gray, 2003). The S3/XO must

also establish digital SOPs to address information management, choices about what information

is sent when due to bandwidth constraints and limited satellite time and/or digital gateways for

satellite information (Kasales & Gray, 2003; Toomey, 2003).

Information Overload. The significant increase in information made available to the staff

can either help or hinder their battle tracking and support of the commander's decision-making,

depending on how information is processed and managed in the tactical operations center. Too

much information is as much or more of a problem as too little. For example, the ability to cut

and paste information, including pictures and graphics, from digital displays enables staffs to

produce longer and longer orders, briefs, and other products which are not necessarily more

informative (Langley, 2004). Moreover, larger files consume more bandwidth, thus slowing

down not only information processing on the part of the recipient but also the transmission of the

file itself (Langley, 2004). The analysts with the skills required to process the large volume of

information produced by digital systems are often not available at the battalion level (Langley,

2004), so the BN XO must ensure that some other means is in place to manage information

coming into and leaving the TOC. The previously described challenges to the XO and S3 to

manage information effectively apply to handling information overload in the digital TOC.

98
Asynchronous Digitization. For obvious reasons, digitization cannot occur throughout the

Army all at one time. The result is that some units are digitized ahead of others. In addition, the

command and control systems of joint forces likely will not have compatible computer networks.

The implication is that units fighting together may not be able to capitalize on digitization

because digital systems are not available to everyone. The responsibility rests with the XO and

S3 to ensure that units with differing digital capability can still be synchronized and employed

according to the commander's intent.

Army Effectiveness - A Proposed Framework for Assessment

As with the previously proposed frameworks for assessment, a red-amber-green rating

system was used as the format for the present proposed framework, and it is illustrated in the

table below. The present criteria used to assess potential impact on Army effectiveness must

evolve to reflect current issues. In addition, these criteria are by no means exhaustive and may

not reflect the areas of greatest joint interest to the Army and S3-XO.net members. A more

thorough assessment of the link between key organizational objectives and the jobs of the

BN/BDE S3/XO was outside the scope of this research and is recommended for future

assessment of S3-XO.net.

Army Effectiveness - Assessment ofS3-XO.net

The impact of S3-XO. net on Army effectiveness was not formally assessed in the present

research for three reasons. First, status ratings of conversation, content, and connections were all

red-amber. Given these status ratings, it is unlikely that S3-XO.net could have a potential impact

status greater than red or red-amber. Second, in the preceding two sections, it was determined

that S3-XO.net likely has not yet achieved significant individual- or unit-level impact. Third,

even if social and intellectual capital were actively developed in S3-XO.net, the APF has been

active for less than three years. This time period is likely too short to expect a discernable impact

at the organizational level, especially considering the relatively small percentage (3%) of the

intended target audience of the forum currently holding membership.

Informal assessment of S3-XO.net indicated that some of the topics identified as relevant

to having an impact on Army effectiveness are present in the forum, including, for example,

command post of the future, fires and effects coordination cell, combined arms training in the

unit of action, leadership challenges in transformation, and so on. However, conversation about

these topics tended to be quite sparse.

Army Effectiveness - Recommendations

Our recommendations for generating discussion and knowledge sharing that will have a

greater impact on Army effectiveness are the same as the recommendations for enhancing impact

on member competence and professionalism and unit effectiveness. Facilitators must be

intimately familiar with the needs and interests of the members and organization they are

serving.

99
100
APPENDIX C - CONDUCTING ASSESSMENT OF THE ARMY PROFESSIONAL FORUM

Regular, formal assessment is the means by which an organization ensures the health and

impact of its knowledge-management initiatives (Hanley, 2003; Crager & Lemons, 2003; Lesser

& Storck, 2001; Department of the U.S. Navy, 2001). Assessment enables decision makers to

effectively allocate limited resources to the initiative by (a) identifying the components of the

initiative that require additional investment; and (b) providing insight into how investment could

strengthen vulnerable aspects of the initiative. In addition, assessment reveals the degree to

which investment in the initiative can be expected to payoff in terms of organizational

performance.

Organizational decision makers must resist, however, confusing the means for capturing

impact with impact itself, else the knowledge-sharing initiative will become incapable of

adapting to changes in the organizational environment. As an analogy, a test of academic

achievement may be used to capture the impact of an educational program, but test scores should

never be considered the outcome of the educational program. When this happens (and it often

does), focus shifts away from meeting environmental demands for intellectual development

toward meeting an artificial standard for performance. Distancing from the environment

decreases awareness and adaptability. Metrics for assessing the impact of a knowledge sharinginitiative must be flexible and must always reflect the environmental demands that pressure the

initiative to evolve. The metrics and techniques presented in this report and appendix represent a

starting point for thoughtful analysis of Army Professional Forums (APFs), an example of how it

can be accomplished in response to both current environmental demands and in accordance with

lasting and generalizable best practice in assessment.

The level of understanding about the health and impact of a discussion forum that can be

accomplished in a given time is determined by the focus and efficiency of the assessment

process. Focus is enhanced by a thorough understanding of the forum's online meeting space, the

needs of the population the forum is intended to serve, the needs of the organization, and how the

forum is expected to have an impact. For this reason, individuals conducting assessment should

be educated in knowledge-sharing theory and practice and in the Army's functioning and

organizational structure, particularly as it relates to the intended membership of the forum.

Although independent, assessors should work closely with the forum leadership, facilitators, and

active members of the forum to ensure that assessment directs forum cultivation rather than

imposing top-down requirements. Efficiency is enhanced by standardization and, where possible,

automated data collection. Several of the metrics presented in this report are designed to be

generalizable, and provide standardized criteria for assessing APFs. In the table below, we

present a broad overview of conducting assessment of APFs using our assessment framework.

We address how it is done, how often it needs to be done, how long it takes, and whether or not it

can be automated.

101
How is it done? How often does it [ How long Can it be

need to be done? j does it take? automated?

Statement of Review of plan documentation As events warrant 0.5 hr No

Purpose

Envisioned Impact Review of plan documentation As events warrant 0.5 hr No

Description of Review of plan documentation As events warrant 0.5 hr No

Target Audience _________________ ________

Statement of

Norms, Values, and Review of plan documentation As events warrant 0.5 hr No

Conventions

Specification of

Roles and Review of plan documentation As events warrant 1 hr No

Responsibilities

Functional and

Technical Review of plan documentation As events warrant 1 hr No

Specification

The plan should be assessed before the forum is launched--preferably by multiple assessors

to ensure the objectivity and reliability of the rating--and after significant changes in forum

membership are made (e.g., a sub-community breaks off to form its own forum) or

immediately after significant changes in technical architecture are made (e.g., adding

Comments synchronous collaboration capability). Time estimates assume one assessor. Two assessors

would approximately double the time estimate. After the plan is independently assessed, the

findings should be discussed with the forum leadership and recommendations for

strengthening the plan should be provided. If the initial plan requires significant

modification, assessment should be conducted immediately after the plan is revised and

I prior to forum launch.

_........_PERSONNEL .

Leadership Interviews with forum leaders Every 6 months 2 hr + N hrs No

Core Group Interviews with core group; Every 6 months 4 hr + N hrs No Biographical data collection

Facilitators Interviews with facilitators; Analysis of facilitator activity in the forum Every 6 months 4 hr + N hrs No

Technical Support Interviews with technical support Every 6 months 2 hr + N hrs No staff I

Initial assessment of personnel should be conducted within six months after forum launch.

The time required to conduct the interviews for each personnel category is two hours plus

one hour times the number of people interviewed. This estimate allows one hour for each

interview and two hours to collate and summarize interview findings. Where other data are

collected, two additional hours are estimated to conduct data collection. For quiet forums,

Comments two hours will allow complete analysis. For very active forums, two hours will allow

analysis of a representative sampling of data. Time estimate does not include time devoted

to scheduling interviews or downloading conversation posts into a main database used for

analysis. This could be done automatically with databasing and scripting techniques. After

personnel are independently assessed, the findings should be discussed with the forum

leadership and recommendations for enhancing personnel support should be provided.

_______ ..... FUNCTIONAL REQUIREMENTS',. _____ , _ __... .

Locating the Query forum leaders and/or

Forum facilitators; External search for forum

Joining the Forum Query facilitators; Survey members As events warrant 1.5 hr Yes/No

Tone in the Forum Analysis of online meeting space Every 6 months 1 hr No

Conversation Analysis of online meeting space As events warrant 1 hr No

Support

102
Locating Content Analysis of online meeting space; As events warrant 3 hr Yes/No

Survey members

Locating People Analysis of online meeting space As events warrant 1.5 hr No

Initial assessment of functional requirements, with the exception of surveying members,

should occur prior to forum launch as a sort of usability test. Follow-on assessment should

occur as significant changes are made to the design of the online meeting space (e.g., new

content taxonomy) or to how it is accessed. Surveying members can be automated through

pop-up survey questions administered in the online meeting space. Time estimates presented

above are likely to be shorter for follow-on assessments. Time estimates for surveying

Comments members do not include the time required to automate survey administration or to modify

survey questions from those presented in Appendix C, but do assume time required for data

entry and summarization. Time estimates apply to surveying a representative sample of

forum members. After functional requirements are independently assessed, the findings

should be discussed with the forum leadership and facilitators, and recommendations for

enhancing functional requirements should be provided. If initial functional requirements

need significant modification, assessment should be conducted immediately after they are

revised and prior to forum launch.

Membership 

Composition Analysis of online meeting space Every 6 months 5 hr Yes

Participating Analysis of online meeting space Every 6 months 0.5 hr Yes

Members

Identifying Experts Analysis of online meeting space; Every 6 months 0.5 hr Yes

Survey members

Facilitation of Query facilitators; Survey members Every 6 months 2 hr Yes/No

Connections

Connection Quality Survey members Every 6 months 3 hr Yes

Initial assessment of connections should occur within 6 months of forum launch. All time

estimates assume that assessment is not automated. Time estimate for membership

composition includes time for data downloading and cleaning. Surveying members can be

automated through pop-up survey questions administered in the online meeting space. Time

estimates do not include the time required to automate survey administration or to modify

Comments survey questions from those presented in Appendix C, but do include time for data entry and

summarization. Time estimates apply to surveying a representative sample of forum

members. Data collection on membership composition, participating members and

identifying experts could be automated through databasing and scripting techniques. After

connections are independently assessed, the findings should be discussed with the forum

leadership and facilitators, and recommendations for enhancing connections should be

provided.

.CONTEXT'ýý

Shared Narratives Analysis of online meeting space Every 6 months 0.5 hr No

Shared Codes and Survey members Every 6 months 1 hr Yes

Language

Initial assessment of connections should occur within 6 months of forum launch. Time

estimate for surveying members include time for data entry and summarization. Surveying

members can be automated through pop-up survey questions administered in the online

meeting space. Time estimates do not include the time required to automate survey

Comments administration or to modify survey questions from those presented in Appendix C, but do

include time for data entry and summarization. Time estimates apply to surveying a

representative sample of forum members. After context is independently assessed, the

findings should be discussed with the forum leadership and facilitators, and

recommendations for enhancing context should be provided.

103
Contributing Analysis of online meeting space Every 6 months 0.5 hr Yes

Members

Content Synthesis Analysis of online meeting space Every 6 months 4 hr No

Content Analysis of online meeting space; Every 6 months 1 hr Yes

Organization Survey members

Content Activity Analysis of online meeting space Every 6 months 1 hr Yes

Content Quality Analysis of online meeting space; Every 6 months 5 hr Yes/No

Survey members Every_6_months 5_hr Yes/No

Professional Links Analysis of online meeting space Every 6 months 0.5 hr No

Initial assessment of content should occur within 6 months of forum launch. Time estimate

for contributing members assumes data downloading and cleaning for membership

composition has been completed. Surveying members can be automated through pop-up

survey questions administered in the online meeting space. Time estimates do not include

the time required to automate survey administration or to modify survey questions from

those presented in Appendix C, but do include time for data entry and summarization. Time

estimates apply to surveying a representative sample of forum members. Data collection on

Comments contributing members and content activity could be automated through databasing and

scripting techniques. For quiet forums, the time estimate for content synthesis and content

quality will allow complete analysis. For very active forums, estimates will allow analysis of

a representative sampling of data. Time estimate does not include the time to modify the

rubric presented in this report or to create a new rubric. Where possible, two assessors

should be used to assess content synthesis and content quality and their assessments

compared and combined. After content is independently assessed, the findings should be

discussed with the forum leadership and facilitators, and recommendations for enhancing

content should be provided.

CONVERSATION _________

Participating Analysis of online meeting space Every 6 months 0.5 hr Yes

Members

Conversation Analysis of online meeting space Every 6 months 5 hr Yes/No

Activity_________________________________________

Conversation Analysis of online meeting space; Every 6 months 8 hr Yes/No

Quality Survey members I I

Initial assessment of conversation should occur within 6 months of forum launch. Time

estimate for contributing members assumes data downloading and cleaning for membership

composition has been completed. Surveying members can be automated through pop-up

survey questions administered in the online meeting space The time estimate does not

include the time required to automate survey administration or to modify survey questions

from those presented in Appendix C, but do include time for data entry and summarization.

Time estimates apply to surveying a representative sample of forum members. Data

collection on participating members and conversation activity could be automated through

Comments databasing and scripting techniques. For quiet forums, the time estimate for conversation quality will allow complete analysis. For very active forums, estimates will allow analysis of

a representative sampling of data. Time estimate does not include the time to modify the

rubric presented in this report or to create a new rubric. Where possible, two assessors

should be used to assess conversation quality and their assessments compared and

combined. Time estimates do not include time to download conversation posts into a main

database used for analysis. This could be done automatically with databasing and scripting

techniques. After conversation is independently assessed, the findings should be discussed

with the forum leadership and facilitators, and recommendations for improving conversation

should be provided.

INDIVIDUAL IMPACT_

Indirect Assessment Analysis of online meeting space Every 6 months Unknown No

104
Direct Assessment Special purpose measures; Survey As events warrant Unknown Yes/No

members

The time estimated to conduct indirect assessment is unknown because it is unknown how

long it will take to determine what content should be present in the forum to aid in the

development of individual competence and professionalism. This will be at least slightly

different for each forum. Once relevant content is determined, the estimated time to conduct

assessment will depend on how much content is relevant and sampled. A forum plan with a

solid statement of envisioned impact and facilitators/assessors familiar with the current

Army will facilitate indirect assessment. The time estimated to conduct direct assessment is

Comments unknown because it is unknown how long it will take to develop special purpose measures

(e.g., tactical games, situational judgment scenarios) and implement them in the online

meeting space. Surveying members can be automated through pop-up survey questions

administered in the online meeting space, but survey questions will have to change as

content or emphasis changes. Direct assessment should only be conducted if the findings

from indirect assessment suggest it is warranted. After individual impact is independently

assessed, the findings should be discussed with the forum leadership and facilitators, and

recommendations for improving impact should be provided.

UNIT IMPACTi

Indirect Assessment Analysis of online meeting space Every 6 months Unknown No

Direct Assessment Unit performance observations As events warrant Unknown No

The time estimated to conduct indirect assessment is unknown because it is unknown how

long it will take to determine what content is necessary to aid in the enhancement of unit

effectiveness. This will be different for each forum. Once relevant content is determined, the

estimated time to conduct assessment will depend on how much content is relevant and

sampled. A forum plan with a solid statement of envisioned impact and facilitators/assessors

Comments familiar with the current Army will facilitate indirect assessment. The time estimated to

conduct direct assessment is unknown because it is unknown how long it will take to arrange

and conduct observations. Direct assessment should only be conducted if the findings from

indirect assessment suggest it is warranted. After unit impact is independently assessed, the

findings should be discussed with the forum leadership and facilitators, and

recommendations for improving impact should be provided.

Indirect

Assessment Analysis of online meeting space Every 6 months Unknown No

Direct Assessment Organizational performance observations; Archival data collection As events warrant Unknown No

The time estimated to conduct indirect assessment is unknown because it is unknown how

long it will take to determine what content is necessary to aid in the enhancement of

organizational effectiveness. This will be different for each forum. Once relevant content is

determined, the estimated time to conduct assessment will depend on how much content is

relevant and sampled. A forum plan with a solid statement of envisioned impact and

facilitators/assessors familiar with the current Army will facilitate indirect assessment. The

time estimated to conduct direct assessment is unknown because it is unknown how long it

will take to arrange and conduct observations or collect archival data. Direct assessment

should only be conducted if the findings from indirect assessment suggest it is warranted.

After organizational impact is independently assessed, the findings should be discussed with

the forum leadership and facilitators, and recommendations for improving impact should be

provided.

105
106
004

-0 g.l 00 4 W 0 1.

0 40,~ >

0 0) U0 45

E). -0 -, 4) Q) 4O o0 0 , 000 'o 

4)j 4) 2 > . z 5 'O 0 

4) a CA 4)

0

0~ 00 CA o)

4)~~V C. -40 

4) 4.) a.

x I o co1; 0 ca

4o Jj X 0 " cn 2.0

0) cng-~-4

-c 1

4))

0 .

x 1-4 - c0.1 0V 0o 0

0 0

4)

- .

z-,-3 0e -. ct Q.) A

0 oo

rnw 0 U3',

I- 0 

0. . Z

Cl U V4 C 
v~ '0

V~ 0 4

E al) a..)

g 0 .9 't 'g '

`3 c" t v0 > 0

0 rn

420 tQ) ýý4 n V.)

00 C) w : O 4:

rOco V 000 > 0 

-. a.) 0. 70 (L)

-x ca.0 0~t 00

'8

14 z 0 "~ "% 0

) u. o.b~

-li 0a 0

=C q :J 0 

Ob L4 Ct) 

00

-'S

w 0

co U, 0 00

C) 0

r. Z 0

) a l) 0 0

00

-L 0 c 0
"" 4w

(2 C)C~ . 0 CV) 001

Q 0) u~ E B

(D, 0

0 . 0 1 . LV-0 0

~co

0~ '0.L

0 0

0 0

0 C6 -0 0

0 0 o

,I:.) cn00 '

ct 0 r'. o- 

.0 C13 U

>~t>

0~~ 0 z .0 X Ca

P. coQ 0 

00

0 PL 

Q0) g

US a

0 -cg 0

te.

(L 0v- -'

*cJ 0a 0 05co. r 00 V "r -6

0~0

0 0th i. 0 0 0

0o U U ;

-)= t-.- ;
C

'-4
APPENDIX E - ACRONYMS

AKO

APF

ATLDP Army Training and Leader Development Panel

BCKS Battle Command Knowledge System

BCT brigade combat team

BDE brigade

BN battalion

BOS battlefield operating system

BTB brigade troops battalion

C2 command and control

CAC Combined Arms Center

CALL Center for Army Lessons Learned

CCIR commander's critical information requirements

CG commanding general

CGSC Command and General Staff College

COA course of action

CoP community of practice

DOA U.S. Department of the Army

DSM decision support matrix

DST decision support template

FECC fires and effects coordination cell

FM field manual

FMI field manual interim

FORSCOM U.S. Army Forces Command

ISR intelligence, surveillance, and reconnaissance

KM knowledge management

MDMP military decision-making process

METL mission essential task list

MI military intelligence

MTDA modification table of distribution and allowances

MTOE modification table of organization and equipment

NCO noncommissioned officer

OIF MHAT Operation Iraqi Freedom Mental Health Advisory Team

RSTA reconnaissance, surveillance, and target acquisition

111
S3 operations officer

SBCT Stryker brigade combat team

SOP standing operating procedure

TACSOP tactical standing operating procedures

TOC tactical operations center

UE unit of employment

USAIS U.S. Army Infantry School

XO executive officer

112